Sankofa/docs/DEPLOYMENT_NEXT_STEPS.md

Deployment Next Steps

Date: 2025-12-09
Status: ⚠️ LOCK ISSUE - MANUAL RESOLUTION REQUIRED

---

Current Situation

✅ Completed

1. Provider Configuration: ✅ Verified and working
2. VM Resource Created: ✅ basic-vm-001 (VMID 100)
3. Deployment Initiated: ✅ VM created in Proxmox

⚠️ Blocking Issue

VM Lock Timeout: Configuration update blocked by Proxmox lock file

Error: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout

---

Immediate Action Required

Step 1: Resolve Lock on Proxmox Node

Access the Proxmox node and clear the lock:

# Connect to Proxmox node (replace with actual IP/hostname)
ssh root@<proxmox-node-ip>

# Check VM status
qm status 100

# Unlock the VM
qm unlock 100

# If unlock doesn't work, remove lock file
rm -f /var/lock/qemu-server/lock-100.conf

# Verify lock is cleared
ls -la /var/lock/qemu-server/lock-100.conf

Note: If you don't have direct SSH access, you may need to:

• Use Proxmox web UI
• Access via console
• Use another method to access the node

Step 2: Verify Image Availability

While on the Proxmox node, verify the image exists:

# Check for image
find /var/lib/vz/template/iso -name "ubuntu-22.04-cloud.img"
pvesm list local-lvm | grep ubuntu-22.04-cloud

# If missing, download it
cd /var/lib/vz/template/iso
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
mv jammy-server-cloudimg-amd64.img ubuntu-22.04-cloud.img

Step 3: Monitor Automatic Retry

After clearing the lock, the provider will automatically retry:

# Watch VM status
kubectl get proxmoxvm basic-vm-001 -w

# Watch provider logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f

Expected Timeline: 1-5 minutes after lock is cleared

---

After Lock Resolution

Expected Sequence

1. Provider retries configuration update (automatic)
2. VM configuration completes successfully
3. Image import (if needed) completes
4. Boot order set correctly
5. Cloud-init configured
6. VM boots successfully
7. VM reaches "running" state
8. IP address assigned
9. Ready condition becomes "True"

Verification Steps

Once VM is running:

# Get VM IP
IP=$(kubectl get proxmoxvm basic-vm-001 -o jsonpath='{.status.networkInterfaces[0].ipAddress}')

# Check cloud-init logs
ssh admin@$IP "cat /var/log/cloud-init-output.log | tail -50"

# Verify services
ssh admin@$IP "systemctl status qemu-guest-agent chrony unattended-upgrades"

# Test SSH access
ssh admin@$IP "hostname && uptime"

---

If Lock Resolution Fails

Alternative: Delete and Redeploy

If the lock cannot be cleared:

# 1. Delete Kubernetes resource
kubectl delete proxmoxvm basic-vm-001

# 2. On Proxmox node, force delete VM
ssh root@<proxmox-node> "qm destroy 100 --purge --skiplock"

# 3. Clean up locks
ssh root@<proxmox-node> "rm -f /var/lock/qemu-server/lock-100.conf"

# 4. Wait for cleanup
sleep 10

# 5. Redeploy
kubectl apply -f examples/production/basic-vm.yaml

---

Long-term Solutions

1. Code Enhancement

Add lock handling to provider code:

• Detect lock errors in UpdateVM
• Automatically call qm unlock before retry
• Increase timeout for lock operations
• Add exponential backoff for lock retries

File: crossplane-provider-proxmox/pkg/proxmox/client.go

2. Pre-deployment Checks

Add validation before VM creation:

• Check for existing locks on target node
• Verify no conflicting operations
• Ensure Proxmox node is healthy

3. Deployment Strategy

For full deployment:

• Deploy VMs sequentially (not in parallel)
• Add delays between deployments (30-60 seconds)
• Monitor each deployment before proceeding
• Implement retry logic with lock handling

---

Full Deployment Plan (After Test Success)

Phase 1: Infrastructure (2 VMs)

1. nginx-proxy-vm.yaml
2. cloudflare-tunnel-vm.yaml

Phase 2: SMOM-DBIS-138 Core (8 VMs)

3-6. validator-01 through validator-04 7-10. sentry-01 through sentry-04

Phase 3: SMOM-DBIS-138 Services (8 VMs)

11-14. rpc-node-01 through rpc-node-04 15. services.yaml 16. blockscout.yaml 17. monitoring.yaml 18. management.yaml

Phase 4: Phoenix VMs (8 VMs)

19-26. All Phoenix VMs

Phase 5: Template VMs (2 VMs - Optional)

27. medium-vm.yaml
28. large-vm.yaml

Total: 28 additional VMs after test VM

---

Summary

Current Status

• ✅ Provider: Working
• ✅ VM Created: Yes (VMID 100)
• ⚠️ Configuration: Blocked by lock
• ⚠️ State: Stopped

Required Action

Manual lock resolution on Proxmox node

After Resolution

• Provider will automatically retry
• VM should complete configuration
• VM should boot successfully
• Full deployment can proceed

---

Last Updated: 2025-12-09
Status: ⚠️ WAITING FOR MANUAL LOCK RESOLUTION