Sankofa/docs/proxmox/archive/REMAINING_BLOCKERS_GUIDE.md

Remaining Blockers - Complete Step-by-Step Guide

Last Updated: 2024-12-19

Overview

This document provides detailed step-by-step instructions to resolve all remaining blockers for deployment.

Priority Order

Important: Resolve blockers in this order for optimal workflow:

1. SSH Access (Blocker 2) - Required first to verify/download images
2. Image Verification (Blocker 3) - Depends on SSH, needed before VM deployment
3. Kubernetes Cluster (Blocker 1) - Needed for provider deployment and VM creation

Rationale: SSH access is needed to verify and download images, which must be ready before deploying VMs via Crossplane. Kubernetes can be set up in parallel, but images should be verified first.

Blocker 2: SSH Access to Proxmox Nodes (PRIORITY 1)

Resolve this first - Required for image verification and download

Required For

• TASK-009: Build and test Crossplane provider
• TASK-010: Deploy Crossplane provider to Kubernetes
• TASK-011: Create ProviderConfig resource
• TASK-014: Set up monitoring dashboards
• TASK-015: Deploy test VMs via Crossplane

Step-by-Step Instructions

Option A: Using kind (Kubernetes in Docker) - Recommended for Local Development

Step 1: Install kind

# On Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# Verify installation
kind version

Step 2: Install kubectl

# On Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

# Verify installation
kubectl version --client

Step 3: Create kind cluster

# Create cluster configuration
cat > kind-config.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: sankofa
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080
  - containerPort: 30443
    hostPort: 30443
EOF

# Create cluster
kind create cluster --name sankofa --config kind-config.yaml

# Verify cluster
kubectl cluster-info --context kind-sankofa
kubectl get nodes

Step 4: Configure kubeconfig

# Set context
kubectl config use-context kind-sankofa

# Verify
kubectl config current-context

Step 5: Install Crossplane

# Add Crossplane Helm repository
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

# Install Crossplane
helm install crossplane \
  crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace \
  --wait

# Verify installation
kubectl get pods -n crossplane-system

Step 6: Verify cluster is ready

# Check all pods are running
kubectl get pods --all-namespaces

# Test cluster connectivity
kubectl get nodes

Option B: Using minikube

Step 1: Install minikube

# On Linux
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Verify installation
minikube version

Step 2: Start minikube

# Start cluster
minikube start --driver=docker

# Verify
minikube status
kubectl get nodes

Step 3: Install Crossplane (same as kind, Step 5 above)

Option C: Using Existing Kubernetes Cluster

Step 1: Verify cluster access

# Check current context
kubectl config current-context

# Verify connectivity
kubectl cluster-info
kubectl get nodes

Step 2: Install Crossplane (same as kind, Step 5 above)

Verification Checklist

• kind/minikube installed OR existing cluster accessible
• kubectl installed and configured
• Cluster created and nodes ready
• Crossplane installed in crossplane-system namespace
• All Crossplane pods running

---

Blocker 2: SSH Access to Proxmox Nodes

Required For

• TASK-012: Deploy Prometheus exporters
• TASK-013: Configure Cloudflare tunnels
• TASK-030: Generate Cloudflare tunnel credentials
• Image verification and download

Step-by-Step Instructions

Step 1: Generate SSH Key Pair (if not exists)

# Generate SSH key
ssh-keygen -t ed25519 -C "sankofa-proxmox" -f ~/.ssh/sankofa_proxmox

# Or use existing key
# Skip this step if you already have an SSH key

Step 2: Copy Public Key to ML110-01

Option A: Using ssh-copy-id

# Copy key to ML110-01
ssh-copy-id -i ~/.ssh/sankofa_proxmox.pub root@192.168.11.10

# Test connection
ssh -i ~/.ssh/sankofa_proxmox root@192.168.11.10 'echo "SSH working"'

Option B: Manual Copy

# Display public key
cat ~/.ssh/sankofa_proxmox.pub

# SSH to ML110-01 with password
ssh root@192.168.11.10

# On ML110-01, add key to authorized_keys
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "YOUR_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
exit

Step 3: Copy Public Key to R630-01

# Copy key to R630-01
ssh-copy-id -i ~/.ssh/sankofa_proxmox.pub root@192.168.11.11

# Test connection
ssh -i ~/.ssh/sankofa_proxmox root@192.168.11.11 'echo "SSH working"'

Step 4: Configure SSH Config (Optional but Recommended)

# Add to ~/.ssh/config
cat >> ~/.ssh/config <<EOF

Host ml110-01
    HostName 192.168.11.10
    User root
    IdentityFile ~/.ssh/sankofa_proxmox
    StrictHostKeyChecking no

Host r630-01
    HostName 192.168.11.11
    User root
    IdentityFile ~/.ssh/sankofa_proxmox
    StrictHostKeyChecking no
EOF

# Test connections
ssh ml110-01 'hostname'
ssh r630-01 'hostname'

Step 5: Update .env with SSH Key Path (Optional)

# Add to .env file
echo "SSH_KEY=~/.ssh/sankofa_proxmox" >> .env
echo "SSH_USER=root" >> .env

Verification Checklist

• SSH key pair generated
• Public key copied to ML110-01
• Public key copied to R630-01
• SSH connection works to ML110-01 (passwordless)
• SSH connection works to R630-01 (passwordless)
• SSH config file updated (optional)

---

Blocker 1: Kubernetes Cluster Setup (PRIORITY 3)

Can be done in parallel - Needed for provider deployment and VM creation

Required For

• TASK-015: Deploy test VMs via Crossplane
• All VM deployment tasks

Step-by-Step Instructions

Step 1: Verify Images via SSH

# Check ML110-01
ssh ml110-01 'pveam list local | grep ubuntu'

# Check R630-01
ssh r630-01 'pveam list local | grep ubuntu'

Step 2: Download Images if Missing

Option A: Using pveam (Proxmox Template Downloader)

# On ML110-01
ssh ml110-01 <<EOF
# List available templates
pveam available | grep ubuntu-22.04

# Download Ubuntu 22.04 template
pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz

# Verify download
pveam list local | grep ubuntu-22.04
EOF

# On R630-01
ssh r630-01 <<EOF
# Download Ubuntu 22.04 template
pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz

# Verify download
pveam list local | grep ubuntu-22.04
EOF

Option B: Download Cloud Image and Upload

# Download Ubuntu 22.04 Cloud Image
wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img

# Upload to ML110-01 via SCP
scp ubuntu-22.04-server-cloudimg-amd64.img ml110-01:/var/lib/vz/template/iso/

# Upload to R630-01 via SCP
scp ubuntu-22.04-server-cloudimg-amd64.img r630-01:/var/lib/vz/template/iso/

# On each node, rename if needed
ssh ml110-01 'mv /var/lib/vz/template/iso/ubuntu-22.04-server-cloudimg-amd64.img /var/lib/vz/template/iso/ubuntu-22.04-cloud.img'
ssh r630-01 'mv /var/lib/vz/template/iso/ubuntu-22.04-server-cloudimg-amd64.img /var/lib/vz/template/iso/ubuntu-22.04-cloud.img'

Option C: Using Proxmox Web UI

1. Log in to ML110-01: https://ml110-01.sankofa.nexus:8006
2. Go to: Datacenter → Storage → local → Content
3. Click Templates → Download
4. Search for: ubuntu-22.04-standard
5. Click Download
6. Repeat for R630-01

Step 3: Verify Image Names Match Manifests

# Check actual image names on ML110-01
ssh ml110-01 'pveam list local'

# Check actual image names on R630-01
ssh r630-01 'pveam list local'

# If image name differs from "ubuntu-22.04-cloud", update manifests:
# - crossplane-provider-proxmox/examples/test-vm-instance-1.yaml
# - crossplane-provider-proxmox/examples/test-vm-instance-2.yaml
# - crossplane-provider-proxmox/examples/vm-example.yaml

Step 4: Test Image (Optional)

# Create a test VM via Proxmox Web UI or API to verify image works
# This ensures the image is valid before using it with Crossplane

Verification Checklist

• Images verified on ML110-01
• Images verified on R630-01
• Image names match manifest references (ubuntu-22.04-cloud)
• Images accessible from storage pools
• (Optional) Test VM created successfully

---

Complete Deployment Sequence

Once all blockers are resolved, follow this sequence:

Phase 1: SSH Setup (Blocker 2) - DO THIS FIRST

1. ✅ Generate SSH keys
2. ✅ Copy keys to both Proxmox nodes
3. ✅ Test passwordless SSH
4. ✅ Configure SSH config

Phase 2: Image Preparation (Blocker 3) - DO THIS SECOND

1. ✅ Verify images exist
2. ✅ Download missing images
3. ✅ Verify image names match manifests
4. ✅ Test image (optional)

Phase 3: Kubernetes Setup (Blocker 1) - CAN BE DONE IN PARALLEL

1. ✅ Install kind/minikube or access existing cluster
2. ✅ Install kubectl
3. ✅ Create/verify cluster
4. ✅ Install Crossplane
5. ✅ Verify Crossplane pods running

Phase 2: SSH Setup (Blocker 2)

1. ✅ Generate SSH keys
2. ✅ Copy keys to both Proxmox nodes
3. ✅ Test passwordless SSH
4. ✅ Configure SSH config

Phase 3: Image Preparation (Blocker 3)

1. ✅ Verify images exist
2. ✅ Download missing images
3. ✅ Verify image names match manifests
4. ✅ Test image (optional)

Phase 4: Provider Deployment

1. Build Crossplane provider
2. Deploy provider to Kubernetes
3. Create ProviderConfig secret
4. Apply ProviderConfig
5. Verify provider connectivity

Phase 5: Test Deployment

1. Deploy test VM on ML110-01
2. Deploy test VM on R630-01
3. Verify VM lifecycle operations

Phase 6: Monitoring Setup

1. Deploy Prometheus exporters (via SSH)
2. Configure Grafana dashboards
3. Set up alerts

Phase 7: Cloudflare Tunnels

1. Generate tunnel credentials
2. Deploy tunnels to nodes (via SSH)
3. Verify tunnel connectivity

Quick Reference Commands

Kubernetes

# Create cluster
kind create cluster --name sankofa

# Install Crossplane
helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace

# Verify
kubectl get pods -n crossplane-system

SSH

# Test connections
ssh ml110-01 'hostname'
ssh r630-01 'hostname'

Images

# Check images
ssh ml110-01 'pveam list local | grep ubuntu'
ssh r630-01 'pveam list local | grep ubuntu'

# Download images
ssh ml110-01 'pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz'
ssh r630-01 'pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz'

Troubleshooting

Kubernetes Issues

• Cluster won't start: Check Docker is running (for kind/minikube)
• Crossplane won't install: Check cluster has sufficient resources
• Pods not running: Check kubectl describe pod for errors

SSH Issues

• Permission denied: Verify public key is in ~/.ssh/authorized_keys
• Connection timeout: Check firewall rules and network connectivity
• Host key verification failed: Add StrictHostKeyChecking no to SSH config

Image Issues

• Image not found: Verify storage pool name and image location
• Download fails: Check internet connectivity and Proxmox template repository
• Name mismatch: Update manifests or rename image files

Related Documentation

• Deployment Readiness Final
• Kubernetes Deployment Status
• Image Inventory
• Task List