Sankofa/docs/vm/VM_CREATION_PROCEDURE.md

# VM Creation Procedure - Complete Guide

**Last Updated**: 2025-12-11  
**Status**: ✅ Complete with Guest Agent Configuration

---

## Overview

This document provides step-by-step procedures for creating VMs in the Sankofa Phoenix infrastructure using Crossplane and Proxmox. All procedures ensure proper QEMU Guest Agent configuration.

---

## Prerequisites

1. **Crossplane Provider Installed**
   - Provider configured and connected to Proxmox
   - Provider config secret created

2. **Proxmox Access**
   - Valid credentials in `.env` file
   - Network access to Proxmox nodes

3. **Kubernetes Access**
   - `kubectl` configured
   - Access to target namespace

---

## Method 1: Using Production Templates (Recommended)

### Step 1: Choose Template

Available templates in `examples/production/`:
- `basic-vm.yaml` - 2 CPU, 4Gi RAM, 50Gi disk
- `medium-vm.yaml` - 4 CPU, 8Gi RAM, 100Gi disk
- `large-vm.yaml` - 8 CPU, 16Gi RAM, 200Gi disk

### Step 2: Customize Template

**Required changes:**
- `metadata.name` - Unique VM name
- `spec.forProvider.name` - VM name in Proxmox
- `spec.forProvider.node` - Proxmox node (ml110-01 or r630-01)
- `spec.forProvider.site` - Site identifier
- `userData.users[0].ssh_authorized_keys` - Your SSH public key

**Example:**
```yaml
apiVersion: proxmox.sankofa.nexus/v1alpha1
kind: ProxmoxVM
metadata:
  name: my-vm-001
  namespace: default
spec:
  forProvider:
    node: "ml110-01"
    name: "my-vm-001"
    cpu: 2
    memory: "4Gi"
    disk: "50Gi"
    # ... rest of config
    userData: |
      #cloud-config
      users:
        - name: admin
          ssh_authorized_keys:
            - ssh-rsa YOUR_PUBLIC_KEY_HERE
      # ... rest of cloud-init
```

### Step 3: Apply Template

```bash
kubectl apply -f examples/production/basic-vm.yaml
```

### Step 4: Monitor Creation

```bash
# Watch VM status
kubectl get proxmoxvm -w

# Check events
kubectl describe proxmoxvm <vm-name>

# Check logs (if provider logs available)
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox
```

### Step 5: Verify Guest Agent

**Wait for cloud-init (1-2 minutes), then:**

```bash
# On Proxmox node
ssh root@<proxmox-node>

# Get VMID
VMID=$(qm list | grep <vm-name> | awk '{print $1}')

# Check guest agent
qm config $VMID | grep agent
qm guest exec $VMID -- systemctl status qemu-guest-agent
```

---

## Method 2: Using Crossplane Examples

### Step 1: Use Example Template

```bash
# Copy example
cp crossplane-provider-proxmox/examples/vm-example.yaml my-vm.yaml

# Edit and customize
vim my-vm.yaml
```

### Step 2: Apply

```bash
kubectl apply -f my-vm.yaml
```

### Step 3: Verify

Same as Method 1, Step 5.

---

## Method 3: Using GitOps Templates

### Step 1: Use Template

Templates in `gitops/templates/vm/`:
- `ubuntu-22.04.yaml`
- `ubuntu-20.04.yaml`
- `debian-12.yaml`

### Step 2: Render with Values

**Create values file:**
```yaml
name: my-vm
namespace: default
node: ml110-01
cpu: 2
memory: 4Gi
disk: 50Gi
site: us-sfvalley
```

**Render template:**
```bash
# Using helm or similar tool
helm template my-vm gitops/templates/vm/ubuntu-22.04.yaml -f values.yaml
```

### Step 3: Apply

```bash
kubectl apply -f rendered-template.yaml
```

---

## Guest Agent Configuration

### Automatic Configuration

✅ **All templates include:**
- `qemu-guest-agent` package in cloud-init
- Service enablement and startup
- Verification with retry logic
- Error handling and automatic installation

✅ **Crossplane provider automatically:**
- Sets `agent: 1` in Proxmox VM config
- Enables guest agent communication channel

### Manual Verification

**After VM creation (wait 1-2 minutes for cloud-init):**

```bash
# On Proxmox node
VMID=<vm-id>

# Check Proxmox config
qm config $VMID | grep agent
# Expected: agent: 1

# Check package
qm guest exec $VMID -- dpkg -l | grep qemu-guest-agent
# Expected: Package listed as installed

# Check service
qm guest exec $VMID -- systemctl status qemu-guest-agent
# Expected: active (running)
```

### Manual Fix (if needed)

**If guest agent not working:**

```bash
# 1. Enable in Proxmox
qm set $VMID --agent 1

# 2. Install in guest (via console/SSH)
apt-get update
apt-get install -y qemu-guest-agent
systemctl enable qemu-guest-agent
systemctl start qemu-guest-agent

# 3. Restart VM
qm shutdown $VMID  # Graceful
# OR
qm stop $VMID && qm start $VMID  # Force
```

---

## Post-Creation Checklist

- [ ] VM created successfully
- [ ] VM is running (`qm status <VMID>`)
- [ ] Guest agent enabled in Proxmox (`agent: 1`)
- [ ] Guest agent package installed
- [ ] Guest agent service running
- [ ] Cloud-init completed (check `/var/log/cloud-init-output.log`)
- [ ] SSH access working
- [ ] Network connectivity verified
- [ ] Time synchronization working (NTP)
- [ ] Security updates configured

---

## Troubleshooting

### VM Not Created

**Check:**
```bash
# Provider status
kubectl get providerconfig

# VM resource status
kubectl describe proxmoxvm <vm-name>

# Provider logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox
```

**Common issues:**
- Provider not connected
- Invalid credentials
- Resource quota exceeded
- Network connectivity issues

### VM Created But Not Starting

**Check:**
```bash
# VM status
qm status <VMID>

# VM config
qm config <VMID>

# Boot order
qm config <VMID> | grep boot

# Disk
qm config <VMID> | grep disk
```

**Common issues:**
- Missing boot disk
- Incorrect boot order
- Disk not imported
- Network configuration issues

### Guest Agent Not Working

**See:** `docs/GUEST_AGENT_COMPLETE_PROCEDURE.md`

**Quick fix:**
```bash
# Enable in Proxmox
qm set <VMID> --agent 1

# Install in guest
qm guest exec <VMID> -- apt-get install -y qemu-guest-agent
qm guest exec <VMID> -- systemctl start qemu-guest-agent

# Restart VM
qm shutdown <VMID>
```

---

## Best Practices

### 1. Always Use Templates

- Use production templates from `examples/production/`
- Templates include all required configurations
- Ensures consistency across VMs

### 2. Include Guest Agent

- All templates include guest agent configuration
- Verify after creation
- Monitor service status

### 3. Use Proper Naming

- Follow naming conventions
- Include environment/tenant identifiers
- Use descriptive names

### 4. Configure SSH Keys

- Always include SSH public keys in cloud-init
- Use `ssh_authorized_keys` in userData
- Disable password authentication

### 5. Monitor Resources

- Check resource quotas before creation
- Monitor disk usage
- Set appropriate resource limits

### 6. Document Exceptions

- Document any custom configurations
- Note any deviations from templates
- Record troubleshooting steps

---

## Related Documents

- `docs/GUEST_AGENT_COMPLETE_PROCEDURE.md` - Guest agent setup
- `docs/VM_100_GUEST_AGENT_FIXED.md` - Specific VM troubleshooting
- `examples/production/` - Production templates
- `crossplane-provider-proxmox/examples/` - Provider examples

---

## Quick Reference

**Create VM:**
```bash
kubectl apply -f examples/production/basic-vm.yaml
```

**Check status:**
```bash
kubectl get proxmoxvm
qm list
```

**Verify guest agent:**
```bash
qm config <VMID> | grep agent
qm guest exec <VMID> -- systemctl status qemu-guest-agent
```

**Access VM:**
```bash
# Get IP
qm guest exec <VMID> -- hostname -I

# SSH
ssh admin@<vm-ip>
```

---

**Last Updated**: 2025-12-11