Sankofa/scripts/README.md

# Installation Scripts

Automated installation scripts for deploying the hybrid cloud control plane.

## Structure

```
scripts/
├── bootstrap-cluster.sh          # Kubernetes cluster bootstrap
├── install-components.sh        # Control plane components installation
├── setup-proxmox-agents.sh      # Proxmox site agent setup
├── configure-cloudflare.sh      # Cloudflare tunnel configuration
├── validate.sh                  # Post-install validation
├── enable-guest-agent-existing-vms.sh  # Enable guest agent on all VMs
├── verify-guest-agent.sh        # Verify guest agent status on all VMs
└── ansible/                     # Ansible playbooks
    ├── site-playbook.yml        # Multi-site deployment
    ├── inventory.example        # Inventory template
    └── roles/                   # Ansible roles
```

## Usage

### Quick Start

```bash
# 1. Bootstrap Kubernetes cluster
./bootstrap-cluster.sh

# 2. Install control plane components
./install-components.sh

# 3. Setup Proxmox agents (run on each Proxmox node)
./setup-proxmox-agents.sh --site us-sfvalley --node ML110-01

# 4. Configure Cloudflare tunnels
./configure-cloudflare.sh

# 5. Validate installation
./validate.sh
```

### Ansible Deployment

For multi-site deployments, use Ansible:

```bash
cd ansible
cp inventory.example inventory
# Edit inventory with your hosts
ansible-playbook -i inventory site-playbook.yml
```

## Prerequisites

- Linux-based systems (Ubuntu 22.04+, RHEL 8+, Debian 11+)
- Root or sudo access
- Internet connectivity
- Kubernetes cluster (for component installation)
- Proxmox VE 8+ (for agent setup)
- Cloudflare account (for tunnel configuration)

## Script Details

### bootstrap-cluster.sh
Installs and configures Kubernetes cluster (RKE2 or k3s):
- System preparation
- Container runtime installation
- Kubernetes installation
- Network plugin configuration
- Storage class setup

### install-components.sh
Installs all control plane components:
- ArgoCD
- Rancher
- Crossplane
- Vault
- Monitoring stack
- Portal

### setup-proxmox-agents.sh
Configures Proxmox nodes:
- cloudflared installation
- Prometheus exporter installation

### enable-guest-agent-existing-vms.sh
Enables QEMU guest agent on all existing VMs:
- Automatically discovers all nodes on each Proxmox site
- Discovers all VMs on each node
- Checks if guest agent is already enabled
- Enables guest agent on VMs that need it
- Provides summary statistics

**Usage:**
```bash
./scripts/enable-guest-agent-existing-vms.sh
```

**Features:**
- Dynamic node and VM discovery (no hardcoded VMIDs)
- Supports API token and password authentication
- Skips VMs that already have guest agent enabled
- Provides detailed progress and summary reports

### verify-guest-agent.sh
Verifies guest agent status on all VMs:
- Lists all VMs with their guest agent status
- Shows which VMs have guest agent enabled/disabled
- Provides per-node and per-site summaries

**Usage:**
```bash
./scripts/verify-guest-agent.sh
```

**Note:** New VMs created with the updated Crossplane provider automatically have guest agent enabled in Proxmox configuration (`agent=1`). The guest agent package is also automatically installed via cloud-init userData.

### setup-dns-records.sh
Creates DNS records for Proxmox instances via Cloudflare API:
- A records for primary FQDNs
- CNAME records for API and metrics endpoints
- Automated record creation and verification

### create-proxmox-secret.sh
Creates Kubernetes secrets for Proxmox credentials:
- Interactive credential input
- Secret creation in crossplane-system namespace
- Verification of secret creation

### verify-provider-deployment.sh
Verifies Crossplane provider deployment:
- CRD existence check
- Provider deployment status
- Pod health and logs
- ProviderConfig status
- Credentials secret verification

### test-proxmox-connectivity.sh
Tests Proxmox instance connectivity:
- DNS resolution testing
- HTTP connectivity testing
- Authentication testing (with credentials)
- Version information retrieval

### deploy-crossplane-provider.sh
Automated deployment of Crossplane provider:
- Builds provider (optional)
- Installs CRDs
- Deploys provider to Kubernetes
- Verifies deployment status

### deploy-test-vms.sh
Deploys test VMs to both Proxmox instances:
- Deploys VM to Instance 1 (ML110-01)
- Deploys VM to Instance 2 (R630-01)
- Waits for VM creation
- Displays VM status

### setup-monitoring.sh
Sets up Prometheus and Grafana for Proxmox:
- Creates ServiceMonitor for Prometheus
- Configures scrape targets
- Creates alert rules
- Imports Grafana dashboards

### quick-deploy.sh
Interactive quick deployment script:
- Guides through all deployment steps
- Runs all deployment scripts in sequence
- Interactive prompts for each step
- Custom agent installation
- Service configuration

### configure-cloudflare.sh
Sets up Cloudflare tunnels:
- Tunnel creation
- Configuration deployment
- Service startup
- Health checks

### validate.sh
Validates installation:
- Component health checks
- API connectivity tests
- Resource availability
- Network connectivity