Sankofa/crossplane-provider-proxmox/README.md

# Crossplane Provider for Proxmox

A custom Crossplane provider that enables provisioning and management of Proxmox VE resources through Kubernetes.

## Features

- **Virtual Machine Management**: Create, update, delete VMs
- **Storage Management**: Manage storage pools and volumes
- **Network Management**: Configure network bridges and interfaces
- **Multi-Site Support**: Manage multiple Proxmox clusters
- **Status Reporting**: Real-time VM status and IP addresses
- **Reconciliation**: Automatic drift detection and correction
- **Retry Logic**: Automatic retry for transient failures
- **Error Handling**: Comprehensive error handling and reporting

## Architecture

```
crossplane-provider-proxmox/
├── apis/                    # CRD API definitions
│   └── v1alpha1/           # API version
├── pkg/                     # Provider implementation
│   ├── controller/         # Crossplane controllers
│   ├── proxmox/            # Proxmox API client
│   └── managed/            # Managed resource types
├── config/                 # Deployment manifests
│   └── crd/                # CRD definitions
└── examples/               # Usage examples
```

## Installation

### Prerequisites

- Kubernetes cluster with Crossplane installed
- Proxmox VE cluster with API access
- Go 1.21+ for building

### Build and Install

```bash
# Build the provider
make build

# Install CRDs
kubectl apply -f config/crd/bases/

# Deploy the provider
kubectl apply -f config/provider.yaml

# Create ProviderConfig
kubectl apply -f examples/provider-config.yaml
```

## Configuration

### Module Path

**IMPORTANT**: Before building, update the module path in `go.mod`:

```go
module github.com/sankofa/crossplane-provider-proxmox
```

### Provider Configuration

```yaml
apiVersion: proxmox.sankofa.nexus/v1alpha1
kind: ProviderConfig
metadata:
  name: proxmox-provider-config
spec:
  credentials:
    source: Secret
    secretRef:
      name: proxmox-credentials
      namespace: crossplane-system
      key: credentials.json
  sites:
    - name: us-sfvalley
      endpoint: https://ml110-01.sankofa.nexus:8006
      node: ML110-01
    - name: eu-west-1
      endpoint: https://r630-01.sankofa.nexus:8006
      node: R630-01
```

### Create a Virtual Machine

```yaml
apiVersion: proxmox.sankofa.nexus/v1alpha1
kind: ProxmoxVM
metadata:
  name: web-server-01
spec:
  forProvider:
    node: pve1
    name: web-server-01
    cpu: 4
    memory: 8Gi
    disk: 100Gi
    storage: local-lvm
    network: vmbr0
    image: ubuntu-22.04-cloud
    site: us-sfvalley
  providerConfigRef:
    name: proxmox-provider-config
```

## API Reference

### ProxmoxVM

Manages a Proxmox virtual machine.

**Spec:**
- `node`: Proxmox node to deploy on (required)
- `name`: VM name (required, see validation rules below)
- `cpu`: Number of CPU cores (required, min: 1, max: 1024, default: 2)
- `memory`: Memory size (required, see validation rules below)
- `disk`: Disk size (required, see validation rules below)
- `storage`: Storage pool name (default: "local-lvm")
- `network`: Network bridge (default: "vmbr0", see validation rules below)
- `image`: OS template/image (required, see validation rules below)
- `site`: Site identifier (required, must match ProviderConfig)
- `userData`: Optional cloud-init user data in YAML format

**Status:**
- `vmId`: Proxmox VM ID
- `state`: VM state (running, stopped, etc.)
- `ipAddress`: VM IP address
- `conditions`: Resource conditions

### Validation Rules

The provider includes comprehensive input validation:

#### VM Name
- **Length**: 1-100 characters
- **Characters**: Alphanumeric, hyphen, underscore, dot, space
- **Restrictions**: Cannot start or end with spaces
- **Example**: `"web-server-01"`, `"vm.001"`, `"my vm"`

#### Memory
- **Format**: Supports `Gi`, `Mi`, `Ki`, `G`, `M`, `K` or plain numbers (assumed MB)
- **Range**: 128 MB - 2 TB
- **Case-insensitive**: `"4Gi"`, `"4gi"`, `"4GI"` all work
- **Examples**: `"4Gi"`, `"8192Mi"`, `"4096"`

#### Disk
- **Format**: Supports `Ti`, `Gi`, `Mi`, `T`, `G`, `M` or plain numbers (assumed GB)
- **Range**: 1 GB - 100 TB
- **Case-insensitive**: `"50Gi"`, `"50gi"`, `"50GI"` all work
- **Examples**: `"50Gi"`, `"1Ti"`, `"100"`

#### CPU
- **Range**: 1-1024 cores
- **Example**: `2`, `4`, `8`

#### Network Bridge
- **Format**: Alphanumeric, hyphen, underscore
- **Validation**: Bridge must exist on the target node (validated before VM creation)
- **Example**: `"vmbr0"`, `"custom-bridge"`, `"bridge_01"`

#### Image
Three formats are supported:
1. **Template VMID**: Numeric VMID (100-999999999) for template cloning
   - Example: `"100"`, `"1000"`
2. **Volume ID**: `storage:path/to/image` format
   - Example: `"local:iso/ubuntu-22.04.iso"`
3. **Image Name**: Named image in storage
   - Example: `"ubuntu-22.04-cloud"`
   - Maximum length: 255 characters

### Multi-Tenancy

The provider supports tenant isolation through tags:

- **Tenant ID**: Set via Kubernetes label `tenant-id` or `tenant.sankofa.nexus/id`
- **Tag Format**: `tenant_{id}` (underscore separator)
- **Filtering**: Use `ListVMs()` with tenant ID filter
- **Example**:
  ```yaml
  metadata:
    labels:
      tenant-id: "customer-123"
  # Results in Proxmox tag: tenant_customer-123
  ```

## Error Handling and Retry Logic

The provider includes comprehensive error handling and automatic retry logic:

### Error Categories

- **Network Errors**: Automatically retried with exponential backoff
  - Connection failures, timeouts, 502/503 errors
- **Authentication Errors**: Not retried (requires credential fix)
  - Invalid credentials, 401/403 errors
- **Configuration Errors**: Not retried (requires manual intervention)
  - Missing ProviderConfig, invalid site configuration
- **Quota Errors**: Not retried (requires resource adjustment)
  - Resource quota exceeded
- **API Not Supported**: Not retried
  - importdisk API not available (falls back to template cloning)

### Retry Configuration

- **Max Retries**: 3 (configurable)
- **Backoff**: Exponential with jitter
- **Max Delay**: 30 seconds
- **Retryable Errors**: Network, temporary failures

### Error Reporting

Errors are reported via Kubernetes Conditions:
- `ValidationFailed`: Input validation errors
- `ConfigurationError`: Configuration issues
- `NetworkError`: Network connectivity problems
- `AuthenticationError`: Authentication failures
- `QuotaExceeded`: Resource quota violations
- `NodeUnhealthy`: Node health check failures

## Development

### Building

```bash
go mod download
go build -o bin/provider ./cmd/provider
```

### Testing

#### Unit Tests

```bash
# Run all unit tests
go test ./pkg/...

# Run with coverage
go test -cover ./pkg/...
go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out

# Run specific package tests
go test ./pkg/utils/...
go test ./pkg/proxmox/...
go test ./pkg/controller/virtualmachine/...
```

#### Integration Tests

```bash
# Run integration tests (requires Proxmox test environment)
go test -tags=integration ./pkg/controller/virtualmachine/...

# Skip integration tests
go test -short ./pkg/...
```

See [docs/TESTING.md](docs/TESTING.md) for detailed testing guidelines.

### Running Locally

```bash
# Set up local development environment
export PROXMOX_ENDPOINT=https://pve1.local:8006
export PROXMOX_USERNAME=root@pam
export PROXMOX_PASSWORD=your-password

# Run the provider
./bin/provider
```

## Troubleshooting

### Common Issues

#### Validation Errors

**VM Name Invalid**
```
Error: VM name contains invalid characters
```
- **Solution**: Ensure VM name only contains alphanumeric, hyphen, underscore, dot, or space characters

**Memory/Disk Out of Range**
```
Error: memory 64Mi is below minimum of 128 MB
```
- **Solution**: Increase memory/disk values to meet minimum requirements (128 MB memory, 1 GB disk)

**Network Bridge Not Found**
```
Error: network bridge 'vmbr999' does not exist on node 'test-node'
```
- **Solution**: Verify network bridge exists using `pvesh get /nodes/{node}/network` or create the bridge

#### Authentication Errors

**401 Unauthorized**
- **Solution**: Verify credentials in ProviderConfig secret are correct
- Check API token format: `PVEAPIToken ${token}` (space, not equals sign)

#### Image Import Errors

**importdisk API Not Supported**
```
Error: importdisk API is not supported in this Proxmox version
```
- **Solution**: Use template cloning (numeric VMID) or pre-imported images instead
- Or upgrade Proxmox to version 6.0+

#### Network Errors

**Connection Timeout**
- **Solution**: Verify Proxmox API endpoint is accessible
- Check firewall rules and network connectivity

### Debugging

Enable verbose logging:

```bash
# Set log level
export LOG_LEVEL=debug

# Run provider with debug logging
./bin/provider --log-level=debug
```

Check VM status:

```bash
# Get VM details
kubectl get proxmoxvm <vm-name> -o yaml

# Check conditions
kubectl describe proxmoxvm <vm-name>

# View controller logs
kubectl logs -n crossplane-system -l app=provider-proxmox
```

## Additional Resources

- [Testing Guide](docs/TESTING.md) - Comprehensive testing documentation
- [API Examples](examples/) - Usage examples
- [Proxmox API Documentation](https://pve.proxmox.com/pve-docs/api-viewer/)

## License

Apache 2.0