loc_az_hci/docs/deployment/deployment-guide.md

# Deployment Guide

## Prerequisites

Before starting the deployment, ensure you have:

1. **Two Proxmox VE hosts** with:
   - Proxmox VE 7.0+ installed
   - Static IP addresses configured
   - At least 8GB RAM per node
   - Network connectivity between nodes
   - Root or sudo access

2. **Azure Subscription** with:
   - Azure CLI installed and authenticated
   - Contributor role on subscription
   - Resource group creation permissions

3. **Network Requirements**:
   - Static IP addresses for all nodes
   - DNS resolution (or hosts file)
   - Internet access for Azure Arc connectivity
   - NFS server (optional, for shared storage)

4. **Tools Installed**:
   - SSH client
   - kubectl
   - helm (optional)
   - terraform (optional)

5. **Environment Configuration**:
   - Copy `.env.example` to `.env` and fill in all credentials
   - See [Configuration](#configuration) section for details

## Configuration

### Environment Variables Setup

Before starting deployment, configure your environment variables:

1. **Copy the template:**
   ```bash
   cp .env.example .env
   ```

2. **Edit `.env` with your credentials:**
   - Azure credentials: `AZURE_SUBSCRIPTION_ID`, `AZURE_TENANT_ID`
   - Cloudflare: `CLOUDFLARE_API_TOKEN`
   - Proxmox: `PVE_ROOT_PASS` (shared root password for all instances)
   - Proxmox ML110: `PROXMOX_ML110_URL`
   - Proxmox R630: `PROXMOX_R630_URL`
   
   **Note**: The username `root@pam` is implied and should not be stored. For production operations, use RBAC accounts and API tokens instead of root credentials.

3. **Load environment variables:**
   ```bash
   # Source the .env file
   export $(cat .env | grep -v '^#' | xargs)
   ```

**Note**: All scripts in this guide will use environment variables from `.env` if available. You can also set them manually using `export` commands.

## Deployment Phases

### Phase 1: Proxmox Cluster Setup

#### Step 1.1: Configure Network on Both Nodes

On each Proxmox node:

```bash
# Option 1: Use .env file (recommended)
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

# Option 2: Set environment variables manually
export NODE_IP=192.168.1.10  # Use appropriate IP for each node
export NODE_GATEWAY=192.168.1.1
export NODE_NETMASK=24
export NODE_HOSTNAME=pve-node-1  # Use appropriate hostname

# Run network configuration script
cd /path/to/loc_az_hci
./infrastructure/proxmox/network-config.sh
```

**For Node 2**, repeat with appropriate values:
```bash
export NODE_IP=192.168.1.11
export NODE_HOSTNAME=pve-node-2
./infrastructure/proxmox/network-config.sh
```

#### Step 1.2: Update Proxmox Repositories

On both nodes:

```bash
# Update to subscription-free repos
sed -i 's/enterprise/no-subscription/g' /etc/apt/sources.list.d/pve-enterprise.list
apt update && apt dist-upgrade -y
```

#### Step 1.3: Configure Shared Storage (NFS)

**Option A: Using existing NFS server**

On both Proxmox nodes:

```bash
export NFS_SERVER=192.168.1.100
export NFS_PATH=/mnt/proxmox-storage
export STORAGE_NAME=nfs-shared

./infrastructure/proxmox/nfs-storage.sh
```

**Option B: Set up NFS server**

If you need to set up an NFS server, install and configure it on a separate machine or VM.

#### Step 1.4: Create Proxmox Cluster

**On Node 1** (cluster creator):

```bash
export NODE_ROLE=create
export CLUSTER_NAME=hc-cluster

./infrastructure/proxmox/cluster-setup.sh
```

**On Node 2** (join cluster):

```bash
export NODE_ROLE=join
export CLUSTER_NODE_IP=192.168.1.10  # IP of Node 1
export ROOT_PASSWORD=your-root-password  # Optional, will prompt if not set

./infrastructure/proxmox/cluster-setup.sh
```

**Verify cluster**:

```bash
pvecm status
pvecm nodes
```

### Phase 2: Azure Arc Integration

#### Step 2.1: Prepare Azure Environment

```bash
# Load environment variables from .env (if using .env file)
export $(cat .env | grep -v '^#' | xargs)

# Login to Azure
az login

# Set subscription (use from .env or set manually)
az account set --subscription "${AZURE_SUBSCRIPTION_ID:-your-subscription-id}"

# Create resource group (if not exists)
az group create --name "${AZURE_RESOURCE_GROUP:-HC-Stack}" --location "${AZURE_LOCATION:-eastus}"
```

#### Step 2.2: Onboard Proxmox Hosts to Azure Arc

On each Proxmox node:

```bash
# Load environment variables from .env (if using .env file)
export $(cat .env | grep -v '^#' | xargs)

# Set Azure variables (use from .env or get from Azure CLI)
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"
export TAGS="type=proxmox,environment=hybrid"

./scripts/azure-arc/onboard-proxmox-hosts.sh
```

**Verify in Azure Portal**:
- Navigate to: Azure Portal → Azure Arc → Servers
- You should see both Proxmox nodes

#### Step 2.3: Create VMs for Kubernetes and Git

Create VMs in Proxmox web UI or using Terraform:

```bash
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

cd terraform/proxmox
# Create terraform.tfvars from environment variables or edit manually
cat > terraform.tfvars <<EOF
proxmox_host     = "${PROXMOX_ML110_URL#https://}"
proxmox_username = "root@pam"  # Hardcoded, not from env (best practice)
proxmox_password = "${PVE_ROOT_PASS}"
proxmox_node     = "pve-node-1"
EOF

terraform init
terraform plan
terraform apply
```

#### Step 2.4: Onboard VMs to Azure Arc

For each VM:

```bash
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export VM_IP=192.168.1.188
export VM_USER=ubuntu
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"

./scripts/azure-arc/onboard-vms.sh
```

### Phase 3: Kubernetes Setup

#### Step 3.1: Install K3s

On the VM designated for Kubernetes:

```bash
export INSTALL_MODE=local
export K3S_VERSION=latest

./infrastructure/kubernetes/k3s-install.sh
```

**Or install remotely**:

```bash
export INSTALL_MODE=remote
export REMOTE_IP=192.168.1.188
export REMOTE_USER=ubuntu

./infrastructure/kubernetes/k3s-install.sh
```

#### Step 3.2: Onboard Kubernetes to Azure Arc

```bash
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"
export CLUSTER_NAME=proxmox-k3s-cluster

# Ensure kubeconfig is set
export KUBECONFIG=~/.kube/config

./infrastructure/kubernetes/arc-onboard-k8s.sh
```

**Verify in Azure Portal**:
- Navigate to: Azure Portal → Azure Arc → Kubernetes
- You should see your cluster

#### Step 3.3: Install Base Infrastructure

```bash
# Apply namespace and base infrastructure
kubectl apply -f gitops/infrastructure/namespace.yaml
kubectl apply -f gitops/infrastructure/ingress-controller.yaml
kubectl apply -f gitops/infrastructure/cert-manager.yaml
```

### Phase 4: Git/DevOps Setup

#### Option A: Deploy Gitea (Recommended for small deployments)

```bash
export GITEA_DOMAIN=git.local
export GITEA_PORT=3000

./infrastructure/gitops/gitea-deploy.sh
```

Access Gitea at `http://git.local:3000` and complete initial setup.

#### Option B: Deploy GitLab CE

```bash
export GITLAB_DOMAIN=gitlab.local
export GITLAB_PORT=8080

./infrastructure/gitops/gitlab-deploy.sh
```

**Note**: GitLab requires at least 8GB RAM.

#### Option C: Azure DevOps Self-Hosted Agent

On a VM:

```bash
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export AZP_URL="${AZP_URL:-https://dev.azure.com/yourorg}"
export AZP_TOKEN="${AZP_TOKEN:-your-personal-access-token}"
export AZP_AGENT_NAME=proxmox-agent-1
export AZP_POOL=Default

./infrastructure/gitops/azure-devops-agent.sh
```

### Phase 5: Configure GitOps

#### Step 5.1: Create Git Repository

1. Create a new repository in your Git server (Gitea/GitLab)
2. Clone the repository locally
3. Copy the `gitops/` directory to your repository
4. Commit and push:

```bash
git clone http://git.local:3000/user/gitops-repo.git
cd gitops-repo
cp -r /path/to/loc_az_hci/gitops/* .
git add .
git commit -m "Initial GitOps configuration"
git push
```

#### Step 5.2: Connect GitOps to Azure Arc

In Azure Portal:

1. Navigate to: Azure Arc → Kubernetes → Your cluster
2. Go to "GitOps" section
3. Click "Add configuration"
4. Configure:
   - Repository URL: `http://git.local:3000/user/gitops-repo.git`
   - Branch: `main`
   - Path: `gitops/`
   - Authentication: Configure as needed

### Phase 6: Deploy HC Stack Services

#### Option A: Deploy via GitOps (Recommended)

1. Update Helm chart values in your Git repository
2. Commit and push changes
3. Flux will automatically deploy updates

#### Option B: Deploy Manually with Helm

```bash
# Add Helm charts
helm install besu ./gitops/apps/besu -n blockchain
helm install firefly ./gitops/apps/firefly -n blockchain
helm install chainlink-ccip ./gitops/apps/chainlink-ccip -n blockchain
helm install blockscout ./gitops/apps/blockscout -n blockchain
helm install cacti ./gitops/apps/cacti -n monitoring
helm install nginx-proxy ./gitops/apps/nginx-proxy -n hc-stack
```

#### Option C: Deploy with Terraform

```bash
cd terraform/kubernetes
terraform init
terraform plan
terraform apply
```

### Phase 7: Verify Deployment

#### Check Proxmox Cluster

```bash
pvecm status
pvesm status
```

#### Check Azure Arc

```bash
# List Arc-enabled servers
az connectedmachine list --resource-group HC-Stack -o table

# List Arc-enabled Kubernetes clusters
az arc kubernetes list --resource-group HC-Stack -o table
```

#### Check Kubernetes

```bash
kubectl get nodes
kubectl get pods --all-namespaces
kubectl get services --all-namespaces
```

#### Check Applications

```bash
# Check Besu
kubectl get pods -n blockchain -l app=besu

# Check Firefly
kubectl get pods -n blockchain -l app=firefly

# Check all services
kubectl get all --all-namespaces
```

## Post-Deployment Configuration

### 1. Configure Ingress

Update ingress configurations for external access:

```bash
# Edit ingress resources
kubectl edit ingress -n blockchain
```

### 2. Set Up Monitoring

- Configure Cacti to monitor your infrastructure
- Set up Azure Monitor alerts
- Configure log aggregation

### 3. Configure Backup

- Set up Proxmox backup schedules
- Configure Kubernetes backup (Velero)
- Set up Azure Backup for Arc resources

### 4. Security Hardening

- Enable Azure Policy for compliance
- Configure network policies
- Set up RBAC
- Enable Defender for Cloud

## Troubleshooting

### Common Issues

1. **Cluster creation fails**:
   - Check network connectivity between nodes
   - Verify firewall rules
   - Check Corosync configuration

2. **Azure Arc connection fails**:
   - Verify internet connectivity
   - Check Azure credentials
   - Review agent logs: `journalctl -u azcmagent`

3. **Kubernetes pods not starting**:
   - Check resource limits
   - Verify storage classes
   - Review pod logs: `kubectl logs <pod-name>`

4. **GitOps not syncing**:
   - Check Flux logs: `kubectl logs -n flux-system -l app=flux`
   - Verify repository access
   - Check GitOps configuration in Azure Portal

## Next Steps

1. Review architecture documentation
2. Set up monitoring and alerting
3. Configure backup and disaster recovery
4. Implement security policies
5. Plan for scaling and expansion