loc_az_hci/docs/deployment/deployment-guide.md

Deployment Guide

Prerequisites

Before starting the deployment, ensure you have:

1. Two Proxmox VE hosts with:

   • Proxmox VE 7.0+ installed
   • Static IP addresses configured
   • At least 8GB RAM per node
   • Network connectivity between nodes
   • Root or sudo access

2. Azure Subscription with:

   • Azure CLI installed and authenticated
   • Contributor role on subscription
   • Resource group creation permissions

3. Network Requirements:

   • Static IP addresses for all nodes
   • DNS resolution (or hosts file)
   • Internet access for Azure Arc connectivity
   • NFS server (optional, for shared storage)

4. Tools Installed:

   • SSH client
   • kubectl
   • helm (optional)
   • terraform (optional)

5. Environment Configuration:

   • Copy .env.example to .env and fill in all credentials
   • See Configuration section for details

Configuration

Environment Variables Setup

Before starting deployment, configure your environment variables:

1. Copy the template:

   cp .env.example .env
   

2. Edit .env with your credentials:

   • Azure credentials: AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID
   • Cloudflare: CLOUDFLARE_API_TOKEN
   • Proxmox: PVE_ROOT_PASS (shared root password for all instances)
   • Proxmox ML110: PROXMOX_ML110_URL
   • Proxmox R630: PROXMOX_R630_URL

   Note: The username root@pam is implied and should not be stored. For production operations, use RBAC accounts and API tokens instead of root credentials.

3. Load environment variables:

   # Source the .env file
   export $(cat .env | grep -v '^#' | xargs)
   

Note: All scripts in this guide will use environment variables from .env if available. You can also set them manually using export commands.

Deployment Phases

Phase 1: Proxmox Cluster Setup

Step 1.1: Configure Network on Both Nodes

On each Proxmox node:

# Option 1: Use .env file (recommended)
# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

# Option 2: Set environment variables manually
export NODE_IP=192.168.1.10  # Use appropriate IP for each node
export NODE_GATEWAY=192.168.1.1
export NODE_NETMASK=24
export NODE_HOSTNAME=pve-node-1  # Use appropriate hostname

# Run network configuration script
cd /path/to/loc_az_hci
./infrastructure/proxmox/network-config.sh

For Node 2, repeat with appropriate values:

export NODE_IP=192.168.1.11
export NODE_HOSTNAME=pve-node-2
./infrastructure/proxmox/network-config.sh

Step 1.2: Update Proxmox Repositories

On both nodes:

# Update to subscription-free repos
sed -i 's/enterprise/no-subscription/g' /etc/apt/sources.list.d/pve-enterprise.list
apt update && apt dist-upgrade -y

Step 1.3: Configure Shared Storage (NFS)

Option A: Using existing NFS server

On both Proxmox nodes:

export NFS_SERVER=192.168.1.100
export NFS_PATH=/mnt/proxmox-storage
export STORAGE_NAME=nfs-shared

./infrastructure/proxmox/nfs-storage.sh

Option B: Set up NFS server

If you need to set up an NFS server, install and configure it on a separate machine or VM.

Step 1.4: Create Proxmox Cluster

On Node 1 (cluster creator):

export NODE_ROLE=create
export CLUSTER_NAME=hc-cluster

./infrastructure/proxmox/cluster-setup.sh

On Node 2 (join cluster):

export NODE_ROLE=join
export CLUSTER_NODE_IP=192.168.1.10  # IP of Node 1
export ROOT_PASSWORD=your-root-password  # Optional, will prompt if not set

./infrastructure/proxmox/cluster-setup.sh

Verify cluster:

pvecm status
pvecm nodes

Phase 2: Azure Arc Integration

Step 2.1: Prepare Azure Environment

# Load environment variables from .env (if using .env file)
export $(cat .env | grep -v '^#' | xargs)

# Login to Azure
az login

# Set subscription (use from .env or set manually)
az account set --subscription "${AZURE_SUBSCRIPTION_ID:-your-subscription-id}"

# Create resource group (if not exists)
az group create --name "${AZURE_RESOURCE_GROUP:-HC-Stack}" --location "${AZURE_LOCATION:-eastus}"

Step 2.2: Onboard Proxmox Hosts to Azure Arc

On each Proxmox node:

# Load environment variables from .env (if using .env file)
export $(cat .env | grep -v '^#' | xargs)

# Set Azure variables (use from .env or get from Azure CLI)
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"
export TAGS="type=proxmox,environment=hybrid"

./scripts/azure-arc/onboard-proxmox-hosts.sh

Verify in Azure Portal:

• Navigate to: Azure Portal → Azure Arc → Servers
• You should see both Proxmox nodes

Step 2.3: Create VMs for Kubernetes and Git

Create VMs in Proxmox web UI or using Terraform:

# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

cd terraform/proxmox
# Create terraform.tfvars from environment variables or edit manually
cat > terraform.tfvars <<EOF
proxmox_host     = "${PROXMOX_ML110_URL#https://}"
proxmox_username = "root@pam"  # Hardcoded, not from env (best practice)
proxmox_password = "${PVE_ROOT_PASS}"
proxmox_node     = "pve-node-1"
EOF

terraform init
terraform plan
terraform apply

Step 2.4: Onboard VMs to Azure Arc

For each VM:

# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export VM_IP=192.168.1.188
export VM_USER=ubuntu
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"

./scripts/azure-arc/onboard-vms.sh

Phase 3: Kubernetes Setup

Step 3.1: Install K3s

On the VM designated for Kubernetes:

export INSTALL_MODE=local
export K3S_VERSION=latest

./infrastructure/kubernetes/k3s-install.sh

Or install remotely:

export INSTALL_MODE=remote
export REMOTE_IP=192.168.1.188
export REMOTE_USER=ubuntu

./infrastructure/kubernetes/k3s-install.sh

Step 3.2: Onboard Kubernetes to Azure Arc

# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
export LOCATION="${AZURE_LOCATION:-eastus}"
export CLUSTER_NAME=proxmox-k3s-cluster

# Ensure kubeconfig is set
export KUBECONFIG=~/.kube/config

./infrastructure/kubernetes/arc-onboard-k8s.sh

Verify in Azure Portal:

• Navigate to: Azure Portal → Azure Arc → Kubernetes
• You should see your cluster

Step 3.3: Install Base Infrastructure

# Apply namespace and base infrastructure
kubectl apply -f gitops/infrastructure/namespace.yaml
kubectl apply -f gitops/infrastructure/ingress-controller.yaml
kubectl apply -f gitops/infrastructure/cert-manager.yaml

Phase 4: Git/DevOps Setup

Option A: Deploy Gitea (Recommended for small deployments)

export GITEA_DOMAIN=git.local
export GITEA_PORT=3000

./infrastructure/gitops/gitea-deploy.sh

Access Gitea at http://git.local:3000 and complete initial setup.

Option B: Deploy GitLab CE

export GITLAB_DOMAIN=gitlab.local
export GITLAB_PORT=8080

./infrastructure/gitops/gitlab-deploy.sh

Note: GitLab requires at least 8GB RAM.

Option C: Azure DevOps Self-Hosted Agent

On a VM:

# Load environment variables from .env
export $(cat .env | grep -v '^#' | xargs)

export AZP_URL="${AZP_URL:-https://dev.azure.com/yourorg}"
export AZP_TOKEN="${AZP_TOKEN:-your-personal-access-token}"
export AZP_AGENT_NAME=proxmox-agent-1
export AZP_POOL=Default

./infrastructure/gitops/azure-devops-agent.sh

Phase 5: Configure GitOps

Step 5.1: Create Git Repository

1. Create a new repository in your Git server (Gitea/GitLab)
2. Clone the repository locally
3. Copy the gitops/ directory to your repository
4. Commit and push:

git clone http://git.local:3000/user/gitops-repo.git
cd gitops-repo
cp -r /path/to/loc_az_hci/gitops/* .
git add .
git commit -m "Initial GitOps configuration"
git push

Step 5.2: Connect GitOps to Azure Arc

In Azure Portal:

1. Navigate to: Azure Arc → Kubernetes → Your cluster
2. Go to "GitOps" section
3. Click "Add configuration"
4. Configure:
   • Repository URL: http://git.local:3000/user/gitops-repo.git
   • Branch: main
   • Path: gitops/
   • Authentication: Configure as needed

Phase 6: Deploy HC Stack Services

Option A: Deploy via GitOps (Recommended)

1. Update Helm chart values in your Git repository
2. Commit and push changes
3. Flux will automatically deploy updates

Option B: Deploy Manually with Helm

# Add Helm charts
helm install besu ./gitops/apps/besu -n blockchain
helm install firefly ./gitops/apps/firefly -n blockchain
helm install chainlink-ccip ./gitops/apps/chainlink-ccip -n blockchain
helm install blockscout ./gitops/apps/blockscout -n blockchain
helm install cacti ./gitops/apps/cacti -n monitoring
helm install nginx-proxy ./gitops/apps/nginx-proxy -n hc-stack

Option C: Deploy with Terraform

cd terraform/kubernetes
terraform init
terraform plan
terraform apply

Phase 7: Verify Deployment

Check Proxmox Cluster

pvecm status
pvesm status

Check Azure Arc

# List Arc-enabled servers
az connectedmachine list --resource-group HC-Stack -o table

# List Arc-enabled Kubernetes clusters
az arc kubernetes list --resource-group HC-Stack -o table

Check Kubernetes

kubectl get nodes
kubectl get pods --all-namespaces
kubectl get services --all-namespaces

Check Applications

# Check Besu
kubectl get pods -n blockchain -l app=besu

# Check Firefly
kubectl get pods -n blockchain -l app=firefly

# Check all services
kubectl get all --all-namespaces

Post-Deployment Configuration

1. Configure Ingress

Update ingress configurations for external access:

# Edit ingress resources
kubectl edit ingress -n blockchain

2. Set Up Monitoring

• Configure Cacti to monitor your infrastructure
• Set up Azure Monitor alerts
• Configure log aggregation

3. Configure Backup

• Set up Proxmox backup schedules
• Configure Kubernetes backup (Velero)
• Set up Azure Backup for Arc resources

4. Security Hardening

• Enable Azure Policy for compliance
• Configure network policies
• Set up RBAC
• Enable Defender for Cloud

Troubleshooting

Common Issues

1. Cluster creation fails:

   • Check network connectivity between nodes
   • Verify firewall rules
   • Check Corosync configuration

2. Azure Arc connection fails:

   • Verify internet connectivity
   • Check Azure credentials
   • Review agent logs: journalctl -u azcmagent

3. Kubernetes pods not starting:

   • Check resource limits
   • Verify storage classes
   • Review pod logs: kubectl logs <pod-name>

4. GitOps not syncing:

   • Check Flux logs: kubectl logs -n flux-system -l app=flux
   • Verify repository access
   • Check GitOps configuration in Azure Portal

Next Steps

1. Review architecture documentation
2. Set up monitoring and alerting
3. Configure backup and disaster recovery
4. Implement security policies
5. Plan for scaling and expansion