smom-dbis-138/docs/deployment/DEPLOYMENT_CHECKLIST.md

Complete Deployment Checklist - Chain-138 Multi-Region Network

Current Status

• ✅ Cloud for Sovereignty foundation deployed (37 regions)
• ✅ Terraform configuration optimized (48 validators across 24 regions)
• ✅ Quota analysis complete (240 vCPUs available)
• ⏳ Infrastructure deployment pending
• ⏳ Besu network deployment pending
• ⏳ Contract deployment pending

Phase 1: Infrastructure Deployment

1.1 Verify Prerequisites

• Verify Azure subscription access
• Verify quota availability in all 24 regions
• Verify Terraform is installed and configured
• Verify Azure CLI is authenticated
• Verify .env file has all required variables

1.2 Deploy Cloud for Sovereignty Infrastructure

cd terraform/well-architected/cloud-sovereignty
terraform init
terraform plan -out=tfplan-240vpu
terraform apply tfplan-240vpu

• Deploy resource groups (24 regions)
• Deploy virtual networks (24 regions)
• Deploy Key Vaults (24 regions)
• Deploy Log Analytics workspaces (24 regions)
• Deploy storage accounts (24 regions)
• Verify all foundation resources deployed

1.3 Deploy AKS Clusters

• Deploy AKS clusters in 24 regions
• Verify system node pools (3 nodes per region = 72 total)
• Verify cluster connectivity
• Configure kubectl contexts for all regions
• Verify Azure Monitor integration

1.4 Deploy Validator Node Pools

• Deploy validator node pools (2 per region = 48 total)
• Verify validator nodes are running
• Verify node labels and taints
• Verify node connectivity

1.5 Verify Infrastructure

• Verify all 24 AKS clusters operational
• Verify all 72 system nodes running
• Verify all 48 validator nodes running
• Verify network connectivity between regions
• Verify quota usage (240 vCPUs total)

Phase 2: Kubernetes Configuration

2.1 Configure Namespaces

• Create besu-network namespace in all clusters
• Create monitoring namespace in all clusters
• Configure RBAC for namespaces

2.2 Configure Storage

• Create StorageClasses for persistent volumes
• Create PVCs for Besu data (validators)
• Verify storage provisioning

2.3 Configure Networking

• Configure Network Policies
• Configure LoadBalancers for RPC endpoints
• Configure Ingress controllers
• Verify cross-region connectivity

2.4 Configure Secrets

• Create Kubernetes secrets for validator keys
• Create secrets for CCIP credentials
• Create secrets for monitoring credentials
• Verify secrets are accessible

Phase 3: Besu Network Deployment

3.1 Generate Genesis File

• Generate genesis.json with 48 validators
• Configure IBFT 2.0 consensus parameters
• Add validator addresses to genesis
• Verify genesis file is valid

3.2 Generate Validator Keys

• Generate 48 validator key pairs
• Store keys securely (Key Vault)
• Create Kubernetes secrets for keys
• Verify key accessibility

3.3 Deploy Besu Validators

• Deploy Besu validator StatefulSets (48 validators)
• Configure validator pods with node selectors
• Verify validators are starting
• Verify validator connectivity

3.4 Deploy Besu Sentries (Pods)

• Deploy Besu sentry Deployments (24-48 pods)
• Configure sentry pods on system nodes
• Configure P2P networking
• Verify sentry connectivity

3.5 Configure Peering

• Create static-nodes.json for all validators
• Configure validator-to-sentry peering
• Configure sentry-to-sentry peering
• Verify P2P connections established

3.6 Start Consensus Network

• Verify all 48 validators are running
• Verify consensus is active
• Verify blocks are being produced
• Verify network synchronization

Phase 4: Smart Contract Deployment

4.1 Ethereum Mainnet Contracts

• Deploy CCIPLogger to Ethereum Mainnet
• Verify CCIPLogger deployment
• Update .env with CCIPLogger address
• Fund CCIPLogger with LINK tokens

4.2 Chain-138 Contracts

• Deploy CCIPTxReporter to Chain-138
• Verify CCIPTxReporter deployment
• Update .env with CCIPTxReporter address
• Fund CCIPTxReporter with native tokens

4.3 Bridge Contracts

• Deploy CCIPWETH9Bridge to Ethereum Mainnet
• Deploy CCIPWETH9Bridge to Chain-138
• Deploy CCIPWETH10Bridge to Ethereum Mainnet
• Deploy CCIPWETH10Bridge to Chain-138
• Verify all bridge deployments
• Update .env with bridge addresses

4.4 Configure Bridges

• Configure WETH9 bridge destinations
• Configure WETH10 bridge destinations
• Enable bridge destinations
• Verify bridge configuration

Phase 5: CCIP Integration

5.1 Configure CCIP Routers

• Identify Ethereum Mainnet CCIP Router address
• Identify Chain-138 CCIP Router address
• Update .env with router addresses
• Verify router connectivity

5.2 Configure Chain Selectors

• Get Ethereum Mainnet chain selector
• Get Chain-138 chain selector
• Update .env with chain selectors
• Verify chain selector configuration

5.3 Fund CCIP Contracts

• Fund CCIPLogger with LINK tokens
• Fund CCIPTxReporter with native tokens
• Fund bridges with LINK tokens
• Verify sufficient funding

5.4 Test CCIP Integration

• Test CCIP message sending (Chain-138 → Ethereum)
• Test CCIP message receiving (Ethereum → Chain-138)
• Verify message delivery
• Verify message verification

Phase 6: Monitoring & Observability

6.1 Deploy Monitoring Stack

• Deploy Prometheus to all regions
• Deploy Grafana to all regions
• Configure Prometheus scraping
• Configure Grafana dashboards

6.2 Configure Alerts

• Configure Azure Monitor alerts
• Configure Prometheus alerts
• Configure Grafana alerts
• Test alert delivery

6.3 Configure Logging

• Configure Log Analytics integration
• Configure log aggregation
• Configure log retention
• Verify log accessibility

6.4 Create Dashboards

• Create validator status dashboard
• Create network health dashboard
• Create CCIP message tracking dashboard
• Create cost monitoring dashboard

Phase 7: Testing & Verification

7.1 Network Testing

• Test validator consensus
• Test block production
• Test network synchronization
• Test cross-region connectivity

7.2 Contract Testing

• Test WETH9 cross-chain transfers
• Test WETH10 cross-chain transfers
• Test CCIP message delivery
• Test bridge functionality

7.3 Performance Testing

• Test transaction throughput
• Test block time consistency
• Test network latency
• Test CCIP message latency

7.4 Security Testing

• Test validator failover
• Test network partition handling
• Test consensus under load
• Test CCIP security

Phase 8: Documentation & Handoff

8.1 Update Documentation

• Update deployment procedures
• Update configuration guides
• Update operational runbooks
• Update troubleshooting guides

8.2 Create Runbooks

• Create validator maintenance runbook
• Create network troubleshooting runbook
• Create CCIP troubleshooting runbook
• Create disaster recovery runbook

8.3 Training

• Train operations team
• Document operational procedures
• Create knowledge base
• Schedule regular reviews

Phase 9: Production Readiness

9.1 Security Review

• Complete security audit
• Review access controls
• Review network security
• Review contract security

9.2 Performance Optimization

• Optimize validator performance
• Optimize network performance
• Optimize CCIP performance
• Optimize cost

9.3 Backup & Recovery

• Configure backup procedures
• Test disaster recovery
• Document recovery procedures
• Schedule regular backups

9.4 Go-Live Checklist

• All validators operational
• All contracts deployed
• All monitoring configured
• All documentation complete
• Team trained
• Security reviewed
• Performance tested
• Backup procedures tested

Quick Start Commands

Deploy Infrastructure

cd terraform/well-architected/cloud-sovereignty
terraform init
terraform plan -out=tfplan
terraform apply tfplan

Verify Deployment

# Check all clusters
for region in northeurope uksouth francecentral; do
  az aks get-credentials --resource-group az-p-${region}-rg-comp-001 --name az-p-${region}-aks-main
  kubectl get nodes
done

Deploy Besu Network

# Deploy validators
kubectl apply -f k8s/besu/validators/ -n besu-network

# Deploy sentries
kubectl apply -f k8s/besu/sentries/ -n besu-network

Deploy Contracts

# Deploy to Ethereum Mainnet
cd scripts/deployment
./deploy-ccip-logger-mainnet.sh

# Deploy to Chain-138
./deploy-ccip-reporter-chain138.sh

Estimated Timeline

• Phase 1 (Infrastructure): 2-4 hours
• Phase 2 (Kubernetes): 1-2 hours
• Phase 3 (Besu Network): 2-3 hours
• Phase 4 (Contracts): 1-2 hours
• Phase 5 (CCIP): 1-2 hours
• Phase 6 (Monitoring): 1-2 hours
• Phase 7 (Testing): 2-4 hours
• Phase 8 (Documentation): 2-3 hours
• Phase 9 (Production): 2-4 hours

Total Estimated Time: 14-26 hours

Critical Dependencies

1. Quota Availability: All 24 regions must have 10+ vCPUs available
2. Network Connectivity: Cross-region connectivity required
3. Validator Keys: 48 validator key pairs must be generated securely
4. CCIP Funding: LINK tokens required for CCIP operations
5. Contract Deployment: ETH required for Mainnet deployments

Risk Mitigation

1. Quota Issues: Monitor quota usage, request increases if needed
2. Network Issues: Test connectivity before deployment
3. Key Management: Use Azure Key Vault for secure key storage
4. Funding: Ensure sufficient LINK and ETH before deployment
5. Rollback Plan: Document rollback procedures for each phase