smom-dbis-138/docs/deployment/DEPLOYMENT_ORDER.md

Deployment Order - Complete Task List

This document defines the proper order for deploying the DeFi Oracle Meta Mainnet (ChainID 138) infrastructure and services.

📋 Deployment Phases

The deployment is organized into 8 phases, each building on the previous:

1. Prerequisites & Setup - Environment and tooling
2. Foundation - Core Azure infrastructure
3. Networking - Network infrastructure and security
4. Compute - AKS cluster and node pools
5. Storage & Secrets - Storage accounts and Key Vault
6. Application - Kubernetes workloads
7. External Services - DNS, SSL, and monitoring
8. Contracts & Integration - Smart contracts and external integrations

---

Phase 1: Prerequisites & Setup

1.1 Azure Authentication & Configuration

• Install Azure CLI
• Login to Azure (az login)
• Verify subscription access
• Set default subscription
• Verify Azure CLI version

1.2 Environment Configuration

• Create .env file
• Set AZURE_SUBSCRIPTION_ID
• Set AZURE_TENANT_ID
• Set AZURE_LOCATION=westeurope
• Set AZURE_RESOURCE_GROUP (or use default)
• Set CLOUDFLARE_ZONE_ID
• Set CLOUDFLARE_API_TOKEN
• Verify environment variables

1.3 Prerequisites Verification

• Run ./scripts/azure/check-azure-prerequisites.sh
• Verify resource providers are registered
• Check quotas for westeurope region
• Verify Terraform backend storage account exists
• Verify all required tools are installed (terraform, kubectl, helm, forge)

1.4 Key Generation

• Generate validator keys (./scripts/key-management/generate-validator-keys.sh 4)
• Generate oracle keys (./scripts/key-management/generate-oracle-keys.sh)
• Generate genesis file (./scripts/generate-genesis.sh)
• Verify keys are generated correctly

---

Phase 2: Foundation Infrastructure

2.1 Terraform Initialization

• Navigate to terraform/ directory
• Initialize Terraform (terraform init)
• Verify backend configuration
• Verify Terraform version (>= 1.0)

2.2 Terraform Configuration

• Copy terraform.tfvars.example to terraform.tfvars
• Set environment = "prod"
• Set location = "westeurope"
• Set cluster_name (following naming convention)
• Configure node counts and VM sizes
• Review and adjust tags

2.3 Resource Groups

• Create network resource group (az-p-we-rg-net-001)
• Create compute resource group (az-p-we-rg-comp-001)
• Create storage resource group (az-p-we-rg-stor-001)
• Create security resource group (az-p-we-rg-sec-001)
• Verify resource groups created

2.4 Terraform Planning

• Run terraform plan
• Review planned resources
• Verify naming convention compliance
• Check for any errors or warnings
• Save plan output for review

---

Phase 3: Networking Infrastructure

3.1 Virtual Network

• Deploy virtual network (az-p-we-vnet-main)
• Configure address space (10.0.0.0/16)
• Verify VNet created

3.2 Subnets

• Create AKS subnet (az-p-we-snet-aks)
• Create validator subnet (az-p-we-snet-valid)
• Create sentry subnet (az-p-we-snet-sent)
• Create RPC subnet (az-p-we-snet-rpc)
• Create Application Gateway subnet (az-p-we-snet-agw)
• Configure service endpoints where needed
• Verify all subnets created

3.3 Network Security Groups

• Create validator NSG (az-p-we-nsg-valid)
• Create sentry NSG (az-p-we-nsg-sent)
• Create RPC NSG (az-p-we-nsg-rpc)
• Configure NSG rules (allow/deny)
• Associate NSGs with subnets
• Verify NSG rules

3.4 Public IPs and Load Balancers

• Create Application Gateway public IP (az-p-we-pip-agw)
• Verify public IP created
• Note public IP address for DNS configuration

---

Phase 4: Compute Infrastructure

4.1 Key Vault Setup

• Create Key Vault (az-p-we-kv-secrets-001)
• Configure Key Vault access policies or RBAC
• Enable soft delete and purge protection
• Store validator keys in Key Vault
• Store oracle keys in Key Vault
• Verify Key Vault access

4.2 Log Analytics Workspace

• Create Log Analytics workspace (az-p-we-law-main)
• Configure retention period (90 days for prod)
• Verify workspace created

4.3 AKS Cluster

• Deploy AKS cluster (az-p-we-aks-main)
• Configure network plugin (Azure CNI)
• Configure network policy (Azure)
• Enable Azure Monitor
• Enable Azure Policy
• Configure Key Vault secrets provider
• Verify cluster is running

4.4 Node Pools

• Verify system node pool is created
• Create validator node pool (az-p-we-aks-node-valid)
• Create sentry node pool (az-p-we-aks-node-sent)
• Create RPC node pool (az-p-we-aks-node-rpc)
• Configure node labels and taints
• Verify all node pools are running

4.5 kubectl Configuration

• Get AKS credentials (az aks get-credentials)
• Verify kubectl access
• Test kubectl connection
• Verify node access

---

Phase 5: Storage & Secrets

5.1 Storage Accounts

• Create backup storage account (az-p-we-st-backup-001)
• Create shared storage account (az-p-we-st-shared-001)
• Configure storage account security
• Enable versioning and soft delete
• Verify storage accounts

5.2 Storage Containers

• Create chaindata container
• Create config container
• Configure container access policies
• Verify containers

5.3 Key Vault Secrets

• Store all validator private keys
• Store oracle private key
• Store database passwords
• Store API keys
• Verify secrets are accessible from AKS

---

Phase 6: Application Deployment

6.1 Kubernetes Namespace

• Create besu-network namespace
• Create monitoring namespace
• Configure namespace labels
• Verify namespaces

6.2 ConfigMaps and Secrets

• Create genesis config map
• Create static-nodes config map
• Create application config maps
• Create Kubernetes secrets from Key Vault
• Verify ConfigMaps and secrets

6.3 Validator Deployment

• Deploy validator StatefulSet
• Configure validator pods
• Verify validators are running
• Check validator logs
• Verify validators are syncing

6.4 Sentry Deployment

• Deploy sentry StatefulSet
• Configure sentry pods
• Verify sentries are running
• Check sentry logs
• Verify P2P connectivity

6.5 RPC Node Deployment

• Deploy RPC StatefulSet
• Configure RPC pods
• Verify RPC nodes are running
• Check RPC node logs
• Test RPC endpoint locally

6.6 Application Gateway

• Deploy Application Gateway (az-p-we-agw-main)
• Configure backend pools
• Configure HTTP settings
• Configure listeners
• Configure routing rules
• Configure WAF rules
• Verify Application Gateway is running

---

Phase 7: External Services

7.1 DNS Configuration

• Get Application Gateway public IP
• Configure Cloudflare DNS records:
  • A record for root domain (d-bis.org)
  • A record for www.d-bis.org
  • A record for rpc.d-bis.org
  • A record for rpc2.d-bis.org
  • A record for explorer.d-bis.org
• Wait for DNS propagation (5-15 minutes)
• Verify DNS resolution

7.2 SSL/TLS Configuration

• Enable Cloudflare SSL/TLS (Full or Full Strict)
• Verify SSL certificates
• Test HTTPS access
• Configure certificate auto-renewal

7.3 Monitoring Setup

• Deploy Prometheus
• Deploy Grafana (optional)
• Configure alert rules
• Set up alert notifications
• Verify monitoring is collecting metrics

7.4 Blockscout Deployment

• Deploy PostgreSQL database for Blockscout
• Wait for database to be ready
• Deploy Blockscout application
• Run database migrations
• Configure Blockscout settings
• Verify Blockscout is accessible
• Configure CORS headers

---

Phase 8: Contracts & Integration

8.1 Contract Deployment Preparation

• Set RPC_URL in .env
• Set PRIVATE_KEY in .env (deployment key)
• Verify RPC endpoint is accessible
• Test RPC connection

8.2 Smart Contract Deployment

• Deploy WETH contract
• Deploy Multicall contract
• Deploy Oracle Aggregator contract
• Deploy CCIP Router contract (optional)
• Verify all contracts deployed
• Save contract addresses

8.3 Token List Update

• Update token list with contract addresses
• Add token metadata
• Validate token list JSON
• Commit token list changes

8.4 Deployment Verification

• Run deployment verification script
• Test RPC endpoints (public)
• Test Blockscout explorer
• Test contract interactions
• Verify block production
• Check validator health
• Generate verification report

8.5 External Integration (Post-Deployment)

• Submit Ethereum-Lists PR
• Submit token list to CoinGecko
• Submit token list to Uniswap
• Verify MetaMask integration
• Test token auto-detection

---

Quick Reference Commands

Phase 1: Prerequisites

./scripts/deployment/azure-login.sh
./scripts/deployment/populate-env.sh
./scripts/azure/check-azure-prerequisites.sh
./scripts/key-management/generate-validator-keys.sh 4

Phase 2-4: Infrastructure

cd terraform
terraform init
terraform plan
terraform apply

Phase 5: Storage & Secrets

./scripts/key-management/azure-keyvault-setup.sh

Phase 6: Kubernetes

az aks get-credentials --resource-group az-p-we-rg-comp-001 --name az-p-we-aks-main
kubectl apply -f k8s/base/namespace.yaml
helm install besu-validators ./helm/besu-network -f helm/besu-network/values-validators.yaml -n besu-network

Phase 7: External Services

./scripts/deployment/get-app-gateway-ip.sh
./scripts/deployment/cloudflare-dns.sh --zone-id $CLOUDFLARE_ZONE_ID --api-token $CLOUDFLARE_API_TOKEN --ip <gateway-ip>

Phase 8: Contracts

./scripts/deployment/deploy-weth.sh
./scripts/deployment/deploy-multicall.sh
./scripts/deployment/verify-deployment.sh

Verification Checklist

After each phase, verify:

• All resources created successfully
• Naming convention followed
• Tags applied correctly
• No errors in logs
• Resources accessible
• Security configured properly

Rollback Procedures

If deployment fails at any phase:

1. Document the failure point
2. Review error logs
3. Fix the issue
4. Rollback if necessary (terraform destroy for failed resources)
5. Re-run from the failed phase

Estimated Timeline

• Phase 1: 30 minutes
• Phase 2: 1-2 hours
• Phase 3: 30 minutes
• Phase 4: 1-2 hours
• Phase 5: 30 minutes
• Phase 6: 2-3 hours
• Phase 7: 1 hour
• Phase 8: 1-2 hours

Total: ~8-12 hours for complete deployment

Dependencies

Phase 1 (Prerequisites)
  └─> Phase 2 (Foundation)
      └─> Phase 3 (Networking)
          └─> Phase 4 (Compute)
              └─> Phase 5 (Storage)
                  └─> Phase 6 (Application)
                      └─> Phase 7 (External)
                          └─> Phase 8 (Contracts)

Notes

• Each phase must be completed before moving to the next
• Some steps within a phase can be done in parallel
• Always verify each phase before proceeding
• Keep backups of configurations and keys
• Document any deviations from the standard process