the_order/docs/deployment/DEPLOYMENT_GUIDE.md

The Order - Complete Deployment Guide

Last Updated: 2025-01-27
Target Platform: Azure (West Europe)
Deployment Method: Kubernetes (AKS)
Policy: No US Commercial or Government regions
Naming Convention: See NAMING_CONVENTION.md

🚀 Automated Deployment: Use the deployment automation scripts for faster, repeatable deployments:

./scripts/deploy/deploy.sh --all --environment dev

See scripts/deploy/README.md for automation documentation.

---

Table of Contents

1. Prerequisites
2. Azure Infrastructure Setup
3. Entra ID Configuration
4. Database & Storage Setup
5. Container Registry Setup
6. Application Build & Package
7. Database Migrations
8. Secrets Configuration
9. Infrastructure Services Deployment
10. Backend Services Deployment
11. Frontend Applications Deployment
12. Networking & Gateways
13. Monitoring & Observability
14. Testing & Validation
15. Production Hardening

---

Phase 1: Prerequisites

Estimated Time: 1-2 days
Dependencies: None

1.1 Development Environment Setup

• Install Required Tools

  # Node.js >= 18.0.0
  node --version
  
  # pnpm >= 8.0.0
  pnpm --version
  
  # Azure CLI
  az --version
  
  # Terraform >= 1.5.0
  terraform --version
  
  # kubectl
  kubectl version --client
  
  # Docker (for local development)
  docker --version
  

• Clone Repository

  git clone <repository-url>
  cd the-order
  git submodule update --init --recursive
  

• Install Dependencies

  pnpm install --frozen-lockfile
  

• Build All Packages

  pnpm build
  

1.2 Azure Account Setup

• Create Azure Subscription (if not exists)

  • Go to Azure Portal
  • Create new subscription
  • Note subscription ID

• Login to Azure CLI

  az login
  az account set --subscription <subscription-id>
  az account show
  

• Verify Permissions

  • Subscription Contributor or Owner role required
  • Ability to create resource groups
  • Ability to register resource providers

1.3 Local Development Services (Optional for Testing)

• Start Local Services

  docker-compose up -d
  

  This starts:

  • PostgreSQL (port 5432)
  • Redis (port 6379)
  • OpenSearch (port 9200)
  • OpenSearch Dashboards (port 5601)

---

Phase 2: Azure Infrastructure Setup

Estimated Time: 4-6 weeks
Dependencies: Phase 1 complete
Critical Path: Must complete before any deployments

2.1 Azure Subscription Preparation

• Run Azure Setup Scripts

  # From project root
  ./infra/scripts/azure-setup.sh
  

  This will:

  • List all non-US Azure regions
  • Set default region to West Europe
  • Register resource providers
  • Check quotas
  • Generate reports

• Register Resource Providers

  ./infra/scripts/azure-register-providers.sh
  

  Required providers (13 total):

  • Microsoft.ContainerService
  • Microsoft.KeyVault
  • Microsoft.Storage
  • Microsoft.Network
  • Microsoft.Compute
  • Microsoft.DBforPostgreSQL
  • Microsoft.ContainerRegistry
  • Microsoft.ManagedIdentity
  • Microsoft.Insights
  • Microsoft.Logic
  • Microsoft.OperationalInsights
  • Microsoft.Authorization
  • Microsoft.Resources

• Review Quotas

  ./infra/scripts/azure-check-quotas.sh
  cat azure-quotas-all-regions.txt
  

  Ensure sufficient quotas for:

  • VM cores (for AKS nodes)
  • Storage accounts
  • Network resources

2.2 Terraform Infrastructure Deployment

• Initialize Terraform

  cd infra/terraform
  terraform init
  

• Create Initial Infrastructure (State Storage)

  # Create resource groups and storage for Terraform state
  terraform plan -target=azurerm_resource_group.terraform_state \
                 -target=azurerm_storage_account.terraform_state \
                 -target=azurerm_storage_container.terraform_state
  
  terraform apply -target=azurerm_resource_group.terraform_state \
                  -target=azurerm_storage_account.terraform_state \
                  -target=azurerm_storage_container.terraform_state
  

• Configure Remote State Backend

  # Get storage account name
  terraform output terraform_state_storage_account_name
  
  # Update versions.tf - uncomment and configure backend block
  # Then re-initialize
  terraform init -migrate-state
  

• Plan Full Infrastructure

  terraform plan -out=tfplan
  terraform show tfplan
  

• Deploy Core Infrastructure (Resource Groups, Storage)

  terraform apply tfplan
  

• Deploy AKS Cluster (To be added to Terraform)

  • Create AKS cluster configuration
  • Configure Azure CNI networking
  • Set up node pools
  • Configure Azure Disk CSI driver
  • Deploy cluster

• Deploy Azure Database for PostgreSQL (To be added to Terraform)

  • Create PostgreSQL server
  • Configure firewall rules
  • Set up databases (dev, stage, prod)
  • Configure backup and retention

• Deploy Azure Key Vault (To be added to Terraform)

  • Create Key Vault instances (dev, stage, prod)
  • Configure access policies
  • Enable soft delete and purge protection

• Deploy Azure Container Registry (To be added to Terraform)

  • Create ACR instance
  • Configure admin user or managed identity
  • Enable geo-replication (optional)

• Deploy Virtual Network (To be added to Terraform)

  • Create VNet with subnets
  • Configure Network Security Groups
  • Set up private endpoints (if needed)

• Deploy Application Gateway / Load Balancer (To be added to Terraform)

  • Create Application Gateway
  • Configure SSL certificates
  • Set up routing rules

2.3 Kubernetes Configuration

• Configure AKS Access

  az aks get-credentials --resource-group the-order-dev-rg \
                         --name the-order-dev-aks
  kubectl get nodes
  

• Set Up Azure CNI Networking

  • Verify CNI is configured
  • Test pod networking

• Configure Azure Key Vault Provider for Secrets Store CSI

  # Install External Secrets Operator
  kubectl apply -f https://external-secrets.io/latest/deploy/
  
  # Configure Azure Key Vault integration
  # (Configuration to be added)
  

• Configure Azure Container Registry Integration

  # Attach ACR to AKS
  az aks update -n the-order-dev-aks \
                -g the-order-dev-rg \
                --attach-acr <acr-name>
  

• Set Up Azure Monitor for Containers

  • Enable container insights
  • Configure Log Analytics workspace
  • Set up alerts

---

Phase 3: Entra ID Configuration

Estimated Time: 1-2 days
Dependencies: Phase 1 complete
Can run in parallel with Phase 2

3.1 Azure AD App Registration

• Create App Registration

  • Go to Azure Portal → Azure Active Directory → App registrations
  • Create new registration
  • Note Application (client) ID
  • Note Directory (tenant) ID

• Configure API Permissions

  • Add permission: Verifiable Credentials Service - VerifiableCredential.Create.All
  • Add permission: Verifiable Credentials Service - VerifiableCredential.Verify.All
  • Grant admin consent

• Create Client Secret

  • Go to Certificates & secrets
  • Create new client secret
  • IMPORTANT: Save secret value immediately (only shown once)
  • Store securely in Azure Key Vault

• Configure Redirect URIs

  • Add callback URLs for portal applications
  • Add logout URLs

3.2 Microsoft Entra VerifiedID Setup

• Enable Verified ID Service

  • Go to Azure Portal → Verified ID
  • Enable the service (may require tenant admin approval)
  • Wait for service activation

• Create Credential Manifest

  • Go to Azure Portal → Verified ID → Credential manifests
  • Create new credential manifest
  • Define credential type
  • Define claims schema
  • Note Manifest ID

• Verify Issuer DID

  • Format: did:web:{tenant-id}.verifiedid.msidentity.com
  • Verify DID is accessible
  • Test DID resolution

3.3 Azure Logic Apps Setup (Optional)

• Create Logic App Workflows

  • Create workflow for eIDAS verification
  • Create workflow for VC issuance
  • Create workflow for document processing
  • Note workflow URLs

• Configure Access

  • Generate access keys OR
  • Configure managed identity
  • Grant necessary permissions

• Test Workflow Triggers

  • Test eIDAS verification workflow
  • Test VC issuance workflow
  • Verify callbacks work

---

Phase 4: Database & Storage Setup

Estimated Time: 1-2 days
Dependencies: Phase 2 (Terraform infrastructure) complete

4.1 PostgreSQL Database Setup

• Create Databases

  -- For each environment (dev, stage, prod)
  CREATE DATABASE theorder_dev;
  CREATE DATABASE theorder_stage;
  CREATE DATABASE theorder_prod;
  

• Configure Database Users

  CREATE USER theorder_app WITH PASSWORD '<secure-password>';
  GRANT ALL PRIVILEGES ON DATABASE theorder_dev TO theorder_app;
  

• Configure Firewall Rules

  az postgres server firewall-rule create \
    --resource-group the-order-dev-rg \
    --server-name <server-name> \
    --name AllowAKS \
    --start-ip-address <aks-ip-range-start> \
    --end-ip-address <aks-ip-range-end>
  

• Test Database Connection

  psql -h <server-name>.postgres.database.azure.com \
       -U theorder_app \
       -d theorder_dev
  

4.2 Storage Account Setup

• Verify Storage Accounts Created

  az storage account list --resource-group the-order-dev-rg
  

• Create Storage Containers

  # Application data containers
  az storage container create \
    --name intake-documents \
    --account-name <storage-account-name>
  
  az storage container create \
    --name dataroom-deals \
    --account-name <storage-account-name>
  
  az storage container create \
    --name credentials \
    --account-name <storage-account-name>
  

• Configure Storage Access

  • Set up managed identity access
  • Configure CORS (if needed)
  • Enable versioning and soft delete

4.3 Redis Cache Setup (If using Azure Cache for Redis)

• Create Redis Cache (To be added to Terraform)
  • Create Azure Cache for Redis instance
  • Configure firewall rules
  • Set up access keys
  • Test connection

4.4 OpenSearch Setup (If using managed service)

• Create OpenSearch Service (To be added to Terraform)
  • Create managed OpenSearch cluster
  • Configure access
  • Set up indices
  • Test connection

---

Phase 5: Container Registry Setup

Estimated Time: 1 day
Dependencies: Phase 2 (ACR created)

5.1 Azure Container Registry Configuration

• Verify ACR Created

  az acr list --resource-group the-order-dev-rg
  

• Configure ACR Access

  # Enable admin user (or use managed identity)
  az acr update --name <acr-name> --admin-enabled true
  
  # Get credentials
  az acr credential show --name <acr-name>
  

• Attach ACR to AKS

  az aks update -n the-order-dev-aks \
                -g the-order-dev-rg \
                --attach-acr <acr-name>
  

• Test ACR Access from AKS

  kubectl run test-pull --image=<acr-name>.azurecr.io/test:latest \
                        --restart=Never \
                        --rm -i --tty
  

---

Phase 6: Application Build & Package

Estimated Time: 2-4 hours
Dependencies: Phase 1, Phase 5 (ACR ready)

6.1 Build All Packages

• Build Shared Packages

  # From project root
  pnpm build
  
  # Or build individually
  pnpm --filter @the-order/ui build
  pnpm --filter @the-order/auth build
  pnpm --filter @the-order/api-client build
  pnpm --filter @the-order/database build
  pnpm --filter @the-order/storage build
  pnpm --filter @the-order/crypto build
  pnpm --filter @the-order/schemas build
  

6.2 Build Frontend Applications

• Build Portal Public

  pnpm --filter portal-public build
  

• Build Portal Internal

  pnpm --filter portal-internal build
  

6.3 Build Backend Services

• Build Identity Service

  pnpm --filter @the-order/identity build
  

• Build Intake Service

  pnpm --filter @the-order/intake build
  

• Build Finance Service

  pnpm --filter @the-order/finance build
  

• Build Dataroom Service

  pnpm --filter @the-order/dataroom build
  

6.4 Create Docker Images

Note: Dockerfiles need to be created for each service/app

• Create Dockerfiles (To be created)

  • services/identity/Dockerfile
  • services/intake/Dockerfile
  • services/finance/Dockerfile
  • services/dataroom/Dockerfile
  • apps/portal-public/Dockerfile
  • apps/portal-internal/Dockerfile

• Build and Push Images to ACR

  # Login to ACR
  az acr login --name <acr-name>
  
  # Build and push each service
  # Identity Service
  docker build -t <acr-name>.azurecr.io/identity:latest \
               -t <acr-name>.azurecr.io/identity:$(git rev-parse --short HEAD) \
               -f services/identity/Dockerfile .
  docker push <acr-name>.azurecr.io/identity:latest
  docker push <acr-name>.azurecr.io/identity:$(git rev-parse --short HEAD)
  
  # Intake Service
  docker build -t <acr-name>.azurecr.io/intake:latest \
               -t <acr-name>.azurecr.io/intake:$(git rev-parse --short HEAD) \
               -f services/intake/Dockerfile .
  docker push <acr-name>.azurecr.io/intake:latest
  docker push <acr-name>.azurecr.io/intake:$(git rev-parse --short HEAD)
  
  # Finance Service
  docker build -t <acr-name>.azurecr.io/finance:latest \
               -t <acr-name>.azurecr.io/finance:$(git rev-parse --short HEAD) \
               -f services/finance/Dockerfile .
  docker push <acr-name>.azurecr.io/finance:latest
  docker push <acr-name>.azurecr.io/finance:$(git rev-parse --short HEAD)
  
  # Dataroom Service
  docker build -t <acr-name>.azurecr.io/dataroom:latest \
               -t <acr-name>.azurecr.io/dataroom:$(git rev-parse --short HEAD) \
               -f services/dataroom/Dockerfile .
  docker push <acr-name>.azurecr.io/dataroom:latest
  docker push <acr-name>.azurecr.io/dataroom:$(git rev-parse --short HEAD)
  
  # Portal Public
  docker build -t <acr-name>.azurecr.io/portal-public:latest \
               -t <acr-name>.azurecr.io/portal-public:$(git rev-parse --short HEAD) \
               -f apps/portal-public/Dockerfile .
  docker push <acr-name>.azurecr.io/portal-public:latest
  docker push <acr-name>.azurecr.io/portal-public:$(git rev-parse --short HEAD)
  
  # Portal Internal
  docker build -t <acr-name>.azurecr.io/portal-internal:latest \
               -t <acr-name>.azurecr.io/portal-internal:$(git rev-parse --short HEAD) \
               -f apps/portal-internal/Dockerfile .
  docker push <acr-name>.azurecr.io/portal-internal:latest
  docker push <acr-name>.azurecr.io/portal-internal:$(git rev-parse --short HEAD)
  

• Sign Images with Cosign (Security best practice)

  # Generate signing key (one-time)
  cosign generate-key-pair
  
  # Sign each image
  cosign sign --key cosign.key <acr-name>.azurecr.io/identity:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/intake:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/finance:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/dataroom:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/portal-public:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/portal-internal:latest
  

---

Phase 7: Database Migrations

Estimated Time: 1-2 hours
Dependencies: Phase 4 (Database created), Phase 6 (Packages built)

7.1 Run Database Migrations

• Run Migrations for Each Environment

  # Development
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_dev"
  pnpm --filter @the-order/database migrate up
  
  # Staging
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_stage"
  pnpm --filter @the-order/database migrate up
  
  # Production
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_prod"
  pnpm --filter @the-order/database migrate up
  

• Verify Schema Created

  \dt  -- List tables
  \d+ <table-name>  -- Describe table
  

• Seed Initial Data (If needed)

  # Run seed scripts if they exist
  pnpm --filter @the-order/database seed
  

---

Phase 8: Secrets Configuration

Estimated Time: 2-4 hours
Dependencies: Phase 2 (Key Vault created), Phase 3 (Entra ID configured)

8.1 Store Secrets in Azure Key Vault

• Store Database Credentials

  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "database-url-dev" \
    --value "postgresql://user:pass@host:5432/theorder_dev"
  

• Store Entra ID Secrets

  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-tenant-id" \
    --value "<tenant-id>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-client-id" \
    --value "<client-id>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-client-secret" \
    --value "<client-secret>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-credential-manifest-id" \
    --value "<manifest-id>"
  

• Store Storage Credentials

  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "storage-account-name" \
    --value "<storage-account-name>"
  

• Store JWT Secrets

  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "jwt-secret" \
    --value "<secure-random-string>"
  

• Store KMS Keys

  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "kms-key-id" \
    --value "<kms-key-id>"
  

• Store Other Service Secrets

  # Payment gateway
  az keyvault secret set --vault-name <key-vault-name> --name "payment-gateway-api-key" --value "..."
  
  # OCR service
  az keyvault secret set --vault-name <key-vault-name> --name "ocr-service-api-key" --value "..."
  
  # eIDAS
  az keyvault secret set --vault-name <key-vault-name> --name "eidas-api-key" --value "..."
  

8.2 Configure External Secrets Operator

• Create SecretStore for Azure Key Vault

  # infra/k8s/base/external-secrets-store.yaml (to be created)
  apiVersion: external-secrets.io/v1beta1
  kind: SecretStore
  metadata:
    name: azure-keyvault
  spec:
    provider:
      azurekv:
        vaultUrl: https://<key-vault-name>.vault.azure.net
        authType: WorkloadIdentity
        serviceAccountRef:
          name: external-secrets-sa
  

• Create ExternalSecret Resources

  # infra/k8s/base/external-secrets.yaml (to be created)
  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: the-order-secrets
  spec:
    refreshInterval: 1h
    secretStoreRef:
      name: azure-keyvault
      kind: SecretStore
    target:
      name: the-order-secrets
      creationPolicy: Owner
    data:
      - secretKey: DATABASE_URL
        remoteRef:
          key: database-url-dev
      - secretKey: ENTRA_TENANT_ID
        remoteRef:
          key: entra-tenant-id
      # ... more secrets
  

• Apply External Secrets Configuration

  kubectl apply -f infra/k8s/base/external-secrets-store.yaml
  kubectl apply -f infra/k8s/base/external-secrets.yaml
  

---

Phase 9: Infrastructure Services Deployment

Estimated Time: 1-2 days
Dependencies: Phase 2, Phase 8 (Secrets configured)

9.1 Deploy External Secrets Operator

• Install External Secrets Operator

  kubectl apply -f https://external-secrets.io/latest/deploy/
  kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=external-secrets -n external-secrets-system
  

9.2 Deploy Monitoring Stack

• Deploy Prometheus (To be configured)

  # Using Helm or manifests
  helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  helm install prometheus prometheus-community/kube-prometheus-stack
  

• Deploy Grafana (To be configured)

  # Usually included with Prometheus stack
  # Access via port-forward or ingress
  kubectl port-forward svc/prometheus-grafana 3000:80
  

• Configure OpenTelemetry (To be configured)

  • Deploy OpenTelemetry Collector
  • Configure exporters
  • Set up trace collection

9.3 Deploy Logging Stack

• Deploy OpenSearch (If not using managed service)

  # Deploy OpenSearch operator or Helm chart
  # Configuration to be added
  

• Configure Log Aggregation

  • Set up Fluent Bit or Fluentd
  • Configure log forwarding
  • Set up log retention policies

---

Phase 10: Backend Services Deployment

Estimated Time: 2-4 days
Dependencies: Phase 6 (Images built), Phase 7 (Migrations run), Phase 8 (Secrets configured), Phase 9 (Infrastructure ready)

10.1 Create Kubernetes Manifests

• Create Base Manifests (To be created)
  • infra/k8s/base/identity/deployment.yaml
  • infra/k8s/base/identity/service.yaml
  • infra/k8s/base/intake/deployment.yaml
  • infra/k8s/base/intake/service.yaml
  • infra/k8s/base/finance/deployment.yaml
  • infra/k8s/base/finance/service.yaml
  • infra/k8s/base/dataroom/deployment.yaml
  • infra/k8s/base/dataroom/service.yaml

10.2 Deploy Identity Service

• Deploy Identity Service

  kubectl apply -k infra/k8s/overlays/dev
  # Or for specific service
  kubectl apply -f infra/k8s/base/identity/
  

• Verify Deployment

  kubectl get pods -l app=identity -n the-order-dev
  kubectl logs -l app=identity -n the-order-dev
  kubectl get svc identity -n the-order-dev
  

• Test Health Endpoint

  kubectl port-forward svc/identity 4002:4002
  curl http://localhost:4002/health
  

10.3 Deploy Intake Service

• Deploy Intake Service

  kubectl apply -f infra/k8s/base/intake/
  

• Verify Deployment

  kubectl get pods -l app=intake -n the-order-dev
  kubectl logs -l app=intake -n the-order-dev
  

• Test Health Endpoint

  kubectl port-forward svc/intake 4001:4001
  curl http://localhost:4001/health
  

10.4 Deploy Finance Service

• Deploy Finance Service

  kubectl apply -f infra/k8s/base/finance/
  

• Verify Deployment

  kubectl get pods -l app=finance -n the-order-dev
  kubectl logs -l app=finance -n the-order-dev
  

• Test Health Endpoint

  kubectl port-forward svc/finance 4003:4003
  curl http://localhost:4003/health
  

10.5 Deploy Dataroom Service

• Deploy Dataroom Service

  kubectl apply -f infra/k8s/base/dataroom/
  

• Verify Deployment

  kubectl get pods -l app=dataroom -n the-order-dev
  kubectl logs -l app=dataroom -n the-order-dev
  

• Test Health Endpoint

  kubectl port-forward svc/dataroom 4004:4004
  curl http://localhost:4004/health
  

10.6 Verify Service-to-Service Communication

• Test Internal Service Communication

  # From within cluster
  kubectl run test-pod --image=curlimages/curl --rm -it --restart=Never -- \
    curl http://identity:4002/health
  

---

Phase 11: Frontend Applications Deployment

Estimated Time: 1-2 days
Dependencies: Phase 6 (Images built), Phase 10 (Backend services deployed)

11.1 Deploy Portal Public

• Create Kubernetes Manifests (To be created)

  • infra/k8s/base/portal-public/deployment.yaml
  • infra/k8s/base/portal-public/service.yaml
  • infra/k8s/base/portal-public/ingress.yaml

• Deploy Portal Public

  kubectl apply -f infra/k8s/base/portal-public/
  

• Verify Deployment

  kubectl get pods -l app=portal-public -n the-order-dev
  kubectl logs -l app=portal-public -n the-order-dev
  

• Test Application

  kubectl port-forward svc/portal-public 3000:3000
  # Open http://localhost:3000 in browser
  

11.2 Deploy Portal Internal

• Create Kubernetes Manifests (To be created)

  • infra/k8s/base/portal-internal/deployment.yaml
  • infra/k8s/base/portal-internal/service.yaml
  • infra/k8s/base/portal-internal/ingress.yaml

• Deploy Portal Internal

  kubectl apply -f infra/k8s/base/portal-internal/
  

• Verify Deployment

  kubectl get pods -l app=portal-internal -n the-order-dev
  kubectl logs -l app=portal-internal -n the-order-dev
  

• Test Application

  kubectl port-forward svc/portal-internal 3001:3001
  # Open http://localhost:3001 in browser
  

---

Phase 12: Networking & Gateways

Estimated Time: 2-3 days
Dependencies: Phase 10, Phase 11 (Services and apps deployed)

12.1 Configure Ingress

• Deploy NGINX Ingress Controller (If not using Application Gateway)

  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  helm install ingress-nginx ingress-nginx/ingress-nginx
  

• Create Ingress Resources

  # infra/k8s/base/ingress.yaml (to be created)
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: the-order-ingress
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
  spec:
    tls:
      - hosts:
          - api.theorder.org
          - portal.theorder.org
          - admin.theorder.org
        secretName: the-order-tls
    rules:
      - host: api.theorder.org
        http:
          paths:
            - path: /identity
              pathType: Prefix
              backend:
                service:
                  name: identity
                  port:
                    number: 4002
      # ... more rules
  

• Apply Ingress Configuration

  kubectl apply -f infra/k8s/base/ingress.yaml
  

12.2 Configure Application Gateway (If using)

• Create Application Gateway Backend Pools

  az network application-gateway address-pool create \
    --resource-group the-order-dev-rg \
    --gateway-name <gateway-name> \
    --name identity-backend \
    --servers <aks-node-ip>
  

• Configure Routing Rules

  • Set up path-based routing
  • Configure SSL termination
  • Set up health probes

12.3 Configure DNS

• Create DNS Records

  # For each domain
  # api.theorder.org -> Application Gateway IP
  # portal.theorder.org -> Application Gateway IP
  # admin.theorder.org -> Application Gateway IP
  

• Verify DNS Resolution

  nslookup api.theorder.org
  nslookup portal.theorder.org
  nslookup admin.theorder.org
  

12.4 Configure SSL/TLS Certificates

• Obtain SSL Certificates

  • Use Let's Encrypt (cert-manager)
  • Or Azure Key Vault certificates
  • Or import existing certificates

• Configure cert-manager (If using Let's Encrypt)

  kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
  

• Create ClusterIssuer

  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-prod
  spec:
    acme:
      server: https://acme-v02.api.letsencrypt.org/directory
      email: admin@theorder.org
      privateKeySecretRef:
        name: letsencrypt-prod
      solvers:
        - http01:
            ingress:
              class: nginx
  

12.5 Configure WAF Rules

• Configure Azure WAF (If using Application Gateway)
  • Set up OWASP rules
  • Configure custom rules
  • Set up rate limiting
  • Configure IP allow/deny lists

---

Phase 13: Monitoring & Observability

Estimated Time: 2-3 days
Dependencies: Phase 9, Phase 10, Phase 11 (Services deployed)

13.1 Configure Application Insights

• Create Application Insights Resources

  az monitor app-insights component create \
    --app the-order-dev \
    --location westeurope \
    --resource-group the-order-dev-rg
  

• Configure Application Insights in Services

  • Add instrumentation keys to services
  • Configure custom metrics
  • Set up alerts

13.2 Configure Log Analytics

• Create Log Analytics Workspace

  az monitor log-analytics workspace create \
    --resource-group the-order-dev-rg \
    --workspace-name the-order-dev-logs
  

• Configure Log Collection

  • Set up container insights
  • Configure log forwarding
  • Set up log queries

13.3 Set Up Alerts

• Create Alert Rules

  # High error rate
  az monitor metrics alert create \
    --name "high-error-rate" \
    --resource-group the-order-dev-rg \
    --scopes <resource-id> \
    --condition "avg Percentage > 5" \
    --window-size 5m \
    --evaluation-frequency 1m
  

• Configure Alert Actions

  • Set up email notifications
  • Configure webhook actions
  • Set up PagerDuty integration (if needed)

13.4 Configure Dashboards

• Create Grafana Dashboards

  • Service health dashboard
  • Performance metrics dashboard
  • Business metrics dashboard
  • Error tracking dashboard

• Configure Azure Dashboards

  • Create custom dashboards
  • Set up shared dashboards
  • Configure access permissions

---

Phase 14: Testing & Validation

Estimated Time: 3-5 days
Dependencies: All previous phases complete

14.1 Health Checks

• Verify All Services Healthy

  # Check all pods
  kubectl get pods -n the-order-dev
  
  # Check service endpoints
  for svc in identity intake finance dataroom portal-public portal-internal; do
    kubectl exec -it deployment/$svc -n the-order-dev -- curl http://localhost/health
  done
  

14.2 Integration Testing

• Test API Endpoints

  # Identity Service
  curl https://api.theorder.org/identity/health
  curl https://api.theorder.org/identity/vc/issue/entra
  
  # Intake Service
  curl https://api.theorder.org/intake/health
  
  # Finance Service
  curl https://api.theorder.org/finance/health
  
  # Dataroom Service
  curl https://api.theorder.org/dataroom/health
  

• Test Frontend Applications

  • Portal Public accessible
  • Portal Internal accessible
  • Authentication flow works
  • API integration works
  • Forms submit correctly

14.3 End-to-End Testing

• Test Complete User Flows
  • User registration flow
  • Application submission flow
  • Credential issuance flow
  • Payment processing flow
  • Document upload flow

14.4 Performance Testing

• Load Testing

  # Use tools like k6, Apache Bench, or JMeter
  k6 run load-test.js
  

• Verify Performance Metrics

  • Response times acceptable
  • Throughput meets requirements
  • Resource usage within limits

14.5 Security Testing

• Run Security Scans

  # Trivy scan
  trivy k8s cluster --severity HIGH,CRITICAL
  
  # Check for exposed secrets
  kubectl get secrets -n the-order-dev
  

• Verify Security Controls

  • Network policies configured
  • RBAC properly set up
  • Secrets not exposed
  • TLS/SSL working
  • Authentication required

---

Phase 15: Production Hardening

Estimated Time: 2-3 days
Dependencies: Phase 14 (Testing complete)

15.1 Production Configuration

• Update Replica Counts

  # Update kustomization for production
  # Set appropriate replica counts
  kubectl scale deployment identity --replicas=3 -n the-order-prod
  

• Configure Resource Limits

  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "512Mi"
      cpu: "500m"
  

• Configure Liveness and Readiness Probes

  livenessProbe:
    httpGet:
      path: /health
      port: 4002
    initialDelaySeconds: 30
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /health
      port: 4002
    initialDelaySeconds: 5
    periodSeconds: 5
  

15.2 Backup Configuration

• Configure Database Backups

  az postgres server backup create \
    --resource-group the-order-prod-rg \
    --server-name <server-name> \
    --backup-name daily-backup
  

• Configure Storage Backups

  • Enable blob versioning
  • Configure retention policies
  • Set up geo-replication (if needed)

15.3 Disaster Recovery

• Create Backup Procedures

  • Document backup process
  • Test restore procedures
  • Set up automated backups

• Configure Failover

  • Set up multi-region deployment (if needed)
  • Configure DNS failover
  • Test disaster recovery procedures

15.4 Documentation

• Update Deployment Documentation

  • Document all configuration
  • Create runbooks
  • Document troubleshooting steps

• Create Operational Runbooks

  • Incident response procedures
  • Common troubleshooting
  • Escalation procedures

---

Deployment Checklist Summary

Pre-Deployment (Phases 1-5)

• Prerequisites installed
• Azure account setup
• Infrastructure deployed
• Entra ID configured
• Database and storage ready
• Container registry ready

Build & Configure (Phases 6-8)

• Applications built
• Docker images created and pushed
• Database migrations run
• Secrets configured

Deploy (Phases 9-12)

• Infrastructure services deployed
• Backend services deployed
• Frontend applications deployed
• Networking configured

Validate & Harden (Phases 13-15)

• Monitoring configured
• Testing complete
• Production hardening done

---

Environment-Specific Deployment

Development Environment

# Deploy to dev
kubectl apply -k infra/k8s/overlays/dev

Staging Environment

# Deploy to staging
kubectl apply -k infra/k8s/overlays/stage

Production Environment

# Deploy to production (after approval)
kubectl apply -k infra/k8s/overlays/prod

---

Rollback Procedures

Rollback Application Deployment

# Rollback to previous version
kubectl rollout undo deployment/<service-name> -n the-order-prod

Rollback Infrastructure

# Rollback Terraform changes
terraform plan -destroy
terraform apply -target=<resource>

---

Troubleshooting

Common Issues

1. Pods Not Starting

   kubectl describe pod <pod-name> -n the-order-dev
   kubectl logs <pod-name> -n the-order-dev
   

2. Service Not Accessible

   kubectl get svc -n the-order-dev
   kubectl get ingress -n the-order-dev
   

3. Database Connection Issues

   # Check firewall rules
   az postgres server firewall-rule list --server-name <name>
   
   # Test connection
   psql -h <host> -U <user> -d <database>
   

---

Estimated Timeline

Phase	Duration	Dependencies
Phase 1: Prerequisites	1-2 days	None
Phase 2: Azure Infrastructure	4-6 weeks	Phase 1
Phase 3: Entra ID	1-2 days	Phase 1
Phase 4: Database & Storage	1-2 days	Phase 2
Phase 5: Container Registry	1 day	Phase 2
Phase 6: Build & Package	2-4 hours	Phase 1, 5
Phase 7: Database Migrations	1-2 hours	Phase 4, 6
Phase 8: Secrets Configuration	2-4 hours	Phase 2, 3
Phase 9: Infrastructure Services	1-2 days	Phase 2, 8
Phase 10: Backend Services	2-4 days	Phase 6, 7, 8, 9
Phase 11: Frontend Apps	1-2 days	Phase 6, 10
Phase 12: Networking	2-3 days	Phase 10, 11
Phase 13: Monitoring	2-3 days	Phase 9, 10, 11
Phase 14: Testing	3-5 days	All previous
Phase 15: Production Hardening	2-3 days	Phase 14

Total Estimated Time: 8-12 weeks (with parallel work on Phases 2-3)

---

Quick Reference Commands

# Infrastructure
./infra/scripts/azure-setup.sh
terraform init && terraform plan && terraform apply

# Build
pnpm build
docker build -t <image> -f <Dockerfile> .

# Deploy
kubectl apply -k infra/k8s/overlays/dev
kubectl get pods -n the-order-dev
kubectl logs -f <pod-name> -n the-order-dev

# Verify
kubectl get all -n the-order-dev
kubectl port-forward svc/<service> <port>:<port>
curl http://localhost:<port>/health

---

See individual phase sections for detailed instructions.