the_order/docs/deployment/DEPLOYMENT_GUIDE.md

# The Order - Complete Deployment Guide

**Last Updated**: 2025-01-27  
**Target Platform**: Azure (West Europe)  
**Deployment Method**: Kubernetes (AKS)  
**Policy**: No US Commercial or Government regions  
**Naming Convention**: See [NAMING_CONVENTION.md](../governance/NAMING_CONVENTION.md)

> **🚀 Automated Deployment**: Use the deployment automation scripts for faster, repeatable deployments:
> ```bash
> ./scripts/deploy/deploy.sh --all --environment dev
> ```
> See [scripts/deploy/README.md](../../scripts/deploy/README.md) for automation documentation.

---

## Table of Contents

1. [Prerequisites](#phase-1-prerequisites)
2. [Azure Infrastructure Setup](#phase-2-azure-infrastructure-setup)
3. [Entra ID Configuration](#phase-3-entra-id-configuration)
4. [Database & Storage Setup](#phase-4-database--storage-setup)
5. [Container Registry Setup](#phase-5-container-registry-setup)
6. [Application Build & Package](#phase-6-application-build--package)
7. [Database Migrations](#phase-7-database-migrations)
8. [Secrets Configuration](#phase-8-secrets-configuration)
9. [Infrastructure Services Deployment](#phase-9-infrastructure-services-deployment)
10. [Backend Services Deployment](#phase-10-backend-services-deployment)
11. [Frontend Applications Deployment](#phase-11-frontend-applications-deployment)
12. [Networking & Gateways](#phase-12-networking--gateways)
13. [Monitoring & Observability](#phase-13-monitoring--observability)
14. [Testing & Validation](#phase-14-testing--validation)
15. [Production Hardening](#phase-15-production-hardening)

---

## Phase 1: Prerequisites

**Estimated Time**: 1-2 days  
**Dependencies**: None

### 1.1 Development Environment Setup

- [ ] **Install Required Tools**
  ```bash
  # Node.js >= 18.0.0
  node --version
  
  # pnpm >= 8.0.0
  pnpm --version
  
  # Azure CLI
  az --version
  
  # Terraform >= 1.5.0
  terraform --version
  
  # kubectl
  kubectl version --client
  
  # Docker (for local development)
  docker --version
  ```

- [ ] **Clone Repository**
  ```bash
  git clone <repository-url>
  cd the-order
  git submodule update --init --recursive
  ```

- [ ] **Install Dependencies**
  ```bash
  pnpm install --frozen-lockfile
  ```

- [ ] **Build All Packages**
  ```bash
  pnpm build
  ```

### 1.2 Azure Account Setup

- [ ] **Create Azure Subscription** (if not exists)
  - Go to Azure Portal
  - Create new subscription
  - Note subscription ID

- [ ] **Login to Azure CLI**
  ```bash
  az login
  az account set --subscription <subscription-id>
  az account show
  ```

- [ ] **Verify Permissions**
  - Subscription Contributor or Owner role required
  - Ability to create resource groups
  - Ability to register resource providers

### 1.3 Local Development Services (Optional for Testing)

- [ ] **Start Local Services**
  ```bash
  docker-compose up -d
  ```
  
  This starts:
  - PostgreSQL (port 5432)
  - Redis (port 6379)
  - OpenSearch (port 9200)
  - OpenSearch Dashboards (port 5601)

---

## Phase 2: Azure Infrastructure Setup

**Estimated Time**: 4-6 weeks  
**Dependencies**: Phase 1 complete  
**Critical Path**: Must complete before any deployments

### 2.1 Azure Subscription Preparation

- [ ] **Run Azure Setup Scripts**
  ```bash
  # From project root
  ./infra/scripts/azure-setup.sh
  ```
  
  This will:
  - List all non-US Azure regions
  - Set default region to West Europe
  - Register resource providers
  - Check quotas
  - Generate reports

- [ ] **Register Resource Providers**
  ```bash
  ./infra/scripts/azure-register-providers.sh
  ```
  
  Required providers (13 total):
  - Microsoft.ContainerService
  - Microsoft.KeyVault
  - Microsoft.Storage
  - Microsoft.Network
  - Microsoft.Compute
  - Microsoft.DBforPostgreSQL
  - Microsoft.ContainerRegistry
  - Microsoft.ManagedIdentity
  - Microsoft.Insights
  - Microsoft.Logic
  - Microsoft.OperationalInsights
  - Microsoft.Authorization
  - Microsoft.Resources

- [ ] **Review Quotas**
  ```bash
  ./infra/scripts/azure-check-quotas.sh
  cat azure-quotas-all-regions.txt
  ```
  
  Ensure sufficient quotas for:
  - VM cores (for AKS nodes)
  - Storage accounts
  - Network resources

### 2.2 Terraform Infrastructure Deployment

- [ ] **Initialize Terraform**
  ```bash
  cd infra/terraform
  terraform init
  ```

- [ ] **Create Initial Infrastructure (State Storage)**
  ```bash
  # Create resource groups and storage for Terraform state
  terraform plan -target=azurerm_resource_group.terraform_state \
                 -target=azurerm_storage_account.terraform_state \
                 -target=azurerm_storage_container.terraform_state
  
  terraform apply -target=azurerm_resource_group.terraform_state \
                  -target=azurerm_storage_account.terraform_state \
                  -target=azurerm_storage_container.terraform_state
  ```

- [ ] **Configure Remote State Backend**
  ```bash
  # Get storage account name
  terraform output terraform_state_storage_account_name
  
  # Update versions.tf - uncomment and configure backend block
  # Then re-initialize
  terraform init -migrate-state
  ```

- [ ] **Plan Full Infrastructure**
  ```bash
  terraform plan -out=tfplan
  terraform show tfplan
  ```

- [ ] **Deploy Core Infrastructure** (Resource Groups, Storage)
  ```bash
  terraform apply tfplan
  ```

- [ ] **Deploy AKS Cluster** (To be added to Terraform)
  - [ ] Create AKS cluster configuration
  - [ ] Configure Azure CNI networking
  - [ ] Set up node pools
  - [ ] Configure Azure Disk CSI driver
  - [ ] Deploy cluster

- [ ] **Deploy Azure Database for PostgreSQL** (To be added to Terraform)
  - [ ] Create PostgreSQL server
  - [ ] Configure firewall rules
  - [ ] Set up databases (dev, stage, prod)
  - [ ] Configure backup and retention

- [ ] **Deploy Azure Key Vault** (To be added to Terraform)
  - [ ] Create Key Vault instances (dev, stage, prod)
  - [ ] Configure access policies
  - [ ] Enable soft delete and purge protection

- [ ] **Deploy Azure Container Registry** (To be added to Terraform)
  - [ ] Create ACR instance
  - [ ] Configure admin user or managed identity
  - [ ] Enable geo-replication (optional)

- [ ] **Deploy Virtual Network** (To be added to Terraform)
  - [ ] Create VNet with subnets
  - [ ] Configure Network Security Groups
  - [ ] Set up private endpoints (if needed)

- [ ] **Deploy Application Gateway / Load Balancer** (To be added to Terraform)
  - [ ] Create Application Gateway
  - [ ] Configure SSL certificates
  - [ ] Set up routing rules

### 2.3 Kubernetes Configuration

- [ ] **Configure AKS Access**
  ```bash
  az aks get-credentials --resource-group the-order-dev-rg \
                         --name the-order-dev-aks
  kubectl get nodes
  ```

- [ ] **Set Up Azure CNI Networking**
  - [ ] Verify CNI is configured
  - [ ] Test pod networking

- [ ] **Configure Azure Key Vault Provider for Secrets Store CSI**
  ```bash
  # Install External Secrets Operator
  kubectl apply -f https://external-secrets.io/latest/deploy/
  
  # Configure Azure Key Vault integration
  # (Configuration to be added)
  ```

- [ ] **Configure Azure Container Registry Integration**
  ```bash
  # Attach ACR to AKS
  az aks update -n the-order-dev-aks \
                -g the-order-dev-rg \
                --attach-acr <acr-name>
  ```

- [ ] **Set Up Azure Monitor for Containers**
  - [ ] Enable container insights
  - [ ] Configure Log Analytics workspace
  - [ ] Set up alerts

---

## Phase 3: Entra ID Configuration

**Estimated Time**: 1-2 days  
**Dependencies**: Phase 1 complete  
**Can run in parallel with Phase 2**

### 3.1 Azure AD App Registration

- [ ] **Create App Registration**
  - Go to Azure Portal → Azure Active Directory → App registrations
  - Create new registration
  - Note **Application (client) ID**
  - Note **Directory (tenant) ID**

- [ ] **Configure API Permissions**
  - Add permission: `Verifiable Credentials Service - VerifiableCredential.Create.All`
  - Add permission: `Verifiable Credentials Service - VerifiableCredential.Verify.All`
  - Grant admin consent

- [ ] **Create Client Secret**
  - Go to Certificates & secrets
  - Create new client secret
  - **IMPORTANT**: Save secret value immediately (only shown once)
  - Store securely in Azure Key Vault

- [ ] **Configure Redirect URIs**
  - Add callback URLs for portal applications
  - Add logout URLs

### 3.2 Microsoft Entra VerifiedID Setup

- [ ] **Enable Verified ID Service**
  - Go to Azure Portal → Verified ID
  - Enable the service (may require tenant admin approval)
  - Wait for service activation

- [ ] **Create Credential Manifest**
  - Go to Azure Portal → Verified ID → Credential manifests
  - Create new credential manifest
  - Define credential type
  - Define claims schema
  - Note **Manifest ID**

- [ ] **Verify Issuer DID**
  - Format: `did:web:{tenant-id}.verifiedid.msidentity.com`
  - Verify DID is accessible
  - Test DID resolution

### 3.3 Azure Logic Apps Setup (Optional)

- [ ] **Create Logic App Workflows**
  - Create workflow for eIDAS verification
  - Create workflow for VC issuance
  - Create workflow for document processing
  - Note workflow URLs

- [ ] **Configure Access**
  - Generate access keys OR
  - Configure managed identity
  - Grant necessary permissions

- [ ] **Test Workflow Triggers**
  - Test eIDAS verification workflow
  - Test VC issuance workflow
  - Verify callbacks work

---

## Phase 4: Database & Storage Setup

**Estimated Time**: 1-2 days  
**Dependencies**: Phase 2 (Terraform infrastructure) complete

### 4.1 PostgreSQL Database Setup

- [ ] **Create Databases**
  ```sql
  -- For each environment (dev, stage, prod)
  CREATE DATABASE theorder_dev;
  CREATE DATABASE theorder_stage;
  CREATE DATABASE theorder_prod;
  ```

- [ ] **Configure Database Users**
  ```sql
  CREATE USER theorder_app WITH PASSWORD '<secure-password>';
  GRANT ALL PRIVILEGES ON DATABASE theorder_dev TO theorder_app;
  ```

- [ ] **Configure Firewall Rules**
  ```bash
  az postgres server firewall-rule create \
    --resource-group the-order-dev-rg \
    --server-name <server-name> \
    --name AllowAKS \
    --start-ip-address <aks-ip-range-start> \
    --end-ip-address <aks-ip-range-end>
  ```

- [ ] **Test Database Connection**
  ```bash
  psql -h <server-name>.postgres.database.azure.com \
       -U theorder_app \
       -d theorder_dev
  ```

### 4.2 Storage Account Setup

- [ ] **Verify Storage Accounts Created**
  ```bash
  az storage account list --resource-group the-order-dev-rg
  ```

- [ ] **Create Storage Containers**
  ```bash
  # Application data containers
  az storage container create \
    --name intake-documents \
    --account-name <storage-account-name>
  
  az storage container create \
    --name dataroom-deals \
    --account-name <storage-account-name>
  
  az storage container create \
    --name credentials \
    --account-name <storage-account-name>
  ```

- [ ] **Configure Storage Access**
  - Set up managed identity access
  - Configure CORS (if needed)
  - Enable versioning and soft delete

### 4.3 Redis Cache Setup (If using Azure Cache for Redis)

- [ ] **Create Redis Cache** (To be added to Terraform)
  - Create Azure Cache for Redis instance
  - Configure firewall rules
  - Set up access keys
  - Test connection

### 4.4 OpenSearch Setup (If using managed service)

- [ ] **Create OpenSearch Service** (To be added to Terraform)
  - Create managed OpenSearch cluster
  - Configure access
  - Set up indices
  - Test connection

---

## Phase 5: Container Registry Setup

**Estimated Time**: 1 day  
**Dependencies**: Phase 2 (ACR created)

### 5.1 Azure Container Registry Configuration

- [ ] **Verify ACR Created**
  ```bash
  az acr list --resource-group the-order-dev-rg
  ```

- [ ] **Configure ACR Access**
  ```bash
  # Enable admin user (or use managed identity)
  az acr update --name <acr-name> --admin-enabled true
  
  # Get credentials
  az acr credential show --name <acr-name>
  ```

- [ ] **Attach ACR to AKS**
  ```bash
  az aks update -n the-order-dev-aks \
                -g the-order-dev-rg \
                --attach-acr <acr-name>
  ```

- [ ] **Test ACR Access from AKS**
  ```bash
  kubectl run test-pull --image=<acr-name>.azurecr.io/test:latest \
                        --restart=Never \
                        --rm -i --tty
  ```

---

## Phase 6: Application Build & Package

**Estimated Time**: 2-4 hours  
**Dependencies**: Phase 1, Phase 5 (ACR ready)

### 6.1 Build All Packages

- [ ] **Build Shared Packages**
  ```bash
  # From project root
  pnpm build
  
  # Or build individually
  pnpm --filter @the-order/ui build
  pnpm --filter @the-order/auth build
  pnpm --filter @the-order/api-client build
  pnpm --filter @the-order/database build
  pnpm --filter @the-order/storage build
  pnpm --filter @the-order/crypto build
  pnpm --filter @the-order/schemas build
  ```

### 6.2 Build Frontend Applications

- [ ] **Build Portal Public**
  ```bash
  pnpm --filter portal-public build
  ```

- [ ] **Build Portal Internal**
  ```bash
  pnpm --filter portal-internal build
  ```

### 6.3 Build Backend Services

- [ ] **Build Identity Service**
  ```bash
  pnpm --filter @the-order/identity build
  ```

- [ ] **Build Intake Service**
  ```bash
  pnpm --filter @the-order/intake build
  ```

- [ ] **Build Finance Service**
  ```bash
  pnpm --filter @the-order/finance build
  ```

- [ ] **Build Dataroom Service**
  ```bash
  pnpm --filter @the-order/dataroom build
  ```

### 6.4 Create Docker Images

**Note**: Dockerfiles need to be created for each service/app

- [ ] **Create Dockerfiles** (To be created)
  - [ ] `services/identity/Dockerfile`
  - [ ] `services/intake/Dockerfile`
  - [ ] `services/finance/Dockerfile`
  - [ ] `services/dataroom/Dockerfile`
  - [ ] `apps/portal-public/Dockerfile`
  - [ ] `apps/portal-internal/Dockerfile`

- [ ] **Build and Push Images to ACR**
  ```bash
  # Login to ACR
  az acr login --name <acr-name>
  
  # Build and push each service
  # Identity Service
  docker build -t <acr-name>.azurecr.io/identity:latest \
               -t <acr-name>.azurecr.io/identity:$(git rev-parse --short HEAD) \
               -f services/identity/Dockerfile .
  docker push <acr-name>.azurecr.io/identity:latest
  docker push <acr-name>.azurecr.io/identity:$(git rev-parse --short HEAD)
  
  # Intake Service
  docker build -t <acr-name>.azurecr.io/intake:latest \
               -t <acr-name>.azurecr.io/intake:$(git rev-parse --short HEAD) \
               -f services/intake/Dockerfile .
  docker push <acr-name>.azurecr.io/intake:latest
  docker push <acr-name>.azurecr.io/intake:$(git rev-parse --short HEAD)
  
  # Finance Service
  docker build -t <acr-name>.azurecr.io/finance:latest \
               -t <acr-name>.azurecr.io/finance:$(git rev-parse --short HEAD) \
               -f services/finance/Dockerfile .
  docker push <acr-name>.azurecr.io/finance:latest
  docker push <acr-name>.azurecr.io/finance:$(git rev-parse --short HEAD)
  
  # Dataroom Service
  docker build -t <acr-name>.azurecr.io/dataroom:latest \
               -t <acr-name>.azurecr.io/dataroom:$(git rev-parse --short HEAD) \
               -f services/dataroom/Dockerfile .
  docker push <acr-name>.azurecr.io/dataroom:latest
  docker push <acr-name>.azurecr.io/dataroom:$(git rev-parse --short HEAD)
  
  # Portal Public
  docker build -t <acr-name>.azurecr.io/portal-public:latest \
               -t <acr-name>.azurecr.io/portal-public:$(git rev-parse --short HEAD) \
               -f apps/portal-public/Dockerfile .
  docker push <acr-name>.azurecr.io/portal-public:latest
  docker push <acr-name>.azurecr.io/portal-public:$(git rev-parse --short HEAD)
  
  # Portal Internal
  docker build -t <acr-name>.azurecr.io/portal-internal:latest \
               -t <acr-name>.azurecr.io/portal-internal:$(git rev-parse --short HEAD) \
               -f apps/portal-internal/Dockerfile .
  docker push <acr-name>.azurecr.io/portal-internal:latest
  docker push <acr-name>.azurecr.io/portal-internal:$(git rev-parse --short HEAD)
  ```

- [ ] **Sign Images with Cosign** (Security best practice)
  ```bash
  # Generate signing key (one-time)
  cosign generate-key-pair
  
  # Sign each image
  cosign sign --key cosign.key <acr-name>.azurecr.io/identity:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/intake:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/finance:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/dataroom:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/portal-public:latest
  cosign sign --key cosign.key <acr-name>.azurecr.io/portal-internal:latest
  ```

---

## Phase 7: Database Migrations

**Estimated Time**: 1-2 hours  
**Dependencies**: Phase 4 (Database created), Phase 6 (Packages built)

### 7.1 Run Database Migrations

- [ ] **Run Migrations for Each Environment**
  ```bash
  # Development
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_dev"
  pnpm --filter @the-order/database migrate up
  
  # Staging
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_stage"
  pnpm --filter @the-order/database migrate up
  
  # Production
  export DATABASE_URL="postgresql://user:pass@host:5432/theorder_prod"
  pnpm --filter @the-order/database migrate up
  ```

- [ ] **Verify Schema Created**
  ```sql
  \dt  -- List tables
  \d+ <table-name>  -- Describe table
  ```

- [ ] **Seed Initial Data** (If needed)
  ```bash
  # Run seed scripts if they exist
  pnpm --filter @the-order/database seed
  ```

---

## Phase 8: Secrets Configuration

**Estimated Time**: 2-4 hours  
**Dependencies**: Phase 2 (Key Vault created), Phase 3 (Entra ID configured)

### 8.1 Store Secrets in Azure Key Vault

- [ ] **Store Database Credentials**
  ```bash
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "database-url-dev" \
    --value "postgresql://user:pass@host:5432/theorder_dev"
  ```

- [ ] **Store Entra ID Secrets**
  ```bash
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-tenant-id" \
    --value "<tenant-id>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-client-id" \
    --value "<client-id>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-client-secret" \
    --value "<client-secret>"
  
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "entra-credential-manifest-id" \
    --value "<manifest-id>"
  ```

- [ ] **Store Storage Credentials**
  ```bash
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "storage-account-name" \
    --value "<storage-account-name>"
  ```

- [ ] **Store JWT Secrets**
  ```bash
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "jwt-secret" \
    --value "<secure-random-string>"
  ```

- [ ] **Store KMS Keys**
  ```bash
  az keyvault secret set \
    --vault-name <key-vault-name> \
    --name "kms-key-id" \
    --value "<kms-key-id>"
  ```

- [ ] **Store Other Service Secrets**
  ```bash
  # Payment gateway
  az keyvault secret set --vault-name <key-vault-name> --name "payment-gateway-api-key" --value "..."
  
  # OCR service
  az keyvault secret set --vault-name <key-vault-name> --name "ocr-service-api-key" --value "..."
  
  # eIDAS
  az keyvault secret set --vault-name <key-vault-name> --name "eidas-api-key" --value "..."
  ```

### 8.2 Configure External Secrets Operator

- [ ] **Create SecretStore for Azure Key Vault**
  ```yaml
  # infra/k8s/base/external-secrets-store.yaml (to be created)
  apiVersion: external-secrets.io/v1beta1
  kind: SecretStore
  metadata:
    name: azure-keyvault
  spec:
    provider:
      azurekv:
        vaultUrl: https://<key-vault-name>.vault.azure.net
        authType: WorkloadIdentity
        serviceAccountRef:
          name: external-secrets-sa
  ```

- [ ] **Create ExternalSecret Resources**
  ```yaml
  # infra/k8s/base/external-secrets.yaml (to be created)
  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: the-order-secrets
  spec:
    refreshInterval: 1h
    secretStoreRef:
      name: azure-keyvault
      kind: SecretStore
    target:
      name: the-order-secrets
      creationPolicy: Owner
    data:
      - secretKey: DATABASE_URL
        remoteRef:
          key: database-url-dev
      - secretKey: ENTRA_TENANT_ID
        remoteRef:
          key: entra-tenant-id
      # ... more secrets
  ```

- [ ] **Apply External Secrets Configuration**
  ```bash
  kubectl apply -f infra/k8s/base/external-secrets-store.yaml
  kubectl apply -f infra/k8s/base/external-secrets.yaml
  ```

---

## Phase 9: Infrastructure Services Deployment

**Estimated Time**: 1-2 days  
**Dependencies**: Phase 2, Phase 8 (Secrets configured)

### 9.1 Deploy External Secrets Operator

- [ ] **Install External Secrets Operator**
  ```bash
  kubectl apply -f https://external-secrets.io/latest/deploy/
  kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=external-secrets -n external-secrets-system
  ```

### 9.2 Deploy Monitoring Stack

- [ ] **Deploy Prometheus** (To be configured)
  ```bash
  # Using Helm or manifests
  helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  helm install prometheus prometheus-community/kube-prometheus-stack
  ```

- [ ] **Deploy Grafana** (To be configured)
  ```bash
  # Usually included with Prometheus stack
  # Access via port-forward or ingress
  kubectl port-forward svc/prometheus-grafana 3000:80
  ```

- [ ] **Configure OpenTelemetry** (To be configured)
  - Deploy OpenTelemetry Collector
  - Configure exporters
  - Set up trace collection

### 9.3 Deploy Logging Stack

- [ ] **Deploy OpenSearch** (If not using managed service)
  ```bash
  # Deploy OpenSearch operator or Helm chart
  # Configuration to be added
  ```

- [ ] **Configure Log Aggregation**
  - Set up Fluent Bit or Fluentd
  - Configure log forwarding
  - Set up log retention policies

---

## Phase 10: Backend Services Deployment

**Estimated Time**: 2-4 days  
**Dependencies**: Phase 6 (Images built), Phase 7 (Migrations run), Phase 8 (Secrets configured), Phase 9 (Infrastructure ready)

### 10.1 Create Kubernetes Manifests

- [ ] **Create Base Manifests** (To be created)
  - [ ] `infra/k8s/base/identity/deployment.yaml`
  - [ ] `infra/k8s/base/identity/service.yaml`
  - [ ] `infra/k8s/base/intake/deployment.yaml`
  - [ ] `infra/k8s/base/intake/service.yaml`
  - [ ] `infra/k8s/base/finance/deployment.yaml`
  - [ ] `infra/k8s/base/finance/service.yaml`
  - [ ] `infra/k8s/base/dataroom/deployment.yaml`
  - [ ] `infra/k8s/base/dataroom/service.yaml`

### 10.2 Deploy Identity Service

- [ ] **Deploy Identity Service**
  ```bash
  kubectl apply -k infra/k8s/overlays/dev
  # Or for specific service
  kubectl apply -f infra/k8s/base/identity/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=identity -n the-order-dev
  kubectl logs -l app=identity -n the-order-dev
  kubectl get svc identity -n the-order-dev
  ```

- [ ] **Test Health Endpoint**
  ```bash
  kubectl port-forward svc/identity 4002:4002
  curl http://localhost:4002/health
  ```

### 10.3 Deploy Intake Service

- [ ] **Deploy Intake Service**
  ```bash
  kubectl apply -f infra/k8s/base/intake/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=intake -n the-order-dev
  kubectl logs -l app=intake -n the-order-dev
  ```

- [ ] **Test Health Endpoint**
  ```bash
  kubectl port-forward svc/intake 4001:4001
  curl http://localhost:4001/health
  ```

### 10.4 Deploy Finance Service

- [ ] **Deploy Finance Service**
  ```bash
  kubectl apply -f infra/k8s/base/finance/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=finance -n the-order-dev
  kubectl logs -l app=finance -n the-order-dev
  ```

- [ ] **Test Health Endpoint**
  ```bash
  kubectl port-forward svc/finance 4003:4003
  curl http://localhost:4003/health
  ```

### 10.5 Deploy Dataroom Service

- [ ] **Deploy Dataroom Service**
  ```bash
  kubectl apply -f infra/k8s/base/dataroom/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=dataroom -n the-order-dev
  kubectl logs -l app=dataroom -n the-order-dev
  ```

- [ ] **Test Health Endpoint**
  ```bash
  kubectl port-forward svc/dataroom 4004:4004
  curl http://localhost:4004/health
  ```

### 10.6 Verify Service-to-Service Communication

- [ ] **Test Internal Service Communication**
  ```bash
  # From within cluster
  kubectl run test-pod --image=curlimages/curl --rm -it --restart=Never -- \
    curl http://identity:4002/health
  ```

---

## Phase 11: Frontend Applications Deployment

**Estimated Time**: 1-2 days  
**Dependencies**: Phase 6 (Images built), Phase 10 (Backend services deployed)

### 11.1 Deploy Portal Public

- [ ] **Create Kubernetes Manifests** (To be created)
  - [ ] `infra/k8s/base/portal-public/deployment.yaml`
  - [ ] `infra/k8s/base/portal-public/service.yaml`
  - [ ] `infra/k8s/base/portal-public/ingress.yaml`

- [ ] **Deploy Portal Public**
  ```bash
  kubectl apply -f infra/k8s/base/portal-public/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=portal-public -n the-order-dev
  kubectl logs -l app=portal-public -n the-order-dev
  ```

- [ ] **Test Application**
  ```bash
  kubectl port-forward svc/portal-public 3000:3000
  # Open http://localhost:3000 in browser
  ```

### 11.2 Deploy Portal Internal

- [ ] **Create Kubernetes Manifests** (To be created)
  - [ ] `infra/k8s/base/portal-internal/deployment.yaml`
  - [ ] `infra/k8s/base/portal-internal/service.yaml`
  - [ ] `infra/k8s/base/portal-internal/ingress.yaml`

- [ ] **Deploy Portal Internal**
  ```bash
  kubectl apply -f infra/k8s/base/portal-internal/
  ```

- [ ] **Verify Deployment**
  ```bash
  kubectl get pods -l app=portal-internal -n the-order-dev
  kubectl logs -l app=portal-internal -n the-order-dev
  ```

- [ ] **Test Application**
  ```bash
  kubectl port-forward svc/portal-internal 3001:3001
  # Open http://localhost:3001 in browser
  ```

---

## Phase 12: Networking & Gateways

**Estimated Time**: 2-3 days  
**Dependencies**: Phase 10, Phase 11 (Services and apps deployed)

### 12.1 Configure Ingress

- [ ] **Deploy NGINX Ingress Controller** (If not using Application Gateway)
  ```bash
  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  helm install ingress-nginx ingress-nginx/ingress-nginx
  ```

- [ ] **Create Ingress Resources**
  ```yaml
  # infra/k8s/base/ingress.yaml (to be created)
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: the-order-ingress
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
  spec:
    tls:
      - hosts:
          - api.theorder.org
          - portal.theorder.org
          - admin.theorder.org
        secretName: the-order-tls
    rules:
      - host: api.theorder.org
        http:
          paths:
            - path: /identity
              pathType: Prefix
              backend:
                service:
                  name: identity
                  port:
                    number: 4002
      # ... more rules
  ```

- [ ] **Apply Ingress Configuration**
  ```bash
  kubectl apply -f infra/k8s/base/ingress.yaml
  ```

### 12.2 Configure Application Gateway (If using)

- [ ] **Create Application Gateway Backend Pools**
  ```bash
  az network application-gateway address-pool create \
    --resource-group the-order-dev-rg \
    --gateway-name <gateway-name> \
    --name identity-backend \
    --servers <aks-node-ip>
  ```

- [ ] **Configure Routing Rules**
  - Set up path-based routing
  - Configure SSL termination
  - Set up health probes

### 12.3 Configure DNS

- [ ] **Create DNS Records**
  ```bash
  # For each domain
  # api.theorder.org -> Application Gateway IP
  # portal.theorder.org -> Application Gateway IP
  # admin.theorder.org -> Application Gateway IP
  ```

- [ ] **Verify DNS Resolution**
  ```bash
  nslookup api.theorder.org
  nslookup portal.theorder.org
  nslookup admin.theorder.org
  ```

### 12.4 Configure SSL/TLS Certificates

- [ ] **Obtain SSL Certificates**
  - Use Let's Encrypt (cert-manager)
  - Or Azure Key Vault certificates
  - Or import existing certificates

- [ ] **Configure cert-manager** (If using Let's Encrypt)
  ```bash
  kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
  ```

- [ ] **Create ClusterIssuer**
  ```yaml
  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-prod
  spec:
    acme:
      server: https://acme-v02.api.letsencrypt.org/directory
      email: admin@theorder.org
      privateKeySecretRef:
        name: letsencrypt-prod
      solvers:
        - http01:
            ingress:
              class: nginx
  ```

### 12.5 Configure WAF Rules

- [ ] **Configure Azure WAF** (If using Application Gateway)
  - Set up OWASP rules
  - Configure custom rules
  - Set up rate limiting
  - Configure IP allow/deny lists

---

## Phase 13: Monitoring & Observability

**Estimated Time**: 2-3 days  
**Dependencies**: Phase 9, Phase 10, Phase 11 (Services deployed)

### 13.1 Configure Application Insights

- [ ] **Create Application Insights Resources**
  ```bash
  az monitor app-insights component create \
    --app the-order-dev \
    --location westeurope \
    --resource-group the-order-dev-rg
  ```

- [ ] **Configure Application Insights in Services**
  - Add instrumentation keys to services
  - Configure custom metrics
  - Set up alerts

### 13.2 Configure Log Analytics

- [ ] **Create Log Analytics Workspace**
  ```bash
  az monitor log-analytics workspace create \
    --resource-group the-order-dev-rg \
    --workspace-name the-order-dev-logs
  ```

- [ ] **Configure Log Collection**
  - Set up container insights
  - Configure log forwarding
  - Set up log queries

### 13.3 Set Up Alerts

- [ ] **Create Alert Rules**
  ```bash
  # High error rate
  az monitor metrics alert create \
    --name "high-error-rate" \
    --resource-group the-order-dev-rg \
    --scopes <resource-id> \
    --condition "avg Percentage > 5" \
    --window-size 5m \
    --evaluation-frequency 1m
  ```

- [ ] **Configure Alert Actions**
  - Set up email notifications
  - Configure webhook actions
  - Set up PagerDuty integration (if needed)

### 13.4 Configure Dashboards

- [ ] **Create Grafana Dashboards**
  - Service health dashboard
  - Performance metrics dashboard
  - Business metrics dashboard
  - Error tracking dashboard

- [ ] **Configure Azure Dashboards**
  - Create custom dashboards
  - Set up shared dashboards
  - Configure access permissions

---

## Phase 14: Testing & Validation

**Estimated Time**: 3-5 days  
**Dependencies**: All previous phases complete

### 14.1 Health Checks

- [ ] **Verify All Services Healthy**
  ```bash
  # Check all pods
  kubectl get pods -n the-order-dev
  
  # Check service endpoints
  for svc in identity intake finance dataroom portal-public portal-internal; do
    kubectl exec -it deployment/$svc -n the-order-dev -- curl http://localhost/health
  done
  ```

### 14.2 Integration Testing

- [ ] **Test API Endpoints**
  ```bash
  # Identity Service
  curl https://api.theorder.org/identity/health
  curl https://api.theorder.org/identity/vc/issue/entra
  
  # Intake Service
  curl https://api.theorder.org/intake/health
  
  # Finance Service
  curl https://api.theorder.org/finance/health
  
  # Dataroom Service
  curl https://api.theorder.org/dataroom/health
  ```

- [ ] **Test Frontend Applications**
  - [ ] Portal Public accessible
  - [ ] Portal Internal accessible
  - [ ] Authentication flow works
  - [ ] API integration works
  - [ ] Forms submit correctly

### 14.3 End-to-End Testing

- [ ] **Test Complete User Flows**
  - [ ] User registration flow
  - [ ] Application submission flow
  - [ ] Credential issuance flow
  - [ ] Payment processing flow
  - [ ] Document upload flow

### 14.4 Performance Testing

- [ ] **Load Testing**
  ```bash
  # Use tools like k6, Apache Bench, or JMeter
  k6 run load-test.js
  ```

- [ ] **Verify Performance Metrics**
  - Response times acceptable
  - Throughput meets requirements
  - Resource usage within limits

### 14.5 Security Testing

- [ ] **Run Security Scans**
  ```bash
  # Trivy scan
  trivy k8s cluster --severity HIGH,CRITICAL
  
  # Check for exposed secrets
  kubectl get secrets -n the-order-dev
  ```

- [ ] **Verify Security Controls**
  - Network policies configured
  - RBAC properly set up
  - Secrets not exposed
  - TLS/SSL working
  - Authentication required

---

## Phase 15: Production Hardening

**Estimated Time**: 2-3 days  
**Dependencies**: Phase 14 (Testing complete)

### 15.1 Production Configuration

- [ ] **Update Replica Counts**
  ```bash
  # Update kustomization for production
  # Set appropriate replica counts
  kubectl scale deployment identity --replicas=3 -n the-order-prod
  ```

- [ ] **Configure Resource Limits**
  ```yaml
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "512Mi"
      cpu: "500m"
  ```

- [ ] **Configure Liveness and Readiness Probes**
  ```yaml
  livenessProbe:
    httpGet:
      path: /health
      port: 4002
    initialDelaySeconds: 30
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /health
      port: 4002
    initialDelaySeconds: 5
    periodSeconds: 5
  ```

### 15.2 Backup Configuration

- [ ] **Configure Database Backups**
  ```bash
  az postgres server backup create \
    --resource-group the-order-prod-rg \
    --server-name <server-name> \
    --backup-name daily-backup
  ```

- [ ] **Configure Storage Backups**
  - Enable blob versioning
  - Configure retention policies
  - Set up geo-replication (if needed)

### 15.3 Disaster Recovery

- [ ] **Create Backup Procedures**
  - Document backup process
  - Test restore procedures
  - Set up automated backups

- [ ] **Configure Failover**
  - Set up multi-region deployment (if needed)
  - Configure DNS failover
  - Test disaster recovery procedures

### 15.4 Documentation

- [ ] **Update Deployment Documentation**
  - Document all configuration
  - Create runbooks
  - Document troubleshooting steps

- [ ] **Create Operational Runbooks**
  - Incident response procedures
  - Common troubleshooting
  - Escalation procedures

---

## Deployment Checklist Summary

### Pre-Deployment (Phases 1-5)
- [x] Prerequisites installed
- [x] Azure account setup
- [x] Infrastructure deployed
- [x] Entra ID configured
- [x] Database and storage ready
- [x] Container registry ready

### Build & Configure (Phases 6-8)
- [x] Applications built
- [x] Docker images created and pushed
- [x] Database migrations run
- [x] Secrets configured

### Deploy (Phases 9-12)
- [x] Infrastructure services deployed
- [x] Backend services deployed
- [x] Frontend applications deployed
- [x] Networking configured

### Validate & Harden (Phases 13-15)
- [x] Monitoring configured
- [x] Testing complete
- [x] Production hardening done

---

## Environment-Specific Deployment

### Development Environment

```bash
# Deploy to dev
kubectl apply -k infra/k8s/overlays/dev
```

### Staging Environment

```bash
# Deploy to staging
kubectl apply -k infra/k8s/overlays/stage
```

### Production Environment

```bash
# Deploy to production (after approval)
kubectl apply -k infra/k8s/overlays/prod
```

---

## Rollback Procedures

### Rollback Application Deployment

```bash
# Rollback to previous version
kubectl rollout undo deployment/<service-name> -n the-order-prod
```

### Rollback Infrastructure

```bash
# Rollback Terraform changes
terraform plan -destroy
terraform apply -target=<resource>
```

---

## Troubleshooting

### Common Issues

1. **Pods Not Starting**
   ```bash
   kubectl describe pod <pod-name> -n the-order-dev
   kubectl logs <pod-name> -n the-order-dev
   ```

2. **Service Not Accessible**
   ```bash
   kubectl get svc -n the-order-dev
   kubectl get ingress -n the-order-dev
   ```

3. **Database Connection Issues**
   ```bash
   # Check firewall rules
   az postgres server firewall-rule list --server-name <name>
   
   # Test connection
   psql -h <host> -U <user> -d <database>
   ```

---

## Estimated Timeline

| Phase | Duration | Dependencies |
|-------|----------|--------------|
| Phase 1: Prerequisites | 1-2 days | None |
| Phase 2: Azure Infrastructure | 4-6 weeks | Phase 1 |
| Phase 3: Entra ID | 1-2 days | Phase 1 |
| Phase 4: Database & Storage | 1-2 days | Phase 2 |
| Phase 5: Container Registry | 1 day | Phase 2 |
| Phase 6: Build & Package | 2-4 hours | Phase 1, 5 |
| Phase 7: Database Migrations | 1-2 hours | Phase 4, 6 |
| Phase 8: Secrets Configuration | 2-4 hours | Phase 2, 3 |
| Phase 9: Infrastructure Services | 1-2 days | Phase 2, 8 |
| Phase 10: Backend Services | 2-4 days | Phase 6, 7, 8, 9 |
| Phase 11: Frontend Apps | 1-2 days | Phase 6, 10 |
| Phase 12: Networking | 2-3 days | Phase 10, 11 |
| Phase 13: Monitoring | 2-3 days | Phase 9, 10, 11 |
| Phase 14: Testing | 3-5 days | All previous |
| Phase 15: Production Hardening | 2-3 days | Phase 14 |

**Total Estimated Time**: 8-12 weeks (with parallel work on Phases 2-3)

---

## Quick Reference Commands

```bash
# Infrastructure
./infra/scripts/azure-setup.sh
terraform init && terraform plan && terraform apply

# Build
pnpm build
docker build -t <image> -f <Dockerfile> .

# Deploy
kubectl apply -k infra/k8s/overlays/dev
kubectl get pods -n the-order-dev
kubectl logs -f <pod-name> -n the-order-dev

# Verify
kubectl get all -n the-order-dev
kubectl port-forward svc/<service> <port>:<port>
curl http://localhost:<port>/health
```

---

**See individual phase sections for detailed instructions.**