proxmox/docs/03-deployment/PHOENIX_DEPLOYMENT_RUNBOOK.md

Phoenix Deployment Runbook

Target System: Phoenix Core (VLAN 160)
Target Host: r630-01 (192.168.11.11)
VMID Range: 8600-8699
Version: 1.0.0
Last Updated: 2026-01-09 Status: Active Documentation

---

Decision Summary

Phoenix Core uses VMID range 8600-8699 (not 7800-7803) to avoid conflicts with existing legacy containers. This enables parallel deployment with DNS-based cutover.

Phoenix Core Components:

• VMID 8600: Phoenix API (10.160.0.10)
• VMID 8601: Phoenix Portal (10.160.0.11)
• VMID 8602: Phoenix Keycloak (10.160.0.12)
• VMID 8603: Phoenix PostgreSQL (10.160.0.13)

---

Table of Contents

1. Pre-Flight Checks
2. Network Readiness Verification
3. Phase 1: PostgreSQL Deployment (VMID 8603)
4. Phase 2: Keycloak Deployment (VMID 8602)
5. Phase 3: Phoenix API Deployment (VMID 8600)
6. Phase 4: Phoenix Portal Deployment (VMID 8601)
7. Validation Gates
8. Troubleshooting
9. Rollback Procedures

---

Pre-Flight Checks

Before starting deployment, verify the following prerequisites:

1. SSH Access to r630-01

ssh root@192.168.11.11

Verification:

ssh -o StrictHostKeyChecking=no root@192.168.11.11 "pvecm status >/dev/null 2>&1 && echo '✓ Connected' || echo '✗ Connection failed'"

2. Storage Availability

ssh root@192.168.11.11 "pvesm status | grep thin1"

Expected: thin1 storage available with sufficient space (minimum 180GB free for all 4 containers).

3. Source Project Availability

ls -la /home/intlc/projects/Sankofa/api
ls -la /home/intlc/projects/Sankofa/portal

Required: Both api/ and portal/ directories must exist.

4. VMID Availability

ssh root@192.168.11.11 "pct list | grep -E '^860[0-3]'"

Expected: No containers with VMIDs 8600-8603 should exist.

5. IP Address Availability

ssh root@192.168.11.11 "pct list | grep -E '10\.160\.0\.(10|11|12|13)'"

Expected: IPs 10.160.0.10-13 should not be in use.

---

Network Readiness Verification

Step 1: Verify VLAN 160 Configuration

# Check if VLAN 160 exists on the switch/router
ssh root@192.168.11.1 "ip addr show | grep '160' || echo 'VLAN 160 not configured'"

Expected: VLAN 160 interface should exist on the gateway/router.

Step 2: Verify Proxmox Bridge Configuration

# Check bridge configuration
ssh root@192.168.11.11 "cat /etc/network/interfaces | grep -A 5 vmbr0"

Expected: Bridge should support VLAN tagging.

If VLAN-aware bridge needed:

ssh root@192.168.11.11 "cat /etc/network/interfaces.d/vmbr0"
# Should contain: bridge-vlan-aware yes

Step 3: Verify Gateway Accessibility

# Test gateway connectivity
ping -c 3 10.160.0.1

Expected: Gateway (10.160.0.1) should respond to ping.

Step 4: Verify IP Addresses Not in Use

# Test each IP
for ip in 10.160.0.10 10.160.0.11 10.160.0.12 10.160.0.13; do
    ping -c 1 -W 1 $ip 2>&1 | grep -q "100% packet loss" && echo "$ip: Available" || echo "$ip: In use"
done

Expected: All IPs should show "Available".

---

Phase 1: PostgreSQL Deployment (VMID 8603)

Order: Must be deployed first (database is required by other services)

Step 1: Create Container

# On r630-01, create PostgreSQL container
ssh root@192.168.11.11 "pct create 8603 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-postgres-1 \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.13/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"

Step 2: Start Container

ssh root@192.168.11.11 "pct start 8603"
sleep 10

Step 3: Verify Container Status

ssh root@192.168.11.11 "pct status 8603"

Expected: Status should be "running".

Step 4: Install PostgreSQL 16

ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    apt-get update -qq && \
    apt-get install -y -qq wget ca-certificates gnupg lsb-release curl git build-essential sudo'"

ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - && \
    echo \"deb http://apt.postgresql.org/pub/repos/apt \$(lsb_release -cs)-pgdg main\" > /etc/apt/sources.list.d/pgdg.list && \
    apt-get update -qq && \
    apt-get install -y -qq postgresql-16 postgresql-contrib-16'"

Step 5: Configure PostgreSQL

# Generate secure password
DB_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated DB_PASSWORD: $DB_PASSWORD"
# Save this password - you'll need it for the next steps!

# Enable and start PostgreSQL
ssh root@192.168.11.11 "pct exec 8603 -- systemctl enable postgresql && \
    pct exec 8603 -- systemctl start postgresql"

# Wait for PostgreSQL to start
sleep 5

# Create database and user
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sudo -u postgres psql << 'EOF'
CREATE USER phoenix WITH PASSWORD '$DB_PASSWORD';
CREATE DATABASE phoenix OWNER phoenix ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE phoenix TO phoenix;
\\c phoenix
GRANT ALL ON SCHEMA public TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO phoenix;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
CREATE EXTENSION IF NOT EXISTS \"pg_stat_statements\";
EOF\""

Step 6: Configure Network Access

# Allow connections from VLAN 160 subnet
ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'echo \"host    all             all             10.160.0.0/22           md5\" >> /etc/postgresql/16/main/pg_hba.conf'"

# Enable network listening
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sed -i \\\"s/#listen_addresses = 'localhost'/listen_addresses = '*'/\\\" /etc/postgresql/16/main/postgresql.conf\""

# Restart PostgreSQL
ssh root@192.168.11.11 "pct exec 8603 -- systemctl restart postgresql"
sleep 3

Step 7: Verify PostgreSQL

# Test connection
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT version();'\""

Expected: Should return PostgreSQL version information.

---

Phase 2: Keycloak Deployment (VMID 8602)

Order: Deploy after PostgreSQL (requires database)

Step 1: Create Container

ssh root@192.168.11.11 "pct create 8602 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-keycloak-1 \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:30 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.12/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"

Step 2: Start Container and Install Dependencies

ssh root@192.168.11.11 "pct start 8602"
sleep 10

# Install Java 21 and dependencies
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    apt-get update -qq && \
    apt-get install -y -qq openjdk-21-jdk wget curl unzip'"

# Set JAVA_HOME
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'echo \"export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64\" >> /etc/profile'"

Step 3: Create Keycloak Database

# Generate Keycloak database password
KEYCLOAK_DB_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated KEYCLOAK_DB_PASSWORD: $KEYCLOAK_DB_PASSWORD"

# Create database on PostgreSQL container (8603)
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sudo -u postgres psql << 'EOF'
CREATE USER keycloak WITH PASSWORD '$KEYCLOAK_DB_PASSWORD';
CREATE DATABASE keycloak OWNER keycloak ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
EOF\""

Step 4: Download and Install Keycloak

# Download Keycloak 24.0.0
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt && \
    wget -q https://github.com/keycloak/keycloak/releases/download/24.0.0/keycloak-24.0.0.tar.gz && \
    tar -xzf keycloak-24.0.0.tar.gz && \
    mv keycloak-24.0.0 keycloak && \
    rm keycloak-24.0.0.tar.gz && \
    chmod +x keycloak/bin/kc.sh'"

# Build Keycloak (may take several minutes)
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt/keycloak && \
    export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 && \
    ./bin/kc.sh build --db postgres'"

Step 5: Configure Keycloak Service

# Generate admin password
KEYCLOAK_ADMIN_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated KEYCLOAK_ADMIN_PASSWORD: $KEYCLOAK_ADMIN_PASSWORD"

# Generate client secrets
KEYCLOAK_CLIENT_SECRET_API=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-32)
KEYCLOAK_CLIENT_SECRET_PORTAL=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-32)

# Create systemd service
ssh root@192.168.11.11 "pct exec 8602 -- bash -c \"cat > /etc/systemd/system/keycloak.service << 'EOF'
[Unit]
Description=Keycloak Authorization Server
After=network.target

[Service]
Type=idle
User=root
WorkingDirectory=/opt/keycloak
Environment=\\\"JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64\\\"
Environment=\\\"KC_DB=postgres\\\"
Environment=\\\"KC_DB_URL_HOST=10.160.0.13\\\"
Environment=\\\"KC_DB_URL_DATABASE=keycloak\\\"
Environment=\\\"KC_DB_USERNAME=keycloak\\\"
Environment=\\\"KC_DB_PASSWORD=$KEYCLOAK_DB_PASSWORD\\\"
Environment=\\\"KC_HTTP_ENABLED=true\\\"
Environment=\\\"KC_HOSTNAME_STRICT=false\\\"
Environment=\\\"KC_HOSTNAME_PORT=8080\\\"
Environment=\\\"KC_HTTP_PORT=8080\\\"
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
ExecStop=/bin/kill -TERM \\\$MAINPID
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF\""

# Start Keycloak
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'systemctl daemon-reload && \
    systemctl enable keycloak && \
    systemctl start keycloak'"

# Wait for Keycloak to start (may take 1-2 minutes)
echo "Waiting for Keycloak to start..."
sleep 60

# Check if Keycloak is ready
for i in {1..30}; do
    if ssh root@192.168.11.11 "pct exec 8602 -- curl -s -f http://localhost:8080/health/ready >/dev/null 2>&1"; then
        echo "✓ Keycloak is ready"
        break
    fi
    echo "Waiting for Keycloak... ($i/30)"
    sleep 5
done

Step 6: Create Admin User and Clients

# Create admin user (first-time setup only)
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt/keycloak && \
    export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 && \
    ./bin/kc.sh config credentials --server http://localhost:8080 --realm master --user admin --password admin 2>/dev/null || \
    ./bin/kc.sh add-user-keycloak --realm master --username admin --password $KEYCLOAK_ADMIN_PASSWORD'"

# Wait for Keycloak to fully start
sleep 30

# Get admin token and create clients
ssh root@192.168.11.11 "pct exec 8602 -- bash -c \"
TOKEN=\\\$(curl -s -X POST \\\"http://localhost:8080/realms/master/protocol/openid-connect/token\\\" \\
  -H \\\"Content-Type: application/x-www-form-urlencoded\\\" \\
  -d \\\"username=admin\\\" \\
  -d \\\"password=\\\$KEYCLOAK_ADMIN_PASSWORD\\\" \\
  -d \\\"grant_type=password\\\" \\
  -d \\\"client_id=admin-cli\\\" | jq -r '.access_token')

# Create phoenix-api client
curl -s -X POST \\\"http://localhost:8080/admin/realms/master/clients\\\" \\
  -H \\\"Authorization: Bearer \\\$TOKEN\\\" \\
  -H \\\"Content-Type: application/json\\\" \\
  -d '{
    \\\"clientId\\\": \\\"phoenix-api\\\",
    \\\"enabled\\\": true,
    \\\"clientAuthenticatorType\\\": \\\"client-secret\\\",
    \\\"secret\\\": \\\"$KEYCLOAK_CLIENT_SECRET_API\\\",
    \\\"protocol\\\": \\\"openid-connect\\\",
    \\\"publicClient\\\": false,
    \\\"standardFlowEnabled\\\": true,
    \\\"directAccessGrantsEnabled\\\": true,
    \\\"serviceAccountsEnabled\\\": true
  }'

# Create portal-client
curl -s -X POST \\\"http://localhost:8080/admin/realms/master/clients\\\" \\
  -H \\\"Authorization: Bearer \\\$TOKEN\\\" \\
  -H \\\"Content-Type: application/json\\\" \\
  -d '{
    \\\"clientId\\\": \\\"portal-client\\\",
    \\\"enabled\\\": true,
    \\\"clientAuthenticatorType\\\": \\\"client-secret\\\",
    \\\"secret\\\": \\\"$KEYCLOAK_CLIENT_SECRET_PORTAL\\\",
    \\\"protocol\\\": \\\"openid-connect\\\",
    \\\"publicClient\\\": false,
    \\\"standardFlowEnabled\\\": true,
    \\\"directAccessGrantsEnabled\\\": true
  }'
\""

Note: Save these passwords and secrets:

• KEYCLOAK_ADMIN_PASSWORD
• KEYCLOAK_CLIENT_SECRET_API
• KEYCLOAK_CLIENT_SECRET_PORTAL
• KEYCLOAK_DB_PASSWORD

---

Phase 3: Phoenix API Deployment (VMID 8600)

Order: Deploy after PostgreSQL and Keycloak

Step 1: Create Container

ssh root@192.168.11.11 "pct create 8600 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-api-1 \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.10/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"

Step 2: Start Container and Install Node.js

ssh root@192.168.11.11 "pct start 8600"
sleep 10

# Install Node.js 18
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -y -qq nodejs'"

# Install pnpm
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'npm install -g pnpm'"

Step 3: Copy API Project Files

# Create app directory
ssh root@192.168.11.11 "pct exec 8600 -- mkdir -p /opt/phoenix-api"

# Copy API directory (assuming source is on deployment machine)
# If source is on r630-01, adjust path accordingly
# If source is remote, use rsync or scp
rsync -avz --exclude node_modules --exclude .git \
    /home/intlc/projects/Sankofa/api/ \
    root@192.168.11.11:/tmp/phoenix-api-source/

ssh root@192.168.11.11 "pct push 8600 /tmp/phoenix-api-source /opt/phoenix-api --recursive"

Step 4: Install Dependencies and Configure

# Install dependencies
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm install --frozen-lockfile'"

# Create environment file (use the passwords/secrets generated earlier)
ssh root@192.168.11.11 "pct exec 8600 -- bash -c \"cat > /opt/phoenix-api/.env << 'EOF'
# Database
DB_HOST=10.160.0.13
DB_PORT=5432
DB_NAME=phoenix
DB_USER=phoenix
DB_PASSWORD=$DB_PASSWORD

# Keycloak
KEYCLOAK_URL=http://10.160.0.12:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=phoenix-api
KEYCLOAK_CLIENT_SECRET=$KEYCLOAK_CLIENT_SECRET_API
KEYCLOAK_MULTI_REALM=false

# API
API_PORT=4000
JWT_SECRET=$(openssl rand -base64 32)
NODE_ENV=production

# Multi-Tenancy
ENABLE_MULTI_TENANT=true
EOF\""

Step 5: Run Migrations and Build

# Run database migrations
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm db:migrate'"

# Build API
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm build'"

Step 6: Create Systemd Service

ssh root@192.168.11.11 "pct exec 8600 -- bash -c \"cat > /etc/systemd/system/phoenix-api.service << 'EOF'
[Unit]
Description=Phoenix API Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/phoenix-api
Environment=\\\"NODE_ENV=production\\\"
EnvironmentFile=/opt/phoenix-api/.env
ExecStart=/usr/bin/node /opt/phoenix-api/dist/server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF\""

# Start service
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'systemctl daemon-reload && \
    systemctl enable phoenix-api && \
    systemctl start phoenix-api'"

sleep 10

# Verify service is running
ssh root@192.168.11.11 "pct exec 8600 -- systemctl status phoenix-api --no-pager | head -10"

---

Phase 4: Phoenix Portal Deployment (VMID 8601)

Order: Deploy last (depends on API)

Step 1: Create Container

ssh root@192.168.11.11 "pct create 8601 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-portal-1 \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.11/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"

Step 2: Start Container and Install Node.js

ssh root@192.168.11.11 "pct start 8601"
sleep 10

# Install Node.js 18
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -y -qq nodejs'"

# Install pnpm
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'npm install -g pnpm'"

Step 3: Copy Portal Project Files

# Copy portal directory
rsync -avz --exclude node_modules --exclude .git --exclude .next \
    /home/intlc/projects/Sankofa/portal/ \
    root@192.168.11.11:/tmp/phoenix-portal-source/

ssh root@192.168.11.11 "pct push 8601 /tmp/phoenix-portal-source /opt/phoenix-portal --recursive"

Step 4: Install Dependencies and Configure

# Install dependencies
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm install --frozen-lockfile'"

# Create environment file
ssh root@192.168.11.11 "pct exec 8601 -- bash -c \"cat > /opt/phoenix-portal/.env.local << 'EOF'
# Keycloak
KEYCLOAK_URL=http://10.160.0.12:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=portal-client
KEYCLOAK_CLIENT_SECRET=$KEYCLOAK_CLIENT_SECRET_PORTAL

# API
NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://10.160.0.10:4000/graphql
NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://10.160.0.10:4000/graphql-ws

# NextAuth
NEXTAUTH_URL=http://10.160.0.11:3000
NEXTAUTH_SECRET=$(openssl rand -base64 32)

# App
NEXT_PUBLIC_APP_URL=http://10.160.0.11:3000
NODE_ENV=production
EOF\""

Step 5: Build Portal

# Build Portal (may take several minutes)
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm build'"

Step 6: Create Systemd Service

ssh root@192.168.11.11 "pct exec 8601 -- bash -c \"cat > /etc/systemd/system/phoenix-portal.service << 'EOF'
[Unit]
Description=Phoenix Portal
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/phoenix-portal
Environment=\\\"NODE_ENV=production\\\"
EnvironmentFile=/opt/phoenix-portal/.env.local
ExecStart=/usr/bin/node /opt/phoenix-portal/node_modules/.bin/next start
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF\""

# Start service
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'systemctl daemon-reload && \
    systemctl enable phoenix-portal && \
    systemctl start phoenix-portal'"

sleep 15

# Verify service is running
ssh root@192.168.11.11 "pct exec 8601 -- systemctl status phoenix-portal --no-pager | head -10"

---

Validation Gates

Phoenix is NOT "live" until all validation gates pass:

Gate 1: Container Status

for vmid in 8600 8601 8602 8603; do
    status=$(ssh root@192.168.11.11 "pct status $vmid" 2>/dev/null | awk '{print $2}')
    echo "VMID $vmid: $status"
done

Expected: All containers should show "running".

Gate 2: PostgreSQL Database

ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT 1;'\""

Expected: Should return "1" without errors.

Gate 3: Keycloak Health

ssh root@192.168.11.11 "pct exec 8602 -- curl -s http://localhost:8080/health/ready"

Expected: Should return JSON with status "UP".

Gate 4: Keycloak Token Issuance

ssh root@192.168.11.11 "pct exec 8602 -- curl -s -X POST 'http://localhost:8080/realms/master/protocol/openid-connect/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin' \
  -d 'password=$KEYCLOAK_ADMIN_PASSWORD' \
  -d 'grant_type=password' \
  -d 'client_id=admin-cli' | jq -r '.access_token' | head -c 50"

Expected: Should return an access token (JWT string).

Gate 5: API Health Endpoint

curl -s http://10.160.0.10:4000/health

Expected: Should return healthy status (may be JSON or plain text).

Gate 6: API Token Validation

# Get token from Keycloak
TOKEN=$(ssh root@192.168.11.11 "pct exec 8602 -- curl -s -X POST 'http://localhost:8080/realms/master/protocol/openid-connect/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin' \
  -d 'password=$KEYCLOAK_ADMIN_PASSWORD' \
  -d 'grant_type=password' \
  -d 'client_id=admin-cli' | jq -r '.access_token'")

# Test API with token
curl -s -H "Authorization: Bearer $TOKEN" http://10.160.0.10:4000/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ __typename }"}'

Expected: Should return GraphQL response.

Gate 7: Portal Accessibility

curl -s -I http://10.160.0.11:3000 | head -1

Expected: Should return HTTP 200 or 302 (redirect).

Gate 8: Database Persistence

# Restart PostgreSQL container
ssh root@192.168.11.11 "pct reboot 8603"
sleep 30

# Test database after restart
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT 1;'\""

Expected: Database should be accessible after restart.

Gate 9: Service Survivability

# Reboot host (if in maintenance window)
# ssh root@192.168.11.11 "reboot"
# Wait for host to come back up, then verify all services start automatically

# Check all services are active
for vmid in 8600 8601 8602 8603; do
    ssh root@192.168.11.11 "pct status $vmid"
done

Expected: All containers should auto-start and services should be active.

Gate 10: No Dependency on 192.168.11.x

# Verify no hardcoded references to management network
ssh root@192.168.11.11 "pct exec 8600 -- env | grep -i '192.168.11' || echo 'No 192.168.11.x dependencies'"
ssh root@192.168.11.11 "pct exec 8601 -- env | grep -i '192.168.11' || echo 'No 192.168.11.x dependencies'"

Expected: Should show "No 192.168.11.x dependencies".

---

Troubleshooting

Container Won't Start

Symptoms: Container status shows "stopped" after pct start.

Diagnosis:

ssh root@192.168.11.11 "pct status 8600"
ssh root@192.168.11.11 "journalctl -u pve-container@8600 -n 50"

Common Causes:

• Network configuration error
• Storage full
• Template not available

Solution:

• Check network config: ssh root@192.168.11.11 "pct config 8600"
• Check storage: ssh root@192.168.11.11 "pvesm status"
• Check template: ssh root@192.168.11.11 "pvesm list local"

PostgreSQL Connection Issues

Symptoms: API cannot connect to database.

Diagnosis:

# From API container
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'PGPASSWORD=password psql -h 10.160.0.13 -U phoenix -d phoenix -c \"SELECT 1;\"'"

Common Causes:

• Firewall blocking port 5432
• PostgreSQL not listening on network interface
• Wrong password

Solution:

• Check pg_hba.conf: ssh root@192.168.11.11 "pct exec 8603 -- cat /etc/postgresql/16/main/pg_hba.conf | grep 10.160.0.0/22"
• Check postgresql.conf: ssh root@192.168.11.11 "pct exec 8603 -- grep listen_addresses /etc/postgresql/16/main/postgresql.conf"
• Verify password matches

Keycloak Not Starting

Symptoms: Keycloak service fails to start or health check fails.

Diagnosis:

ssh root@192.168.11.11 "pct exec 8602 -- journalctl -u keycloak -n 100 --no-pager"
ssh root@192.168.11.11 "pct exec 8602 -- ps aux | grep keycloak"

Common Causes:

• Java not found
• Database connection failed
• Port 8080 already in use

Solution:

• Check Java: ssh root@192.168.11.11 "pct exec 8602 -- java -version"
• Check database connectivity from Keycloak container
• Check port: ssh root@192.168.11.11 "pct exec 8602 -- netstat -tlnp | grep 8080"

API Service Issues

Symptoms: API service fails to start or health check fails.

Diagnosis:

ssh root@192.168.11.11 "pct exec 8600 -- journalctl -u phoenix-api -n 100 --no-pager"
ssh root@192.168.11.11 "pct exec 8600 -- systemctl status phoenix-api --no-pager"

Common Causes:

• Database connection failed
• Keycloak connection failed
• Build errors
• Missing environment variables

Solution:

• Check environment file: ssh root@192.168.11.11 "pct exec 8600 -- cat /opt/phoenix-api/.env"
• Verify database connection
• Verify Keycloak is accessible: curl http://10.160.0.12:8080/health/ready

Portal Build Failures

Symptoms: Portal build fails or service won't start.

Diagnosis:

ssh root@192.168.11.11 "pct exec 8601 -- journalctl -u phoenix-portal -n 100 --no-pager"
# Check build logs (if available)
ssh root@192.168.11.11 "pct exec 8601 -- cat /opt/phoenix-portal/.next/build.log 2>/dev/null || echo 'No build log'"

Common Causes:

• Build errors
• Missing environment variables
• API endpoint unreachable

Solution:

• Rebuild: ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm build'"
• Check environment: ssh root@192.168.11.11 "pct exec 8601 -- cat /opt/phoenix-portal/.env.local"
• Verify API is accessible: curl http://10.160.0.10:4000/health

---

Rollback Procedures

Scenario 1: Rollback Before DNS Cutover

If issues are discovered before DNS cutover, rollback is simple:

1. Stop all Phoenix containers:

for vmid in 8600 8601 8602 8603; do
    ssh root@192.168.11.11 "pct stop $vmid"
done

2. Do NOT delete containers (they may contain valuable debugging information)

3. Legacy services (7800-series) remain operational - no action needed

Scenario 2: Rollback After DNS Cutover

If issues are discovered after DNS cutover:

1. Revert DNS records (see DNS template document for exact records)
2. Stop Phoenix containers (as above)
3. Legacy services become active again via DNS

Scenario 3: Partial Rollback

If only one service has issues:

1. Stop only the problematic container
2. Other services continue running
3. Re-deploy the problematic service after fixing issues

Data Preservation

Important: Database data is preserved in VMID 8603. If rolling back:

• Option 1: Keep container stopped (data preserved)
• Option 2: Export data before deletion: pg_dump -h 10.160.0.13 -U phoenix phoenix > backup.sql
• Option 3: Backup entire container: vzdump 8603

---

Post-Deployment Checklist

• All validation gates passed
• All services running and accessible
• Database backups configured
• Log rotation configured (prevent disk growth)
• Monitoring configured (optional)
• Firewall rules applied (see firewall rules document)
• DNS records ready (see DNS template document)
• Documentation updated
• Team notified of deployment

---

Next Steps

After successful deployment:

1. Configure DNS (see PHOENIX_DNS_ZONE_TEMPLATE.md)
2. Configure Firewall Rules (see PHOENIX_VLAN160_FIREWALL_RULES.md)
3. Set up monitoring (optional)
4. Configure backups for database
5. Document credentials securely
6. Plan DNS cutover (when ready to go live)

---

Last Updated: 2026-01-09
Status: Ready for Deployment