proxmox/docs/04-configuration/SANKOFA_CUTOVER_PLAN.md

Sankofa Cutover Plan

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-20
Status: Cutover Plan for Sankofa Services Deployment
Purpose: Step-by-step plan to cutover Sankofa domains from temporary Blockscout routing to actual Sankofa services

---

Current State

Sankofa Domains (5 Total)

Domain	SSL Cert ID	NPMplus Proxy Host ID	Current Backend	Status
sankofa.nexus	57	21	192.168.11.140:80 (Blockscout)	⚠️ Temporary
www.sankofa.nexus	64	22	192.168.11.140:80 (Blockscout)	⚠️ Temporary
phoenix.sankofa.nexus	51	23	192.168.11.140:80 (Blockscout)	⚠️ Temporary
www.phoenix.sankofa.nexus	63	24	192.168.11.140:80 (Blockscout)	⚠️ Temporary
the-order.sankofa.nexus	60	25	192.168.11.140:80 (Blockscout)	⚠️ Temporary

Current Issue: All 5 Sankofa domains route to Blockscout (VMID 5000) but Sankofa services are NOT deployed.

SSL Certificates: All certificates exist and are valid until 2026-04-16.

NPMplus Proxy Hosts: All proxy hosts exist and are configured, but point to wrong backend.

---

Pre-Cutover Checklist

Before proceeding with cutover, verify:

1. Sankofa Services Deployed and Validated

• Sankofa services deployed on Proxmox VMs
• VMIDs assigned and documented
• IP addresses assigned and documented
• Services running and healthy
• Health endpoints responding

Verification Commands:

# Check if Sankofa VMs exist
for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
done

# Test health endpoints
for ip in <SANKOFA_IPS>; do
    curl -I http://$ip:80
done

2. VMIDs and IPs Assigned

Document the following before cutover:

Domain	Target VMID	Target IP	Target Port	Service Type	Notes
sankofa.nexus	7801	192.168.11.51	3000	Portal	Sankofa Portal (Microsoft Website)
www.sankofa.nexus	7801	192.168.11.51	3000	Portal	Sankofa Portal (Microsoft Website)
phoenix.sankofa.nexus	7800	192.168.11.50	4000	API	Phoenix API (Azure-like Portal)
www.phoenix.sankofa.nexus	7800	192.168.11.50	4000	API	Phoenix API (Azure-like Portal)
the-order.sankofa.nexus	⚠️ TBD	⚠️ TBD	⚠️ TBD	⚠️ TBD	To be determined

Note: Replace ⚠️ TBD with actual values once Sankofa services are deployed.

3. Health Endpoints Verified

• All Sankofa services have health endpoints
• Health endpoints return HTTP 200 or appropriate status
• Health endpoints respond within acceptable timeout

Example Health Check:

# Test health endpoint
curl -I http://<SANKOFA_IP>:<PORT>/health
# Expected: HTTP 200

4. Internal Connectivity Tested

• NPMplus can reach Sankofa services internally
• Test from NPMplus container: curl -I http://<SANKOFA_IP>:<PORT>
• Verify no firewall rules blocking internal traffic

Verification Command:

NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://<SANKOFA_IP>:<PORT>"

5. SSL Certificates Valid

• Verify certificates exist for all 5 Sankofa domains
• Check certificate expiration dates
• Renew certificates if expired (or allow auto-renewal)

Verification Command:

bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains

---

Cutover Steps

Step 1: Verify Current State

Before making changes, document current state:

1. Export Current NPMplus Configuration:

bash scripts/verify/export-npmplus-config.sh

2. Verify Current Routing:

# Check current proxy hosts for Sankofa domains
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Get current proxy host for sankofa.nexus (example)
curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | \
    jq '.[] | select(.domain_names[] == "sankofa.nexus")'

3. Document Current State:
   • All 5 Sankofa domains currently route to 192.168.11.140:80 (Blockscout)
   • SSL certificates exist (IDs: 51, 57, 60, 63, 64)
   • Proxy hosts exist (IDs: 21-25)

---

Step 2: Deploy Sankofa Services

Complete Sankofa service deployment:

1. Deploy Services:

   • Deploy Sankofa services on Proxmox VMs
   • Assign VMIDs and IP addresses
   • Configure services (nginx if needed, application config, etc.)

2. Verify Services Running:

for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
    ssh root@<PROXMOX_HOST> "pct exec $vmid -- curl -I http://localhost:<PORT>"
done

3. Document Actual IPs/Ports:
   • Update the TBD table above with actual values
   • Record VMIDs, IPs, ports, and service types

---

Step 3: Update NPMplus Proxy Hosts

Update all 5 Sankofa proxy hosts to point to actual services:

Option A: Update via NPMplus API

NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Update sankofa.nexus (example)
# Replace <TARGET_IP> and <TARGET_PORT> with actual values
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "<TARGET_IP>",
        "forward_port": <TARGET_PORT>,
        "allow_websocket_upgrade": false
    }'

Option B: Update via NPMplus Web UI

1. Access NPMplus: https://192.168.11.166:81
2. Navigate to: Proxy Hosts
3. For each Sankofa domain:
   • Click Edit
   • Update Forward Hostname/IP: <TARGET_IP>
   • Update Forward Port: <TARGET_PORT>
   • Update Forward Scheme: http (or https if needed)
   • Save

Domains to Update:

Domain	Proxy Host ID	Current Backend	New Backend
sankofa.nexus	21	192.168.11.140:80	192.168.11.51:3000
www.sankofa.nexus	22	192.168.11.140:80	192.168.11.51:3000
phoenix.sankofa.nexus	23	192.168.11.140:80	192.168.11.50:4000
www.phoenix.sankofa.nexus	24	192.168.11.140:80	192.168.11.50:4000
the-order.sankofa.nexus	25	192.168.11.140:80	⚠️ TBD (to be determined)

Note: the-order.sankofa.nexus target service needs to be determined.

---

Step 4: Verify SSL Certificates

Check certificate validity:

1. Verify Certificates Exist:

bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains

2. Check Certificate Expiration:

# For each Sankofa domain
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo | openssl s_client -connect "$domain:443" -servername "$domain" 2>/dev/null | \
        openssl x509 -noout -dates
done

3. Renew if Expired (if needed):
   • Access NPMplus UI
   • Navigate to SSL Certificates
   • Select certificate and click Renew
   • Or wait for auto-renewal (if enabled)

Expected: All certificates valid until 2026-04-16 (auto-renewal enabled).

---

Step 5: Test End-to-End Routing

Verify complete routing flow:

1. Test DNS Resolution:

for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    dig +short "$domain" @8.8.8.8
done

2. Test HTTPS Requests:

for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    curl -s -o /dev/null -w "%{http_code}" "https://$domain"
    echo ""
done

3. Test Internal Connectivity:

NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

for ip in <SANKOFA_IPS>; do
    ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://$ip:<PORT>"
done

Expected Results:

• DNS resolves to 76.53.10.36
• HTTPS requests return HTTP 200 (or appropriate status)
• SSL certificates valid and match domains
• Internal connectivity succeeds

---

Step 6: Update Source-of-Truth Data

Update canonical data model:

1. Run Verification:

bash scripts/verify/run-full-verification.sh

2. Review Generated JSON:

cat docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json | jq '.backend_vms[] | select(.public_domains[] | contains("sankofa"))'

3. Update Status Fields:
   • Change status: "documented" to status: "verified" for Sankofa domains
   • Update backend_vms array with actual Sankofa VM details
   • Remove Sankofa from issues array (or mark as resolved)

---

Step 7: Update Baseline Documentation

Update reference docs with actual values:

1. Update Comprehensive Architecture Doc:

   • File: docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md
   • Replace TBD values with actual Sankofa VM details
   • Update status from ⚠️ to ✅

2. Update Streamlined Table Doc:

   • File: docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md
   • Replace TBD values with actual Sankofa VM details
   • Update status from ⚠️ Not Deployed to ✅ Active

---

Rollback Plan

If cutover fails or issues occur:

Immediate Rollback (Within 5 Minutes)

1. Restore NPMplus Proxy Hosts:

# Restore sankofa.nexus to Blockscout (example)
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "192.168.11.140",
        "forward_port": 80,
        "allow_websocket_upgrade": false
    }'

2. Repeat for All 5 Domains:

   • Update all 5 proxy hosts back to 192.168.11.140:80

3. Verify Rollback:

# Test domains
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    curl -I "https://$domain"
done

Complete Rollback (If Needed)

1. Restore from Backup:

   • If NPMplus backup exists, restore database
   • Or manually update all proxy hosts via UI

2. Verify Services:

   • Verify Blockscout still responds on 192.168.11.140:80
   • Test all Sankofa domains route to Blockscout

---

Post-Cutover Validation Checklist

After cutover, verify:

1. DNS Resolution

• All Sankofa domains resolve to 76.53.10.36
• DNS propagation complete (test from external DNS servers)

2. SSL Certificates

• All certificates valid and match domains
• No certificate errors in browser
• Certificate chain valid

3. HTTPS Requests

• All Sankofa domains respond with HTTP 200 (or appropriate status)
• No 502 Bad Gateway errors
• No 503 Service Unavailable errors

4. Service Functionality

• Sankofa services accessible via HTTPS
• All features working correctly
• Health endpoints responding

5. Internal Connectivity

• NPMplus can reach all Sankofa services
• Internal HTTP requests succeed
• No firewall issues

6. Monitoring

• Set up monitoring for Sankofa services
• Configure alerts for service downtime
• Monitor certificate expiration

---

Verification Scripts

Run full verification after cutover:

# Full verification
bash scripts/verify/run-full-verification.sh

# Specific tests
bash scripts/verify/verify-end-to-end-routing.sh
bash scripts/verify/verify-backend-vms.sh
bash scripts/verify/export-npmplus-config.sh

---

Timeline Estimate

Task	Estimated Time	Notes
Pre-cutover verification	30 minutes	Verify services deployed
Update NPMplus proxy hosts	15 minutes	5 domains to update
Verify SSL certificates	10 minutes	Check expiration
Test end-to-end routing	20 minutes	Test all domains
Update documentation	15 minutes	Update source-of-truth and baseline docs
Total	90 minutes	Without rollback

---

Success Criteria

Cutover is successful if:

1. ✅ All 5 Sankofa domains route to actual Sankofa services (not Blockscout)
2. ✅ All SSL certificates valid and working
3. ✅ All HTTPS requests return HTTP 200 (or appropriate status)
4. ✅ Internal connectivity verified
5. ✅ Source-of-truth JSON updated
6. ✅ Baseline documentation updated

---

Related Documentation

• Verification Runbook: docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md
• Comprehensive Architecture: docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md
• Streamlined Table: docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md
• NPMplus API Scripts: scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh

---

Last Updated: 2026-01-20
Maintained By: Infrastructure Team
Status: Cutover Plan Ready - Awaiting Sankofa Service Deployment