proxmox/docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md

Ingress Architecture Verification Runbook

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-20
Status: Complete Verification Procedures
Purpose: Operator-grade verification procedures for ingress architecture

---

Overview

This runbook provides step-by-step verification procedures for the complete ingress architecture:

Cloudflare DNS → UDM Pro port-forward → NPMplus (reverse proxy + SSL termination) → Backend VMs/services (nginx or direct ports)

Prerequisites

Access Credentials

Ensure the following are configured in .env:

# Cloudflare API
CLOUDFLARE_API_TOKEN="your-token"
# OR
# CLOUDFLARE_EMAIL="your-email"  # Set in .env file
# CLOUDFLARE_API_KEY="your-key"  # Set in .env file

# NPMplus API
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
NPM_PASSWORD="your-password"

# Network
PUBLIC_IP="76.53.10.36"
NPMPLUS_IP="192.168.11.166"

Tools Required

• curl - HTTP/HTTPS requests
• jq - JSON parsing
• dig - DNS queries
• openssl - SSL certificate inspection
• ssh - Remote access to Proxmox hosts and VMs
• wscat (optional) - WebSocket testing: npm install -g wscat

Proxmox Host Access

• SSH access to Proxmox hosts:
  • r630-01 (192.168.11.11)
  • r630-02 (192.168.11.12)
  • ml110 (192.168.11.10)

---

Quick Start

Run the full verification:

cd /home/intlc/projects/proxmox
bash scripts/verify/run-full-verification.sh

This will:

1. Verify Cloudflare DNS records
2. Verify UDM Pro port forwarding (with manual steps)
3. Export and verify NPMplus configuration
4. Verify all backend VMs
5. Test end-to-end routing
6. Generate source-of-truth JSON

---

Verification Steps

Step 1: Cloudflare DNS Verification

Objective: Verify all 19 DNS records match documented state

Automated Script:

bash scripts/verify/export-cloudflare-dns-records.sh

Manual Steps (if script unavailable):

1. Export DNS Records via Cloudflare API:

# Set credentials
export CLOUDFLARE_API_TOKEN="your-token"

# Get zone IDs
for zone in d-bis.org mim4u.org sankofa.nexus defi-oracle.io; do
    curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$zone" \
        -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
        -H "Content-Type: application/json" | jq '.result[0].id'
done

# Get all A records for each zone
ZONE_ID="your-zone-id"
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=A" \
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
    -H "Content-Type: application/json" | jq '.result[]'

2. Verify Each Record:

   • Type: Should be A
   • Content: Should be 76.53.10.36
   • Proxied: Should be false (DNS Only / gray cloud)
   • TTL: Capture actual value

3. Test DNS Resolution:

for domain in explorer.d-bis.org mim4u.org sankofa.nexus; do
    dig +short "$domain" @8.8.8.8
done

Expected Results:

• All 19 domains resolve to 76.53.10.36
• All records are DNS Only (proxied: false)
• TTL values are reasonable

Evidence to Capture:

• JSON export: docs/04-configuration/verification-evidence/dns-verification-*/all_dns_records.json
• Verification report: docs/04-configuration/verification-evidence/dns-verification-*/verification_report.md

---

Step 2: UDM Pro Port Forwarding Verification

Objective: Verify WAN IP and NAT port forwarding rules

Automated Script:

bash scripts/verify/verify-udm-pro-port-forwarding.sh

Manual Verification Steps (Required):

1. Access UDM Pro Web Interface:

   • URL: https://192.168.0.1 (or your UDM Pro IP)
   • Login with admin credentials

2. Navigate to Port Forwarding:

   • Settings → Firewall & Security → Port Forwarding
   • Or: Networks → Port Forwarding Rules

3. Verify Rules Exist:

   Rule 1: NPMplus HTTPS

   • Name: NPMplus HTTPS (or similar)
   • Source: Any (or specific IP)
   • Destination IP: 76.53.10.36
   • Destination Port: 443
   • Forward to IP: 192.168.11.166
   • Forward to Port: 443
   • Protocol: TCP
   • Interface: WAN

   Rule 2: NPMplus HTTP

   • Name: NPMplus HTTP (or similar)
   • Source: Any (or specific IP)
   • Destination IP: 76.53.10.36
   • Destination Port: 80
   • Forward to IP: 192.168.11.166
   • Forward to Port: 80
   • Protocol: TCP
   • Interface: WAN

4. Capture Evidence:

   • Take screenshot of port forwarding rules
   • Save as: docs/04-configuration/verification-evidence/udm-pro-verification-*/udm-pro-port-forwarding-screenshot.png
   • Export UDM Pro config (if available): Settings → Maintenance → Download Backup

5. Test Internal Connectivity:

# Test HTTP
curl -I http://192.168.11.166:80

# Test HTTPS
curl -I -k https://192.168.11.166:443

Expected Results:

• Both rules exist and are enabled
• Internal connectivity succeeds (HTTP 200 or similar)
• Screenshots captured

Evidence to Capture:

• Screenshot of port forwarding rules
• UDM Pro config export (if available)
• Internal connectivity test results

---

Step 3: NPMplus Configuration Verification

Objective: Verify NPMplus configuration matches documentation

Automated Script:

bash scripts/verify/export-npmplus-config.sh

Manual Steps (if script unavailable):

1. Authenticate to NPMplus API:

NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
NPM_PASSWORD="your-password"

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

2. Export Proxy Hosts:

curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > proxy_hosts.json

3. Export Certificates:

curl -s -k -X GET "$NPM_URL/api/nginx/certificates" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > certificates.json

4. Verify Certificate Files:

NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- ls -la /data/tls/certbot/live/"

Expected Results:

• 26 proxy hosts exported
• 19 certificates exported
• Certificate files exist on disk
• Certificate expiration dates match API data

Evidence to Capture:

• JSON export: docs/04-configuration/verification-evidence/npmplus-verification-*/proxy_hosts.json
• JSON export: docs/04-configuration/verification-evidence/npmplus-verification-*/certificates.json
• Verification report: docs/04-configuration/verification-evidence/npmplus-verification-*/verification_report.md

---

Step 4: Backend VMs Verification

Objective: Verify each backend VM matches documented configuration

Automated Script:

bash scripts/verify/verify-backend-vms.sh

Manual Steps (per VM):

VMs to Verify (8 total):

• With Nginx: 5000, 7810, 10130, 2400
• Without Nginx: 2101, 2201, 10150, 10151

Example Verification for VMID 5000:

VMID=5000
HOST=192.168.11.12

# Check status
ssh root@"$HOST" "pct status $VMID"

# Get IP
ssh root@"$HOST" "pct config $VMID | grep ip"

# Check nginx
ssh root@"$HOST" "pct exec $VMID -- systemctl status nginx"

# Check listening ports
ssh root@"$HOST" "pct exec $VMID -- ss -lntp"

# Check nginx config
ssh root@"$HOST" "pct exec $VMID -- ls -la /etc/nginx/sites-enabled/"

# Test health endpoint
curl -I http://192.168.11.140:80

Expected Results:

• All VMs are running
• IP addresses match expected values
• Services are active (nginx/Besu/Node.js)
• Expected ports are listening
• Health endpoints respond

Evidence to Capture:

• VM verification JSON: docs/04-configuration/verification-evidence/backend-vms-verification-*/vmid_*_verification.json
• Listening ports output: docs/04-configuration/verification-evidence/backend-vms-verification-*/vmid_*_listening_ports.txt
• Verification report: docs/04-configuration/verification-evidence/backend-vms-verification-*/verification_report.md

---

Step 5: End-to-End Routing Verification

Objective: Verify complete request flow from external to backend

Automated Script:

bash scripts/verify/verify-end-to-end-routing.sh

When RPC is via Cloudflare Tunnel (Option B), the 6 RPC hostnames resolve to Cloudflare IPs; use ACCEPT_ANY_DNS=1 for a full E2E pass: ACCEPT_ANY_DNS=1 bash scripts/verify/verify-end-to-end-routing.sh. See OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md.

Manual Steps (per domain):

1. Test DNS Resolution:

dig +short explorer.d-bis.org @8.8.8.8
# Expected: 76.53.10.36 (or Cloudflare IPs for RPC hostnames when using Option B tunnel)

2. Test SSL Certificate:

echo | openssl s_client -connect explorer.d-bis.org:443 -servername explorer.d-bis.org 2>/dev/null | \
    openssl x509 -noout -subject -issuer -dates -ext subjectAltName

3. Test HTTPS Request:

curl -vI https://explorer.d-bis.org
# Check:
# - HTTP response code (200/301/302)
# - SSL certificate CN/SAN matches domain
# - Headers: HSTS, CSP, X-Frame-Options

4. Test RPC Endpoint (for RPC domains):

curl -X POST https://rpc-http-pub.d-bis.org \
    -H 'Content-Type: application/json' \
    -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'

5. Test WebSocket (for RPC WS domains, requires wscat):

wscat -c wss://rpc-ws-pub.d-bis.org
# Then send: {"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}

Expected Results:

• Web/api domains resolve to 76.53.10.36; RPC hostnames may resolve to Cloudflare IPs when using Option B tunnel
• SSL certificates valid and match domains
• HTTP/HTTPS responses return valid codes
• RPC endpoints respond with JSON-RPC results

Evidence to Capture:

• E2E test results: docs/04-configuration/verification-evidence/e2e-verification-*/all_e2e_results.json
• HTTP headers: docs/04-configuration/verification-evidence/e2e-verification-*/*_https_headers.txt
• Verification report: docs/04-configuration/verification-evidence/e2e-verification-*/verification_report.md

---

Evidence Collection Checklist

DNS Verification

• JSON export of all DNS records
• Verification report showing status per domain
• Screenshot of Cloudflare DNS dashboard (optional)

UDM Pro Verification

• Screenshot of port forwarding rules
• UDM Pro config export (if available)
• Internal connectivity test results

NPMplus Verification

• JSON export of proxy hosts
• JSON export of certificates
• Certificate file verification results
• Verification report

Backend VMs Verification

• VM status for each VMID
• IP addresses
• Service status (nginx/systemd/docker)
• Listening ports output
• Config file paths (for nginx VMs)
• Health check results

E2E Verification

• DNS resolution results per domain
• SSL certificate details per domain
• HTTP/HTTPS response codes and headers
• RPC test results (if applicable)
• WebSocket test results (if applicable)

---

Troubleshooting

DNS Resolution Fails

Symptoms: Domain doesn't resolve to 76.53.10.36

Solutions:

• Check Cloudflare DNS records via API or UI
• Verify DNS propagation: dig +short domain @8.8.8.8
• Check if record is proxied (should be DNS Only)

SSL Certificate Invalid

Symptoms: Browser shows certificate error

Solutions:

• Check certificate expiration: openssl s_client -connect domain:443 | openssl x509 -noout -dates
• Verify certificate in NPMplus: Check certificate list in API export
• Renew certificate if expired: NPMplus UI → SSL Certificates → Renew

Internal Connectivity Fails

Symptoms: Cannot connect to NPMplus or backend VMs

Solutions:

• Verify VM is running: pct status <VMID>
• Check firewall rules on Proxmox host
• Verify network configuration: pct config <VMID> | grep ip
• Test from different network segment

Backend Service Not Responding

Symptoms: HTTP 502 Bad Gateway or connection timeout

Solutions:

• Check service status: systemctl status <service>
• Verify port is listening: ss -lntp | grep <port>
• Check service logs: journalctl -u <service> -n 50
• Verify NPMplus proxy host configuration points to correct IP:port

RPC Endpoint Not Working

Symptoms: RPC requests fail or timeout

Solutions:

• Verify Besu node is running and synced
• Check RPC port is listening: ss -lntp | grep 8545
• Test direct connection: curl -X POST http://<VM_IP>:8545 -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'
• Verify NPMplus proxy host allows WebSocket upgrade (for WS endpoints)

---

Expected Results vs Actual Results

Expected Configuration

• DNS Records: 19 A records, all pointing to 76.53.10.36, all DNS Only
• Port Forwarding: 2 rules (80/443 → NPMplus)
• NPMplus: 26 proxy hosts, 19 SSL certificates
• Backend VMs: 8 VMs (4 with nginx, 4 direct access)
• E2E: All active domains should respond correctly

Common Discrepancies

1. Sankofa domains: Will fail until services are deployed (known issue)
2. Certificate expiration: All certificates expire 2026-04-16
3. Nginx config paths: Some VMs have TBD config paths

---

Updating Source-of-Truth

After completing verification:

1. Run source-of-truth generator:

bash scripts/verify/generate-source-of-truth.sh

2. Review generated JSON:

cat docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json | jq '.'

3. Update status fields:

   • Change status: "documented" to status: "verified" for verified components
   • Add verified_at timestamps
   • Update any incorrect values

4. Update baseline documentation:

   • Update DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md with verified data
   • Update DNS_NPMPLUS_VM_STREAMLINED_TABLE.md with verified data

---

Related Documentation

• Comprehensive Architecture: docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md
• Streamlined Table: docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md
• NPMplus Backup/Restore: docs/04-configuration/NPMPLUS_BACKUP_RESTORE.md
• Sankofa Cutover Plan: docs/04-configuration/SANKOFA_CUTOVER_PLAN.md
• Risks and Hardening: docs/04-configuration/INGRESS_RISKS_AND_HARDENING.md

---

Last Updated: 2026-01-20
Maintained By: Infrastructure Team
Status: Complete Verification Procedures