proxmox/docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md

# Ingress Architecture Verification Runbook

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date**: 2026-01-20  
**Status**: Complete Verification Procedures  
**Purpose**: Operator-grade verification procedures for ingress architecture

---

## Overview

This runbook provides step-by-step verification procedures for the complete ingress architecture:

**Cloudflare DNS → UDM Pro port-forward → NPMplus (reverse proxy + SSL termination) → Backend VMs/services (nginx or direct ports)**

## Prerequisites

**Production note:** Example values below (e.g. `your-token`, `your-password`) are placeholders. In production, source real values from your `.env` file only; do not commit secrets.

### Access Credentials

Ensure the following are configured in `.env`:

```bash
# Cloudflare API
CLOUDFLARE_API_TOKEN="your-token"
# OR
# CLOUDFLARE_EMAIL="your-email"  # Set in .env file
# CLOUDFLARE_API_KEY="your-key"  # Set in .env file

# NPMplus API
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
NPM_PASSWORD="your-password"

# Network
PUBLIC_IP="76.53.10.36"
NPMPLUS_IP="192.168.11.166"
```

### Tools Required

- `curl` - HTTP/HTTPS requests
- `jq` - JSON parsing
- `dig` - DNS queries
- `openssl` - SSL certificate inspection
- `ssh` - Remote access to Proxmox hosts and VMs
- `wscat` (optional) - WebSocket testing: `npm install -g wscat`

### Proxmox Host Access

- SSH access to Proxmox hosts:
  - `r630-01` (192.168.11.11)
  - `r630-02` (192.168.11.12)
  - `ml110` (192.168.11.10)

---

## Quick Start

Run the full verification:

```bash
cd /home/intlc/projects/proxmox
bash scripts/verify/run-full-verification.sh
```

This will:
1. Verify Cloudflare DNS records
2. Verify UDM Pro port forwarding (with manual steps)
3. Export and verify NPMplus configuration
4. Verify all backend VMs
5. Test end-to-end routing
6. Generate source-of-truth JSON

---

## Verification Steps

### Step 1: Cloudflare DNS Verification

**Objective**: Verify all 19 DNS records match documented state

**Automated Script**:
```bash
bash scripts/verify/export-cloudflare-dns-records.sh
```

**Manual Steps** (if script unavailable):

1. **Export DNS Records via Cloudflare API**:
```bash
# Set credentials
export CLOUDFLARE_API_TOKEN="your-token"

# Get zone IDs
for zone in d-bis.org mim4u.org sankofa.nexus defi-oracle.io; do
    curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$zone" \
        -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
        -H "Content-Type: application/json" | jq '.result[0].id'
done

# Get all A records for each zone
ZONE_ID="your-zone-id"
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=A" \
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
    -H "Content-Type: application/json" | jq '.result[]'
```

2. **Verify Each Record**:
   - Type: Should be `A`
   - Content: Should be `76.53.10.36`
   - Proxied: Should be `false` (DNS Only / gray cloud)
   - TTL: Capture actual value

3. **Test DNS Resolution**:
```bash
for domain in explorer.d-bis.org mim4u.org sankofa.nexus; do
    dig +short "$domain" @8.8.8.8
done
```

**Expected Results**:
- All 19 domains resolve to `76.53.10.36`
- All records are DNS Only (proxied: false)
- TTL values are reasonable

**Evidence to Capture**:
- JSON export: `docs/04-configuration/verification-evidence/dns-verification-*/all_dns_records.json`
- Verification report: `docs/04-configuration/verification-evidence/dns-verification-*/verification_report.md`

---

### Step 2: UDM Pro Port Forwarding Verification

**Objective**: Verify WAN IP and NAT port forwarding rules

**Automated Script**:
```bash
bash scripts/verify/verify-udm-pro-port-forwarding.sh
```

**Manual Verification Steps** (Required):

1. **Access UDM Pro Web Interface**:
   - URL: `https://192.168.0.1` (or your UDM Pro IP)
   - Login with admin credentials

2. **Navigate to Port Forwarding**:
   - Settings → Firewall & Security → Port Forwarding
   - Or: Networks → Port Forwarding Rules

3. **Verify Rules Exist**:

   **Rule 1: NPMplus HTTPS**
   - Name: NPMplus HTTPS (or similar)
   - Source: Any (or specific IP)
   - Destination IP: `76.53.10.36`
   - Destination Port: `443`
   - Forward to IP: `192.168.11.166`
   - Forward to Port: `443`
   - Protocol: `TCP`
   - Interface: `WAN`

   **Rule 2: NPMplus HTTP**
   - Name: NPMplus HTTP (or similar)
   - Source: Any (or specific IP)
   - Destination IP: `76.53.10.36`
   - Destination Port: `80`
   - Forward to IP: `192.168.11.166`
   - Forward to Port: `80`
   - Protocol: `TCP`
   - Interface: `WAN`

4. **Capture Evidence**:
   - Take screenshot of port forwarding rules
   - Save as: `docs/04-configuration/verification-evidence/udm-pro-verification-*/udm-pro-port-forwarding-screenshot.png`
   - Export UDM Pro config (if available): Settings → Maintenance → Download Backup

5. **Test Internal Connectivity**:
```bash
# Test HTTP
curl -I http://192.168.11.166:80

# Test HTTPS
curl -I -k https://192.168.11.166:443
```

**Expected Results**:
- Both rules exist and are enabled
- Internal connectivity succeeds (HTTP 200 or similar)
- Screenshots captured

**Evidence to Capture**:
- Screenshot of port forwarding rules
- UDM Pro config export (if available)
- Internal connectivity test results

---

### Step 3: NPMplus Configuration Verification

**Objective**: Verify NPMplus configuration matches documentation

**Automated Script**:
```bash
bash scripts/verify/export-npmplus-config.sh
```

**Manual Steps** (if script unavailable):

1. **Authenticate to NPMplus API**:
```bash
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
NPM_PASSWORD="your-password"

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')
```

2. **Export Proxy Hosts**:
```bash
curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > proxy_hosts.json
```

3. **Export Certificates**:
```bash
curl -s -k -X GET "$NPM_URL/api/nginx/certificates" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > certificates.json
```

4. **Verify Certificate Files**:
```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- ls -la /data/tls/certbot/live/"
```

**Expected Results**:
- 26 proxy hosts exported
- 19 certificates exported
- Certificate files exist on disk
- Certificate expiration dates match API data

**Evidence to Capture**:
- JSON export: `docs/04-configuration/verification-evidence/npmplus-verification-*/proxy_hosts.json`
- JSON export: `docs/04-configuration/verification-evidence/npmplus-verification-*/certificates.json`
- Verification report: `docs/04-configuration/verification-evidence/npmplus-verification-*/verification_report.md`

---

### Step 4: Backend VMs Verification

**Objective**: Verify each backend VM matches documented configuration

**Automated Script**:
```bash
bash scripts/verify/verify-backend-vms.sh
```

**Manual Steps** (per VM):

**VMs to Verify** (8 total):

- **With Nginx**: 5000, 7810, 10130, 2400
- **Without Nginx**: 2101, 2201, 10150, 10151

**Example Verification for VMID 5000**:

```bash
VMID=5000
HOST=192.168.11.12

# Check status
ssh root@"$HOST" "pct status $VMID"

# Get IP
ssh root@"$HOST" "pct config $VMID | grep ip"

# Check nginx
ssh root@"$HOST" "pct exec $VMID -- systemctl status nginx"

# Check listening ports
ssh root@"$HOST" "pct exec $VMID -- ss -lntp"

# Check nginx config
ssh root@"$HOST" "pct exec $VMID -- ls -la /etc/nginx/sites-enabled/"

# Test health endpoint
curl -I http://192.168.11.140:80
```

**Expected Results**:
- All VMs are running
- IP addresses match expected values
- Services are active (nginx/Besu/Node.js)
- Expected ports are listening
- Health endpoints respond

**Evidence to Capture**:
- VM verification JSON: `docs/04-configuration/verification-evidence/backend-vms-verification-*/vmid_*_verification.json`
- Listening ports output: `docs/04-configuration/verification-evidence/backend-vms-verification-*/vmid_*_listening_ports.txt`
- Verification report: `docs/04-configuration/verification-evidence/backend-vms-verification-*/verification_report.md`

---

### Step 5: End-to-End Routing Verification

**Objective**: Verify complete request flow from external to backend

**Automated Script**:
```bash
bash scripts/verify/verify-end-to-end-routing.sh
```
When RPC is via **Cloudflare Tunnel** (Option B), the 6 RPC hostnames resolve to Cloudflare IPs; use `ACCEPT_ANY_DNS=1` for a full E2E pass: `ACCEPT_ANY_DNS=1 bash scripts/verify/verify-end-to-end-routing.sh`. See [OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md).

**Manual Steps** (per domain):

1. **Test DNS Resolution**:
```bash
dig +short explorer.d-bis.org @8.8.8.8
# Expected: 76.53.10.36 (or Cloudflare IPs for RPC hostnames when using Option B tunnel)
```

2. **Test SSL Certificate**:
```bash
echo | openssl s_client -connect explorer.d-bis.org:443 -servername explorer.d-bis.org 2>/dev/null | \
    openssl x509 -noout -subject -issuer -dates -ext subjectAltName
```

3. **Test HTTPS Request**:
```bash
curl -vI https://explorer.d-bis.org
# Check:
# - HTTP response code (200/301/302)
# - SSL certificate CN/SAN matches domain
# - Headers: HSTS, CSP, X-Frame-Options
```

4. **Test RPC Endpoint** (for RPC domains):
```bash
curl -X POST https://rpc-http-pub.d-bis.org \
    -H 'Content-Type: application/json' \
    -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'
```

5. **Test WebSocket** (for RPC WS domains, requires wscat):
```bash
wscat -c wss://rpc-ws-pub.d-bis.org
# Then send: {"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}
```

**Expected Results**:
- Web/api domains resolve to `76.53.10.36`; RPC hostnames may resolve to Cloudflare IPs when using Option B tunnel
- SSL certificates valid and match domains
- HTTP/HTTPS responses return valid codes
- RPC endpoints respond with JSON-RPC results

**Evidence to Capture**:
- E2E test results: `docs/04-configuration/verification-evidence/e2e-verification-*/all_e2e_results.json`
- HTTP headers: `docs/04-configuration/verification-evidence/e2e-verification-*/*_https_headers.txt`
- Verification report: `docs/04-configuration/verification-evidence/e2e-verification-*/verification_report.md`

---

## Evidence Collection Checklist

### DNS Verification
- [ ] JSON export of all DNS records
- [ ] Verification report showing status per domain
- [ ] Screenshot of Cloudflare DNS dashboard (optional)

### UDM Pro Verification
- [ ] Screenshot of port forwarding rules
- [ ] UDM Pro config export (if available)
- [ ] Internal connectivity test results

### NPMplus Verification
- [ ] JSON export of proxy hosts
- [ ] JSON export of certificates
- [ ] Certificate file verification results
- [ ] Verification report

### Backend VMs Verification
- [ ] VM status for each VMID
- [ ] IP addresses
- [ ] Service status (nginx/systemd/docker)
- [ ] Listening ports output
- [ ] Config file paths (for nginx VMs)
- [ ] Health check results

### E2E Verification
- [ ] DNS resolution results per domain
- [ ] SSL certificate details per domain
- [ ] HTTP/HTTPS response codes and headers
- [ ] RPC test results (if applicable)
- [ ] WebSocket test results (if applicable)

---

## Troubleshooting

### DNS Resolution Fails

**Symptoms**: Domain doesn't resolve to `76.53.10.36`

**Solutions**:
- Check Cloudflare DNS records via API or UI
- Verify DNS propagation: `dig +short domain @8.8.8.8`
- Check if record is proxied (should be DNS Only)

### SSL Certificate Invalid

**Symptoms**: Browser shows certificate error

**Solutions**:
- Check certificate expiration: `openssl s_client -connect domain:443 | openssl x509 -noout -dates`
- Verify certificate in NPMplus: Check certificate list in API export
- Renew certificate if expired: NPMplus UI → SSL Certificates → Renew

### Public URL Timeout (000) — DNS OK but explorer.d-bis.org unreachable

**Symptoms**: `curl https://explorer.d-bis.org` times out; `dig explorer.d-bis.org` returns 76.53.10.36.

**Cause**: Often NAT hairpin (client on LAN; router does not loop 76.53.10.36 back to NPMplus), or firewall blocking 443.

**Solutions**: See [EXPLORER_PUBLIC_URL_UNREACHABLE_FIX.md](../05-network/EXPLORER_PUBLIC_URL_UNREACHABLE_FIX.md): enable hairpin on UDM Pro, or use hosts entry `192.168.11.167 explorer.d-bis.org` for LAN; verify port forward; test from external network.

### Internal Connectivity Fails

**Symptoms**: Cannot connect to NPMplus or backend VMs

**Solutions**:
- Verify VM is running: `pct status <VMID>`
- Check firewall rules on Proxmox host
- Verify network configuration: `pct config <VMID> | grep ip`
- Test from different network segment

### Backend Service Not Responding

**Symptoms**: HTTP 502 Bad Gateway or connection timeout

**Solutions**:
- Check service status: `systemctl status <service>`
- Verify port is listening: `ss -lntp | grep <port>`
- Check service logs: `journalctl -u <service> -n 50`
- Verify NPMplus proxy host configuration points to correct IP:port

### RPC Endpoint Not Working

**Symptoms**: RPC requests fail or timeout

**Solutions**:
- Verify Besu node is running and synced
- Check RPC port is listening: `ss -lntp | grep 8545`
- Test direct connection: `curl -X POST http://<VM_IP>:8545 -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'`
- Verify NPMplus proxy host allows WebSocket upgrade (for WS endpoints)

---

## Expected Results vs Actual Results

### Expected Configuration

- **DNS Records**: 19 A records, all pointing to `76.53.10.36`, all DNS Only
- **Port Forwarding**: 2 rules (80/443 → NPMplus)
- **NPMplus**: 26 proxy hosts, 19 SSL certificates
- **Backend VMs**: 8 VMs (4 with nginx, 4 direct access)
- **E2E**: All active domains should respond correctly

### Common Discrepancies

1. **Sankofa domains**: Will fail until services are deployed (known issue)
2. **Certificate expiration**: All certificates expire 2026-04-16
3. **Nginx config paths**: Some VMs have TBD config paths

---

## Updating Source-of-Truth

After completing verification:

1. **Run source-of-truth generator**:
```bash
bash scripts/verify/generate-source-of-truth.sh
```

2. **Review generated JSON**:
```bash
cat docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json | jq '.'
```

3. **Update status fields**:
   - Change `status: "documented"` to `status: "verified"` for verified components
   - Add `verified_at` timestamps
   - Update any incorrect values

4. **Update baseline documentation**:
   - Update `DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md` with verified data
   - Update `DNS_NPMPLUS_VM_STREAMLINED_TABLE.md` with verified data

---

## Related Documentation

- **Comprehensive Architecture**: `docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md`
- **Streamlined Table**: `docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md`
- **NPMplus Backup/Restore**: `docs/04-configuration/NPMPLUS_BACKUP_RESTORE.md`
- **Sankofa Cutover Plan**: `docs/04-configuration/SANKOFA_CUTOVER_PLAN.md`
- **Risks and Hardening**: `docs/04-configuration/INGRESS_RISKS_AND_HARDENING.md`

---

**Last Updated**: 2026-01-20  
**Maintained By**: Infrastructure Team  
**Status**: Complete Verification Procedures