proxmox/docs/04-configuration/NPMPLUS_BACKEND_SERVICES_RESOLUTION.md

NPMplus Backend Services Resolution

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Status: Network Routing Issue Identified

Completed Tasks ✅

1. Backend Service Diagnosis ✅

   • Created scripts/diagnose-npmplus-backend-services.sh
   • Verified all 7 backend services are running and responding
   • Identified network routing issue between NPMplus and backend services

2. Backend Service Fix Script ✅

   • Created scripts/fix-npmplus-backend-services.sh
   • Attempted to start stopped containers (3 containers have config issues)
   • All backend services are actually running and accessible from their hosts

3. HTTPS Domain Verification ✅

   • Created scripts/verify-npmplus-domains-https.sh
   • Tested all 19 production domains
   • All domains returning 502 errors due to network routing issue

Root Cause Analysis

Problem: NPMplus container (10233) cannot reach backend services on 192.168.11.0/24 network.

Current Network Configuration:

• NPMplus Container:

  • Currently on VLAN 11 (tag=11) but not receiving IP address
  • Previously on 192.168.0.0/24 (default network)
  • Needs access to 192.168.11.0/24 for backend services

• Backend Services:

  • All on 192.168.11.0/24 (VLAN 11 - MGMT-LAN)
  • Services are running and responding:
    • VMID 5000 (blockscout-1): 192.168.11.140:80 ✅
    • VMID 10130 (dbis-frontend): 192.168.11.130:80 ✅
    • VMID 10150 (dbis-api-primary): 192.168.11.155:3000 ✅
    • VMID 10151 (dbis-api-secondary): 192.168.11.156:3000 ✅
    • VMID 7811 (mim-api-1): 192.168.11.36:80 ✅
    • Current Active VMIDs:
      • VMID 2501 (besu-rpc-2): 192.168.11.251:443 ✅ (Currently active)
      • VMID 2502 (besu-rpc-3): 192.168.11.252:443 ✅ (Currently active)
    • Planned New VMIDs (Scripts Updated, VMIDs Not Yet Created):
      • VMID 2101 (besu-rpc-core-1): 192.168.11.211:443 ⚠️ (Planned - not yet created)
      • VMID 2201 (besu-rpc-public-1): 192.168.11.221:443 ⚠️ (Planned - not yet created)
      • VMID 2301 (besu-rpc-private-1): 192.168.11.231:443 ⚠️ (Planned - not yet created)
      • VMID 2302 (besu-rpc-private-2): 192.168.11.232:443 ⚠️ (Planned - not yet created)

Required Fixes

Option 1: Configure NPMplus with Static IP on VLAN 11 (Recommended)

1. Assign static IP to NPMplus container:

   ssh root@192.168.11.11
   pct set 10233 -net0 name=eth0,bridge=vmbr0,tag=11,ip=192.168.11.166/24,gw=192.168.11.1
   pct stop 10233
   pct start 10233
   

2. Verify connectivity:

   pct exec 10233 -- ping -c 2 192.168.11.140
   pct exec 10233 -- curl -I http://192.168.11.140:80
   

3. Update NPMplus port forwarding:

   • Update UDM Pro port forwarding rules to point to new IP (192.168.11.166)
   • Or keep existing IP (192.168.0.166) if dual-homed

Option 2: Configure UDM Pro Inter-VLAN Routing

1. Enable routing between 192.168.0.0/24 and 192.168.11.0/24:

   • Access UDM Pro web UI: https://192.168.11.1
   • Navigate to: Settings → Networks → Routing
   • Add static route:
     • Destination: 192.168.11.0/24
     • Gateway: 192.168.11.1
     • Interface: VLAN 11

2. Configure firewall rules:

   • Allow traffic from 192.168.0.0/24 to 192.168.11.0/24
   • Allow return traffic

3. Revert NPMplus VLAN assignment:

   ssh root@192.168.11.11
   pct set 10233 -net0 name=eth0,bridge=vmbr0,hwaddr=BC:24:11:97:68:6C,ip=dhcp,type=veth
   pct stop 10233
   pct start 10233
   

Option 3: Dual-Home NPMplus (Both Networks)

1. Add second network interface:

   ssh root@192.168.11.11
   pct set 10233 -net1 name=eth1,bridge=vmbr0,tag=11,ip=192.168.11.166/24
   pct stop 10233
   pct start 10233
   

2. Configure routing inside container:

   pct exec 10233 -- ip route add 192.168.11.0/24 dev eth1
   

Verification Steps

After implementing one of the above options:

1. Test backend connectivity from NPMplus:

   ssh root@192.168.11.11
   pct exec 10233 -- curl -I http://192.168.11.140:80
   pct exec 10233 -- curl -I http://192.168.11.130:80
   pct exec 10233 -- curl -I http://192.168.11.155:3000
   

2. Test HTTPS domains:

   bash scripts/verify-npmplus-domains-https.sh
   

3. Check NPMplus logs:

   ssh root@192.168.11.11
   pct exec 10233 -- docker logs npmplus --tail 50
   

Remaining Tasks

1. Fix Network Routing (Choose one option above)
2. Fix Quirks Mode (Backend HTML templates need <!DOCTYPE html>)
   • This requires updating backend application code
   • Affected services: All frontend applications
   • See: docs/04-configuration/NPMPLUS_CSP_QUIRKS_MODE_FIX.md

Scripts Created

• scripts/diagnose-npmplus-backend-services.sh - Diagnose backend service status
• scripts/fix-npmplus-backend-services.sh - Start stopped services
• scripts/verify-npmplus-domains-https.sh - Verify HTTPS domain accessibility

Next Steps

1. Immediate: Choose and implement one of the network routing options above
2. After routing fixed: Re-run scripts/verify-npmplus-domains-https.sh to verify 502 errors resolved
3. Backend fixes: Update backend HTML templates to include <!DOCTYPE html> for Quirks Mode fix

---

Last Updated: 2025-01-20
Status:

• ✅ Scripts updated with new VMID mappings (2101, 2201, 2301, 2302)
• ⚠️ New VMIDs not yet created on system (still using 2501, 2502)
• ⚠️ Network routing configuration required