proxmox/docs/04-configuration/NPMPLUS_NETWORK_ROUTING_ISSUE.md

NPMplus Network Routing Issue - Root Cause Analysis

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Container: 10233 (NPMplus)
IP: 192.168.11.166
Issue: Container cannot reach backend services on 192.168.11.0/24

---

Current Status

✅ What's Working

• Container has correct IP address: 192.168.11.166/24
• Container can reach gateway: 192.168.11.1 (UDM Pro)
• Routing table is correct: 192.168.11.0/24 dev eth0
• Proxmox host CAN reach backend services
• Backend services are running and responding

❌ What's Not Working

• Container CANNOT ping backend services (all 7 services fail)
• All HTTPS domains return 502 errors
• Network connectivity from container to 192.168.11.0/24 is blocked

---

Root Cause Analysis

Finding 1: Proxmox Bridge VLAN Configuration

• Container veth interface: veth10233i0 is configured with VLAN 1 (PVID), not VLAN 11
• Container config: Shows tag=11 but veth interface doesn't reflect this
• Bridge status: vmbr0 has VLAN 11 sub-interface (vmbr0v11) but container veth is on VLAN 1

Finding 2: Network Isolation

• Container is on VLAN 11 network (192.168.11.166)
• Backend services are on VLAN 11 network (192.168.11.0/24)
• Both should be on same VLAN, but connectivity fails
• This suggests either:
  1. UDM Pro firewall blocking inter-VLAN communication (even within same VLAN)
  2. Proxmox bridge VLAN tagging not working correctly
  3. ARP/neighbor discovery failing

Finding 3: Proxmox Host Can Reach Backends

• Proxmox host (192.168.11.11) CAN ping backend services
• This confirms backend services are reachable
• Issue is container-specific networking

---

Diagnostic Commands

Check Container Network

ssh root@192.168.11.11 "pct exec 10233 -- ip addr show eth0"
ssh root@192.168.11.11 "pct exec 10233 -- ip route show"
ssh root@192.168.11.11 "pct exec 10233 -- ping -c 2 192.168.11.1"
ssh root@192.168.11.11 "pct exec 10233 -- ping -c 2 192.168.11.140"

Check Proxmox Bridge VLAN

ssh root@192.168.11.11 "bridge vlan show vmbr0 | grep -E '11|10233'"
ssh root@192.168.11.11 "bridge vlan show veth10233i0"

Check UDM Pro Firewall Rules

# Via API
curl -k -X GET "https://192.168.11.1/proxy/network/integration/v1/sites/88f7af54-98f8-306a-a1c7-c9349722b1f6/acl-rules" \
  -H "X-API-KEY: <API_KEY>" \
  -H 'Accept: application/json' | jq '.data[] | select(.enabled == true)'

---

Potential Solutions

Solution 1: Fix Proxmox Bridge VLAN Tagging (Recommended)

The container's veth interface needs to be properly configured for VLAN 11:

# Stop container
ssh root@192.168.11.11 "pct stop 10233"

# Remove VLAN 1 from veth interface
ssh root@192.168.11.11 "bridge vlan del vid 1 dev veth10233i0"

# Add VLAN 11 as PVID
ssh root@192.168.11.11 "bridge vlan add vid 11 pvid untagged dev veth10233i0"

# Start container
ssh root@192.168.11.11 "pct start 10233"

Note: This may not persist across container restarts. May need to configure in Proxmox network configuration.

Solution 2: Check UDM Pro Firewall Rules

UDM Pro may have firewall rules blocking traffic even within the same VLAN:

1. Access UDM Pro web UI: https://192.168.11.1
2. Navigate to: Settings → Firewall & Security → Firewall Rules
3. Check for rules blocking:
   • Source: 192.168.11.166 or 192.168.11.0/24
   • Destination: 192.168.11.0/24
4. Ensure there's an ALLOW rule for same-VLAN communication

Solution 3: Use Proxmox Network Configuration

Instead of manual bridge VLAN configuration, reconfigure container network:

# Remove current network config
ssh root@192.168.11.11 "pct set 10233 -delete net0"

# Add network with proper VLAN tagging
ssh root@192.168.11.11 "pct set 10233 -net0 name=eth0,bridge=vmbr0,tag=11,firewall=1,ip=192.168.11.166/24,gw=192.168.11.1"

# Restart container
ssh root@192.168.11.11 "pct stop 10233 && pct start 10233"

Solution 4: Check ARP/Neighbor Discovery

Container may not be able to resolve MAC addresses:

# Check ARP table in container
ssh root@192.168.11.11 "pct exec 10233 -- arp -a"

# Try to resolve gateway MAC
ssh root@192.168.11.11 "pct exec 10233 -- arp -s 192.168.11.1 <GATEWAY_MAC>"

---

Next Steps

1. Immediate: Check UDM Pro firewall rules via web UI
2. If firewall is OK: Fix Proxmox bridge VLAN configuration
3. Verify: Test connectivity after fixes
4. Document: Update configuration documentation

---

Related Files

• scripts/check-npmplus-network-connectivity.sh - Diagnostic script
• scripts/diagnose-npmplus-backend-services.sh - Backend service check
• docs/04-configuration/NPMPLUS_BACKEND_SERVICES_RESOLUTION.md - Related documentation

---

Status: 🔴 BLOCKED - Network routing issue preventing backend connectivity