Files
proxmox/docs/11-references/NETWORK_CONFIGURATION_MASTER.md

276 lines
13 KiB
Markdown
Raw Normal View History

# Network Configuration Master
**Last Updated:** 2026-02-12
**Status:** 🟢 Active Master Reference
**Purpose:** Single source of truth for all network configurations (UDM Pro edge, Proxmox hosts, NPMplus, port forwarding)
**Recent:** Option B (RPC via Cloudflare Tunnel) active for 6 RPC hostnames. E2E: [05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md](../05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md); Option B: [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md).
---
## Network Overview
### Primary Network
- **Subnet:** 192.168.11.0/24
- **Gateway:** 192.168.11.1
- **Netmask:** 255.255.255.0
- **VLAN:** 11 (MGMT-LAN)
- **DNS Servers:** 8.8.8.8, 8.8.4.4
### Proxmox Hosts (192.168.11.1112; ml110 repurposed)
| Host | IP Address | Role | Status |
|------|------------|------|--------|
| r630-01 | 192.168.11.11 | Infrastructure, RPC, Services, **CCIP Relay** | ✅ Active |
| r630-02 | 192.168.11.12 | Firefly, NPMplus secondary, MIM4U | ✅ Active |
**ML110 (192.168.11.10) repurposed:** ML110 Gen9 is being converted to **OPNsense/pfSense** with 812 GbE, acting as **WAN aggregator** between 610 Spectrum cable modems and the 2× UDM Pro gateways. After repurpose, .10 is the firewall appliance (not Proxmox). See [ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md](ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md). **Before repurpose:** Migrate all containers/VMs off ml110 to r630-01/r630-02 (or other R630s); cluster will be 2 nodes until r630-03+ join.
**CCIP Relay (r630-01):** Host service at `/opt/smom-dbis-138/services/relay`; relays Chain 138 → Mainnet; uses VMID 2201 (192.168.11.221) for RPC. See [07-ccip/CCIP_RELAY_DEPLOYMENT.md](../07-ccip/CCIP_RELAY_DEPLOYMENT.md).
**Four NPMplus instances (one per public IP):** 76.53.10.36, 76.53.10.37, 76.53.10.38, 76.53.10.40. See [04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md](../04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md).
**NPMplus #1 (76.53.10.36, LXC VMID 10233):** 192.168.11.166 (eth0) and 192.168.11.167 (eth1). Only **192.168.11.167** is used in UDM Pro port forwarding: 76.53.10.36:80 → 192.168.11.167:80, 76.53.10.36:443 → 192.168.11.167:443. Main d-bis.org, explorer, Option B RPC (6 hostnames), MIM4U, etc.
**NPMplus #3 (76.53.10.38, LXC VMID 10235):** 192.168.11.169 (single NIC). Port forwarding: 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443. **Nathan's core-2 RPC, All Mainnet (Alltra), and HYBX** nodes and services route here. Designated public IP: 76.53.10.42. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md).
**NPMplus #4 (76.53.10.40, LXC VMID 10236):** 192.168.11.170. Port forwarding: 76.53.10.40:80/81/443 → 192.168.11.170:80/81/443; optional 22 → 192.168.11.59 (dev VM). **Dev/Codespaces:** Gitea, Cursor Remote SSH, Proxmox admin panels (pve.r630-01, pve.r630-02). Dedicated Cloudflare Tunnel. *(ml110 repurposed to OPNsense/pfSense WAN aggregator; no longer Proxmox.)* See [04-configuration/DEV_CODESPACES_76_53_10_40.md](../04-configuration/DEV_CODESPACES_76_53_10_40.md) and [04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md](../04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md).
**Dev VM (VMID 5700):** 192.168.11.59. Shared Cursor dev environment, four users, Gitea (private GitOps). See [04-configuration/DEV_VM_GITOPS_PLAN.md](../04-configuration/DEV_VM_GITOPS_PLAN.md).
**IP reference format:** Use `IP (VMID)` or `VMID (IP)` consistently. Full registry: [02-architecture/VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md).
### Fixed Permanent VMID → IP (Do Not Change)
| VMID | Hostname | IP Address | Purpose |
|------|----------|------------|---------|
| 2101 | besu-rpc-core-1 | 192.168.11.211 | Admin, contract deployment (RPC_CORE_1) |
| 2102 | besu-rpc-core-2 | 192.168.11.212 | Nathan RPC, SFValley2 tunnel (RPC_CORE_2) |
| **2201** | besu-rpc-public-1 | **192.168.11.221** | Bridge, monitoring, public-facing (RPC_PUBLIC_1) |
| 5000 | blockscout-1 | 192.168.11.140 | Explorer (IP_BLOCKSCOUT); web:80, API:4000 |
These IPs are **fixed and permanent**. Scripts and configs must use these values. Source: `config/ip-addresses.conf`.
---
## IP Address Ranges by Service Type
### Infrastructure Services (192.168.11.20-39)
- **Range:** 192.168.11.20 - 192.168.11.39
- **Purpose:** Proxmox infrastructure, monitoring, gateways
- **VMIDs:** 100-130, 3500-3501
### MIM4U Services (192.168.11.36-37)
- **Range:** 192.168.11.36 - 192.168.11.37
- **Purpose:** MIM4U web and API services
- **VMIDs:** 7810-7811
### Sankofa/Phoenix Services (192.168.11.50-59)
- **Range:** 192.168.11.50 - 192.168.11.59
- **Purpose:** Sankofa and Phoenix services
- **VMIDs:** 7800-7803
### Machine Learning (192.168.11.60-69)
- **Range:** 192.168.11.60 - 192.168.11.69
- **Purpose:** ML nodes, Hyperledger services
- **VMIDs:** 3000-3003, 6000, 6400
### Monitoring (192.168.11.80-89)
- **Range:** 192.168.11.80 - 192.168.11.89
- **Purpose:** Monitoring and telemetry
- **VMIDs:** 5200
### RPC Translator Services (192.168.11.110-112)
- **Range:** 192.168.11.110 - 192.168.11.112
- **Purpose:** RPC translator supporting services
- **VMIDs:** 106-108
### Besu Validators (192.168.11.100-109)
- **Range:** 192.168.11.100 - 192.168.11.109
- **Purpose:** Besu validator nodes
- **VMIDs:** 1000-1004, 10100-10101
### Besu Sentries (192.168.11.150-159, 192.168.11.213-214)
- **Range:** 192.168.11.150 - 192.168.11.159, 192.168.11.213 - 192.168.11.214
- **Purpose:** Besu sentry nodes (1505-1506 moved from .170/.171 for CCIP range 2026-02-01)
- **VMIDs:** 1500-1506
### DBIS Services (192.168.11.120-159)
- **Range:** 192.168.11.120 - 192.168.11.159
- **Purpose:** DBIS Core services
- **VMIDs:** 10120, 10130, 10150-10151
### RPC Nodes & Phoenix Vault (192.168.11.200-243)
- **Range:** 192.168.11.200 - 192.168.11.243 (excl. 192.168.11.170-212 reserved for CCIP interim)
- **Purpose:** Besu RPC nodes, Phoenix Vault (8641 at .215 as of 2026-02-01)
- **VMIDs:** 2101, 2201, 2301-2308, 2400-2403, 2500-2505 (Besu RPC; 2506-2508 destroyed 2026-02-08), 8640, 8641, 8642
### Explorer & Public (192.168.11.140-149)
- **Range:** 192.168.11.140 - 192.168.11.149
- **Purpose:** Public-facing services
- **VMIDs:** 5000
### NPMplus & Order (192.168.11.160-170)
- **Range:** 192.168.11.160 - 192.168.11.170
- **Purpose:** NPMplus proxy (10233: .166/.167), NPMplus secondary (10234: .168), NPMplus Alltra/HYBX (10235: .169), NPMplus Fourth (10236: .170 — dev/Codespaces)
- **VMIDs:** 10233-10236
### Dev VM (192.168.11.59)
- **VMID:** 5700 (dev-vm)
- **Purpose:** Shared Cursor dev, four users, Gitea (private GitOps). Access via fourth NPMplus and 76.53.10.40.
### CCIP Interim (192.168.11.171-212) - Reserved for CCIP Fleet
- **Range:** 192.168.11.171 - 192.168.11.212 (170 = NPMplus Fourth)
- **Purpose:** CCIP Ops/Admin, Monitoring, Commit, Execute, RMN
- **Status:** ✅ Cleared 2026-02-01 (1505, 1506, 8641 relocated)
### Order Services (192.168.11.40-49)
- **Range:** 192.168.11.40 - 192.168.11.49
- **Purpose:** Order services
- **VMIDs:** 10000-10001
---
## VLAN Configuration
### Current (Flat Network)
- **VLAN 11:** All services (192.168.11.0/24)
- **Status:** Active, all services on single VLAN
### Planned (Future Migration)
- **VLAN 110:** BESU-VAL (10.110.0.0/24) - Validators
- **VLAN 111:** BESU-SEN (10.111.0.0/24) - Sentries
- **VLAN 112:** BESU-RPC (10.112.0.0/24) - RPC nodes
- **VLAN 120:** BLOCKSCOUT (10.120.0.0/24) - Explorer
- **VLAN 160:** SANKOFA-SVC (10.160.0.0/22) - Sankofa services
- **VLAN 200-203:** Sovereign tenants (10.200.0.0/20 each)
---
## Port Assignments
### Standard Besu Ports
- **8545:** HTTP JSON-RPC
- **8546:** WebSocket JSON-RPC
- **30303:** P2P networking (TCP/UDP)
- **9545:** Prometheus metrics
### Standard Application Ports
- **80:** HTTP
- **443:** HTTPS
- **3000:** Node.js API
- **4000:** Blockscout API (VMID 5000 @ 192.168.11.140)
- **3080:** Forge Verification Proxy (for Blockscout contract verification)
- **5432:** PostgreSQL
- **6379:** Redis
- **8006:** Proxmox Web UI
- **8080:** Keycloak
- **8200:** Vault
- **9000:** Web3Signer
---
## Public IP Configuration
### Block #1 (Spectrum) - 76.53.10.32/28
- **Gateway:** 76.53.10.33 (Spectrum CPE; nmap shows 21, 22, 23, 80, 110, 143, 443, 3389 **filtered** on .33)
- **UDM Pro:** 76.53.10.34 (replaced ER605; edge router)
- **Port forwarding:** 76.53.10.36:80/443 → 192.168.11.167:80/443 (NPMplus). **Origin for public traffic** = 76.53.10.36. Verify 76.53.10.36:80 and :443 are **open from the internet** before using Fastly or direct; see [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md).
- **NPMplus Alltra/HYBX:** 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443 (port forward); 76.53.10.42 designated public IP. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md).
- **NPMplus Fourth (dev/Codespaces):** 76.53.10.40:80/81/443 → 192.168.11.170; optional 22 → 192.168.11.59. See [04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md](../04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md).
- **Usable:** 76.53.10.35-46 (13 IPs)
- **Status:** ✅ Active
### Blocks #2-#6
- **Status:** To be configured
- **Purpose:** Role-based egress NAT pools
---
## Network Access Patterns
### Public Internet Access
**Primary path (web/api):** DNS (Cloudflare) → Fastly or A 76.53.10.36 → UDM Pro (76.53.10.36:80/443) → NPMplus (192.168.11.167) → internal services. **Option B (RPC):** The 6 RPC HTTP hostnames use Cloudflare Tunnel (CNAME to cfargotunnel.com); cloudflared (e.g. VMID 102) → NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md). Verify 76.53.10.36:80/443 for direct/Fastly: [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md).
```
Internet
Cloudflare DNS (optional proxy) → Fastly or 76.53.10.36
UDM Pro (76.53.10.36:80/443 port forward)
NPMplus (VMID 10233: 192.168.11.167:443)
Internal Services
```
### Internal RPC Access
```
Internal Network (192.168.11.0/24)
Direct to RPC Nodes (192.168.11.211-243:8545/8546)
```
---
## Firewall Rules
### P2P Communication
- **Port:** 30303 (TCP/UDP)
- **Allowed:** Between Besu nodes
- **Status:** ✅ Enabled
### RPC Access
- **Ports:** 8545 (HTTP), 8546 (WebSocket)
- **Allowed IPs:** 0.0.0.0/0 (public access)
- **Status:** ✅ Enabled
### Metrics Scraping
- **Port:** 9545
- **Allowed:** Monitoring systems
- **Status:** ✅ Enabled
---
## DNS Configuration
### Internal DNS
- **Primary:** 8.8.8.8
- **Secondary:** 8.8.4.4
- **Internal Domains:** sankofa.nexus (internal)
### Public DNS
- **Provider:** Cloudflare (retained for all public hostnames)
- **Domains:** d-bis.org, mim4u.org, defi-oracle.io, etc.
- **Public path:** Web/api: CNAME to Fastly (Option A) or A to 76.53.10.36 (Option C). **RPC (Option B):** The 6 RPC HTTP hostnames use CNAME to <tunnel-id>.cfargotunnel.com (Proxied); tunnel connector → NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md).
---
## Centralized IP Configuration
**Configuration File:** `config/ip-addresses.conf`
**Purpose:** Centralized IP address definitions for all scripts
**Status:** ✅ Active - 8+ scripts updated to use centralized config
**Automation:** `scripts/centralize-ip-addresses.sh` - Automated IP centralization
---
## Related Documents
- **[NETWORK_CONFIGURATION_MASTER.md](NETWORK_CONFIGURATION_MASTER.md)** (this doc) - IP matrix above
- **[HARDWARE_INVENTORY_MASTER.md](HARDWARE_INVENTORY_MASTER.md)** - 13× R630, 3× R750, 2× Dell 7920, 2× UDM Pro, 2× UniFi XG 10G, ml110
- **[13_NODE_NETWORK_AND_CABLING_CHECKLIST.md](13_NODE_NETWORK_AND_CABLING_CHECKLIST.md)** - VLANs, topology, XG port mapping
- **[13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md](13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md)** - Bring-online order for R630/R750/7920/UDM Pro #2
- **[VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md)** - VMID master inventory
- **[VMID_IP_FIXED_REFERENCE.md](VMID_IP_FIXED_REFERENCE.md)** - Fixed VMID→IP (2101, 2201, 5000)
- **[BLOCKSCOUT_FIX_RUNBOOK.md](../03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md)** - Blockscout (VMID 5000) troubleshooting
- **[NETWORK_ARCHITECTURE.md](../02-architecture/NETWORK_ARCHITECTURE.md)** - Detailed architecture
---
**Last Updated:** 2026-02-06
**Maintainer:** System Administrator
**Update Frequency:** On network configuration changes
**Current Status:** ✅ Up to date - Option B (RPC via tunnel) documented; Blockscout API :4000, Forge Verification Proxy :3080