# Network Configuration Master **Last Updated:** 2026-04-03 **Status:** 🟒 Active Master Reference **Purpose:** Single source of truth for all network configurations (UDM Pro edge, Proxmox hosts, NPMplus, port forwarding) **Recent:** Option B (RPC via Cloudflare Tunnel) active for 6 RPC hostnames. E2E: [05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md](../05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md); Option B: [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md). **Proxmox cluster (verified 2026-04-02):** Five nodes, **quorate** (`pvecm`): **ml110** `192.168.11.10`, **r630-01** `.11`, **r630-02** `.12`, **r630-03** `.13`, **r630-04** `.14` (`r630-04.sankofa.nexus`). **r630-03** / **r630-04** remain **empty of guests**; workload stays on `.10`–`.12`. **Template vs live (read-only):** `bash scripts/verify/audit-proxmox-operational-template.sh` now SSHs **all five** IPs by default (`config/ip-addresses.conf`); ML110 may skip if SSH is down or host repurposed. **2026-04-08:** `config/proxmox-operational-template.json` + `ALL_VMIDS_ENDPOINTS.md` include Order **VMID 10000/10001/10020** (Postgres primary/replica + Redis on r630-01). **Package baseline (operator run):** all five nodes upgraded toward **pve-manager 9.1.7** and kernel **6.17.13-2-pve** (`apt full-upgrade`, **one node at a time**, reboot where a new kernel was installed). **r630-03** and **r630-04** had **no-subscription** apt sources applied first (they previously hit **401** on `enterprise.proxmox.com` without a subscription). **Shared LVM thin storage:** `data` / `local-lvm` in `/etc/pve/storage.cfg` include **ml110,r630-01,r630-03,r630-04**; **r630-04** uses dual SSDs in VG `pve` (~467β€―GiB thin data) plus Ceph OSDs on four SSDs; **r630-03** uses **sda3+sdb** in VG `pve` (~1β€―TiB thin data); **r630-03** **sdc–sdh** are **LVM thin pools** **`thin1-r630-03`** … **`thin6-r630-03`** (~226β€―GiB each; provision script in repo). **Other workstations:** if SSH to **r630-04** fails with **host key changed**, run `bash scripts/verify/refresh-proxmox-host-key-r630-04.sh` (or `ssh-keygen -R 192.168.11.14`) after confirming the new key out-of-band. --- ## Network Overview ### Primary Network - **Subnet:** 192.168.11.0/24 - **Gateway:** 192.168.11.1 - **Netmask:** 255.255.255.0 - **VLAN:** 11 (MGMT-LAN) - **DNS Servers:** 8.8.8.8, 8.8.4.4 ### Proxmox Hosts (five-node cluster; ml110 still PVE until WAN-aggregator cutover) | Host (short) | **Canonical FQDN** | IP Address | Role | Status | |--------------|----------------------|------------|------|--------| | ml110 | **ml110.sankofa.nexus** | 192.168.11.10 | Besu validators/RPC (Chain 138); **still Proxmox** in cluster | βœ… Active | | r630-01 | **r630-01.sankofa.nexus** | 192.168.11.11 | Infrastructure, RPC, Services, **CCIP Relay** | βœ… Active | | r630-02 | **r630-02.sankofa.nexus** | 192.168.11.12 | Firefly, NPMplus secondary, MIM4U, Blockscout | βœ… Active | | r630-03 | **r630-03.sankofa.nexus** | 192.168.11.13 | **Spare** (no LXCs/VMs); **pve** ~1β€―TiB + **thin1-r630-03**…**thin6-r630-03** on 6Γ—SSD | βœ… Active | | r630-04 | **r630-04.sankofa.nexus** | 192.168.11.14 | **Spare** (no LXCs/VMs); **pve** thin ~467β€―GiB + Ceph OSDs | βœ… Active | **Naming:** Proxmox hypervisor **management DNS** uses **`short-hostname.sankofa.nexus`** (same label as the Host column + `.sankofa.nexus`; see `config/ip-addresses.conf` `PROXMOX_FQDN_*`). Use FQDN for SSH, TLS cert SANs, and docs; IPs remain the wire target on VLAN 11. **Verify / bootstrap:** `bash scripts/verify/check-proxmox-mgmt-fqdn.sh` (`--print-hosts` for `/etc/hosts`); `bash scripts/security/ensure-proxmox-ssh-access.sh` (`--fqdn` when DNS exists). **ML110 (192.168.11.10) repurposed:** ML110 Gen9 is being converted to **OPNsense/pfSense** with 8–12 GbE, acting as **WAN aggregator** between 6–10 Spectrum cable modems and the 2Γ— UDM Pro gateways. After repurpose, .10 is the firewall appliance (not Proxmox). See [ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md](ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md). **Before repurpose:** Migrate all containers/VMs off ml110 to r630-01/r630-02 (or other R630s). **r630-03/04** are available as migration targets (no guests; local **data**/**local-lvm** storage live as of 2026-04-02). **ml110 LVM hygiene (2026-04-02):** Stale **thin** LVs on **ml110** named **`vm-2503-disk-0`**, **`vm-6201-disk-0`**, **`vm-9000-*`** were **removed** after cluster config check: **2503** / **6201** live disks are on **r630-01** / **r630-02** (`/etc/pve/nodes/.../lxc/*.conf`); **9000** had no **`vmlist`** entry. **ml110 `pve-guests.service`:** can stay **`activating (start)`** for days if **`startall`** wedges (historical **`cfs-lock` / `vzstart`** timeouts). That blocks **`apt`** during **`pve-manager` postinst** ( **`systemctl reload-or-restart pvescheduler`** waits on **`pve-guests`** ). **Unblock:** `systemctl list-jobs`, then **`systemctl cancel `** for **`pve-guests.service`** and **`pvescheduler.service`**, then **`dpkg --configure -a`** if needed. After a host reboot, confirm **validators** **1003** / **1004** are **running** (`pct start` if not). **CCIP Relay (r630-01):** Host service at `/opt/smom-dbis-138/services/relay`; relays Chain 138 β†’ Mainnet; uses VMID 2201 (192.168.11.221) for RPC. See [07-ccip/CCIP_RELAY_DEPLOYMENT.md](../07-ccip/CCIP_RELAY_DEPLOYMENT.md). **Four NPMplus instances (one per public IP):** 76.53.10.36, 76.53.10.37, 76.53.10.38, 76.53.10.40. See [04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md](../04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md). **NPMplus #1 (76.53.10.36, LXC VMID 10233):** 192.168.11.166 (eth0) and 192.168.11.167 (eth1). Only **192.168.11.167** is used in UDM Pro port forwarding: 76.53.10.36:80 β†’ 192.168.11.167:80, 76.53.10.36:443 β†’ 192.168.11.167:443. Main d-bis.org, explorer, Option B RPC (6 hostnames), MIM4U, etc. **NPMplus #3 (76.53.10.38, LXC VMID 10235):** 192.168.11.169 (single NIC). Port forwarding: 76.53.10.38:80/81/443 β†’ 192.168.11.169:80/81/443. **Nathan's core-2 RPC, All Mainnet (Alltra), and HYBX** nodes and services route here. Designated public IP: 76.53.10.42. Public service names are intended to use the Cloudflare tunnel / proxied `CNAME` path first, with the direct edge kept as management or fallback. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md). **NPMplus #4 (76.53.10.40, LXC VMID 10236):** 192.168.11.170. Port forwarding: 76.53.10.40:80/81/443 β†’ 192.168.11.170:80/81/443; optional 22 β†’ 192.168.11.59 (dev VM). **Dev/Codespaces:** Gitea, Cursor Remote SSH, Proxmox admin panels (pve.r630-01, pve.r630-02). Dedicated Cloudflare Tunnel. *(ml110 repurposed to OPNsense/pfSense WAN aggregator; no longer Proxmox.)* See [04-configuration/DEV_CODESPACES_76_53_10_40.md](../04-configuration/DEV_CODESPACES_76_53_10_40.md) and [04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md](../04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md). **Dev VM (VMID 5700):** 192.168.11.59. Shared Cursor dev environment, four users, Gitea (private GitOps). See [04-configuration/DEV_VM_GITOPS_PLAN.md](../04-configuration/DEV_VM_GITOPS_PLAN.md). **IP reference format:** Use `IP (VMID)` or `VMID (IP)` consistently. Full registry: [02-architecture/VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md). ### Fixed Permanent VMID β†’ IP (Do Not Change) | VMID | Hostname | IP Address | Purpose | |------|----------|------------|---------| | 2101 | besu-rpc-core-1 | 192.168.11.211 | Admin, contract deployment (RPC_CORE_1) | | 2102 | besu-rpc-core-2 | 192.168.11.212 | Nathan RPC, SFValley2 tunnel (RPC_CORE_2) | | **2201** | besu-rpc-public-1 | **192.168.11.221** | Bridge, monitoring, public-facing (RPC_PUBLIC_1) | | 5000 | blockscout-1 | 192.168.11.140 | Explorer (IP_BLOCKSCOUT); web:80, API:4000 | These IPs are **fixed and permanent**. Scripts and configs must use these values. Source: `config/ip-addresses.conf`. --- ## IP Address Ranges by Service Type ### Infrastructure Services (192.168.11.20-39) - **Range:** 192.168.11.20 - 192.168.11.39 - **Purpose:** Proxmox infrastructure, monitoring, gateways - **VMIDs:** 100-130, 3500-3501 ### MIM4U Services (192.168.11.36-37) - **Range:** 192.168.11.36 - 192.168.11.37 - **Purpose:** MIM4U web and API services - **VMIDs:** 7810-7811 ### Sankofa/Phoenix Services (192.168.11.50-59) - **Range:** 192.168.11.50 - 192.168.11.59 - **Purpose:** Sankofa and Phoenix services - **VMIDs:** 7800-7803 ### Machine Learning (192.168.11.60-69) - **Range:** 192.168.11.60 - 192.168.11.69 - **Purpose:** ML nodes, Hyperledger services - **VMIDs:** 3000-3003, 6000, 6400 ### Monitoring (192.168.11.80-89) - **Range:** 192.168.11.80 - 192.168.11.89 - **Purpose:** Monitoring and telemetry - **VMIDs:** 5200 ### RPC Translator Services (192.168.11.110-112) - **Range:** 192.168.11.110 - 192.168.11.112 - **Purpose:** RPC translator supporting services - **VMIDs:** 106-108 ### Besu Validators (192.168.11.100-109) - **Range:** 192.168.11.100 - 192.168.11.109 - **Purpose:** Besu validator nodes - **VMIDs:** 1000-1004, 10100-10101 ### Besu Sentries (192.168.11.150-159, 192.168.11.213-214) - **Range:** 192.168.11.150 - 192.168.11.159, 192.168.11.213 - 192.168.11.214 - **Purpose:** Besu sentry nodes (1505-1506 moved from .170/.171 for CCIP range 2026-02-01) - **VMIDs:** 1500-1506 ### DBIS Services (192.168.11.120-159) - **Range:** 192.168.11.120 - 192.168.11.159 - **Purpose:** DBIS Core services - **VMIDs:** 10120, 10130, 10150-10151 - **10120 dbis-redis:** live/static IP **192.168.11.125** (`DBIS_REDIS_IP` in `config/ip-addresses.conf`); older docs may still say .120. ### RPC Nodes & Phoenix Vault (192.168.11.200-243) - **Range:** 192.168.11.200 - 192.168.11.243 (excl. 192.168.11.170-212 reserved for CCIP interim) - **Purpose:** Besu RPC nodes, Phoenix Vault (8641 at .215 as of 2026-02-01) - **VMIDs:** 2101, 2201, 2301-2308, 2400-2403, 2500-2505 (Besu RPC; 2506-2508 destroyed 2026-02-08), 8640, 8641, 8642 ### Explorer & Public (192.168.11.140-149) - **Range:** 192.168.11.140 - 192.168.11.149 - **Purpose:** Public-facing services - **VMIDs:** 5000 ### NPMplus & Order (192.168.11.160-170) - **Range:** 192.168.11.160 - 192.168.11.170 - **Purpose:** NPMplus proxy (10233: .166/.167), NPMplus secondary (10234: .168), NPMplus Alltra/HYBX (10235: .169), NPMplus Fourth (10236: .170 β€” dev/Codespaces) - **VMIDs:** 10233-10236 ### Dev VM (192.168.11.59) - **VMID:** 5700 (dev-vm) - **Purpose:** Shared Cursor dev, four users, Gitea (private GitOps). Access via fourth NPMplus and 76.53.10.40. ### CCIP Interim (192.168.11.171-212) - Reserved for CCIP Fleet - **Range:** 192.168.11.171 - 192.168.11.212 (170 = NPMplus Fourth) - **Purpose:** CCIP Ops/Admin, Monitoring, Commit, Execute, RMN - **Status:** βœ… Cleared 2026-02-01 (1505, 1506, 8641 relocated) ### Order Services (192.168.11.40-49) - **Range:** 192.168.11.40 - 192.168.11.49 - **Purpose:** Order services - **VMIDs:** 10000-10001 --- ## VLAN Configuration ### Current (Flat Network) - **VLAN 11:** All services (192.168.11.0/24) - **Status:** Active, all services on single VLAN ### Planned (Future Migration) - **VLAN 110:** BESU-VAL (10.110.0.0/24) - Validators - **VLAN 111:** BESU-SEN (10.111.0.0/24) - Sentries - **VLAN 112:** BESU-RPC (10.112.0.0/24) - RPC nodes - **VLAN 120:** BLOCKSCOUT (10.120.0.0/24) - Explorer - **VLAN 160:** SANKOFA-SVC (10.160.0.0/22) - Sankofa services - **VLAN 200-203:** Sovereign tenants (10.200.0.0/20 each) --- ## Port Assignments ### Standard Besu Ports - **8545:** HTTP JSON-RPC - **8546:** WebSocket JSON-RPC - **30303:** P2P networking (TCP/UDP) - **9545:** Prometheus metrics ### Standard Application Ports - **80:** HTTP - **443:** HTTPS - **3000:** Node.js API - **4000:** Blockscout API (VMID 5000 @ 192.168.11.140) - **3080:** Forge Verification Proxy (for Blockscout contract verification) - **5432:** PostgreSQL - **6379:** Redis - **8006:** Proxmox Web UI - **8080:** Keycloak - **8200:** Vault - **9000:** Web3Signer --- ## Public IP Configuration ### Block #1 (Spectrum) - 76.53.10.32/28 - **Gateway:** 76.53.10.33 (Spectrum CPE; nmap shows 21, 22, 23, 80, 110, 143, 443, 3389 **filtered** on .33) - **UDM Pro:** 76.53.10.34 (replaced ER605; edge router) - **Port forwarding:** 76.53.10.36:80/443 β†’ 192.168.11.167:80/443 (NPMplus). **Origin for public traffic** = 76.53.10.36. Verify 76.53.10.36:80 and :443 are **open from the internet** before using Fastly or direct; see [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md). - **NPMplus Alltra/HYBX:** 76.53.10.38:80/81/443 β†’ 192.168.11.169:80/81/443 (port forward); 76.53.10.42 designated public IP. Public DNS for Alltra/HYBX services should prefer proxied Cloudflare tunnel `CNAME`s rather than direct `A` records to the designated IP. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md). - **NPMplus Fourth (dev/Codespaces):** 76.53.10.40:80/81/443 β†’ 192.168.11.170; optional 22 β†’ 192.168.11.59. See [04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md](../04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md). - **Usable:** 76.53.10.35-46 (13 IPs) - **Status:** βœ… Active ### Blocks #2-#6 - **Status:** To be configured - **Purpose:** Role-based egress NAT pools --- ## Network Access Patterns ### Public Internet Access **Primary path (web/api):** DNS (Cloudflare) β†’ Fastly or A 76.53.10.36 β†’ UDM Pro (76.53.10.36:80/443) β†’ NPMplus (192.168.11.167) β†’ internal services. **Option B (RPC):** The 6 RPC HTTP hostnames use Cloudflare Tunnel (CNAME to cfargotunnel.com); cloudflared (e.g. VMID 102) β†’ NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md). Verify 76.53.10.36:80/443 for direct/Fastly: [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md). ``` Internet ↓ Cloudflare DNS (optional proxy) β†’ Fastly or 76.53.10.36 ↓ UDM Pro (76.53.10.36:80/443 port forward) ↓ NPMplus (VMID 10233: 192.168.11.167:443) ↓ Internal Services ``` ### Internal RPC Access ``` Internal Network (192.168.11.0/24) ↓ Direct to RPC Nodes (192.168.11.211-243:8545/8546) ``` --- ## Firewall Rules ### P2P Communication - **Port:** 30303 (TCP/UDP) - **Allowed:** Between Besu nodes - **Status:** βœ… Enabled ### RPC Access - **Ports:** 8545 (HTTP), 8546 (WebSocket) - **Allowed IPs:** 0.0.0.0/0 (public access) - **Status:** βœ… Enabled ### Metrics Scraping - **Port:** 9545 - **Allowed:** Monitoring systems - **Status:** βœ… Enabled --- ## DNS Configuration ### Internal DNS - **Primary:** 8.8.8.8 - **Secondary:** 8.8.4.4 - **Internal Domains:** sankofa.nexus (internal) ### Public DNS - **Provider:** Cloudflare (retained for all public hostnames) - **Domains:** d-bis.org, mim4u.org, defi-oracle.io, etc. - **Public path:** Web/api: CNAME to Fastly (Option A) or A to 76.53.10.36 (Option C). **RPC (Option B):** The 6 RPC HTTP hostnames use CNAME to <tunnel-id>.cfargotunnel.com (Proxied); tunnel connector β†’ NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md). --- ## Centralized IP Configuration **Configuration File:** `config/ip-addresses.conf` **Purpose:** Centralized IP address definitions for all scripts **Status:** βœ… Active - 8+ scripts updated to use centralized config **Automation:** `scripts/centralize-ip-addresses.sh` - Automated IP centralization --- ## Related Documents - **[NETWORK_CONFIGURATION_MASTER.md](NETWORK_CONFIGURATION_MASTER.md)** (this doc) - IP matrix above - **[IT_OPS_EDGE_DISCOVERY_IPS.md](../04-configuration/IT_OPS_EDGE_DISCOVERY_IPS.md)** - LAN discovery IPs (.23, .26 VMID 105 NPM, .2 UDM HA, workstations) for IT IPAM - **[VLAN_FLAT_11_TO_SEGMENTED_RUNBOOK.md](../03-deployment/VLAN_FLAT_11_TO_SEGMENTED_RUNBOOK.md)** - ordered migration from flat VLAN 11 to segmented VLANs (operator checklist) - **[HARDWARE_INVENTORY_MASTER.md](HARDWARE_INVENTORY_MASTER.md)** - 13Γ— R630, 3Γ— R750, 2Γ— Dell 7920, 2Γ— UDM Pro, 2Γ— UniFi XG 10G, ml110 - **[13_NODE_NETWORK_AND_CABLING_CHECKLIST.md](13_NODE_NETWORK_AND_CABLING_CHECKLIST.md)** - VLANs, topology, XG port mapping - **[13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md](13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md)** - Bring-online order for R630/R750/7920/UDM Pro #2 - **[VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md)** - VMID master inventory - **[VMID_IP_FIXED_REFERENCE.md](VMID_IP_FIXED_REFERENCE.md)** - Fixed VMIDβ†’IP (2101, 2201, 5000) - **[BLOCKSCOUT_FIX_RUNBOOK.md](../03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md)** - Blockscout (VMID 5000) troubleshooting - **[NETWORK_ARCHITECTURE.md](../02-architecture/NETWORK_ARCHITECTURE.md)** - Detailed architecture --- **Last Updated:** 2026-02-06 **Maintainer:** System Administrator **Update Frequency:** On network configuration changes **Current Status:** βœ… Up to date - Option B (RPC via tunnel) documented; Blockscout API :4000, Forge Verification Proxy :3080