d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
main
proxmox/docs/11-references/NETWORK_CONFIGURATION_MASTER.md
defiQUG cc6d0705da
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
Details
docs: references, network, besu, CCIP, troubleshooting, archive, quick ref 
Made-with: Cursor
2026-03-27 18:50:28 -07:00

277 lines
13 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Network Configuration Master
**Last Updated:** 2026-02-12
**Status:** 🟢 Active Master Reference
**Purpose:** Single source of truth for all network configurations (UDM Pro edge, Proxmox hosts, NPMplus, port forwarding)
**Recent:** Option B (RPC via Cloudflare Tunnel) active for 6 RPC hostnames. E2E: [05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md](../05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md); Option B: [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md).
---
## Network Overview
### Primary Network
- **Subnet:** 192.168.11.0/24
- **Gateway:** 192.168.11.1
- **Netmask:** 255.255.255.0
- **VLAN:** 11 (MGMT-LAN)
- **DNS Servers:** 8.8.8.8, 8.8.4.4
### Proxmox Hosts (192.168.11.11–12; ml110 repurposed)
| Host | IP Address | Role | Status |
|------|------------|------|--------|
| r630-01 | 192.168.11.11 | Infrastructure, RPC, Services, **CCIP Relay** | ✅ Active |
| r630-02 | 192.168.11.12 | Firefly, NPMplus secondary, MIM4U | ✅ Active |
**ML110 (192.168.11.10) repurposed:** ML110 Gen9 is being converted to **OPNsense/pfSense** with 8–12 GbE, acting as **WAN aggregator** between 6–10 Spectrum cable modems and the 2× UDM Pro gateways. After repurpose, .10 is the firewall appliance (not Proxmox). See [ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md](ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md). **Before repurpose:** Migrate all containers/VMs off ml110 to r630-01/r630-02 (or other R630s); cluster will be 2 nodes until r630-03+ join.
**CCIP Relay (r630-01):** Host service at `/opt/smom-dbis-138/services/relay`; relays Chain 138 → Mainnet; uses VMID 2201 (192.168.11.221) for RPC. See [07-ccip/CCIP_RELAY_DEPLOYMENT.md](../07-ccip/CCIP_RELAY_DEPLOYMENT.md).
**Four NPMplus instances (one per public IP):** 76.53.10.36, 76.53.10.37, 76.53.10.38, 76.53.10.40. See [04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md](../04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md).
**NPMplus #1 (76.53.10.36, LXC VMID 10233):** 192.168.11.166 (eth0) and 192.168.11.167 (eth1). Only **192.168.11.167** is used in UDM Pro port forwarding: 76.53.10.36:80 → 192.168.11.167:80, 76.53.10.36:443 → 192.168.11.167:443. Main d-bis.org, explorer, Option B RPC (6 hostnames), MIM4U, etc.
**NPMplus #3 (76.53.10.38, LXC VMID 10235):** 192.168.11.169 (single NIC). Port forwarding: 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443. **Nathan's core-2 RPC, All Mainnet (Alltra), and HYBX** nodes and services route here. Designated public IP: 76.53.10.42. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md).
**NPMplus #4 (76.53.10.40, LXC VMID 10236):** 192.168.11.170. Port forwarding: 76.53.10.40:80/81/443 → 192.168.11.170:80/81/443; optional 22 → 192.168.11.59 (dev VM). **Dev/Codespaces:** Gitea, Cursor Remote SSH, Proxmox admin panels (pve.r630-01, pve.r630-02). Dedicated Cloudflare Tunnel. *(ml110 repurposed to OPNsense/pfSense WAN aggregator; no longer Proxmox.)* See [04-configuration/DEV_CODESPACES_76_53_10_40.md](../04-configuration/DEV_CODESPACES_76_53_10_40.md) and [04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md](../04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md).
**Dev VM (VMID 5700):** 192.168.11.59. Shared Cursor dev environment, four users, Gitea (private GitOps). See [04-configuration/DEV_VM_GITOPS_PLAN.md](../04-configuration/DEV_VM_GITOPS_PLAN.md).
**IP reference format:** Use `IP (VMID)` or `VMID (IP)` consistently. Full registry: [02-architecture/VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md).
### Fixed Permanent VMID → IP (Do Not Change)
| VMID | Hostname | IP Address | Purpose |
|------|----------|------------|---------|
| 2101 | besu-rpc-core-1 | 192.168.11.211 | Admin, contract deployment (RPC_CORE_1) |
| 2102 | besu-rpc-core-2 | 192.168.11.212 | Nathan RPC, SFValley2 tunnel (RPC_CORE_2) |
| **2201** | besu-rpc-public-1 | **192.168.11.221** | Bridge, monitoring, public-facing (RPC_PUBLIC_1) |
| 5000 | blockscout-1 | 192.168.11.140 | Explorer (IP_BLOCKSCOUT); web:80, API:4000 |
These IPs are **fixed and permanent**. Scripts and configs must use these values. Source: `config/ip-addresses.conf`.
---
## IP Address Ranges by Service Type
### Infrastructure Services (192.168.11.20-39)
- **Range:** 192.168.11.20 - 192.168.11.39
- **Purpose:** Proxmox infrastructure, monitoring, gateways
- **VMIDs:** 100-130, 3500-3501
### MIM4U Services (192.168.11.36-37)
- **Range:** 192.168.11.36 - 192.168.11.37
- **Purpose:** MIM4U web and API services
- **VMIDs:** 7810-7811
### Sankofa/Phoenix Services (192.168.11.50-59)
- **Range:** 192.168.11.50 - 192.168.11.59
- **Purpose:** Sankofa and Phoenix services
- **VMIDs:** 7800-7803
### Machine Learning (192.168.11.60-69)
- **Range:** 192.168.11.60 - 192.168.11.69
- **Purpose:** ML nodes, Hyperledger services
- **VMIDs:** 3000-3003, 6000, 6400
### Monitoring (192.168.11.80-89)
- **Range:** 192.168.11.80 - 192.168.11.89
- **Purpose:** Monitoring and telemetry
- **VMIDs:** 5200
### RPC Translator Services (192.168.11.110-112)
- **Range:** 192.168.11.110 - 192.168.11.112
- **Purpose:** RPC translator supporting services
- **VMIDs:** 106-108
### Besu Validators (192.168.11.100-109)
- **Range:** 192.168.11.100 - 192.168.11.109
- **Purpose:** Besu validator nodes
- **VMIDs:** 1000-1004, 10100-10101
### Besu Sentries (192.168.11.150-159, 192.168.11.213-214)
- **Range:** 192.168.11.150 - 192.168.11.159, 192.168.11.213 - 192.168.11.214
- **Purpose:** Besu sentry nodes (1505-1506 moved from .170/.171 for CCIP range 2026-02-01)
- **VMIDs:** 1500-1506
### DBIS Services (192.168.11.120-159)
- **Range:** 192.168.11.120 - 192.168.11.159
- **Purpose:** DBIS Core services
- **VMIDs:** 10120, 10130, 10150-10151
- **10120 dbis-redis:** live/static IP **192.168.11.125** (`DBIS_REDIS_IP` in `config/ip-addresses.conf`); older docs may still say .120.
### RPC Nodes & Phoenix Vault (192.168.11.200-243)
- **Range:** 192.168.11.200 - 192.168.11.243 (excl. 192.168.11.170-212 reserved for CCIP interim)
- **Purpose:** Besu RPC nodes, Phoenix Vault (8641 at .215 as of 2026-02-01)
- **VMIDs:** 2101, 2201, 2301-2308, 2400-2403, 2500-2505 (Besu RPC; 2506-2508 destroyed 2026-02-08), 8640, 8641, 8642
### Explorer & Public (192.168.11.140-149)
- **Range:** 192.168.11.140 - 192.168.11.149
- **Purpose:** Public-facing services
- **VMIDs:** 5000
### NPMplus & Order (192.168.11.160-170)
- **Range:** 192.168.11.160 - 192.168.11.170
- **Purpose:** NPMplus proxy (10233: .166/.167), NPMplus secondary (10234: .168), NPMplus Alltra/HYBX (10235: .169), NPMplus Fourth (10236: .170 — dev/Codespaces)
- **VMIDs:** 10233-10236
### Dev VM (192.168.11.59)
- **VMID:** 5700 (dev-vm)
- **Purpose:** Shared Cursor dev, four users, Gitea (private GitOps). Access via fourth NPMplus and 76.53.10.40.
### CCIP Interim (192.168.11.171-212) - Reserved for CCIP Fleet
- **Range:** 192.168.11.171 - 192.168.11.212 (170 = NPMplus Fourth)
- **Purpose:** CCIP Ops/Admin, Monitoring, Commit, Execute, RMN
- **Status:** ✅ Cleared 2026-02-01 (1505, 1506, 8641 relocated)
### Order Services (192.168.11.40-49)
- **Range:** 192.168.11.40 - 192.168.11.49
- **Purpose:** Order services
- **VMIDs:** 10000-10001
---
## VLAN Configuration
### Current (Flat Network)
- **VLAN 11:** All services (192.168.11.0/24)
- **Status:** Active, all services on single VLAN
### Planned (Future Migration)
- **VLAN 110:** BESU-VAL (10.110.0.0/24) - Validators
- **VLAN 111:** BESU-SEN (10.111.0.0/24) - Sentries
- **VLAN 112:** BESU-RPC (10.112.0.0/24) - RPC nodes
- **VLAN 120:** BLOCKSCOUT (10.120.0.0/24) - Explorer
- **VLAN 160:** SANKOFA-SVC (10.160.0.0/22) - Sankofa services
- **VLAN 200-203:** Sovereign tenants (10.200.0.0/20 each)
---
## Port Assignments
### Standard Besu Ports
- **8545:** HTTP JSON-RPC
- **8546:** WebSocket JSON-RPC
- **30303:** P2P networking (TCP/UDP)
- **9545:** Prometheus metrics
### Standard Application Ports
- **80:** HTTP
- **443:** HTTPS
- **3000:** Node.js API
- **4000:** Blockscout API (VMID 5000 @ 192.168.11.140)
- **3080:** Forge Verification Proxy (for Blockscout contract verification)
- **5432:** PostgreSQL
- **6379:** Redis
- **8006:** Proxmox Web UI
- **8080:** Keycloak
- **8200:** Vault
- **9000:** Web3Signer
---
## Public IP Configuration
### Block #1 (Spectrum) - 76.53.10.32/28
- **Gateway:** 76.53.10.33 (Spectrum CPE; nmap shows 21, 22, 23, 80, 110, 143, 443, 3389 **filtered** on .33)
- **UDM Pro:** 76.53.10.34 (replaced ER605; edge router)
- **Port forwarding:** 76.53.10.36:80/443 → 192.168.11.167:80/443 (NPMplus). **Origin for public traffic** = 76.53.10.36. Verify 76.53.10.36:80 and :443 are **open from the internet** before using Fastly or direct; see [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md).
- **NPMplus Alltra/HYBX:** 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443 (port forward); 76.53.10.42 designated public IP. See [04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md).
- **NPMplus Fourth (dev/Codespaces):** 76.53.10.40:80/81/443 → 192.168.11.170; optional 22 → 192.168.11.59. See [04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md](../04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md).
- **Usable:** 76.53.10.35-46 (13 IPs)
- **Status:** ✅ Active
### Blocks #2-#6
- **Status:** To be configured
- **Purpose:** Role-based egress NAT pools
---
## Network Access Patterns
### Public Internet Access
**Primary path (web/api):** DNS (Cloudflare) → Fastly or A 76.53.10.36 → UDM Pro (76.53.10.36:80/443) → NPMplus (192.168.11.167) → internal services. **Option B (RPC):** The 6 RPC HTTP hostnames use Cloudflare Tunnel (CNAME to cfargotunnel.com); cloudflared (e.g. VMID 102) → NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md). Verify 76.53.10.36:80/443 for direct/Fastly: [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md).
```
Internet
↓
Cloudflare DNS (optional proxy) → Fastly or 76.53.10.36
↓
UDM Pro (76.53.10.36:80/443 port forward)
↓
NPMplus (VMID 10233: 192.168.11.167:443)
↓
Internal Services
```
### Internal RPC Access
```
Internal Network (192.168.11.0/24)
↓
Direct to RPC Nodes (192.168.11.211-243:8545/8546)
```
---
## Firewall Rules
### P2P Communication
- **Port:** 30303 (TCP/UDP)
- **Allowed:** Between Besu nodes
- **Status:** ✅ Enabled
### RPC Access
- **Ports:** 8545 (HTTP), 8546 (WebSocket)
- **Allowed IPs:** 0.0.0.0/0 (public access)
- **Status:** ✅ Enabled
### Metrics Scraping
- **Port:** 9545
- **Allowed:** Monitoring systems
- **Status:** ✅ Enabled
---
## DNS Configuration
### Internal DNS
- **Primary:** 8.8.8.8
- **Secondary:** 8.8.4.4
- **Internal Domains:** sankofa.nexus (internal)
### Public DNS
- **Provider:** Cloudflare (retained for all public hostnames)
- **Domains:** d-bis.org, mim4u.org, defi-oracle.io, etc.
- **Public path:** Web/api: CNAME to Fastly (Option A) or A to 76.53.10.36 (Option C). **RPC (Option B):** The 6 RPC HTTP hostnames use CNAME to <tunnel-id>.cfargotunnel.com (Proxied); tunnel connector → NPMplus https://192.168.11.167:443. See [05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md](../05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md).
---
## Centralized IP Configuration
**Configuration File:** `config/ip-addresses.conf`
**Purpose:** Centralized IP address definitions for all scripts
**Status:** ✅ Active - 8+ scripts updated to use centralized config
**Automation:** `scripts/centralize-ip-addresses.sh` - Automated IP centralization
---
## Related Documents
- **[NETWORK_CONFIGURATION_MASTER.md](NETWORK_CONFIGURATION_MASTER.md)** (this doc) - IP matrix above
- **[HARDWARE_INVENTORY_MASTER.md](HARDWARE_INVENTORY_MASTER.md)** - 13× R630, 3× R750, 2× Dell 7920, 2× UDM Pro, 2× UniFi XG 10G, ml110
- **[13_NODE_NETWORK_AND_CABLING_CHECKLIST.md](13_NODE_NETWORK_AND_CABLING_CHECKLIST.md)** - VLANs, topology, XG port mapping
- **[13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md](13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md)** - Bring-online order for R630/R750/7920/UDM Pro #2
- **[VMID_ALLOCATION_FINAL.md](../02-architecture/VMID_ALLOCATION_FINAL.md)** - VMID master inventory
- **[VMID_IP_FIXED_REFERENCE.md](VMID_IP_FIXED_REFERENCE.md)** - Fixed VMID→IP (2101, 2201, 5000)
- **[BLOCKSCOUT_FIX_RUNBOOK.md](../03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md)** - Blockscout (VMID 5000) troubleshooting
- **[NETWORK_ARCHITECTURE.md](../02-architecture/NETWORK_ARCHITECTURE.md)** - Detailed architecture
---
**Last Updated:** 2026-02-06
**Maintainer:** System Administrator
**Update Frequency:** On network configuration changes
**Current Status:** ✅ Up to date - Option B (RPC via tunnel) documented; Blockscout API :4000, Forge Verification Proxy :3080
Reference in New Issue View Git Blame Copy Permalink