# Network Architecture - Enterprise Orchestration Plan **Navigation:** [Home](../01-getting-started/README.md) > [Architecture](README.md) > Network Architecture **Related:** [PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md) | [DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md) | [ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md) | [11-references/NETWORK_CONFIGURATION_MASTER.md](../11-references/NETWORK_CONFIGURATION_MASTER.md) | **Runbooks & VLAN:** [03-deployment/OPERATIONAL_RUNBOOKS.md](../03-deployment/OPERATIONAL_RUNBOOKS.md) (Phase 4, VLAN), [03-deployment/MISSING_CONTAINERS_LIST.md](../03-deployment/MISSING_CONTAINERS_LIST.md), [04-configuration/UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md](../04-configuration/UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md) **Last Updated:** 2026-02-05 **Document Version:** 2.1 **Status:** ๐ŸŸข Active Documentation **Project:** Sankofa / Phoenix / PanTel ยท ChainID 138 ยท Proxmox + Cloudflare DNS + NPMplus (edge: UDM Pro; Fastly or direct to 76.53.10.36) --- ## Overview This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including: - **Hardware role assignments** (2ร— ER605, 3ร— ES216G, 1ร— ML110, 4ร— R630) - **6ร— /28 public IP blocks** with role-based NAT pools - **VLAN orchestration** with private subnet allocations - **Egress segmentation** by role and security plane - **Cloudflare Zero Trust** integration patterns --- ## Architecture Diagrams ### Network Topology (High Level) ```mermaid graph TB Internet[Internet] CF[Cloudflare Zero Trust] UDM[UDM Pro 76.53.10.34] NPM[NPMplus 192.168.11.167] ES1[ES216G-1 Core] ES2[ES216G-2 Compute] ML[ML110 192.168.11.10] R1[R630-01 192.168.11.11] R2[R630-02 192.168.11.12] Internet --> CF CF --> UDM UDM --> NPM NPM --> ES1 ES1 --> ES2 ES2 --> ML ES2 --> R1 ES2 --> R2 ``` ### VLAN Architecture (Selected VLANs) ```mermaid graph TD V11[VLAN 11: MGMT-LAN
192.168.11.0/24] V110[VLAN 110: BESU-VAL
10.110.0.0/24] V111[VLAN 111: BESU-SEN
10.111.0.0/24] V112[VLAN 112: BESU-RPC
10.112.0.0/24] V132[VLAN 132: CCIP-COMMIT
10.132.0.0/24] V133[VLAN 133: CCIP-EXEC
10.133.0.0/24] V134[VLAN 134: CCIP-RMN
10.134.0.0/24] V11 --> V110 V11 --> V111 V11 --> V112 V11 --> V132 V11 --> V133 V11 --> V134 ``` See [VLAN Set (Authoritative)](#31-vlan-set-authoritative) below for the full table. ### Proxmox Cluster (Nodes) ```mermaid graph LR ML[ml110 192.168.11.10] R1[r630-01 .11] R2[r630-02 .12] R3[r630-03 .13] R4[r630-04 .14] ML --- R1 ML --- R2 R1 --- R2 R1 --- R3 R2 --- R4 ``` --- ## Core Principles 1. **No public IPs on Proxmox hosts or LXCs/VMs** (default) 2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary) 3. **Public IPs used for:** - ER605 WAN addressing - **Egress NAT pools** (role-based allowlisting) - **Break-glass** emergency endpoints only 4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops 5. **Deterministic VMID registry** + IPAM that matches --- ## 1. Physical Topology & Hardware Roles > **Reference:** For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)**. ### 1.1 Hardware Role Assignment #### Edge / Routing - **ER605-A (Primary Edge Router)** - WAN1: Spectrum primary with Block #1 - WAN2: ISP #2 (failover/alternate policy) - Role: Active edge router, NAT pools, routing - **ER605-B (Standby Edge Router / Alternate WAN policy)** - Role: Standby router OR dedicated to WAN2 policies/testing - Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA. #### Switching Fabric - **ES216G-1**: Core / uplinks / trunks - **ES216G-2**: Compute rack aggregation - **ES216G-3**: Mgmt + out-of-band / staging #### Compute - **ML110 Gen9**: "Bootstrap & Management" node - IP: 192.168.11.10 - Role: Proxmox mgmt services, Omada controller, Git, monitoring seed - **4ร— Dell R630**: Proxmox compute cluster nodes - Resources: 512GB RAM each, 2ร—600GB boot, 6ร—250GB SSD - Role: Production workloads, CCIP fleet, sovereign tenants, services --- ## 2. ISP & Public IP Plan (6ร— /28) ### Public Block #1 (Known - Spectrum) | Property | Value | Status | |----------|-------|--------| | **Network** | `76.53.10.32/28` | โœ… Configured | | **Gateway** | `76.53.10.33` | โœ… Active | | **Usable Range** | `76.53.10.33โ€“76.53.10.46` | โœ… In Use | | **Broadcast** | `76.53.10.47` | - | | **UDM Pro (edge)** | `76.53.10.34` (replaced ER605) | โœ… Active | | **Available IPs** | 13 (76.53.10.35-46, excluding .34) | โœ… Available | ### Public Blocks #2โ€“#6 (Reserved - To Be Configured) > **Status:** Blocks #2โ€“#6 are reserved. Document actual network/gateway/usable range when assigned by provider, or keep as placeholders until CCIP/Sankofa/Sovereign egress planning is finalized. See [MASTER_PLAN.md](../00-meta/MASTER_PLAN.md) ยง3.1. | Block | Network | Gateway | Usable Range | Broadcast | Designated Use | |-------|--------|---------|--------------|-----------|----------------| | **#2** | `/28` | `` | `` | `` | CCIP Commit egress NAT pool | | **#3** | `/28` | `` | `` | `` | CCIP Execute egress NAT pool | | **#4** | `/28` | `` | `` | `` | RMN egress NAT pool | | **#5** | `/28` | `` | `` | `` | Sankofa/Phoenix/PanTel service egress | | **#6** | `/28` | `` | `` | `` | Sovereign Cloud Band tenant egress | ### 2.1 Public IP Usage Policy (Role-based) | Public /28 Block | Designated Use | Why | |------------------|----------------|-----| | **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency | | **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs | | **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs | | **#4** | RMN egress NAT pool | Independent security-plane egress | | **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation | | **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control | --- ## 3. Layer-2 & VLAN Orchestration Plan ### 3.1 VLAN Set (Authoritative) > **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility. | VLAN ID | VLAN Name | Purpose | Subnet | Gateway | |--------:|-----------|---------|--------|---------| | **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 | | 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 | | 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 | | 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 | | 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 | | 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 | | 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 | | 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 | | 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 | | 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 | | 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 | | 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 | | 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 | | 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 | | 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 | | 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 | | 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 | | 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 | ### 3.2 Switching Configuration (ES216G) - **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A) - **ES216G-2**: **Compute** (trunks to R630s + ML110) - **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band) **All Proxmox uplinks should be 802.1Q trunk ports.** --- ## 4. Routing, NAT, and Egress Segmentation (ER605) ### 4.1 Dual Router Roles - **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1) - **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services) ### 4.2 NAT Policies (Critical) #### Inbound NAT - **Default: none** - Break-glass only (optional): - Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred) - Proxmox admin should remain **LAN-only** #### Outbound NAT (Role-based Pools Using /28 Blocks) | Private Subnet | Role | Egress NAT Pool | Public Block | |----------------|------|-----------------|--------------| | 10.132.0.0/24 | CCIP Commit | **Block #2** `/28` | #2 | | 10.133.0.0/24 | CCIP Execute | **Block #3** `/28` | #3 | | 10.134.0.0/24 | RMN | **Block #4** `/28` | #4 | | 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `/28` | #5 | | 10.200.0.0/20โ€“10.203.0.0/20 | Sovereign tenants | **Block #6** `/28` | #6 | | 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 | This yields **provable separation**, allowlisting, and incident scoping. --- ## 5. Proxmox Cluster Orchestration ### 5.1 Node Layout - **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner - **r630-01..04**: production compute ### 5.2 Proxmox Networking (per host) - **`vmbr0`**: VLAN-aware bridge - Native VLAN: 11 (MGMT) - Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200โ€“203 - **Proxmox host IP** remains on **VLAN 11** only. ### 5.3 Storage Orchestration (R630) **Hardware:** - 2ร—600GB boot (mirror recommended) - 6ร—250GB SSD **Recommended:** - **Boot drives**: ZFS mirror or hardware RAID1 - **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance) - **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas --- ## 6. Public Edge: Fastly or Direct to NPMplus ### 6.1 Fastly or Direct to NPMplus (Primary Public Path) **Public ingress** is **Fastly** (Option A) or **DNS direct to 76.53.10.36** (Option C). Both flow through **UDM Pro** port forward to **NPMplus** (VMID 10233 at 192.168.11.167). Cloudflare Tunnel is **deprecated** for public access (502 errors); Cloudflare DNS is retained for all public hostnames. - **Flow:** Internet โ†’ Cloudflare DNS โ†’ Fastly or 76.53.10.36 โ†’ UDM Pro (76.53.10.36:80/443) โ†’ NPMplus โ†’ internal services (Blockscout, RPC, DBIS, MIM4U, etc.). - **Pre-requisite:** Verify 76.53.10.36:80 and :443 are open from the internet; see [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md). If closed (e.g. Spectrum filtering), use Option B (tunnel or VPS origin). - **Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access or VPN with strict posture/MFA. --- ## 7. Complete VMID and Network Allocation Table | VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) | |-----------:|-------------------|-----------|--------:|------------------------|---------------------------| | **EDGE** | UDM Pro (replaced ER605) | WAN | โ€” | โ€” | **76.53.10.34** *(edge)* | | **EDGE** | Spectrum ISP Gateway | โ€” | โ€” | โ€” | **76.53.10.33** *(ISP gateway)* | | 1000โ€“1499 | **Besu** โ€“ Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) | | 1500โ€“2499 | **Besu** โ€“ Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* | | 2500โ€“3499 | **Besu** โ€“ RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **Via NPMplus** *(Fastly or direct to 76.53.10.36); Alltra/HYBX via 76.53.10.38 or 76.53.10.42)* | | 3500โ€“4299 | **Besu** โ€“ Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None | | 4300โ€“4999 | **Besu** โ€“ Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None | | 5000โ€“5099 | **Blockscout** โ€“ Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **Via NPMplus** *(Fastly or direct to 76.53.10.36)* | | 5200โ€“5299 | **Cacti** โ€“ Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via NPMplus/Fastly if needed)* | | 5400โ€“5401 | **CCIP** โ€“ Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* | | 5402โ€“5403 | **CCIP** โ€“ Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* | | 5410โ€“5425 | **CCIP** โ€“ Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** | | 5440โ€“5455 | **CCIP** โ€“ Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** | | 5470โ€“5476 | **CCIP** โ€“ RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** | | 5480โ€“5599 | **CCIP** โ€“ Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None | | 6000โ€“6099 | **Fabric** โ€“ Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via NPMplus/Fastly if required)* | | 6200โ€“6299 | **FireFly** โ€“ Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary via NPMplus)* | | 6400โ€“7399 | **Indy** โ€“ Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary via NPMplus)* | | 10235 | **NPMplus Alltra/HYBX** | MGMT-LAN | 11 | 192.168.11.0/24 | **76.53.10.38** *(port forward 80/81/443); 76.53.10.42 designated; see [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md))* | | 7800โ€“8999 | **Sankofa / Phoenix / PanTel** โ€“ Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** | | 10000โ€“10999 | **Phoenix Sovereign Cloud Band** โ€“ SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** | | 11000โ€“11999 | **Phoenix Sovereign Cloud Band** โ€“ ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** | | 12000โ€“12999 | **Phoenix Sovereign Cloud Band** โ€“ DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** | | 13000โ€“13999 | **Phoenix Sovereign Cloud Band** โ€“ Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** | --- ## 8. Network Security Model ### 8.1 Access Patterns 1. **No Public Access (Tunnel/VPN Only)** - Besu Validators (VLAN 110) - Besu Archive/Infrastructure (VLAN 113) - CCIP Ops/Admin (VLAN 130) - CCIP Monitoring (VLAN 131) 2. **Fastly or Direct to NPMplus (Primary)** - All public services route through NPMplus (VMID 10233) at 192.168.11.167 - Public origin: 76.53.10.36 (UDM Pro port forwarding to NPMplus) - Blockscout (VLAN 120), Besu RPC (VLAN 112), FireFly (VLAN 141), Indy (VLAN 150), Sankofa/Phoenix/PanTel (VLAN 160) - Via NPMplus - DNS: Cloudflare. Edge: Fastly (Option A) or direct to 76.53.10.36 (Option C). Tunnel deprecated for public ingress. 3. **Role-Based Egress NAT (Allowlistable)** - CCIP Commit (VLAN 132) โ†’ Block #2 - CCIP Execute (VLAN 133) โ†’ Block #3 - RMN (VLAN 134) โ†’ Block #4 - Sankofa/Phoenix/PanTel (VLAN 160) โ†’ Block #5 - Sovereign tenants (VLAN 200-203) โ†’ Block #6 4. **Cloudflare Access / VPN Only** - CCIP Ops/Admin (VLAN 130) - CCIP Monitoring (VLAN 131) - Optional dashboard publishing --- ## 9. Implementation Notes ### 9.1 Gateway Configuration - All private subnets use `.1` as the gateway address - Example: VLAN 110 uses `10.110.0.1` as gateway - VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility) ### 9.2 Subnet Sizing - **/24 subnets:** Standard service VLANs (256 addresses) - **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses) - **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each) ### 9.3 IP Address Allocation - **Private IPs:** - VLAN 11: 192.168.11.0/24 (legacy mgmt) - All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet) - **Public IPs:** 6ร— /28 blocks with role-based NAT pools - **All public access** routes through NPMplus (Fastly or direct to 76.53.10.36) for security and stability ### 9.4 VLAN Tagging - All VLANs are tagged on the Proxmox bridge - Ensure Proxmox bridge is configured for **VLAN-aware mode** - Physical switch must support VLAN tagging (802.1Q) --- ## 10. Configuration Files This architecture should be reflected in: - `config/network.conf` - Network configuration - `config/proxmox.conf` - VMID ranges - Proxmox bridge configuration (VLAN-aware mode) - ER605 router configuration (NAT pools, routing) - Fastly or direct-to-NPMplus configuration (see 05-network routing docs) - ES216G switch configuration (VLAN trunks) --- ## 11. References - [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration) - [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/) - [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918) - [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/) - [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/) --- ## Related Documentation ### Architecture Documents - **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** โญโญโญ - Complete physical hardware inventory and specifications - **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** โญโญโญ - Enterprise deployment orchestration guide - **[VMID_ALLOCATION_FINAL.md](VMID_ALLOCATION_FINAL.md)** โญโญโญ - VMID allocation registry - **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** โญโญ - Domain structure and DNS assignments - **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** โญ - Domain and hostname structure ### Configuration Documents - **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](/docs/04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration - **[../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare Zero Trust setup - **[../05-network/CLOUDFLARE_ROUTING_MASTER.md](../05-network/CLOUDFLARE_ROUTING_MASTER.md)** - Fastly/Direct for web; Option B (tunnel) for RPC ### Deployment Documents - **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** - Deployment orchestration (this directory) - **[../07-ccip/CCIP_DEPLOYMENT_SPEC.md](../07-ccip/CCIP_DEPLOYMENT_SPEC.md)** - CCIP deployment specification --- **Document Status:** Complete (v2.0) **Maintained By:** Infrastructure Team **Review Cycle:** Quarterly **Next Update:** After public blocks #2-6 are assigned --- ## Change Log ### Version 2.0 (2025-01-20) - Added network topology Mermaid diagram - Added VLAN architecture Mermaid diagram - Added ASCII art network topology - Enhanced public IP block matrix with status indicators - Added breadcrumb navigation - Added status indicators ### Version 1.0 (2024-12-15) - Initial version - Basic network architecture documentation