# Network Architecture - Enterprise Orchestration Plan **Last Updated:** 2025-01-20 **Document Version:** 2.0 **Project:** Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare Zero Trust + Dual ISP + 6×/28 --- ## Overview This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including: - **Hardware role assignments** (2× ER605, 3× ES216G, 1× ML110, 4× R630) - **6× /28 public IP blocks** with role-based NAT pools - **VLAN orchestration** with private subnet allocations - **Egress segmentation** by role and security plane - **Cloudflare Zero Trust** integration patterns --- ## Core Principles 1. **No public IPs on Proxmox hosts or LXCs/VMs** (default) 2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary) 3. **Public IPs used for:** - ER605 WAN addressing - **Egress NAT pools** (role-based allowlisting) - **Break-glass** emergency endpoints only 4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops 5. **Deterministic VMID registry** + IPAM that matches --- ## 1. Physical Topology & Hardware Roles ### 1.1 Hardware Role Assignment #### Edge / Routing - **ER605-A (Primary Edge Router)** - WAN1: Spectrum primary with Block #1 - WAN2: ISP #2 (failover/alternate policy) - Role: Active edge router, NAT pools, routing - **ER605-B (Standby Edge Router / Alternate WAN policy)** - Role: Standby router OR dedicated to WAN2 policies/testing - Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA. #### Switching Fabric - **ES216G-1**: Core / uplinks / trunks - **ES216G-2**: Compute rack aggregation - **ES216G-3**: Mgmt + out-of-band / staging #### Compute - **ML110 Gen9**: "Bootstrap & Management" node - IP: 192.168.11.10 - Role: Proxmox mgmt services, Omada controller, Git, monitoring seed - **4× Dell R630**: Proxmox compute cluster nodes - Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD - Role: Production workloads, CCIP fleet, sovereign tenants, services --- ## 2. ISP & Public IP Plan (6× /28) ### Public Block #1 (Known - Spectrum) | Property | Value | |----------|-------| | **Network** | `76.53.10.32/28` | | **Gateway** | `76.53.10.33` | | **Usable Range** | `76.53.10.33–76.53.10.46` | | **Broadcast** | `76.53.10.47` | | **ER605 WAN1 IP** | `76.53.10.34` (router interface) | ### Public Blocks #2–#6 (Placeholders - To Be Configured) | Block | Network | Gateway | Usable Range | Broadcast | Designated Use | |-------|--------|---------|--------------|-----------|----------------| | **#2** | `/28` | `` | `` | `` | CCIP Commit egress NAT pool | | **#3** | `/28` | `` | `` | `` | CCIP Execute egress NAT pool | | **#4** | `/28` | `` | `` | `` | RMN egress NAT pool | | **#5** | `/28` | `` | `` | `` | Sankofa/Phoenix/PanTel service egress | | **#6** | `/28` | `` | `` | `` | Sovereign Cloud Band tenant egress | ### 2.1 Public IP Usage Policy (Role-based) | Public /28 Block | Designated Use | Why | |------------------|----------------|-----| | **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency | | **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs | | **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs | | **#4** | RMN egress NAT pool | Independent security-plane egress | | **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation | | **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control | --- ## 3. Layer-2 & VLAN Orchestration Plan ### 3.1 VLAN Set (Authoritative) > **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility. | VLAN ID | VLAN Name | Purpose | Subnet | Gateway | |--------:|-----------|---------|--------|---------| | **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 | | 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 | | 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 | | 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 | | 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 | | 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 | | 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 | | 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 | | 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 | | 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 | | 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 | | 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 | | 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 | | 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 | | 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 | | 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 | | 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 | | 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 | ### 3.2 Switching Configuration (ES216G) - **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A) - **ES216G-2**: **Compute** (trunks to R630s + ML110) - **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band) **All Proxmox uplinks should be 802.1Q trunk ports.** --- ## 4. Routing, NAT, and Egress Segmentation (ER605) ### 4.1 Dual Router Roles - **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1) - **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services) ### 4.2 NAT Policies (Critical) #### Inbound NAT - **Default: none** - Break-glass only (optional): - Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred) - Proxmox admin should remain **LAN-only** #### Outbound NAT (Role-based Pools Using /28 Blocks) | Private Subnet | Role | Egress NAT Pool | Public Block | |----------------|------|-----------------|--------------| | 10.132.0.0/24 | CCIP Commit | **Block #2** `/28` | #2 | | 10.133.0.0/24 | CCIP Execute | **Block #3** `/28` | #3 | | 10.134.0.0/24 | RMN | **Block #4** `/28` | #4 | | 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `/28` | #5 | | 10.200.0.0/20–10.203.0.0/20 | Sovereign tenants | **Block #6** `/28` | #6 | | 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 | This yields **provable separation**, allowlisting, and incident scoping. --- ## 5. Proxmox Cluster Orchestration ### 5.1 Node Layout - **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner - **r630-01..04**: production compute ### 5.2 Proxmox Networking (per host) - **`vmbr0`**: VLAN-aware bridge - Native VLAN: 11 (MGMT) - Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203 - **Proxmox host IP** remains on **VLAN 11** only. ### 5.3 Storage Orchestration (R630) **Hardware:** - 2×600GB boot (mirror recommended) - 6×250GB SSD **Recommended:** - **Boot drives**: ZFS mirror or hardware RAID1 - **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance) - **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas --- ## 6. Cloudflare Zero Trust Orchestration ### 6.1 cloudflared Gateway Pattern Run **2 cloudflared LXCs** for redundancy: - `cloudflared-1` on ML110 - `cloudflared-2` on an R630 Both run tunnels for: - Blockscout - FireFly - Gitea - Internal admin dashboards (Grafana) behind Cloudflare Access **Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access with strict posture/MFA. --- ## 7. Complete VMID and Network Allocation Table | VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) | |-----------:|-------------------|-----------|--------:|------------------------|---------------------------| | **EDGE** | ER605 WAN1 (Primary) | WAN1 | — | — | **76.53.10.34** *(router WAN IP)* | | **EDGE** | Spectrum ISP Gateway | — | — | — | **76.53.10.33** *(ISP gateway)* | | 1000–1499 | **Besu** – Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) | | 1500–2499 | **Besu** – Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* | | 2500–3499 | **Besu** – RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **76.53.10.36** *(Reserved edge VIP for emergency RPC only; primary is Cloudflare Tunnel)* | | 3500–4299 | **Besu** – Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None | | 4300–4999 | **Besu** – Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None | | 5000–5099 | **Blockscout** – Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **76.53.10.35** *(Reserved edge VIP for emergency UI only; primary is Cloudflare Tunnel)* | | 5200–5299 | **Cacti** – Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via Cloudflare Tunnel if needed)* | | 5400–5401 | **CCIP** – Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* | | 5402–5403 | **CCIP** – Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* | | 5410–5425 | **CCIP** – Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** | | 5440–5455 | **CCIP** – Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** | | 5470–5476 | **CCIP** – RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** | | 5480–5599 | **CCIP** – Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None | | 6000–6099 | **Fabric** – Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via Cloudflare Tunnel if required)* | | 6200–6299 | **FireFly** – Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary is Cloudflare Tunnel)* | | 6400–7399 | **Indy** – Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary is Cloudflare Tunnel)* | | 7800–8999 | **Sankofa / Phoenix / PanTel** – Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** | | 10000–10999 | **Phoenix Sovereign Cloud Band** – SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** | | 11000–11999 | **Phoenix Sovereign Cloud Band** – ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** | | 12000–12999 | **Phoenix Sovereign Cloud Band** – DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** | | 13000–13999 | **Phoenix Sovereign Cloud Band** – Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** | --- ## 8. Network Security Model ### 8.1 Access Patterns 1. **No Public Access (Tunnel/VPN Only)** - Besu Validators (VLAN 110) - Besu Archive/Infrastructure (VLAN 113) - CCIP Ops/Admin (VLAN 130) - CCIP Monitoring (VLAN 131) 2. **Cloudflare Tunnel (Primary)** - Blockscout (VLAN 120) - Emergency VIP: 76.53.10.35 - Besu RPC (VLAN 112) - Emergency VIP: 76.53.10.36 - FireFly (VLAN 141) - Emergency VIP: 76.53.10.37 - Indy (VLAN 150) - Emergency VIP: 76.53.10.39 - Sankofa/Phoenix/PanTel (VLAN 160) - Emergency VIP: 76.53.10.38 3. **Role-Based Egress NAT (Allowlistable)** - CCIP Commit (VLAN 132) → Block #2 - CCIP Execute (VLAN 133) → Block #3 - RMN (VLAN 134) → Block #4 - Sankofa/Phoenix/PanTel (VLAN 160) → Block #5 - Sovereign tenants (VLAN 200-203) → Block #6 4. **Cloudflare Access / VPN Only** - CCIP Ops/Admin (VLAN 130) - CCIP Monitoring (VLAN 131) - Optional dashboard publishing --- ## 9. Implementation Notes ### 9.1 Gateway Configuration - All private subnets use `.1` as the gateway address - Example: VLAN 110 uses `10.110.0.1` as gateway - VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility) ### 9.2 Subnet Sizing - **/24 subnets:** Standard service VLANs (256 addresses) - **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses) - **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each) ### 9.3 IP Address Allocation - **Private IPs:** - VLAN 11: 192.168.11.0/24 (legacy mgmt) - All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet) - **Public IPs:** 6× /28 blocks with role-based NAT pools - **All public access** should route through Cloudflare Tunnel for security ### 9.4 VLAN Tagging - All VLANs are tagged on the Proxmox bridge - Ensure Proxmox bridge is configured for **VLAN-aware mode** - Physical switch must support VLAN tagging (802.1Q) --- ## 10. Configuration Files This architecture should be reflected in: - `config/network.conf` - Network configuration - `config/proxmox.conf` - VMID ranges - Proxmox bridge configuration (VLAN-aware mode) - ER605 router configuration (NAT pools, routing) - Cloudflare Tunnel configuration - ES216G switch configuration (VLAN trunks) --- ## 11. References - [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration) - [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/) - [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918) - [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/) - [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/) --- **Document Status:** Complete (v2.0) **Maintained By:** Infrastructure Team **Review Cycle:** Quarterly **Next Update:** After public blocks #2-6 are assigned