cb47cce074
- Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
16 KiB
16 KiB
Network Architecture - Enterprise Orchestration Plan
Navigation: Home > Architecture > Network Architecture
Last Updated: 2025-01-20
Document Version: 2.0
Status: 🟢 Active Documentation
Project: Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare Zero Trust + Dual ISP + 6×/28
Overview
This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:
- Hardware role assignments (2× ER605, 3× ES216G, 1× ML110, 4× R630)
- 6× /28 public IP blocks with role-based NAT pools
- VLAN orchestration with private subnet allocations
- Egress segmentation by role and security plane
- Cloudflare Zero Trust integration patterns
Core Principles
- No public IPs on Proxmox hosts or LXCs/VMs (default)
- Inbound access = Cloudflare Zero Trust + cloudflared (primary)
- Public IPs used for:
- ER605 WAN addressing
- Egress NAT pools (role-based allowlisting)
- Break-glass emergency endpoints only
- Segmentation by VLAN/VRF: consensus vs services vs sovereign tenants vs ops
- Deterministic VMID registry + IPAM that matches
1. Physical Topology & Hardware Roles
Reference: For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see PHYSICAL_HARDWARE_INVENTORY.md.
1.1 Hardware Role Assignment
Edge / Routing
-
ER605-A (Primary Edge Router)
- WAN1: Spectrum primary with Block #1
- WAN2: ISP #2 (failover/alternate policy)
- Role: Active edge router, NAT pools, routing
-
ER605-B (Standby Edge Router / Alternate WAN policy)
- Role: Standby router OR dedicated to WAN2 policies/testing
- Note: ER605 does not support full stateful HA. This is active/standby operational redundancy, not automatic session-preserving HA.
Switching Fabric
- ES216G-1: Core / uplinks / trunks
- ES216G-2: Compute rack aggregation
- ES216G-3: Mgmt + out-of-band / staging
Compute
-
ML110 Gen9: "Bootstrap & Management" node
- IP: 192.168.11.10
- Role: Proxmox mgmt services, Omada controller, Git, monitoring seed
-
4× Dell R630: Proxmox compute cluster nodes
- Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD
- Role: Production workloads, CCIP fleet, sovereign tenants, services
2. ISP & Public IP Plan (6× /28)
Public Block #1 (Known - Spectrum)
| Property | Value | Status |
|---|---|---|
| Network | 76.53.10.32/28 |
✅ Configured |
| Gateway | 76.53.10.33 |
✅ Active |
| Usable Range | 76.53.10.33–76.53.10.46 |
✅ In Use |
| Broadcast | 76.53.10.47 |
- |
| ER605 WAN1 IP | 76.53.10.34 (router interface) |
✅ Active |
| Available IPs | 13 (76.53.10.35-46, excluding .34) | ✅ Available |
Public Blocks #2–#6 (Placeholders - To Be Configured)
| Block | Network | Gateway | Usable Range | Broadcast | Designated Use |
|---|---|---|---|---|---|
| #2 | <PUBLIC_BLOCK_2>/28 |
<GW2> |
<USABLE2> |
<BCAST2> |
CCIP Commit egress NAT pool |
| #3 | <PUBLIC_BLOCK_3>/28 |
<GW3> |
<USABLE3> |
<BCAST3> |
CCIP Execute egress NAT pool |
| #4 | <PUBLIC_BLOCK_4>/28 |
<GW4> |
<USABLE4> |
<BCAST4> |
RMN egress NAT pool |
| #5 | <PUBLIC_BLOCK_5>/28 |
<GW5> |
<USABLE5> |
<BCAST5> |
Sankofa/Phoenix/PanTel service egress |
| #6 | <PUBLIC_BLOCK_6>/28 |
<GW6> |
<USABLE6> |
<BCAST6> |
Sovereign Cloud Band tenant egress |
2.1 Public IP Usage Policy (Role-based)
| Public /28 Block | Designated Use | Why |
|---|---|---|
| #1 (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency |
| #2 | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs |
| #3 | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs |
| #4 | RMN egress NAT pool | Independent security-plane egress |
| #5 | Sankofa/Phoenix/PanTel service egress | Service-plane separation |
| #6 | Sovereign Cloud Band tenant egress | Per-sovereign policy control |
3. Layer-2 & VLAN Orchestration Plan
3.1 VLAN Set (Authoritative)
Migration Note: Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.
| VLAN ID | VLAN Name | Purpose | Subnet | Gateway |
|---|---|---|---|---|
| 11 | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 |
| 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 |
| 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 |
| 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 |
| 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 |
| 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 |
| 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 |
| 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 |
| 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 |
| 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 |
| 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 |
| 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 |
| 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 |
| 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 |
| 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 |
| 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 |
| 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 |
| 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 |
3.2 Switching Configuration (ES216G)
- ES216G-1: Core (all VLAN trunks to ES216G-2/3 + ER605-A)
- ES216G-2: Compute (trunks to R630s + ML110)
- ES216G-3: Mgmt/OOB (mgmt access ports, staging, out-of-band)
All Proxmox uplinks should be 802.1Q trunk ports.
4. Routing, NAT, and Egress Segmentation (ER605)
4.1 Dual Router Roles
- ER605-A: Active edge router (WAN1 = Spectrum primary with Block #1)
- ER605-B: Standby router OR dedicated to WAN2 policies/testing (no inbound services)
4.2 NAT Policies (Critical)
Inbound NAT
- Default: none
- Break-glass only (optional):
- Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
- Proxmox admin should remain LAN-only
Outbound NAT (Role-based Pools Using /28 Blocks)
| Private Subnet | Role | Egress NAT Pool | Public Block |
|---|---|---|---|
| 10.132.0.0/24 | CCIP Commit | Block #2 <PUBLIC_BLOCK_2>/28 |
#2 |
| 10.133.0.0/24 | CCIP Execute | Block #3 <PUBLIC_BLOCK_3>/28 |
#3 |
| 10.134.0.0/24 | RMN | Block #4 <PUBLIC_BLOCK_4>/28 |
#4 |
| 10.160.0.0/22 | Sankofa/Phoenix/PanTel | Block #5 <PUBLIC_BLOCK_5>/28 |
#5 |
| 10.200.0.0/20–10.203.0.0/20 | Sovereign tenants | Block #6 <PUBLIC_BLOCK_6>/28 |
#6 |
| 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 |
This yields provable separation, allowlisting, and incident scoping.
5. Proxmox Cluster Orchestration
5.1 Node Layout
- ml110 (192.168.11.10): mgmt + seed services + initial automation runner
- r630-01..04: production compute
5.2 Proxmox Networking (per host)
vmbr0: VLAN-aware bridge- Native VLAN: 11 (MGMT)
- Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203
- Proxmox host IP remains on VLAN 11 only.
5.3 Storage Orchestration (R630)
Hardware:
- 2×600GB boot (mirror recommended)
- 6×250GB SSD
Recommended:
- Boot drives: ZFS mirror or hardware RAID1
- Data SSDs: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
- High-write workloads (logs/metrics/indexers) on dedicated dataset with quotas
6. Cloudflare Zero Trust Orchestration
6.1 cloudflared Gateway Pattern
Run 2 cloudflared LXCs for redundancy:
cloudflared-1on ML110cloudflared-2on an R630
Both run tunnels for:
- Blockscout
- FireFly
- Gitea
- Internal admin dashboards (Grafana) behind Cloudflare Access
Keep Proxmox UI LAN-only; if needed, publish via Cloudflare Access with strict posture/MFA.
7. Complete VMID and Network Allocation Table
| VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) |
|---|---|---|---|---|---|
| EDGE | ER605 WAN1 (Primary) | WAN1 | — | — | 76.53.10.34 (router WAN IP) |
| EDGE | Spectrum ISP Gateway | — | — | — | 76.53.10.33 (ISP gateway) |
| 1000–1499 | Besu – Validators | BESU-VAL | 110 | 10.110.0.0/24 | None (no inbound; tunnel/VPN only) |
| 1500–2499 | Besu – Sentries | BESU-SEN | 111 | 10.111.0.0/24 | None (optional later via NAT pool) |
| 2500–3499 | Besu – RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | 76.53.10.36 (Reserved edge VIP for emergency RPC only; primary is Cloudflare Tunnel) |
| 3500–4299 | Besu – Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None |
| 4300–4999 | Besu – Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None |
| 5000–5099 | Blockscout – Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | 76.53.10.35 (Reserved edge VIP for emergency UI only; primary is Cloudflare Tunnel) |
| 5200–5299 | Cacti – Interop middleware | CACTI | 121 | 10.121.0.0/24 | None (publish via Cloudflare Tunnel if needed) |
| 5400–5401 | CCIP – Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None (Cloudflare Access / VPN only) |
| 5402–5403 | CCIP – Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None (optionally publish dashboards via Cloudflare Access) |
| 5410–5425 | CCIP – Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | Egress NAT: Block #2 |
| 5440–5455 | CCIP – Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | Egress NAT: Block #3 |
| 5470–5476 | CCIP – RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | Egress NAT: Block #4 |
| 5480–5599 | CCIP – Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None |
| 6000–6099 | Fabric – Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None (publish via Cloudflare Tunnel if required) |
| 6200–6299 | FireFly – Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | 76.53.10.37 (Reserved edge VIP if ever needed; primary is Cloudflare Tunnel) |
| 6400–7399 | Indy – Identity layer | INDY | 150 | 10.150.0.0/24 | 76.53.10.39 (Reserved edge VIP for DID endpoints if required; primary is Cloudflare Tunnel) |
| 7800–8999 | Sankofa / Phoenix / PanTel – Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | Egress NAT: Block #5 |
| 10000–10999 | Phoenix Sovereign Cloud Band – SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | Egress NAT: Block #6 |
| 11000–11999 | Phoenix Sovereign Cloud Band – ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | Egress NAT: Block #6 |
| 12000–12999 | Phoenix Sovereign Cloud Band – DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | Egress NAT: Block #6 |
| 13000–13999 | Phoenix Sovereign Cloud Band – Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | Egress NAT: Block #6 |
8. Network Security Model
8.1 Access Patterns
-
No Public Access (Tunnel/VPN Only)
- Besu Validators (VLAN 110)
- Besu Archive/Infrastructure (VLAN 113)
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131)
-
Cloudflare Tunnel (Primary)
- Blockscout (VLAN 120) - Emergency VIP: 76.53.10.35
- Besu RPC (VLAN 112) - Emergency VIP: 76.53.10.36
- FireFly (VLAN 141) - Emergency VIP: 76.53.10.37
- Indy (VLAN 150) - Emergency VIP: 76.53.10.39
- Sankofa/Phoenix/PanTel (VLAN 160) - Emergency VIP: 76.53.10.38
-
Role-Based Egress NAT (Allowlistable)
- CCIP Commit (VLAN 132) → Block #2
- CCIP Execute (VLAN 133) → Block #3
- RMN (VLAN 134) → Block #4
- Sankofa/Phoenix/PanTel (VLAN 160) → Block #5
- Sovereign tenants (VLAN 200-203) → Block #6
-
Cloudflare Access / VPN Only
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131) - Optional dashboard publishing
9. Implementation Notes
9.1 Gateway Configuration
- All private subnets use
.1as the gateway address - Example: VLAN 110 uses
10.110.0.1as gateway - VLAN 11 (MGMT) uses
192.168.11.1(legacy compatibility)
9.2 Subnet Sizing
- /24 subnets: Standard service VLANs (256 addresses)
- /22 subnet: Sankofa/Phoenix/PanTel (1024 addresses)
- /20 subnets: Phoenix Sovereign Cloud Bands (4096 addresses each)
9.3 IP Address Allocation
- Private IPs:
- VLAN 11: 192.168.11.0/24 (legacy mgmt)
- All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
- Public IPs: 6× /28 blocks with role-based NAT pools
- All public access should route through Cloudflare Tunnel for security
9.4 VLAN Tagging
- All VLANs are tagged on the Proxmox bridge
- Ensure Proxmox bridge is configured for VLAN-aware mode
- Physical switch must support VLAN tagging (802.1Q)
10. Configuration Files
This architecture should be reflected in:
config/network.conf- Network configurationconfig/proxmox.conf- VMID ranges- Proxmox bridge configuration (VLAN-aware mode)
- ER605 router configuration (NAT pools, routing)
- Cloudflare Tunnel configuration
- ES216G switch configuration (VLAN trunks)
11. References
- Proxmox VLAN Configuration
- Cloudflare Tunnel Documentation
- RFC 1918 - Private Address Space
- ER605 User Guide
- ES216G Configuration Guide
Related Documentation
Architecture Documents
- PHYSICAL_HARDWARE_INVENTORY.md ⭐⭐⭐ - Complete physical hardware inventory and specifications
- ORCHESTRATION_DEPLOYMENT_GUIDE.md ⭐⭐⭐ - Enterprise deployment orchestration guide
- VMID_ALLOCATION_FINAL.md ⭐⭐⭐ - VMID allocation registry
- DOMAIN_STRUCTURE.md ⭐⭐ - Domain structure and DNS assignments
- HOSTNAME_MIGRATION_GUIDE.md ⭐ - Hostname migration procedures
Configuration Documents
- ../04-configuration/ER605_ROUTER_CONFIGURATION.md - Router configuration
- ../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md - Cloudflare Zero Trust setup
- ../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md - Cloudflare tunnel routing
Deployment Documents
- ../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md - Deployment orchestration
- ../07-ccip/CCIP_DEPLOYMENT_SPEC.md - CCIP deployment specification
Document Status: Complete (v2.0)
Maintained By: Infrastructure Team
Review Cycle: Quarterly
Next Update: After public blocks #2-6 are assigned
Change Log
Version 2.0 (2025-01-20)
- Added network topology Mermaid diagram
- Added VLAN architecture Mermaid diagram
- Added ASCII art network topology
- Enhanced public IP block matrix with status indicators
- Added breadcrumb navigation
- Added status indicators
Version 1.0 (2024-12-15)
- Initial version
- Basic network architecture documentation