# Network Architecture - Enterprise Orchestration Plan
**Navigation:** [Home](../01-getting-started/README.md) > [Architecture](README.md) > Network Architecture
**Related:** [PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md) | [DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md) | [ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md) | [11-references/NETWORK_CONFIGURATION_MASTER.md](../11-references/NETWORK_CONFIGURATION_MASTER.md) | **Runbooks & VLAN:** [03-deployment/OPERATIONAL_RUNBOOKS.md](../03-deployment/OPERATIONAL_RUNBOOKS.md) (Phase 4, VLAN), [03-deployment/MISSING_CONTAINERS_LIST.md](../03-deployment/MISSING_CONTAINERS_LIST.md), [04-configuration/UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md](../04-configuration/UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md)
**Last Updated:** 2026-02-05
**Document Version:** 2.1
**Status:** ๐ข Active Documentation
**Project:** Sankofa / Phoenix / PanTel ยท ChainID 138 ยท Proxmox + Cloudflare DNS + NPMplus (edge: UDM Pro; Fastly or direct to 76.53.10.36)
---
## Overview
This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:
- **Hardware role assignments** (2ร ER605, 3ร ES216G, 1ร ML110, 4ร R630)
- **6ร /28 public IP blocks** with role-based NAT pools
- **VLAN orchestration** with private subnet allocations
- **Egress segmentation** by role and security plane
- **Cloudflare Zero Trust** integration patterns
---
## Architecture Diagrams
### Network Topology (High Level)
```mermaid
graph TB
Internet[Internet]
CF[Cloudflare Zero Trust]
UDM[UDM Pro 76.53.10.34]
NPM[NPMplus 192.168.11.167]
ES1[ES216G-1 Core]
ES2[ES216G-2 Compute]
ML[ML110 192.168.11.10]
R1[R630-01 192.168.11.11]
R2[R630-02 192.168.11.12]
Internet --> CF
CF --> UDM
UDM --> NPM
NPM --> ES1
ES1 --> ES2
ES2 --> ML
ES2 --> R1
ES2 --> R2
```
### VLAN Architecture (Selected VLANs)
```mermaid
graph TD
V11[VLAN 11: MGMT-LAN
192.168.11.0/24]
V110[VLAN 110: BESU-VAL
10.110.0.0/24]
V111[VLAN 111: BESU-SEN
10.111.0.0/24]
V112[VLAN 112: BESU-RPC
10.112.0.0/24]
V132[VLAN 132: CCIP-COMMIT
10.132.0.0/24]
V133[VLAN 133: CCIP-EXEC
10.133.0.0/24]
V134[VLAN 134: CCIP-RMN
10.134.0.0/24]
V11 --> V110
V11 --> V111
V11 --> V112
V11 --> V132
V11 --> V133
V11 --> V134
```
See [VLAN Set (Authoritative)](#31-vlan-set-authoritative) below for the full table.
### Proxmox Cluster (Nodes)
```mermaid
graph LR
ML[ml110 192.168.11.10]
R1[r630-01 .11]
R2[r630-02 .12]
R3[r630-03 .13]
R4[r630-04 .14]
ML --- R1
ML --- R2
R1 --- R2
R1 --- R3
R2 --- R4
```
---
## Core Principles
1. **No public IPs on Proxmox hosts or LXCs/VMs** (default)
2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary)
3. **Public IPs used for:**
- ER605 WAN addressing
- **Egress NAT pools** (role-based allowlisting)
- **Break-glass** emergency endpoints only
4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops
5. **Deterministic VMID registry** + IPAM that matches
---
## 1. Physical Topology & Hardware Roles
> **Reference:** For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)**.
### 1.1 Hardware Role Assignment
#### Edge / Routing
- **ER605-A (Primary Edge Router)**
- WAN1: Spectrum primary with Block #1
- WAN2: ISP #2 (failover/alternate policy)
- Role: Active edge router, NAT pools, routing
- **ER605-B (Standby Edge Router / Alternate WAN policy)**
- Role: Standby router OR dedicated to WAN2 policies/testing
- Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA.
#### Switching Fabric
- **ES216G-1**: Core / uplinks / trunks
- **ES216G-2**: Compute rack aggregation
- **ES216G-3**: Mgmt + out-of-band / staging
#### Compute
- **ML110 Gen9**: "Bootstrap & Management" node
- IP: 192.168.11.10
- Role: Proxmox mgmt services, Omada controller, Git, monitoring seed
- **4ร Dell R630**: Proxmox compute cluster nodes
- Resources: 512GB RAM each, 2ร600GB boot, 6ร250GB SSD
- Role: Production workloads, CCIP fleet, sovereign tenants, services
---
## 2. ISP & Public IP Plan (6ร /28)
### Public Block #1 (Known - Spectrum)
| Property | Value | Status |
|----------|-------|--------|
| **Network** | `76.53.10.32/28` | โ
Configured |
| **Gateway** | `76.53.10.33` | โ
Active |
| **Usable Range** | `76.53.10.33โ76.53.10.46` | โ
In Use |
| **Broadcast** | `76.53.10.47` | - |
| **UDM Pro (edge)** | `76.53.10.34` (replaced ER605) | โ
Active |
| **Available IPs** | 13 (76.53.10.35-46, excluding .34) | โ
Available |
### Public Blocks #2โ#6 (Reserved - To Be Configured)
> **Status:** Blocks #2โ#6 are reserved. Document actual network/gateway/usable range when assigned by provider, or keep as placeholders until CCIP/Sankofa/Sovereign egress planning is finalized. See [MASTER_PLAN.md](../00-meta/MASTER_PLAN.md) ยง3.1.
| Block | Network | Gateway | Usable Range | Broadcast | Designated Use |
|-------|--------|---------|--------------|-----------|----------------|
| **#2** | `/28` | `` | `` | `` | CCIP Commit egress NAT pool |
| **#3** | `/28` | `` | `` | `` | CCIP Execute egress NAT pool |
| **#4** | `/28` | `` | `` | `` | RMN egress NAT pool |
| **#5** | `/28` | `` | `` | `` | Sankofa/Phoenix/PanTel service egress |
| **#6** | `/28` | `` | `` | `` | Sovereign Cloud Band tenant egress |
### 2.1 Public IP Usage Policy (Role-based)
| Public /28 Block | Designated Use | Why |
|------------------|----------------|-----|
| **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency |
| **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs |
| **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs |
| **#4** | RMN egress NAT pool | Independent security-plane egress |
| **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation |
| **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control |
---
## 3. Layer-2 & VLAN Orchestration Plan
### 3.1 VLAN Set (Authoritative)
> **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.
| VLAN ID | VLAN Name | Purpose | Subnet | Gateway |
|--------:|-----------|---------|--------|---------|
| **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 |
| 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 |
| 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 |
| 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 |
| 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 |
| 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 |
| 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 |
| 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 |
| 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 |
| 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 |
| 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 |
| 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 |
| 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 |
| 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 |
| 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 |
| 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 |
| 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 |
| 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 |
### 3.2 Switching Configuration (ES216G)
- **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A)
- **ES216G-2**: **Compute** (trunks to R630s + ML110)
- **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band)
**All Proxmox uplinks should be 802.1Q trunk ports.**
---
## 4. Routing, NAT, and Egress Segmentation (ER605)
### 4.1 Dual Router Roles
- **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1)
- **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services)
### 4.2 NAT Policies (Critical)
#### Inbound NAT
- **Default: none**
- Break-glass only (optional):
- Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
- Proxmox admin should remain **LAN-only**
#### Outbound NAT (Role-based Pools Using /28 Blocks)
| Private Subnet | Role | Egress NAT Pool | Public Block |
|----------------|------|-----------------|--------------|
| 10.132.0.0/24 | CCIP Commit | **Block #2** `/28` | #2 |
| 10.133.0.0/24 | CCIP Execute | **Block #3** `/28` | #3 |
| 10.134.0.0/24 | RMN | **Block #4** `/28` | #4 |
| 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `/28` | #5 |
| 10.200.0.0/20โ10.203.0.0/20 | Sovereign tenants | **Block #6** `/28` | #6 |
| 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 |
This yields **provable separation**, allowlisting, and incident scoping.
---
## 5. Proxmox Cluster Orchestration
### 5.1 Node Layout
- **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner
- **r630-01..04**: production compute
### 5.2 Proxmox Networking (per host)
- **`vmbr0`**: VLAN-aware bridge
- Native VLAN: 11 (MGMT)
- Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200โ203
- **Proxmox host IP** remains on **VLAN 11** only.
### 5.3 Storage Orchestration (R630)
**Hardware:**
- 2ร600GB boot (mirror recommended)
- 6ร250GB SSD
**Recommended:**
- **Boot drives**: ZFS mirror or hardware RAID1
- **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
- **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas
---
## 6. Public Edge: Fastly or Direct to NPMplus
### 6.1 Fastly or Direct to NPMplus (Primary Public Path)
**Public ingress** is **Fastly** (Option A) or **DNS direct to 76.53.10.36** (Option C). Both flow through **UDM Pro** port forward to **NPMplus** (VMID 10233 at 192.168.11.167). Cloudflare Tunnel is **deprecated** for public access (502 errors); Cloudflare DNS is retained for all public hostnames.
- **Flow:** Internet โ Cloudflare DNS โ Fastly or 76.53.10.36 โ UDM Pro (76.53.10.36:80/443) โ NPMplus โ internal services (Blockscout, RPC, DBIS, MIM4U, etc.).
- **Pre-requisite:** Verify 76.53.10.36:80 and :443 are open from the internet; see [05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md](../05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md). If closed (e.g. Spectrum filtering), use Option B (tunnel or VPS origin).
- **Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access or VPN with strict posture/MFA.
---
## 7. Complete VMID and Network Allocation Table
| VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) |
|-----------:|-------------------|-----------|--------:|------------------------|---------------------------|
| **EDGE** | UDM Pro (replaced ER605) | WAN | โ | โ | **76.53.10.34** *(edge)* |
| **EDGE** | Spectrum ISP Gateway | โ | โ | โ | **76.53.10.33** *(ISP gateway)* |
| 1000โ1499 | **Besu** โ Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) |
| 1500โ2499 | **Besu** โ Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* |
| 2500โ3499 | **Besu** โ RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **Via NPMplus** *(Fastly or direct to 76.53.10.36); Alltra/HYBX via 76.53.10.38 or 76.53.10.42)* |
| 3500โ4299 | **Besu** โ Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None |
| 4300โ4999 | **Besu** โ Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None |
| 5000โ5099 | **Blockscout** โ Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **Via NPMplus** *(Fastly or direct to 76.53.10.36)* |
| 5200โ5299 | **Cacti** โ Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via NPMplus/Fastly if needed)* |
| 5400โ5401 | **CCIP** โ Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* |
| 5402โ5403 | **CCIP** โ Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* |
| 5410โ5425 | **CCIP** โ Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** |
| 5440โ5455 | **CCIP** โ Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** |
| 5470โ5476 | **CCIP** โ RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** |
| 5480โ5599 | **CCIP** โ Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None |
| 6000โ6099 | **Fabric** โ Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via NPMplus/Fastly if required)* |
| 6200โ6299 | **FireFly** โ Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary via NPMplus)* |
| 6400โ7399 | **Indy** โ Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary via NPMplus)* |
| 10235 | **NPMplus Alltra/HYBX** | MGMT-LAN | 11 | 192.168.11.0/24 | **76.53.10.38** *(port forward 80/81/443); 76.53.10.42 designated; see [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](../04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md))* |
| 7800โ8999 | **Sankofa / Phoenix / PanTel** โ Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** |
| 10000โ10999 | **Phoenix Sovereign Cloud Band** โ SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** |
| 11000โ11999 | **Phoenix Sovereign Cloud Band** โ ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** |
| 12000โ12999 | **Phoenix Sovereign Cloud Band** โ DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** |
| 13000โ13999 | **Phoenix Sovereign Cloud Band** โ Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** |
---
## 8. Network Security Model
### 8.1 Access Patterns
1. **No Public Access (Tunnel/VPN Only)**
- Besu Validators (VLAN 110)
- Besu Archive/Infrastructure (VLAN 113)
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131)
2. **Fastly or Direct to NPMplus (Primary)**
- All public services route through NPMplus (VMID 10233) at 192.168.11.167
- Public origin: 76.53.10.36 (UDM Pro port forwarding to NPMplus)
- Blockscout (VLAN 120), Besu RPC (VLAN 112), FireFly (VLAN 141), Indy (VLAN 150), Sankofa/Phoenix/PanTel (VLAN 160) - Via NPMplus
- DNS: Cloudflare. Edge: Fastly (Option A) or direct to 76.53.10.36 (Option C). Tunnel deprecated for public ingress.
3. **Role-Based Egress NAT (Allowlistable)**
- CCIP Commit (VLAN 132) โ Block #2
- CCIP Execute (VLAN 133) โ Block #3
- RMN (VLAN 134) โ Block #4
- Sankofa/Phoenix/PanTel (VLAN 160) โ Block #5
- Sovereign tenants (VLAN 200-203) โ Block #6
4. **Cloudflare Access / VPN Only**
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131) - Optional dashboard publishing
---
## 9. Implementation Notes
### 9.1 Gateway Configuration
- All private subnets use `.1` as the gateway address
- Example: VLAN 110 uses `10.110.0.1` as gateway
- VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility)
### 9.2 Subnet Sizing
- **/24 subnets:** Standard service VLANs (256 addresses)
- **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses)
- **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each)
### 9.3 IP Address Allocation
- **Private IPs:**
- VLAN 11: 192.168.11.0/24 (legacy mgmt)
- All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
- **Public IPs:** 6ร /28 blocks with role-based NAT pools
- **All public access** routes through NPMplus (Fastly or direct to 76.53.10.36) for security and stability
### 9.4 VLAN Tagging
- All VLANs are tagged on the Proxmox bridge
- Ensure Proxmox bridge is configured for **VLAN-aware mode**
- Physical switch must support VLAN tagging (802.1Q)
---
## 10. Configuration Files
This architecture should be reflected in:
- `config/network.conf` - Network configuration
- `config/proxmox.conf` - VMID ranges
- Proxmox bridge configuration (VLAN-aware mode)
- ER605 router configuration (NAT pools, routing)
- Fastly or direct-to-NPMplus configuration (see 05-network routing docs)
- ES216G switch configuration (VLAN trunks)
---
## 11. References
- [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration)
- [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
- [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918)
- [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/)
- [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/)
---
## Related Documentation
### Architecture Documents
- **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** โญโญโญ - Complete physical hardware inventory and specifications
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** โญโญโญ - Enterprise deployment orchestration guide
- **[VMID_ALLOCATION_FINAL.md](VMID_ALLOCATION_FINAL.md)** โญโญโญ - VMID allocation registry
- **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** โญโญ - Domain structure and DNS assignments
- **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** โญ - Domain and hostname structure
### Configuration Documents
- **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](/docs/04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration
- **[../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare Zero Trust setup
- **[../05-network/CLOUDFLARE_ROUTING_MASTER.md](../05-network/CLOUDFLARE_ROUTING_MASTER.md)** - Fastly/Direct for web; Option B (tunnel) for RPC
### Deployment Documents
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** - Deployment orchestration (this directory)
- **[../07-ccip/CCIP_DEPLOYMENT_SPEC.md](../07-ccip/CCIP_DEPLOYMENT_SPEC.md)** - CCIP deployment specification
---
**Document Status:** Complete (v2.0)
**Maintained By:** Infrastructure Team
**Review Cycle:** Quarterly
**Next Update:** After public blocks #2-6 are assigned
---
## Change Log
### Version 2.0 (2025-01-20)
- Added network topology Mermaid diagram
- Added VLAN architecture Mermaid diagram
- Added ASCII art network topology
- Enhanced public IP block matrix with status indicators
- Added breadcrumb navigation
- Added status indicators
### Version 1.0 (2024-12-15)
- Initial version
- Basic network architecture documentation