# Network Architecture - Enterprise Orchestration Plan

**Navigation:** [Home](/docs/01-getting-started/README.md) > [Architecture](/docs/01-getting-started/README.md) > Network Architecture

**Last Updated:** 2025-01-20  
**Document Version:** 2.0  
**Status:** 🟢 Active Documentation  
**Project:** Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare Zero Trust + Dual ISP + 6×/28

---

## Overview

This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:

- **Hardware role assignments** (2× ER605, 3× ES216G, 1× ML110, 4× R630)
- **6× /28 public IP blocks** with role-based NAT pools
- **VLAN orchestration** with private subnet allocations
- **Egress segmentation** by role and security plane
- **Cloudflare Zero Trust** integration patterns

---

## Core Principles

1. **No public IPs on Proxmox hosts or LXCs/VMs** (default)
2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary)
3. **Public IPs used for:**
   - ER605 WAN addressing
   - **Egress NAT pools** (role-based allowlisting)
   - **Break-glass** emergency endpoints only
4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops
5. **Deterministic VMID registry** + IPAM that matches

---

## 1. Physical Topology & Hardware Roles

> **Reference:** For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)**.

### 1.1 Hardware Role Assignment

#### Edge / Routing
- **ER605-A (Primary Edge Router)**
  - WAN1: Spectrum primary with Block #1
  - WAN2: ISP #2 (failover/alternate policy)
  - Role: Active edge router, NAT pools, routing

- **ER605-B (Standby Edge Router / Alternate WAN policy)**
  - Role: Standby router OR dedicated to WAN2 policies/testing
  - Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA.

#### Switching Fabric
- **ES216G-1**: Core / uplinks / trunks
- **ES216G-2**: Compute rack aggregation
- **ES216G-3**: Mgmt + out-of-band / staging

#### Compute
- **ML110 Gen9**: "Bootstrap & Management" node
  - IP: 192.168.11.10
  - Role: Proxmox mgmt services, Omada controller, Git, monitoring seed

- **4× Dell R630**: Proxmox compute cluster nodes
  - Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD
  - Role: Production workloads, CCIP fleet, sovereign tenants, services

---

## 2. ISP & Public IP Plan (6× /28)

### Public Block #1 (Known - Spectrum)

| Property | Value | Status |
|----------|-------|--------|
| **Network** | `76.53.10.32/28` | ✅ Configured |
| **Gateway** | `76.53.10.33` | ✅ Active |
| **Usable Range** | `76.53.10.33–76.53.10.46` | ✅ In Use |
| **Broadcast** | `76.53.10.47` | - |
| **ER605 WAN1 IP** | `76.53.10.34` (router interface) | ✅ Active |
| **Available IPs** | 13 (76.53.10.35-46, excluding .34) | ✅ Available |

### Public Blocks #2–#6 (Placeholders - To Be Configured)

| Block | Network | Gateway | Usable Range | Broadcast | Designated Use |
|-------|--------|---------|--------------|-----------|----------------|
| **#2** | `<PUBLIC_BLOCK_2>/28` | `<GW2>` | `<USABLE2>` | `<BCAST2>` | CCIP Commit egress NAT pool |
| **#3** | `<PUBLIC_BLOCK_3>/28` | `<GW3>` | `<USABLE3>` | `<BCAST3>` | CCIP Execute egress NAT pool |
| **#4** | `<PUBLIC_BLOCK_4>/28` | `<GW4>` | `<USABLE4>` | `<BCAST4>` | RMN egress NAT pool |
| **#5** | `<PUBLIC_BLOCK_5>/28` | `<GW5>` | `<USABLE5>` | `<BCAST5>` | Sankofa/Phoenix/PanTel service egress |
| **#6** | `<PUBLIC_BLOCK_6>/28` | `<GW6>` | `<USABLE6>` | `<BCAST6>` | Sovereign Cloud Band tenant egress |

### 2.1 Public IP Usage Policy (Role-based)

| Public /28 Block | Designated Use | Why |
|------------------|----------------|-----|
| **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency |
| **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs |
| **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs |
| **#4** | RMN egress NAT pool | Independent security-plane egress |
| **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation |
| **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control |

---

## 3. Layer-2 & VLAN Orchestration Plan

### 3.1 VLAN Set (Authoritative)

> **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.

| VLAN ID | VLAN Name | Purpose | Subnet | Gateway |
|--------:|-----------|---------|--------|---------|
| **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 |
| 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 |
| 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 |
| 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 |
| 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 |
| 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 |
| 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 |
| 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 |
| 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 |
| 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 |
| 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 |
| 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 |
| 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 |
| 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 |
| 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 |
| 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 |
| 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 |
| 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 |

### 3.2 Switching Configuration (ES216G)

- **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A)
- **ES216G-2**: **Compute** (trunks to R630s + ML110)
- **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band)

**All Proxmox uplinks should be 802.1Q trunk ports.**

---

## 4. Routing, NAT, and Egress Segmentation (ER605)

### 4.1 Dual Router Roles

- **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1)
- **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services)

### 4.2 NAT Policies (Critical)

#### Inbound NAT

- **Default: none**
- Break-glass only (optional):
  - Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
  - Proxmox admin should remain **LAN-only**

#### Outbound NAT (Role-based Pools Using /28 Blocks)

| Private Subnet | Role | Egress NAT Pool | Public Block |
|----------------|------|-----------------|--------------|
| 10.132.0.0/24 | CCIP Commit | **Block #2** `<PUBLIC_BLOCK_2>/28` | #2 |
| 10.133.0.0/24 | CCIP Execute | **Block #3** `<PUBLIC_BLOCK_3>/28` | #3 |
| 10.134.0.0/24 | RMN | **Block #4** `<PUBLIC_BLOCK_4>/28` | #4 |
| 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `<PUBLIC_BLOCK_5>/28` | #5 |
| 10.200.0.0/20–10.203.0.0/20 | Sovereign tenants | **Block #6** `<PUBLIC_BLOCK_6>/28` | #6 |
| 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 |

This yields **provable separation**, allowlisting, and incident scoping.

---

## 5. Proxmox Cluster Orchestration

### 5.1 Node Layout

- **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner
- **r630-01..04**: production compute

### 5.2 Proxmox Networking (per host)

- **`vmbr0`**: VLAN-aware bridge
  - Native VLAN: 11 (MGMT)
  - Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203
- **Proxmox host IP** remains on **VLAN 11** only.

### 5.3 Storage Orchestration (R630)

**Hardware:**
- 2×600GB boot (mirror recommended)
- 6×250GB SSD

**Recommended:**
- **Boot drives**: ZFS mirror or hardware RAID1
- **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
- **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas

---

## 6. Cloudflare Zero Trust Orchestration

### 6.1 cloudflared Gateway Pattern

Run **2 cloudflared LXCs** for redundancy:

- `cloudflared-1` on ML110
- `cloudflared-2` on an R630

Both run tunnels for:
- Blockscout
- FireFly
- Gitea
- Internal admin dashboards (Grafana) behind Cloudflare Access

**Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access with strict posture/MFA.

---

## 7. Complete VMID and Network Allocation Table

| VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) |
|-----------:|-------------------|-----------|--------:|------------------------|---------------------------|
| **EDGE** | ER605 WAN1 (Primary) | WAN1 | — | — | **76.53.10.34** *(router WAN IP)* |
| **EDGE** | Spectrum ISP Gateway | — | — | — | **76.53.10.33** *(ISP gateway)* |
| 1000–1499 | **Besu** – Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) |
| 1500–2499 | **Besu** – Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* |
| 2500–3499 | **Besu** – RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **76.53.10.36** *(Reserved edge VIP for emergency RPC only; primary is Cloudflare Tunnel)* |
| 3500–4299 | **Besu** – Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None |
| 4300–4999 | **Besu** – Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None |
| 5000–5099 | **Blockscout** – Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **76.53.10.35** *(Reserved edge VIP for emergency UI only; primary is Cloudflare Tunnel)* |
| 5200–5299 | **Cacti** – Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via Cloudflare Tunnel if needed)* |
| 5400–5401 | **CCIP** – Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* |
| 5402–5403 | **CCIP** – Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* |
| 5410–5425 | **CCIP** – Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** |
| 5440–5455 | **CCIP** – Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** |
| 5470–5476 | **CCIP** – RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** |
| 5480–5599 | **CCIP** – Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None |
| 6000–6099 | **Fabric** – Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via Cloudflare Tunnel if required)* |
| 6200–6299 | **FireFly** – Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary is Cloudflare Tunnel)* |
| 6400–7399 | **Indy** – Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary is Cloudflare Tunnel)* |
| 7800–8999 | **Sankofa / Phoenix / PanTel** – Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** |
| 10000–10999 | **Phoenix Sovereign Cloud Band** – SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** |
| 11000–11999 | **Phoenix Sovereign Cloud Band** – ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** |
| 12000–12999 | **Phoenix Sovereign Cloud Band** – DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** |
| 13000–13999 | **Phoenix Sovereign Cloud Band** – Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** |

---

## 8. Network Security Model

### 8.1 Access Patterns

1. **No Public Access (Tunnel/VPN Only)**
   - Besu Validators (VLAN 110)
   - Besu Archive/Infrastructure (VLAN 113)
   - CCIP Ops/Admin (VLAN 130)
   - CCIP Monitoring (VLAN 131)

2. **Cloudflare Tunnel (Primary)**
   - Blockscout (VLAN 120) - Emergency VIP: 76.53.10.35
   - Besu RPC (VLAN 112) - Emergency VIP: 76.53.10.36
   - FireFly (VLAN 141) - Emergency VIP: 76.53.10.37
   - Indy (VLAN 150) - Emergency VIP: 76.53.10.39
   - Sankofa/Phoenix/PanTel (VLAN 160) - Emergency VIP: 76.53.10.38

3. **Role-Based Egress NAT (Allowlistable)**
   - CCIP Commit (VLAN 132) → Block #2
   - CCIP Execute (VLAN 133) → Block #3
   - RMN (VLAN 134) → Block #4
   - Sankofa/Phoenix/PanTel (VLAN 160) → Block #5
   - Sovereign tenants (VLAN 200-203) → Block #6

4. **Cloudflare Access / VPN Only**
   - CCIP Ops/Admin (VLAN 130)
   - CCIP Monitoring (VLAN 131) - Optional dashboard publishing

---

## 9. Implementation Notes

### 9.1 Gateway Configuration
- All private subnets use `.1` as the gateway address
- Example: VLAN 110 uses `10.110.0.1` as gateway
- VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility)

### 9.2 Subnet Sizing
- **/24 subnets:** Standard service VLANs (256 addresses)
- **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses)
- **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each)

### 9.3 IP Address Allocation
- **Private IPs:**
  - VLAN 11: 192.168.11.0/24 (legacy mgmt)
  - All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
- **Public IPs:** 6× /28 blocks with role-based NAT pools
- **All public access** should route through Cloudflare Tunnel for security

### 9.4 VLAN Tagging
- All VLANs are tagged on the Proxmox bridge
- Ensure Proxmox bridge is configured for **VLAN-aware mode**
- Physical switch must support VLAN tagging (802.1Q)

---

## 10. Configuration Files

This architecture should be reflected in:
- `config/network.conf` - Network configuration
- `config/proxmox.conf` - VMID ranges
- Proxmox bridge configuration (VLAN-aware mode)
- ER605 router configuration (NAT pools, routing)
- Cloudflare Tunnel configuration
- ES216G switch configuration (VLAN trunks)

---

## 11. References

- [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration)
- [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
- [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918)
- [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/)
- [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/)

---

## Related Documentation

### Architecture Documents
- **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** ⭐⭐⭐ - Complete physical hardware inventory and specifications
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** ⭐⭐⭐ - Enterprise deployment orchestration guide
- **[VMID_ALLOCATION_FINAL.md](VMID_ALLOCATION_FINAL.md)** ⭐⭐⭐ - VMID allocation registry
- **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** ⭐⭐ - Domain structure and DNS assignments
- **[HOSTNAME_MIGRATION_GUIDE.md](HOSTNAME_MIGRATION_GUIDE.md)** ⭐ - Hostname migration procedures

### Configuration Documents
- **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](/docs/04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration
- **[../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare Zero Trust setup
- **[../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md](../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md)** - Cloudflare tunnel routing

### Deployment Documents
- **[../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md](../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md)** - Deployment orchestration
- **[../07-ccip/CCIP_DEPLOYMENT_SPEC.md](../07-ccip/CCIP_DEPLOYMENT_SPEC.md)** - CCIP deployment specification

---

**Document Status:** Complete (v2.0)  
**Maintained By:** Infrastructure Team  
**Review Cycle:** Quarterly  
**Next Update:** After public blocks #2-6 are assigned

---

## Change Log

### Version 2.0 (2025-01-20)
- Added network topology Mermaid diagram
- Added VLAN architecture Mermaid diagram
- Added ASCII art network topology
- Enhanced public IP block matrix with status indicators
- Added breadcrumb navigation
- Added status indicators

### Version 1.0 (2024-12-15)
- Initial version
- Basic network architecture documentation
