d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
5aca5e2eebde665b3e8dddd139ef804ab7e71413
proxmox/docs/02-architecture/NETWORK_ARCHITECTURE.md
defiQUG 9c37af10c0 Complete optional next steps: fix references and consolidate duplicates 
- Fixed 104 broken references in 59 files
- Consolidated 40+ duplicate status files
- Archived duplicates to reports/archive/duplicates/
- Created scripts for reference fixing and consolidation
- Updated content inconsistency reports

All optional cleanup tasks complete.
2026-01-06 02:25:38 -08:00

367 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Network Architecture - Enterprise Orchestration Plan
**Navigation:** [Home](/docs/01-getting-started/README.md) > [Architecture](/docs/01-getting-started/README.md) > Network Architecture
**Last Updated:** 2025-01-20
**Document Version:** 2.0
**Status:** 🟢 Active Documentation
**Project:** Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare Zero Trust + Dual ISP + 6×/28
---
## Overview
This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:
- **Hardware role assignments** (2× ER605, 3× ES216G, 1× ML110, 4× R630)
- **6× /28 public IP blocks** with role-based NAT pools
- **VLAN orchestration** with private subnet allocations
- **Egress segmentation** by role and security plane
- **Cloudflare Zero Trust** integration patterns
---
## Core Principles
1. **No public IPs on Proxmox hosts or LXCs/VMs** (default)
2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary)
3. **Public IPs used for:**
- ER605 WAN addressing
- **Egress NAT pools** (role-based allowlisting)
- **Break-glass** emergency endpoints only
4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops
5. **Deterministic VMID registry** + IPAM that matches
---
## 1. Physical Topology & Hardware Roles
> **Reference:** For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)**.
### 1.1 Hardware Role Assignment
#### Edge / Routing
- **ER605-A (Primary Edge Router)**
- WAN1: Spectrum primary with Block #1
- WAN2: ISP #2 (failover/alternate policy)
- Role: Active edge router, NAT pools, routing
- **ER605-B (Standby Edge Router / Alternate WAN policy)**
- Role: Standby router OR dedicated to WAN2 policies/testing
- Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA.
#### Switching Fabric
- **ES216G-1**: Core / uplinks / trunks
- **ES216G-2**: Compute rack aggregation
- **ES216G-3**: Mgmt + out-of-band / staging
#### Compute
- **ML110 Gen9**: "Bootstrap & Management" node
- IP: 192.168.11.10
- Role: Proxmox mgmt services, Omada controller, Git, monitoring seed
- **4× Dell R630**: Proxmox compute cluster nodes
- Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD
- Role: Production workloads, CCIP fleet, sovereign tenants, services
---
## 2. ISP & Public IP Plan (6× /28)
### Public Block #1 (Known - Spectrum)
| Property | Value | Status |
|----------|-------|--------|
| **Network** | `76.53.10.32/28` | ✅ Configured |
| **Gateway** | `76.53.10.33` | ✅ Active |
| **Usable Range** | `76.53.10.33–76.53.10.46` | ✅ In Use |
| **Broadcast** | `76.53.10.47` | - |
| **ER605 WAN1 IP** | `76.53.10.34` (router interface) | ✅ Active |
| **Available IPs** | 13 (76.53.10.35-46, excluding .34) | ✅ Available |
### Public Blocks #2–#6 (Placeholders - To Be Configured)
| Block | Network | Gateway | Usable Range | Broadcast | Designated Use |
|-------|--------|---------|--------------|-----------|----------------|
| **#2** | `<PUBLIC_BLOCK_2>/28` | `<GW2>` | `<USABLE2>` | `<BCAST2>` | CCIP Commit egress NAT pool |
| **#3** | `<PUBLIC_BLOCK_3>/28` | `<GW3>` | `<USABLE3>` | `<BCAST3>` | CCIP Execute egress NAT pool |
| **#4** | `<PUBLIC_BLOCK_4>/28` | `<GW4>` | `<USABLE4>` | `<BCAST4>` | RMN egress NAT pool |
| **#5** | `<PUBLIC_BLOCK_5>/28` | `<GW5>` | `<USABLE5>` | `<BCAST5>` | Sankofa/Phoenix/PanTel service egress |
| **#6** | `<PUBLIC_BLOCK_6>/28` | `<GW6>` | `<USABLE6>` | `<BCAST6>` | Sovereign Cloud Band tenant egress |
### 2.1 Public IP Usage Policy (Role-based)
| Public /28 Block | Designated Use | Why |
|------------------|----------------|-----|
| **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency |
| **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs |
| **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs |
| **#4** | RMN egress NAT pool | Independent security-plane egress |
| **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation |
| **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control |
---
## 3. Layer-2 & VLAN Orchestration Plan
### 3.1 VLAN Set (Authoritative)
> **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.
| VLAN ID | VLAN Name | Purpose | Subnet | Gateway |
|--------:|-----------|---------|--------|---------|
| **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 |
| 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 |
| 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 |
| 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 |
| 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 |
| 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 |
| 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 |
| 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 |
| 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 |
| 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 |
| 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 |
| 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 |
| 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 |
| 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 |
| 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 |
| 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 |
| 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 |
| 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 |
### 3.2 Switching Configuration (ES216G)
- **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A)
- **ES216G-2**: **Compute** (trunks to R630s + ML110)
- **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band)
**All Proxmox uplinks should be 802.1Q trunk ports.**
---
## 4. Routing, NAT, and Egress Segmentation (ER605)
### 4.1 Dual Router Roles
- **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1)
- **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services)
### 4.2 NAT Policies (Critical)
#### Inbound NAT
- **Default: none**
- Break-glass only (optional):
- Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
- Proxmox admin should remain **LAN-only**
#### Outbound NAT (Role-based Pools Using /28 Blocks)
| Private Subnet | Role | Egress NAT Pool | Public Block |
|----------------|------|-----------------|--------------|
| 10.132.0.0/24 | CCIP Commit | **Block #2** `<PUBLIC_BLOCK_2>/28` | #2 |
| 10.133.0.0/24 | CCIP Execute | **Block #3** `<PUBLIC_BLOCK_3>/28` | #3 |
| 10.134.0.0/24 | RMN | **Block #4** `<PUBLIC_BLOCK_4>/28` | #4 |
| 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `<PUBLIC_BLOCK_5>/28` | #5 |
| 10.200.0.0/20–10.203.0.0/20 | Sovereign tenants | **Block #6** `<PUBLIC_BLOCK_6>/28` | #6 |
| 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 |
This yields **provable separation**, allowlisting, and incident scoping.
---
## 5. Proxmox Cluster Orchestration
### 5.1 Node Layout
- **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner
- **r630-01..04**: production compute
### 5.2 Proxmox Networking (per host)
- **`vmbr0`**: VLAN-aware bridge
- Native VLAN: 11 (MGMT)
- Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203
- **Proxmox host IP** remains on **VLAN 11** only.
### 5.3 Storage Orchestration (R630)
**Hardware:**
- 2×600GB boot (mirror recommended)
- 6×250GB SSD
**Recommended:**
- **Boot drives**: ZFS mirror or hardware RAID1
- **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
- **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas
---
## 6. Cloudflare Zero Trust Orchestration
### 6.1 cloudflared Gateway Pattern
Run **2 cloudflared LXCs** for redundancy:
- `cloudflared-1` on ML110
- `cloudflared-2` on an R630
Both run tunnels for:
- Blockscout
- FireFly
- Gitea
- Internal admin dashboards (Grafana) behind Cloudflare Access
**Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access with strict posture/MFA.
---
## 7. Complete VMID and Network Allocation Table
| VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) |
|-----------:|-------------------|-----------|--------:|------------------------|---------------------------|
| **EDGE** | ER605 WAN1 (Primary) | WAN1 | — | — | **76.53.10.34** *(router WAN IP)* |
| **EDGE** | Spectrum ISP Gateway | — | — | — | **76.53.10.33** *(ISP gateway)* |
| 1000–1499 | **Besu** – Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) |
| 1500–2499 | **Besu** – Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* |
| 2500–3499 | **Besu** – RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **76.53.10.36** *(Reserved edge VIP for emergency RPC only; primary is Cloudflare Tunnel)* |
| 3500–4299 | **Besu** – Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None |
| 4300–4999 | **Besu** – Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None |
| 5000–5099 | **Blockscout** – Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **76.53.10.35** *(Reserved edge VIP for emergency UI only; primary is Cloudflare Tunnel)* |
| 5200–5299 | **Cacti** – Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via Cloudflare Tunnel if needed)* |
| 5400–5401 | **CCIP** – Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* |
| 5402–5403 | **CCIP** – Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* |
| 5410–5425 | **CCIP** – Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** |
| 5440–5455 | **CCIP** – Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** |
| 5470–5476 | **CCIP** – RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** |
| 5480–5599 | **CCIP** – Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None |
| 6000–6099 | **Fabric** – Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via Cloudflare Tunnel if required)* |
| 6200–6299 | **FireFly** – Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary is Cloudflare Tunnel)* |
| 6400–7399 | **Indy** – Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary is Cloudflare Tunnel)* |
| 7800–8999 | **Sankofa / Phoenix / PanTel** – Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** |
| 10000–10999 | **Phoenix Sovereign Cloud Band** – SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** |
| 11000–11999 | **Phoenix Sovereign Cloud Band** – ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** |
| 12000–12999 | **Phoenix Sovereign Cloud Band** – DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** |
| 13000–13999 | **Phoenix Sovereign Cloud Band** – Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** |
---
## 8. Network Security Model
### 8.1 Access Patterns
1. **No Public Access (Tunnel/VPN Only)**
- Besu Validators (VLAN 110)
- Besu Archive/Infrastructure (VLAN 113)
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131)
2. **Cloudflare Tunnel (Primary)**
- Blockscout (VLAN 120) - Emergency VIP: 76.53.10.35
- Besu RPC (VLAN 112) - Emergency VIP: 76.53.10.36
- FireFly (VLAN 141) - Emergency VIP: 76.53.10.37
- Indy (VLAN 150) - Emergency VIP: 76.53.10.39
- Sankofa/Phoenix/PanTel (VLAN 160) - Emergency VIP: 76.53.10.38
3. **Role-Based Egress NAT (Allowlistable)**
- CCIP Commit (VLAN 132) → Block #2
- CCIP Execute (VLAN 133) → Block #3
- RMN (VLAN 134) → Block #4
- Sankofa/Phoenix/PanTel (VLAN 160) → Block #5
- Sovereign tenants (VLAN 200-203) → Block #6
4. **Cloudflare Access / VPN Only**
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131) - Optional dashboard publishing
---
## 9. Implementation Notes
### 9.1 Gateway Configuration
- All private subnets use `.1` as the gateway address
- Example: VLAN 110 uses `10.110.0.1` as gateway
- VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility)
### 9.2 Subnet Sizing
- **/24 subnets:** Standard service VLANs (256 addresses)
- **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses)
- **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each)
### 9.3 IP Address Allocation
- **Private IPs:**
- VLAN 11: 192.168.11.0/24 (legacy mgmt)
- All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
- **Public IPs:** 6× /28 blocks with role-based NAT pools
- **All public access** should route through Cloudflare Tunnel for security
### 9.4 VLAN Tagging
- All VLANs are tagged on the Proxmox bridge
- Ensure Proxmox bridge is configured for **VLAN-aware mode**
- Physical switch must support VLAN tagging (802.1Q)
---
## 10. Configuration Files
This architecture should be reflected in:
- `config/network.conf` - Network configuration
- `config/proxmox.conf` - VMID ranges
- Proxmox bridge configuration (VLAN-aware mode)
- ER605 router configuration (NAT pools, routing)
- Cloudflare Tunnel configuration
- ES216G switch configuration (VLAN trunks)
---
## 11. References
- [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration)
- [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
- [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918)
- [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/)
- [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/)
---
## Related Documentation
### Architecture Documents
- **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** ⭐⭐⭐ - Complete physical hardware inventory and specifications
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** ⭐⭐⭐ - Enterprise deployment orchestration guide
- **[VMID_ALLOCATION_FINAL.md](VMID_ALLOCATION_FINAL.md)** ⭐⭐⭐ - VMID allocation registry
- **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** ⭐⭐ - Domain structure and DNS assignments
- **[HOSTNAME_MIGRATION_GUIDE.md](HOSTNAME_MIGRATION_GUIDE.md)** ⭐ - Hostname migration procedures
### Configuration Documents
- **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](/docs/04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration
- **[../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare Zero Trust setup
- **[../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md](../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md)** - Cloudflare tunnel routing
### Deployment Documents
- **[../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md](../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md)** - Deployment orchestration
- **[../07-ccip/CCIP_DEPLOYMENT_SPEC.md](../07-ccip/CCIP_DEPLOYMENT_SPEC.md)** - CCIP deployment specification
---
**Document Status:** Complete (v2.0)
**Maintained By:** Infrastructure Team
**Review Cycle:** Quarterly
**Next Update:** After public blocks #2-6 are assigned
---
## Change Log
### Version 2.0 (2025-01-20)
- Added network topology Mermaid diagram
- Added VLAN architecture Mermaid diagram
- Added ASCII art network topology
- Enhanced public IP block matrix with status indicators
- Added breadcrumb navigation
- Added status indicators
### Version 1.0 (2024-12-15)
- Initial version
- Basic network architecture documentation
Reference in New Issue View Git Blame Copy Permalink