d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
cb47cce074da46f3ce5910b752d435c4a3ee87d0
proxmox/docs/02-architecture/NETWORK_ARCHITECTURE.md

367 lines
16 KiB
Markdown
Raw Normal View History

Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
# Network Architecture - Enterprise Orchestration Plan
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
**Navigation:** [Home](../README.md) > [Architecture](README.md) > Network Architecture
Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
**Last Updated:** 2025-01-20
**Document Version:** 2.0
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
**Status:** 🟢 Active Documentation
Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
**Project:** Sankofa / Phoenix / PanTel · ChainID 138 · Proxmox + Cloudflare Zero Trust + Dual ISP + 6×/28
---
## Overview
This document defines the complete enterprise-grade network architecture for the Sankofa/Phoenix/PanTel Proxmox deployment, including:
- **Hardware role assignments** (2× ER605, 3× ES216G, 1× ML110, 4× R630)
- **6× /28 public IP blocks** with role-based NAT pools
- **VLAN orchestration** with private subnet allocations
- **Egress segmentation** by role and security plane
- **Cloudflare Zero Trust** integration patterns
---
## Core Principles
1. **No public IPs on Proxmox hosts or LXCs/VMs** (default)
2. **Inbound access = Cloudflare Zero Trust + cloudflared** (primary)
3. **Public IPs used for:**
- ER605 WAN addressing
- **Egress NAT pools** (role-based allowlisting)
- **Break-glass** emergency endpoints only
4. **Segmentation by VLAN/VRF**: consensus vs services vs sovereign tenants vs ops
5. **Deterministic VMID registry** + IPAM that matches
---
## 1. Physical Topology & Hardware Roles
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
> **Reference:** For complete physical hardware inventory including IP addresses, credentials, and detailed specifications, see **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)**.
Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
### 1.1 Hardware Role Assignment
#### Edge / Routing
- **ER605-A (Primary Edge Router)**
- WAN1: Spectrum primary with Block #1
- WAN2: ISP #2 (failover/alternate policy)
- Role: Active edge router, NAT pools, routing
- **ER605-B (Standby Edge Router / Alternate WAN policy)**
- Role: Standby router OR dedicated to WAN2 policies/testing
- Note: ER605 does not support full stateful HA. This is **active/standby operational redundancy**, not automatic session-preserving HA.
#### Switching Fabric
- **ES216G-1**: Core / uplinks / trunks
- **ES216G-2**: Compute rack aggregation
- **ES216G-3**: Mgmt + out-of-band / staging
#### Compute
- **ML110 Gen9**: "Bootstrap & Management" node
- IP: 192.168.11.10
- Role: Proxmox mgmt services, Omada controller, Git, monitoring seed
- **4× Dell R630**: Proxmox compute cluster nodes
- Resources: 512GB RAM each, 2×600GB boot, 6×250GB SSD
- Role: Production workloads, CCIP fleet, sovereign tenants, services
---
## 2. ISP & Public IP Plan (6× /28)
### Public Block #1 (Known - Spectrum)
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
| Property | Value | Status |
|----------|-------|--------|
| **Network** | `76.53.10.32/28` | ✅ Configured |
| **Gateway** | `76.53.10.33` | ✅ Active |
| **Usable Range** | `76.53.10.33–76.53.10.46` | ✅ In Use |
| **Broadcast** | `76.53.10.47` | - |
| **ER605 WAN1 IP** | `76.53.10.34` (router interface) | ✅ Active |
| **Available IPs** | 13 (76.53.10.35-46, excluding .34) | ✅ Available |
Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
### Public Blocks #2–#6 (Placeholders - To Be Configured)
| Block | Network | Gateway | Usable Range | Broadcast | Designated Use |
|-------|--------|---------|--------------|-----------|----------------|
| **#2** | `<PUBLIC_BLOCK_2>/28` | `<GW2>` | `<USABLE2>` | `<BCAST2>` | CCIP Commit egress NAT pool |
| **#3** | `<PUBLIC_BLOCK_3>/28` | `<GW3>` | `<USABLE3>` | `<BCAST3>` | CCIP Execute egress NAT pool |
| **#4** | `<PUBLIC_BLOCK_4>/28` | `<GW4>` | `<USABLE4>` | `<BCAST4>` | RMN egress NAT pool |
| **#5** | `<PUBLIC_BLOCK_5>/28` | `<GW5>` | `<USABLE5>` | `<BCAST5>` | Sankofa/Phoenix/PanTel service egress |
| **#6** | `<PUBLIC_BLOCK_6>/28` | `<GW6>` | `<USABLE6>` | `<BCAST6>` | Sovereign Cloud Band tenant egress |
### 2.1 Public IP Usage Policy (Role-based)
| Public /28 Block | Designated Use | Why |
|------------------|----------------|-----|
| **#1** (76.53.10.32/28) | Router WAN + break-glass VIPs | Primary connectivity + emergency |
| **#2** | CCIP Commit egress NAT pool | Allowlistable egress for source RPCs |
| **#3** | CCIP Execute egress NAT pool | Allowlistable egress for destination RPCs |
| **#4** | RMN egress NAT pool | Independent security-plane egress |
| **#5** | Sankofa/Phoenix/PanTel service egress | Service-plane separation |
| **#6** | Sovereign Cloud Band tenant egress | Per-sovereign policy control |
---
## 3. Layer-2 & VLAN Orchestration Plan
### 3.1 VLAN Set (Authoritative)
> **Migration Note:** Currently on flat LAN 192.168.11.0/24. This plan migrates to VLANs while keeping compatibility.
| VLAN ID | VLAN Name | Purpose | Subnet | Gateway |
|--------:|-----------|---------|--------|---------|
| **11** | MGMT-LAN | Proxmox mgmt, switches mgmt, admin endpoints | 192.168.11.0/24 | 192.168.11.1 |
| 110 | BESU-VAL | Validator-only network (no member access) | 10.110.0.0/24 | 10.110.0.1 |
| 111 | BESU-SEN | Sentry mesh | 10.111.0.0/24 | 10.111.0.1 |
| 112 | BESU-RPC | RPC / gateway tier | 10.112.0.0/24 | 10.112.0.1 |
| 120 | BLOCKSCOUT | Explorer + DB | 10.120.0.0/24 | 10.120.0.1 |
| 121 | CACTI | Interop middleware | 10.121.0.0/24 | 10.121.0.1 |
| 130 | CCIP-OPS | Ops/admin | 10.130.0.0/24 | 10.130.0.1 |
| 132 | CCIP-COMMIT | Commit-role DON | 10.132.0.0/24 | 10.132.0.1 |
| 133 | CCIP-EXEC | Execute-role DON | 10.133.0.0/24 | 10.133.0.1 |
| 134 | CCIP-RMN | Risk management network | 10.134.0.0/24 | 10.134.0.1 |
| 140 | FABRIC | Fabric | 10.140.0.0/24 | 10.140.0.1 |
| 141 | FIREFLY | FireFly | 10.141.0.0/24 | 10.141.0.1 |
| 150 | INDY | Identity | 10.150.0.0/24 | 10.150.0.1 |
| 160 | SANKOFA-SVC | Sankofa/Phoenix/PanTel service layer | 10.160.0.0/22 | 10.160.0.1 |
| 200 | PHX-SOV-SMOM | Sovereign tenant | 10.200.0.0/20 | 10.200.0.1 |
| 201 | PHX-SOV-ICCC | Sovereign tenant | 10.201.0.0/20 | 10.201.0.1 |
| 202 | PHX-SOV-DBIS | Sovereign tenant | 10.202.0.0/20 | 10.202.0.1 |
| 203 | PHX-SOV-AR | Absolute Realms tenant | 10.203.0.0/20 | 10.203.0.1 |
### 3.2 Switching Configuration (ES216G)
- **ES216G-1**: **Core** (all VLAN trunks to ES216G-2/3 + ER605-A)
- **ES216G-2**: **Compute** (trunks to R630s + ML110)
- **ES216G-3**: **Mgmt/OOB** (mgmt access ports, staging, out-of-band)
**All Proxmox uplinks should be 802.1Q trunk ports.**
---
## 4. Routing, NAT, and Egress Segmentation (ER605)
### 4.1 Dual Router Roles
- **ER605-A**: Active edge router (WAN1 = Spectrum primary with Block #1)
- **ER605-B**: Standby router OR dedicated to WAN2 policies/testing (no inbound services)
### 4.2 NAT Policies (Critical)
#### Inbound NAT
- **Default: none**
- Break-glass only (optional):
- Jumpbox/SSH (single port, IP allowlist, Cloudflare Access preferred)
- Proxmox admin should remain **LAN-only**
#### Outbound NAT (Role-based Pools Using /28 Blocks)
| Private Subnet | Role | Egress NAT Pool | Public Block |
|----------------|------|-----------------|--------------|
| 10.132.0.0/24 | CCIP Commit | **Block #2** `<PUBLIC_BLOCK_2>/28` | #2 |
| 10.133.0.0/24 | CCIP Execute | **Block #3** `<PUBLIC_BLOCK_3>/28` | #3 |
| 10.134.0.0/24 | RMN | **Block #4** `<PUBLIC_BLOCK_4>/28` | #4 |
| 10.160.0.0/22 | Sankofa/Phoenix/PanTel | **Block #5** `<PUBLIC_BLOCK_5>/28` | #5 |
| 10.200.0.0/20–10.203.0.0/20 | Sovereign tenants | **Block #6** `<PUBLIC_BLOCK_6>/28` | #6 |
| 192.168.11.0/24 | Mgmt | Block #1 (or none; tightly restricted) | #1 |
This yields **provable separation**, allowlisting, and incident scoping.
---
## 5. Proxmox Cluster Orchestration
### 5.1 Node Layout
- **ml110 (192.168.11.10)**: mgmt + seed services + initial automation runner
- **r630-01..04**: production compute
### 5.2 Proxmox Networking (per host)
- **`vmbr0`**: VLAN-aware bridge
- Native VLAN: 11 (MGMT)
- Tagged VLANs: 110,111,112,120,121,130,132,133,134,140,141,150,160,200–203
- **Proxmox host IP** remains on **VLAN 11** only.
### 5.3 Storage Orchestration (R630)
**Hardware:**
- 2×600GB boot (mirror recommended)
- 6×250GB SSD
**Recommended:**
- **Boot drives**: ZFS mirror or hardware RAID1
- **Data SSDs**: ZFS pool (striped mirrors if you can pair, or RAIDZ1/2 depending on risk tolerance)
- **High-write workloads** (logs/metrics/indexers) on dedicated dataset with quotas
---
## 6. Cloudflare Zero Trust Orchestration
### 6.1 cloudflared Gateway Pattern
Run **2 cloudflared LXCs** for redundancy:
- `cloudflared-1` on ML110
- `cloudflared-2` on an R630
Both run tunnels for:
- Blockscout
- FireFly
- Gitea
- Internal admin dashboards (Grafana) behind Cloudflare Access
**Keep Proxmox UI LAN-only**; if needed, publish via Cloudflare Access with strict posture/MFA.
---
## 7. Complete VMID and Network Allocation Table
| VMID Range | Domain / Subdomain | VLAN Name | VLAN ID | Private Subnet (GW .1) | Public IP (Edge VIP / NAT) |
|-----------:|-------------------|-----------|--------:|------------------------|---------------------------|
| **EDGE** | ER605 WAN1 (Primary) | WAN1 | — | — | **76.53.10.34** *(router WAN IP)* |
| **EDGE** | Spectrum ISP Gateway | — | — | — | **76.53.10.33** *(ISP gateway)* |
| 1000–1499 | **Besu** – Validators | BESU-VAL | 110 | 10.110.0.0/24 | **None** (no inbound; tunnel/VPN only) |
| 1500–2499 | **Besu** – Sentries | BESU-SEN | 111 | 10.111.0.0/24 | **None** *(optional later via NAT pool)* |
| 2500–3499 | **Besu** – RPC / Gateways | BESU-RPC | 112 | 10.112.0.0/24 | **76.53.10.36** *(Reserved edge VIP for emergency RPC only; primary is Cloudflare Tunnel)* |
| 3500–4299 | **Besu** – Archive/Snapshots/Mirrors/Telemetry | BESU-INFRA | 113 | 10.113.0.0/24 | None |
| 4300–4999 | **Besu** – Reserved expansion | BESU-RES | 114 | 10.114.0.0/24 | None |
| 5000–5099 | **Blockscout** – Explorer/Indexing | BLOCKSCOUT | 120 | 10.120.0.0/24 | **76.53.10.35** *(Reserved edge VIP for emergency UI only; primary is Cloudflare Tunnel)* |
| 5200–5299 | **Cacti** – Interop middleware | CACTI | 121 | 10.121.0.0/24 | None *(publish via Cloudflare Tunnel if needed)* |
| 5400–5401 | **CCIP** – Ops/Admin | CCIP-OPS | 130 | 10.130.0.0/24 | None *(Cloudflare Access / VPN only)* |
| 5402–5403 | **CCIP** – Monitoring/Telemetry | CCIP-MON | 131 | 10.131.0.0/24 | None *(optionally publish dashboards via Cloudflare Access)* |
| 5410–5425 | **CCIP** – Commit-role oracle nodes (16) | CCIP-COMMIT | 132 | 10.132.0.0/24 | **Egress NAT: Block #2** |
| 5440–5455 | **CCIP** – Execute-role oracle nodes (16) | CCIP-EXEC | 133 | 10.133.0.0/24 | **Egress NAT: Block #3** |
| 5470–5476 | **CCIP** – RMN nodes (7) | CCIP-RMN | 134 | 10.134.0.0/24 | **Egress NAT: Block #4** |
| 5480–5599 | **CCIP** – Reserved expansion | CCIP-RES | 135 | 10.135.0.0/24 | None |
| 6000–6099 | **Fabric** – Enterprise contracts | FABRIC | 140 | 10.140.0.0/24 | None *(publish via Cloudflare Tunnel if required)* |
| 6200–6299 | **FireFly** – Workflow/orchestration | FIREFLY | 141 | 10.141.0.0/24 | **76.53.10.37** *(Reserved edge VIP if ever needed; primary is Cloudflare Tunnel)* |
| 6400–7399 | **Indy** – Identity layer | INDY | 150 | 10.150.0.0/24 | **76.53.10.39** *(Reserved edge VIP for DID endpoints if required; primary is Cloudflare Tunnel)* |
| 7800–8999 | **Sankofa / Phoenix / PanTel** – Service + Cloud + Telecom | SANKOFA-SVC | 160 | 10.160.0.0/22 | **Egress NAT: Block #5** |
| 10000–10999 | **Phoenix Sovereign Cloud Band** – SMOM tenant | PHX-SOV-SMOM | 200 | 10.200.0.0/20 | **Egress NAT: Block #6** |
| 11000–11999 | **Phoenix Sovereign Cloud Band** – ICCC tenant | PHX-SOV-ICCC | 201 | 10.201.0.0/20 | **Egress NAT: Block #6** |
| 12000–12999 | **Phoenix Sovereign Cloud Band** – DBIS tenant | PHX-SOV-DBIS | 202 | 10.202.0.0/20 | **Egress NAT: Block #6** |
| 13000–13999 | **Phoenix Sovereign Cloud Band** – Absolute Realms tenant | PHX-SOV-AR | 203 | 10.203.0.0/20 | **Egress NAT: Block #6** |
---
## 8. Network Security Model
### 8.1 Access Patterns
1. **No Public Access (Tunnel/VPN Only)**
- Besu Validators (VLAN 110)
- Besu Archive/Infrastructure (VLAN 113)
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131)
2. **Cloudflare Tunnel (Primary)**
- Blockscout (VLAN 120) - Emergency VIP: 76.53.10.35
- Besu RPC (VLAN 112) - Emergency VIP: 76.53.10.36
- FireFly (VLAN 141) - Emergency VIP: 76.53.10.37
- Indy (VLAN 150) - Emergency VIP: 76.53.10.39
- Sankofa/Phoenix/PanTel (VLAN 160) - Emergency VIP: 76.53.10.38
3. **Role-Based Egress NAT (Allowlistable)**
- CCIP Commit (VLAN 132) → Block #2
- CCIP Execute (VLAN 133) → Block #3
- RMN (VLAN 134) → Block #4
- Sankofa/Phoenix/PanTel (VLAN 160) → Block #5
- Sovereign tenants (VLAN 200-203) → Block #6
4. **Cloudflare Access / VPN Only**
- CCIP Ops/Admin (VLAN 130)
- CCIP Monitoring (VLAN 131) - Optional dashboard publishing
---
## 9. Implementation Notes
### 9.1 Gateway Configuration
- All private subnets use `.1` as the gateway address
- Example: VLAN 110 uses `10.110.0.1` as gateway
- VLAN 11 (MGMT) uses `192.168.11.1` (legacy compatibility)
### 9.2 Subnet Sizing
- **/24 subnets:** Standard service VLANs (256 addresses)
- **/22 subnet:** Sankofa/Phoenix/PanTel (1024 addresses)
- **/20 subnets:** Phoenix Sovereign Cloud Bands (4096 addresses each)
### 9.3 IP Address Allocation
- **Private IPs:**
- VLAN 11: 192.168.11.0/24 (legacy mgmt)
- All other VLANs: 10.x.0.0/24 or /20 or /22 (VLAN ID maps to second octet)
- **Public IPs:** 6× /28 blocks with role-based NAT pools
- **All public access** should route through Cloudflare Tunnel for security
### 9.4 VLAN Tagging
- All VLANs are tagged on the Proxmox bridge
- Ensure Proxmox bridge is configured for **VLAN-aware mode**
- Physical switch must support VLAN tagging (802.1Q)
---
## 10. Configuration Files
This architecture should be reflected in:
- `config/network.conf` - Network configuration
- `config/proxmox.conf` - VMID ranges
- Proxmox bridge configuration (VLAN-aware mode)
- ER605 router configuration (NAT pools, routing)
- Cloudflare Tunnel configuration
- ES216G switch configuration (VLAN trunks)
---
## 11. References
- [Proxmox VLAN Configuration](https://pve.proxmox.com/wiki/Network_Configuration)
- [Cloudflare Tunnel Documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
- [RFC 1918 - Private Address Space](https://tools.ietf.org/html/rfc1918)
- [ER605 User Guide](https://www.tp-link.com/us/support/download/er605/)
- [ES216G Configuration Guide](https://www.tp-link.com/us/support/download/es216g/)
---
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
## Related Documentation
### Architecture Documents
- **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** ⭐⭐⭐ - Complete physical hardware inventory and specifications
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** ⭐⭐⭐ - Enterprise deployment orchestration guide
- **[VMID_ALLOCATION_FINAL.md](VMID_ALLOCATION_FINAL.md)** ⭐⭐⭐ - VMID allocation registry
- **[DOMAIN_STRUCTURE.md](DOMAIN_STRUCTURE.md)** ⭐⭐ - Domain structure and DNS assignments
- **[HOSTNAME_MIGRATION_GUIDE.md](HOSTNAME_MIGRATION_GUIDE.md)** ⭐ - Hostname migration procedures
### Configuration Documents
- **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](../04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration
- **[../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/cloudflare/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare Zero Trust setup
- **[../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md](../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md)** - Cloudflare tunnel routing
### Deployment Documents
- **[../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md](../03-deployment/ORCHESTRATION_DEPLOYMENT_GUIDE.md)** - Deployment orchestration
- **[../07-ccip/CCIP_DEPLOYMENT_SPEC.md](../07-ccip/CCIP_DEPLOYMENT_SPEC.md)** - CCIP deployment specification
---
Refactor code for improved readability and performance
2025-12-21 22:32:09 -08:00
**Document Status:** Complete (v2.0)
**Maintained By:** Infrastructure Team
**Review Cycle:** Quarterly
**Next Update:** After public blocks #2-6 are assigned
Complete markdown files cleanup and organization - Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
2026-01-06 01:46:25 -08:00
---
## Change Log
### Version 2.0 (2025-01-20)
- Added network topology Mermaid diagram
- Added VLAN architecture Mermaid diagram
- Added ASCII art network topology
- Enhanced public IP block matrix with status indicators
- Added breadcrumb navigation
- Added status indicators
### Version 1.0 (2024-12-15)
- Initial version
- Basic network architecture documentation
Reference in New Issue Copy Permalink