proxmox/docs/02-architecture/PROXMOX_CLUSTER_ARCHITECTURE.md

Proxmox Cluster Architecture

Last Updated: 2025-01-20
Document Version: 1.0
Status: Active Documentation

---

Overview

This document describes the Proxmox cluster architecture, including node configuration, storage setup, network bridges, and VM/container distribution.

---

Cluster Architecture Diagram

graph TB
    Cluster[Proxmox Cluster<br/>Name: h]
    
    ML110[ML110 Management Node<br/>192.168.11.10<br/>6 cores, 125GB RAM]
    R6301[R630-01<br/>192.168.11.11<br/>32 cores, 503GB RAM]
    R6302[R630-02<br/>192.168.11.12<br/>32 cores, 503GB RAM]
    R6303[R630-03<br/>192.168.11.13<br/>32 cores, 512GB RAM]
    R6304[R630-04<br/>192.168.11.14<br/>32 cores, 512GB RAM]
    
    Cluster --> ML110
    Cluster --> R6301
    Cluster --> R6302
    Cluster --> R6303
    Cluster --> R6304
    
    ML110 --> Storage1[local: 94GB<br/>local-lvm: 813GB]
    R6301 --> Storage2[local: 536GB<br/>local-lvm: Available]
    R6302 --> Storage3[local: Available<br/>local-lvm: Available]
    R6303 --> Storage4[Storage: Available]
    R6304 --> Storage5[Storage: Available]
    
    ML110 --> Bridge1[vmbr0<br/>VLAN-aware]
    R6301 --> Bridge2[vmbr0<br/>VLAN-aware]
    R6302 --> Bridge3[vmbr0<br/>VLAN-aware]
    R6303 --> Bridge4[vmbr0<br/>VLAN-aware]
    R6304 --> Bridge5[vmbr0<br/>VLAN-aware]

---

Cluster Nodes

Node Summary

Hostname	IP Address	CPU	RAM	Storage	VMs/Containers	Status
ml110	192.168.11.10	6 cores @ 1.60GHz	125GB	local (94GB), local-lvm (813GB)	34	✅ Active
r630-01	192.168.11.11	32 cores @ 2.40GHz	503GB	local (536GB), local-lvm (available)	0	✅ Active
r630-02	192.168.11.12	32 cores @ 2.40GHz	503GB	local (available), local-lvm (available)	0	✅ Active
r630-03	192.168.11.13	32 cores	512GB	Available	0	✅ Active
r630-04	192.168.11.14	32 cores	512GB	Available	0	✅ Active

---

Storage Configuration

Storage Types

local (Directory Storage):

• Type: Directory-based storage
• Used for: ISO images, container templates, backups
• Location: /var/lib/vz

local-lvm (LVM Thin Storage):

• Type: LVM thin provisioning
• Used for: VM/container disk images
• Benefits: Thin provisioning, snapshots, efficient space usage

Storage by Node

ml110:

• local: 94GB total, 7.4GB used (7.87%)
• local-lvm: 813GB total, 214GB used (26.29%)
• Status: ✅ Active and operational

r630-01:

• local: 536GB total, 0% used
• local-lvm: Available (needs activation)
• Status: ⏳ Storage available, ready for use

r630-02:

• local: Available
• local-lvm: Available (needs activation)
• Status: ⏳ Storage available, ready for use

r630-03/r630-04:

• Storage: Available
• Status: ⏳ Ready for configuration

---

Network Configuration

Network Bridge (vmbr0)

All nodes use VLAN-aware bridge:

# Bridge configuration (all nodes)
auto vmbr0
iface vmbr0 inet static
    address 192.168.11.<HOST_IP>/24
    gateway 192.168.11.1
    bridge-ports <PHYSICAL_INTERFACE>
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 11 110 111 112 120 121 130 132 133 134 140 141 150 160 200 201 202 203

Bridge Features:

• VLAN-aware: Supports multiple VLANs on single bridge
• Native VLAN: 11 (MGMT-LAN)
• Tagged VLANs: All service VLANs (110-203)
• 802.1Q Trunking: Enabled for VLAN support

---

VM/Container Distribution

Current Distribution

ml110 (192.168.11.10):

• Total: 34 containers/VMs
• Services: All current services running here
• Breakdown:
  • Besu validators: 5 (VMIDs 1000-1004)
  • Besu sentries: 4 (VMIDs 1500-1503)
  • Besu RPC: 3+ (VMIDs 2500-2502+)
  • Blockscout: 1 (VMID 5000)
  • DBIS services: Multiple
  • Other services: Various

r630-01, r630-02, r630-03, r630-04:

• Total: 0 containers/VMs
• Status: Ready for VM migration/deployment

---

High Availability

Current Setup

• Cluster Name: "h"
• HA Mode: Active/Standby (manual)
• Quorum: 3+ nodes required for quorum
• Storage: Local storage (not shared)

HA Considerations

Current Limitations:

• No shared storage (each node has local storage)
• Manual VM migration required
• No automatic failover

Future Enhancements:

• Consider shared storage (NFS, Ceph, etc.) for true HA
• Implement automatic VM migration
• Configure HA groups for critical services

---

Resource Allocation

CPU Resources

Node	CPU Cores	CPU Usage	Available
ml110	6 @ 1.60GHz	High	Limited
r630-01	32 @ 2.40GHz	Low	Excellent
r630-02	32 @ 2.40GHz	Low	Excellent
r630-03	32 cores	Low	Excellent
r630-04	32 cores	Low	Excellent

Memory Resources

Node	Total RAM	Used	Available	Usage %
ml110	125GB	94GB	31GB	75% ⚠️
r630-01	503GB	~5GB	~498GB	1% ✅
r630-02	503GB	~5GB	~498GB	1% ✅
r630-03	512GB	Low	High	Low ✅
r630-04	512GB	Low	High	Low ✅

---

Storage Recommendations

For R630 Nodes

Boot Drives (2×600GB):

• Recommended: ZFS mirror or hardware RAID1
• Purpose: Proxmox OS and boot files
• Benefits: Redundancy, data integrity

Data SSDs (6×250GB):

• Option 1: ZFS striped mirrors (3 pairs)

  • Capacity: ~750GB usable
  • Performance: High
  • Redundancy: Good

• Option 2: ZFS RAIDZ1 (5 drives + 1 parity)

  • Capacity: ~1.25TB usable
  • Performance: Good
  • Redundancy: Single drive failure tolerance

• Option 3: ZFS RAIDZ2 (4 drives + 2 parity)

  • Capacity: ~1TB usable
  • Performance: Good
  • Redundancy: Dual drive failure tolerance

---

Network Recommendations

VLAN Configuration

All Proxmox hosts should:

• Use VLAN-aware bridge (vmbr0)
• Support all 19 VLANs
• Maintain native VLAN 11 for management
• Enable 802.1Q trunking on physical interfaces

Network Performance

• Link Speed: Ensure 1Gbps or higher for trunk ports
• Jumbo Frames: Consider enabling if supported
• Bonding: Consider link aggregation for redundancy

---

Related Documentation

• NETWORK_ARCHITECTURE.md ⭐⭐⭐ - Network architecture with VLAN plan
• PHYSICAL_HARDWARE_INVENTORY.md ⭐⭐⭐ - Physical hardware inventory
• PROXMOX_COMPREHENSIVE_REVIEW.md ⭐⭐ - Comprehensive Proxmox review
• ORCHESTRATION_DEPLOYMENT_GUIDE.md ⭐⭐⭐ - Deployment orchestration

---

Last Updated: 2025-01-20
Document Version: 1.0
Review Cycle: Quarterly