# Proxmox Cluster Architecture

**Last Updated:** 2025-01-20  
**Document Version:** 1.0  
**Status:** Active Documentation

---

## Overview

This document describes the Proxmox cluster architecture, including node configuration, storage setup, network bridges, and VM/container distribution.

---

## Cluster Architecture Diagram

```mermaid
graph TB
    Cluster[Proxmox Cluster<br/>Name: h]
    
    ML110[ML110 Management Node<br/>192.168.11.10<br/>6 cores, 125GB RAM]
    R6301[R630-01<br/>192.168.11.11<br/>32 cores, 503GB RAM]
    R6302[R630-02<br/>192.168.11.12<br/>32 cores, 503GB RAM]
    R6303[R630-03<br/>192.168.11.13<br/>32 cores, 512GB RAM]
    R6304[R630-04<br/>192.168.11.14<br/>32 cores, 512GB RAM]
    
    Cluster --> ML110
    Cluster --> R6301
    Cluster --> R6302
    Cluster --> R6303
    Cluster --> R6304
    
    ML110 --> Storage1[local: 94GB<br/>local-lvm: 813GB]
    R6301 --> Storage2[local: 536GB<br/>local-lvm: Available]
    R6302 --> Storage3[local: Available<br/>local-lvm: Available]
    R6303 --> Storage4[Storage: Available]
    R6304 --> Storage5[Storage: Available]
    
    ML110 --> Bridge1[vmbr0<br/>VLAN-aware]
    R6301 --> Bridge2[vmbr0<br/>VLAN-aware]
    R6302 --> Bridge3[vmbr0<br/>VLAN-aware]
    R6303 --> Bridge4[vmbr0<br/>VLAN-aware]
    R6304 --> Bridge5[vmbr0<br/>VLAN-aware]
```

---

## Cluster Nodes

### Node Summary

| Hostname | IP Address | CPU | RAM | Storage | VMs/Containers | Status |
|----------|------------|-----|-----|---------|----------------|--------|
| ml110 | 192.168.11.10 | 6 cores @ 1.60GHz | 125GB | local (94GB), local-lvm (813GB) | 34 | ✅ Active |
| r630-01 | 192.168.11.11 | 32 cores @ 2.40GHz | 503GB | local (536GB), local-lvm (available) | 0 | ✅ Active |
| r630-02 | 192.168.11.12 | 32 cores @ 2.40GHz | 503GB | local (available), local-lvm (available) | 0 | ✅ Active |
| r630-03 | 192.168.11.13 | 32 cores | 512GB | Available | 0 | ✅ Active |
| r630-04 | 192.168.11.14 | 32 cores | 512GB | Available | 0 | ✅ Active |

---

## Storage Configuration

### Storage Types

**local (Directory Storage):**
- Type: Directory-based storage
- Used for: ISO images, container templates, backups
- Location: `/var/lib/vz`

**local-lvm (LVM Thin Storage):**
- Type: LVM thin provisioning
- Used for: VM/container disk images
- Benefits: Thin provisioning, snapshots, efficient space usage

### Storage by Node

**ml110:**
- `local`: 94GB total, 7.4GB used (7.87%)
- `local-lvm`: 813GB total, 214GB used (26.29%)
- Status: ✅ Active and operational

**r630-01:**
- `local`: 536GB total, 0% used
- `local-lvm`: Available (needs activation)
- Status: ⏳ Storage available, ready for use

**r630-02:**
- `local`: Available
- `local-lvm`: Available (needs activation)
- Status: ⏳ Storage available, ready for use

**r630-03/r630-04:**
- Storage: Available
- Status: ⏳ Ready for configuration

---

## Network Configuration

### Network Bridge (vmbr0)

**All nodes use VLAN-aware bridge:**

```bash
# Bridge configuration (all nodes)
auto vmbr0
iface vmbr0 inet static
    address 192.168.11.<HOST_IP>/24
    gateway 192.168.11.1
    bridge-ports <PHYSICAL_INTERFACE>
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 11 110 111 112 120 121 130 132 133 134 140 141 150 160 200 201 202 203
```

**Bridge Features:**
- **VLAN-aware:** Supports multiple VLANs on single bridge
- **Native VLAN:** 11 (MGMT-LAN)
- **Tagged VLANs:** All service VLANs (110-203)
- **802.1Q Trunking:** Enabled for VLAN support

---

## VM/Container Distribution

### Current Distribution

**ml110 (192.168.11.10):**
- **Total:** 34 containers/VMs
- **Services:** All current services running here
- **Breakdown:**
  - Besu validators: 5 (VMIDs 1000-1004)
  - Besu sentries: 4 (VMIDs 1500-1503)
  - Besu RPC: 3+ (VMIDs 2500-2502+)
  - Blockscout: 1 (VMID 5000)
  - DBIS services: Multiple
  - Other services: Various

**r630-01, r630-02, r630-03, r630-04:**
- **Total:** 0 containers/VMs
- **Status:** Ready for VM migration/deployment

---

## High Availability

### Current Setup

- **Cluster Name:** "h"
- **HA Mode:** Active/Standby (manual)
- **Quorum:** 3+ nodes required for quorum
- **Storage:** Local storage (not shared)

### HA Considerations

**Current Limitations:**
- No shared storage (each node has local storage)
- Manual VM migration required
- No automatic failover

**Future Enhancements:**
- Consider shared storage (NFS, Ceph, etc.) for true HA
- Implement automatic VM migration
- Configure HA groups for critical services

---

## Resource Allocation

### CPU Resources

| Node | CPU Cores | CPU Usage | Available |
|------|-----------|-----------|-----------|
| ml110 | 6 @ 1.60GHz | High | Limited |
| r630-01 | 32 @ 2.40GHz | Low | Excellent |
| r630-02 | 32 @ 2.40GHz | Low | Excellent |
| r630-03 | 32 cores | Low | Excellent |
| r630-04 | 32 cores | Low | Excellent |

### Memory Resources

| Node | Total RAM | Used | Available | Usage % |
|------|-----------|------|-----------|---------|
| ml110 | 125GB | 94GB | 31GB | 75% ⚠️ |
| r630-01 | 503GB | ~5GB | ~498GB | 1% ✅ |
| r630-02 | 503GB | ~5GB | ~498GB | 1% ✅ |
| r630-03 | 512GB | Low | High | Low ✅ |
| r630-04 | 512GB | Low | High | Low ✅ |

---

## Storage Recommendations

### For R630 Nodes

**Boot Drives (2×600GB):**
- **Recommended:** ZFS mirror or hardware RAID1
- **Purpose:** Proxmox OS and boot files
- **Benefits:** Redundancy, data integrity

**Data SSDs (6×250GB):**
- **Option 1:** ZFS striped mirrors (3 pairs)
  - Capacity: ~750GB usable
  - Performance: High
  - Redundancy: Good
  
- **Option 2:** ZFS RAIDZ1 (5 drives + 1 parity)
  - Capacity: ~1.25TB usable
  - Performance: Good
  - Redundancy: Single drive failure tolerance
  
- **Option 3:** ZFS RAIDZ2 (4 drives + 2 parity)
  - Capacity: ~1TB usable
  - Performance: Good
  - Redundancy: Dual drive failure tolerance

---

## Network Recommendations

### VLAN Configuration

**All Proxmox hosts should:**
- Use VLAN-aware bridge (vmbr0)
- Support all 19 VLANs
- Maintain native VLAN 11 for management
- Enable 802.1Q trunking on physical interfaces

### Network Performance

- **Link Speed:** Ensure 1Gbps or higher for trunk ports
- **Jumbo Frames:** Consider enabling if supported
- **Bonding:** Consider link aggregation for redundancy

---

## Related Documentation

- **[NETWORK_ARCHITECTURE.md](NETWORK_ARCHITECTURE.md)** ⭐⭐⭐ - Network architecture with VLAN plan
- **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** ⭐⭐⭐ - Physical hardware inventory
- **[PROXMOX_COMPREHENSIVE_REVIEW.md](PROXMOX_COMPREHENSIVE_REVIEW.md)** ⭐⭐ - Comprehensive Proxmox review
- **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** ⭐⭐⭐ - Deployment orchestration

---

**Last Updated:** 2025-01-20  
**Document Version:** 1.0  
**Review Cycle:** Quarterly
