proxmox/reports/storage/STORAGE_REVIEW_SUMMARY.md

# Proxmox Storage Review - Complete Summary and Recommendations

**Date:** January 6, 2026  
**Review Scope:** All Proxmox nodes and storage configurations  
**Status:** ✅ Complete

---

## Executive Summary

This document provides a comprehensive review of all storage across all Proxmox nodes with detailed recommendations for optimization, capacity planning, and performance improvements.

### Key Findings

- **Total Containers:** 51 containers across 3 accessible nodes
- **Critical Issues:** 1 storage pool at 97.78% capacity (r630-02 thin1-r630-02)
- **Storage Distribution:** Uneven - ml110 has 37 containers, others underutilized
- **Available Storage:** ~1.2TB total available across all nodes
- **Unreachable Nodes:** r630-03 and r630-04 (require investigation)

---

## Current Storage Status by Node

### ml110 (192.168.11.10) - Management Node

**Status:** ✅ Operational  
**Containers:** 37  
**CPU:** 6 cores @ 1.60GHz (older, slower)  
**Memory:** 125GB (55GB used, 69GB available - 44% usage)

#### Storage Details

| Storage Name | Type | Status | Total | Used | Available | Usage % |
|--------------|------|--------|-------|------|-----------|---------|
| local | dir | ✅ Active | 94GB | 7.5GB | 85.5GB | 8.02% |
| local-lvm | lvmthin | ✅ Active | 813GB | 227GB | 586GB | 27.92% |
| thin1-thin6 | lvmthin | ❌ Disabled | - | - | - | N/A |

**Volume Group:** `pve` - 930.51GB total, **16GB free** ⚠️

**Thin Pool:** `data` - 794.30GB (27.92% used, 1.13% metadata)

**Physical Disks:**
- sda: 931.5GB
- sdb: 931.5GB

#### Issues Identified

1. ⚠️ **Low Volume Group Free Space:** Only 16GB free in VG (1.7%)
   - **Impact:** Cannot create new VMs/containers without expansion
   - **Recommendation:** Expand VG or migrate VMs to other nodes

2. ⚠️ **Multiple Disabled Storage Pools:** thin1-thin6 are disabled
   - **Impact:** Storage pools configured but not usable
   - **Recommendation:** Clean up unused storage definitions or enable if needed

3. ⚠️ **Overloaded Node:** 37 containers on slower CPU
   - **Impact:** Performance degradation
   - **Recommendation:** Migrate containers to r630-01/r630-02

#### Recommendations

**CRITICAL:**
1. **Expand Volume Group** - Add physical volumes or migrate VMs
2. **Monitor Storage Closely** - Only 16GB free space remaining

**HIGH PRIORITY:**
1. **Migrate Containers** - Move 15-20 containers to r630-01/r630-02
2. **Clean Up Storage Config** - Remove or enable disabled storage pools

**RECOMMENDED:**
1. **Storage Monitoring** - Set alerts at 80% usage
2. **Backup Strategy** - Implement regular backups before migration

---

### r630-01 (192.168.11.11) - Production Node

**Status:** ✅ Operational  
**Containers:** 3  
**CPU:** 32 cores @ 2.40GHz (excellent)  
**Memory:** 503GB (7.5GB used, 496GB available - 1.5% usage)

#### Storage Details

| Storage Name | Type | Status | Total | Used | Available | Usage % |
|--------------|------|--------|-------|------|-----------|---------|
| local | dir | ✅ Active | 536GB | 0.1GB | 536GB | 0.02% |
| local-lvm | lvmthin | ✅ Active | 200GB | 5.8GB | 194GB | 2.92% |
| thin1 | lvmthin | ✅ Active | 208GB | 0GB | 208GB | 0.00% |
| thin2-thin6 | lvmthin | ❌ Disabled | - | - | - | N/A |

**Volume Group:** `pve` - 465.77GB total, **57GB free** ✅

**Thin Pools:**
- `data`: 200GB (2.92% used, 11.42% metadata)
- `thin1`: 208GB (0.00% used, 10.43% metadata)

**Physical Disks:**
- sda, sdb: 558.9GB each (boot drives)
- sdc-sdh: 232.9GB each (6 data drives)

#### Issues Identified

1. ⚠️ **Disabled Storage Pools:** thin2-thin6 are disabled
   - **Impact:** Additional storage not available
   - **Recommendation:** Enable if needed or remove from config

2. ✅ **Excellent Capacity:** 57GB free in VG, 408GB available storage
   - **Status:** Ready for VM deployment

#### Recommendations

**HIGH PRIORITY:**
1. **Enable Additional Storage** - Enable thin2-thin6 if needed (or remove from config)
2. **Migrate VMs from ml110** - This node is ready for 15-20 containers

**RECOMMENDED:**
1. **Storage Optimization** - Consider using thin1 for new deployments
2. **Performance Tuning** - Optimize for high-performance workloads

---

### r630-02 (192.168.11.12) - Production Node

**Status:** ✅ Operational  
**Containers:** 11  
**CPU:** 56 cores @ 2.00GHz (excellent - best CPU)  
**Memory:** 251GB (16GB used, 235GB available - 6.4% usage)

#### Storage Details

| Storage Name | Type | Status | Total | Used | Available | Usage % |
|--------------|------|--------|-------|------|-----------|---------|
| local | dir | ✅ Active | 220GB | 4.0GB | 216GB | 1.81% |
| local-lvm | lvmthin | ❌ Disabled | - | - | - | N/A |
| thin1 | lvmthin | ⚠️ Inactive | - | - | - | 0.00% |
| thin1-r630-02 | lvmthin | 🔴 **CRITICAL** | 226GB | 221GB | **5.0GB** | **97.78%** |
| thin2 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
| thin3 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
| thin4 | lvmthin | ✅ Active | 226GB | 28.7GB | 197GB | 12.69% |
| thin5 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
| thin6 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |

**Volume Groups:**
- thin1-thin6: Each 230.87GB with 0.12GB free

**Thin Pools:**
- `thin1-r630-02`: 226.13GB (**97.78% used**, 3.84% metadata) 🔴 **CRITICAL**
- `thin4`: 226.13GB (12.69% used, 1.15% metadata)
- `thin2, thin3, thin5, thin6`: All empty (0.00% used)

**Physical Disks:**
- sda-sdh: 232.9GB each (8 data drives)

#### Issues Identified

1. 🔴 **CRITICAL: Storage Nearly Full** - thin1-r630-02 at 97.78% capacity
   - **Impact:** Cannot create new VMs/containers on this storage
   - **Action Required:** IMMEDIATE - Migrate VMs or expand storage
   - **Available:** Only 5GB free

2. ⚠️ **Inactive Storage:** thin1 is inactive
   - **Impact:** Storage pool not usable
   - **Recommendation:** Activate or remove from config

3. ⚠️ **Disabled Storage:** local-lvm is disabled
   - **Impact:** Standard storage name not available
   - **Recommendation:** Enable if volume group exists

4. ✅ **Excellent Capacity Available:** thin2, thin3, thin5, thin6 are empty (904GB total)

#### Recommendations

**CRITICAL (IMMEDIATE ACTION REQUIRED):**
1. **Migrate VMs from thin1-r630-02** - Move VMs to thin2, thin3, thin5, or thin6
2. **Expand thin1-r630-02** - If migration not possible, expand the pool
3. **Monitor Closely** - Set alerts for this storage pool

**HIGH PRIORITY:**
1. **Activate thin1** - Enable if needed or remove from config
2. **Enable local-lvm** - If volume group exists, enable for standard naming
3. **Balance Storage Usage** - Distribute VMs across thin2-thin6

**RECOMMENDED:**
1. **Storage Monitoring** - Set up automated alerts
2. **Migration Plan** - Document VM migration procedures

---

### r630-03 (192.168.11.13) - Unknown Status

**Status:** ❌ Not Reachable  
**Action Required:** Investigate connectivity issues

#### Recommendations

1. **Check Network Connectivity** - Verify network connection
2. **Check Power Status** - Verify node is powered on
3. **Check SSH Access** - Verify SSH service is running
4. **Review Storage** - Once accessible, perform full storage review

---

### r630-04 (192.168.11.14) - Unknown Status

**Status:** ❌ Not Reachable  
**Action Required:** Investigate connectivity issues

#### Recommendations

1. **Check Network Connectivity** - Verify network connection
2. **Check Power Status** - Verify node is powered on
3. **Check SSH Access** - Verify SSH service is running
4. **Review Storage** - Once accessible, perform full storage review

---

## Critical Issues Summary

### 🔴 CRITICAL - Immediate Action Required

1. **r630-02 thin1-r630-02 Storage at 97.78% Capacity**
   - **Impact:** Cannot create new VMs/containers
   - **Action:** Migrate VMs to other storage pools (thin2-thin6 available)
   - **Timeline:** IMMEDIATE

2. **ml110 Volume Group Low on Space (16GB free)**
   - **Impact:** Limited capacity for new VMs
   - **Action:** Migrate VMs to r630-01/r630-02 or expand storage
   - **Timeline:** Within 1 week

### ⚠️ HIGH PRIORITY

1. **Uneven Workload Distribution**
   - ml110: 37 containers (overloaded)
   - r630-01: 3 containers (underutilized)
   - r630-02: 11 containers (underutilized)
   - **Action:** Migrate 15-20 containers from ml110 to r630-01/r630-02

2. **Disabled/Inactive Storage Pools**
   - Multiple storage pools disabled across nodes
   - **Action:** Enable if needed or clean up storage.cfg

3. **Unreachable Nodes**
   - r630-03 and r630-04 not accessible
   - **Action:** Investigate and restore connectivity

---

## Storage Capacity Analysis

### Total Storage Capacity

| Node | Total Storage | Used Storage | Available Storage | Usage % |
|------|--------------|--------------|------------------|---------|
| ml110 | 907GB | 234.5GB | 671.5GB | 25.9% |
| r630-01 | 744GB | 5.9GB | 738GB | 0.8% |
| r630-02 | 1,358GB | 253.7GB | 1,104GB | 18.7% |
| **Total** | **3,009GB** | **494GB** | **2,515GB** | **16.4%** |

### Storage Distribution

- **ml110:** 27.92% of local-lvm used (good, but VG low on space)
- **r630-01:** 2.92% of local-lvm used (excellent - ready for deployment)
- **r630-02:** 97.78% of thin1-r630-02 used (CRITICAL), but other pools empty

### Capacity Planning

**Current Capacity:** ~2.5TB available  
**Projected Growth:** Based on current usage patterns  
**Recommendation:** Plan for expansion when total usage reaches 70%

---

## Detailed Recommendations

### 1. Immediate Actions (This Week)

#### r630-02 Storage Crisis

```bash
# 1. List VMs on thin1-r630-02
ssh root@192.168.11.12 "pvesm list thin1-r630-02"

# 2. Migrate VMs to thin2 (or thin3, thin5, thin6)
# Example migration:
pct migrate <VMID> r630-02 --storage thin2

# 3. Verify migration
pvesm status
```

#### ml110 Storage Expansion

**Option A: Migrate VMs (Recommended)**
```bash
# Migrate containers to r630-01
pct migrate <VMID> r630-01 --storage thin1

# Migrate containers to r630-02
pct migrate <VMID> r630-02 --storage thin2
```

**Option B: Expand Volume Group**
```bash
# Add physical volume (if disks available)
pvcreate /dev/sdX
vgextend pve /dev/sdX
lvextend -l +100%FREE pve/data
```

### 2. Storage Optimization (Next 2 Weeks)

#### Enable Disabled Storage Pools

**ml110:**
```bash
# Review and clean up disabled storage
ssh root@192.168.11.10
pvesm status
# Remove unused storage definitions or enable if needed
```

**r630-01:**
```bash
# Enable thin2-thin6 if volume groups exist
ssh root@192.168.11.11
# Check if VGs exist
vgs
# Enable storage pools if VGs exist
for i in thin2 thin3 thin4 thin5 thin6; do
    pvesm set $i --disable 0 2>/dev/null || echo "$i not available"
done
```

**r630-02:**
```bash
# Activate thin1 if needed
ssh root@192.168.11.12
pvesm set thin1 --disable 0
```

#### Balance Workload Distribution

**Migration Plan:**
- **ml110 → r630-01:** Migrate 10-12 medium workload containers
- **ml110 → r630-02:** Migrate 10-12 heavy workload containers
- **r630-02 thin1-r630-02 → thin2-thin6:** Migrate VMs to balance storage

**Target Distribution:**
- ml110: 15-17 containers (management/lightweight)
- r630-01: 15-17 containers (medium workload)
- r630-02: 15-17 containers (heavy workload)

### 3. Long-term Improvements (Next Month)

#### Storage Monitoring

**Set Up Automated Alerts:**
```bash
# Create monitoring script
cat > /usr/local/bin/storage-alert.sh << 'EOF'
#!/bin/bash
# Check storage usage and send alerts
for node in ml110 r630-01 r630-02; do
    ssh root@$node "pvesm status" | awk '$NF > 80 {print "ALERT: $1 on $node at "$NF"%"}'
done
EOF

# Add to crontab (check every hour)
0 * * * * /usr/local/bin/storage-alert.sh
```

#### Backup Strategy

1. **Implement Regular Backups**
   - Daily backups for critical VMs
   - Weekly full backups
   - Monthly archive backups

2. **Backup Storage**
   - Use separate storage for backups
   - Consider NFS for shared backup storage
   - Implement backup rotation (keep 30 days)

#### Performance Optimization

1. **Storage Performance Tuning**
   - Use LVM thin for all VM disks
   - Monitor I/O performance
   - Optimize thin pool metadata size

2. **Network Storage Consideration**
   - Evaluate NFS for shared storage
   - Consider Ceph for high availability
   - Plan for shared storage migration

---

## Storage Type Recommendations

### By Use Case

| Use Case | Recommended Storage | Current Status | Action |
|----------|-------------------|----------------|--------|
| VM/Container Disks | LVM Thin (lvmthin) | ✅ Used | Continue using |
| ISO Images | Directory (dir) | ✅ Used | Continue using |
| Container Templates | Directory (dir) | ✅ Used | Continue using |
| Backups | Directory or NFS | ⚠️ Not configured | Implement |
| High-Performance VMs | LVM Thin or ZFS | ✅ LVM Thin | Consider ZFS for future |

### Storage Performance Best Practices

1. **Use LVM Thin for VM Disks** ✅ Currently implemented
2. **Monitor Thin Pool Metadata** ⚠️ Set up monitoring
3. **Balance Storage Across Nodes** ⚠️ Needs improvement
4. **Implement Backup Storage** ❌ Not implemented

---

## Security Recommendations

1. **Storage Access Control**
   - Review `/etc/pve/storage.cfg` node restrictions
   - Ensure proper node assignments
   - Verify storage permissions

2. **Backup Security**
   - Encrypt backups containing sensitive data
   - Store backups off-site
   - Test backup restoration regularly

---

## Monitoring Recommendations

### Storage Monitoring Metrics

1. **Storage Usage** - Alert at 80%
2. **Thin Pool Metadata** - Alert at 80%
3. **Volume Group Free Space** - Alert at 10%
4. **Storage I/O Performance** - Monitor latency

### Automated Alerts

Set up alerts for:
- Storage usage >80%
- Thin pool metadata >80%
- Volume group free space <10%
- Storage errors or failures

---

## Migration Recommendations

### Workload Distribution Strategy

**Current State:**
- ml110: 37 containers (overloaded, slower CPU)
- r630-01: 3 containers (underutilized, excellent CPU)
- r630-02: 11 containers (underutilized, best CPU)

**Target State:**
- ml110: 15-17 containers (management/lightweight)
- r630-01: 15-17 containers (medium workload)
- r630-02: 15-17 containers (heavy workload)

**Benefits:**
- Better performance (ml110 CPU is slower)
- Better resource utilization
- Improved redundancy
- Better storage distribution

### Migration Priority

1. **CRITICAL:** Migrate VMs from r630-02 thin1-r630-02 (97.78% full)
2. **HIGH:** Migrate 15-20 containers from ml110 to r630-01/r630-02
3. **MEDIUM:** Balance storage usage across all thin pools on r630-02

---

## Action Plan Summary

### Week 1 (Critical)

- [ ] Migrate VMs from r630-02 thin1-r630-02 to thin2-thin6
- [ ] Set up storage monitoring alerts
- [ ] Investigate r630-03 and r630-04 connectivity

### Week 2-3 (High Priority)

- [ ] Migrate 15-20 containers from ml110 to r630-01/r630-02
- [ ] Enable/clean up disabled storage pools
- [ ] Balance storage usage across nodes

### Month 1 (Recommended)

- [ ] Implement backup strategy
- [ ] Set up comprehensive storage monitoring
- [ ] Optimize storage performance
- [ ] Document storage procedures

---

## Conclusion

This comprehensive storage review identifies:

✅ **Current Status:** Storage well configured with LVM thin pools  
⚠️ **Critical Issues:** 1 storage pool at 97.78% capacity  
✅ **Capacity Available:** ~2.5TB total available storage  
⚠️ **Distribution:** Uneven workload distribution  

**Immediate Actions Required:**
1. Migrate VMs from r630-02 thin1-r630-02 (CRITICAL)
2. Migrate containers from ml110 to balance workload
3. Set up storage monitoring and alerts

**Long-term Goals:**
1. Implement backup strategy
2. Optimize storage performance
3. Plan for storage expansion
4. Consider shared storage for HA

---

**Report Generated:** January 6, 2026  
**Next Review:** February 6, 2026 (Monthly)