proxmox/reports/storage/MIGRATION_AND_MONITORING_STATUS.md

# Storage Migration and Monitoring Setup - Status Report

**Date:** January 6, 2026  
**Status:** ✅ In Progress

---

## Executive Summary

Successfully initiated migration of containers from r630-02 `thin1-r630-02` storage pool (97.78% full) to other available thin pools. Storage monitoring system has been set up with automated alerts.

---

## Migration Status

### Source Storage: thin1-r630-02
- **Initial Status:** 97.78% capacity (221GB used, 5GB available) 🔴 CRITICAL
- **Containers to Migrate:** 10 containers
- **Target Storage Pools:** thin2, thin3, thin5, thin6 (all empty, 226GB each)

### Migration Progress

| Container | VMID | Name | Status | Target Storage |
|-----------|------|------|--------|----------------|
| proxmox-mail-gateway | 100 | ✅ Migrated | thin2 |
| proxmox-datacenter-manager | 101 | ✅ Migrated | thin2 |
| cloudflared | 102 | ⏳ Pending | - |
| omada | 103 | ⏳ Pending | - |
| gitea | 104 | ⏳ Pending | - |
| nginxproxymanager | 105 | ⏳ Pending | - |
| monitoring-1 | 130 | ⏳ Pending | - |
| blockscout-1 | 5000 | ⏳ Pending | - |
| firefly-1 | 6200 | ⏳ Pending | - |
| firefly-ali-1 | 6201 | ⏳ Pending | - |

**Progress:** 2/10 containers migrated (20%)

### Migration Script

**Location:** `scripts/migrate-thin1-r630-02.sh`

**Features:**
- ✅ Automatically identifies containers on source storage
- ✅ Skips already migrated containers
- ✅ Distributes containers across target pools (round-robin)
- ✅ Stops containers before migration, restarts after
- ✅ Verifies migration success
- ✅ Comprehensive logging

**Usage:**
```bash
# Interactive mode (with confirmation)
./scripts/migrate-thin1-r630-02.sh

# Non-interactive mode (auto-confirm)
./scripts/migrate-thin1-r630-02.sh --yes
```

**Logs:** `logs/migrations/migrate-thin1-r630-02_*.log`

---

## Storage Monitoring Setup

### Monitoring Script

**Location:** `scripts/storage-monitor.sh`

**Features:**
- ✅ Monitors all Proxmox nodes (ml110, r630-01, r630-02, r630-03, r630-04)
- ✅ Checks storage usage (warning at 80%, critical at 90%)
- ✅ Checks volume group free space (warning at 10GB, critical at 5GB)
- ✅ Generates alerts for issues
- ✅ Logs all status checks
- ✅ Generates daily summaries

**Usage:**
```bash
# Run full monitoring check
./scripts/storage-monitor.sh check

# Show current storage status
./scripts/storage-monitor.sh status

# Show recent alerts
./scripts/storage-monitor.sh alerts
```

**Logs:**
- Alerts: `logs/storage-monitoring/storage_alerts_YYYYMMDD.log`
- Status: `logs/storage-monitoring/storage_status_YYYYMMDD.log`
- Summary: `logs/storage-monitoring/storage_summary_YYYYMMDD.txt`

### Automated Monitoring (Cron Job)

**Status:** ✅ Configured

**Schedule:** Every hour at :00

**Command:**
```bash
0 * * * * /home/intlc/projects/proxmox/scripts/storage-monitor.sh check >> /home/intlc/projects/proxmox/logs/storage-monitoring/cron.log 2>&1
```

**View Cron Jobs:**
```bash
crontab -l
```

**Setup Script:** `scripts/setup-storage-monitoring-cron.sh`

---

## Alert Thresholds

### Storage Usage Alerts
- **Warning:** ≥80% capacity
- **Critical:** ≥90% capacity

### Volume Group Free Space Alerts
- **Warning:** ≤10GB free
- **Critical:** ≤5GB free

### Current Alerts

Based on last monitoring check:
- 🔴 **CRITICAL:** r630-02:thin1-r630-02 is at 97.78% capacity (5GB available)
- ⚠️ **WARNING:** ml110:pve volume group has only 16GB free space

---

## Next Steps

### Immediate Actions

1. **Complete Migration** ⏳
   - Continue migrating remaining 8 containers
   - Run: `./scripts/migrate-thin1-r630-02.sh --yes`
   - Monitor progress in logs

2. **Verify Migration Success** ⏳
   - Check all containers are running after migration
   - Verify storage usage on thin1-r630-02 is reduced
   - Confirm containers are distributed across thin2-thin6

### Short-term (This Week)

1. **Monitor Storage** ✅
   - Automated monitoring is active
   - Review alerts daily
   - Check logs for any issues

2. **Verify Cron Job** ⏳
   - Wait for next hourly run
   - Check `logs/storage-monitoring/cron.log`
   - Verify alerts are being generated

### Long-term (This Month)

1. **Optimize Storage Distribution**
   - Balance containers across all thin pools
   - Plan for future growth
   - Consider expanding storage if needed

2. **Enhance Monitoring**
   - Add email/Slack notifications (optional)
   - Set up dashboard for storage metrics
   - Create automated reports

---

## Commands Reference

### Check Migration Status
```bash
# Check containers on source storage
ssh root@192.168.11.12 "pvesm list thin1-r630-02"

# Check storage usage
ssh root@192.168.11.12 "pvesm status | grep thin"
```

### Run Migration
```bash
# Complete migration (non-interactive)
cd /home/intlc/projects/proxmox
./scripts/migrate-thin1-r630-02.sh --yes
```

### Check Monitoring
```bash
# Run monitoring check
./scripts/storage-monitor.sh check

# View recent alerts
./scripts/storage-monitor.sh alerts

# View current status
./scripts/storage-monitor.sh status
```

### View Logs
```bash
# Migration logs
ls -lh logs/migrations/

# Monitoring logs
ls -lh logs/storage-monitoring/

# Cron logs
tail -f logs/storage-monitoring/cron.log
```

---

## Summary

✅ **Migration Script Created:** Fully functional with error handling and logging  
✅ **Monitoring Script Created:** Comprehensive monitoring with alerts  
✅ **Cron Job Configured:** Automated hourly monitoring  
⏳ **Migration In Progress:** 2/10 containers migrated (20% complete)  
⏳ **Remaining Work:** Complete migration of 8 remaining containers

**Critical Issue Status:** 🔴 Still critical - thin1-r630-02 at 97.78% (migration in progress)

---

**Last Updated:** January 6, 2026  
**Next Review:** After migration completion