proxmox/docs/archive/deployment-reports/STORAGE_INVESTIGATION_RESULTS.md

Storage Investigation Results - thin2 Pool

Date: 2026-01-18
Storage Pool: thin2 on r630-02 (192.168.11.12)
Status: 88.78% Used (210.5 GB / 237.1 GB)
Available: 26.6 GB

---

Summary

The thin2 storage pool is at 88.78% capacity with only 26.6 GB available. The main consumer is VMID 5000 (Blockscout) which uses 96.01% of its 200GB allocation (~192GB actual usage).

---

Storage Breakdown

Containers on thin2:

VMID	Name	Status	Size	Thin2 Usage	Notes
5000	blockscout-1	running	200GB	96.01%	Main consumer (~192GB)
6200	firefly-1	running	50GB	9.77%	~4.9GB used
6201	firefly-ali-1	running	50GB	7.72%	~3.9GB used (also on thin3)

Total allocated: 300GB (200GB + 50GB + 50GB)
Total used on thin2: ~200.8GB

---

Blockscout (VMID 5000) Disk Usage Analysis

Overall Usage:

• Total disk: 196GB
• Used: 7.4GB (4% of container)
• Available: 179GB

Note: The container shows only 7.4GB used, but the LVM thin pool shows 96.01% usage of the 200GB allocation. This indicates thin provisioning - the LVM volume is allocated but not fully written to.

Disk Usage Breakdown:

1. Systemd Journal Logs: 4.0GB ⚠️ MAJOR ISSUE

• Location: /var/log/journal
• Usage: 4.0GB
• Impact: Largest single consumer
• Action: Clean up journal logs

2. Docker Data: 2.7GB

• Location: /var/lib/docker
• Breakdown:
  • PostgreSQL data volume: 1.3GB
  • Docker overlay2: 1.4GB
  • Images: 724.1MB
  • Containers: 689.5KB
  • Volumes: 1.375GB
• Status: Normal usage

3. Other Directories:

• /usr: 747MB
• /opt/blockscout: 32KB
• /tmp: 40KB

---

Root Cause

The systemd journal logs are consuming 4.0GB of space, which is excessive. This is the main contributor to storage usage.

Why is the journal so large?

Systemd journal accumulates logs over time and can grow very large if not properly managed. The Blockscout container has been running for 4+ days without journal cleanup.

---

Recommendations

Immediate Actions:

1. Clean up systemd journal logs (will free ~4GB)

   # On VMID 5000
   journalctl --vacuum-size=100M
   # Or keep last 3 days only
   journalctl --vacuum-time=3d
   

2. Configure journal limits to prevent future issues

   # Limit journal to 100MB
   echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf
   systemctl restart systemd-journald
   

3. Monitor Docker volumes (PostgreSQL data growing over time)

   • Current: 1.3GB
   • Should monitor growth
   • Consider periodic cleanup of old Blockscout data

Long-term Actions:

1. Review LVM thin pool allocation for Blockscout

   • Currently: 200GB allocated
   • Actual usage: ~7.4GB (4%)
   • Could reduce to 50GB or 100GB if needed

2. Implement log rotation for all containers

   • Configure journald limits
   • Set up log rotation for application logs

3. Monitor thin2 pool usage

   • Set up alerts at 80%, 90%, 95%
   • Regular cleanup of journal logs

---

Storage Pool Status

thin2 Pool:

• Total: 237.1 GB
• Used: 210.5 GB (88.78%)
• Available: 26.6 GB
• Status: ⚠️ WARNING - Low space

Other Pools on r630-02:

• thin1-r630-02: 0% used (empty)
• thin3: 1.72% used (4GB / 237GB)
• thin4: 21.12% used (50GB / 237GB)
• thin5: 0% used (empty)
• thin6: 0% used (empty)

---

Cleanup Script

Run this to clean up journal logs:

# On Proxmox host (192.168.11.12)
pct exec 5000 -- journalctl --vacuum-size=100M

# Or keep last 3 days only
pct exec 5000 -- journalctl --vacuum-time=3d

# Configure journal limits
pct exec 5000 -- bash -c "echo 'SystemMaxUse=100M' >> /etc/systemd/journald.conf && systemctl restart systemd-journald"

---

Expected Results After Cleanup

• Before: 7.4GB used (journal: 4.0GB)
• After cleanup: ~3.4GB used (journal: <100MB)
• Freed space: ~4GB
• Thin2 pool: ~84% used (down from 88.78%)

---

Next Steps: Run the cleanup script to free up ~4GB of space.