# Storage Investigation Results - thin2 Pool

**Date**: 2026-01-18  
**Storage Pool**: thin2 on r630-02 (192.168.11.12)  
**Status**: 88.78% Used (210.5 GB / 237.1 GB)  
**Available**: 26.6 GB

---

## Summary

The thin2 storage pool is at **88.78% capacity** with only **26.6 GB available**. The main consumer is **VMID 5000 (Blockscout)** which uses **96.01% of its 200GB allocation** (~192GB actual usage).

---

## Storage Breakdown

### Containers on thin2:

| VMID | Name | Status | Size | Thin2 Usage | Notes |
|------|------|--------|------|-------------|-------|
| **5000** | **blockscout-1** | **running** | **200GB** | **96.01%** | **Main consumer (~192GB)** |
| 6200 | firefly-1 | running | 50GB | 9.77% | ~4.9GB used |
| 6201 | firefly-ali-1 | running | 50GB | 7.72% | ~3.9GB used (also on thin3) |

**Total allocated**: 300GB (200GB + 50GB + 50GB)  
**Total used on thin2**: ~200.8GB

---

## Blockscout (VMID 5000) Disk Usage Analysis

### Overall Usage:
- **Total disk**: 196GB
- **Used**: 7.4GB (4% of container)
- **Available**: 179GB

**Note**: The container shows only 7.4GB used, but the LVM thin pool shows 96.01% usage of the 200GB allocation. This indicates **thin provisioning** - the LVM volume is allocated but not fully written to.

### Disk Usage Breakdown:

#### 1. **Systemd Journal Logs: 4.0GB** ⚠️ **MAJOR ISSUE**
- **Location**: `/var/log/journal`
- **Usage**: 4.0GB
- **Impact**: Largest single consumer
- **Action**: **Clean up journal logs**

#### 2. **Docker Data: 2.7GB**
- **Location**: `/var/lib/docker`
- **Breakdown**:
  - PostgreSQL data volume: **1.3GB**
  - Docker overlay2: **1.4GB**
  - Images: 724.1MB
  - Containers: 689.5KB
  - Volumes: 1.375GB
- **Status**: Normal usage

#### 3. **Other Directories:**
- `/usr`: 747MB
- `/opt/blockscout`: 32KB
- `/tmp`: 40KB

---

## Root Cause

The **systemd journal logs** are consuming **4.0GB** of space, which is excessive. This is the main contributor to storage usage.

### Why is the journal so large?

Systemd journal accumulates logs over time and can grow very large if not properly managed. The Blockscout container has been running for **4+ days** without journal cleanup.

---

## Recommendations

### Immediate Actions:

1. **Clean up systemd journal logs** (will free ~4GB)
   ```bash
   # On VMID 5000
   journalctl --vacuum-size=100M
   # Or keep last 3 days only
   journalctl --vacuum-time=3d
   ```

2. **Configure journal limits** to prevent future issues
   ```bash
   # Limit journal to 100MB
   echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf
   systemctl restart systemd-journald
   ```

3. **Monitor Docker volumes** (PostgreSQL data growing over time)
   - Current: 1.3GB
   - Should monitor growth
   - Consider periodic cleanup of old Blockscout data

### Long-term Actions:

1. **Review LVM thin pool allocation** for Blockscout
   - Currently: 200GB allocated
   - Actual usage: ~7.4GB (4%)
   - Could reduce to 50GB or 100GB if needed

2. **Implement log rotation** for all containers
   - Configure journald limits
   - Set up log rotation for application logs

3. **Monitor thin2 pool usage**
   - Set up alerts at 80%, 90%, 95%
   - Regular cleanup of journal logs

---

## Storage Pool Status

### thin2 Pool:
- **Total**: 237.1 GB
- **Used**: 210.5 GB (88.78%)
- **Available**: 26.6 GB
- **Status**: ⚠️ **WARNING** - Low space

### Other Pools on r630-02:
- **thin1-r630-02**: 0% used (empty)
- **thin3**: 1.72% used (4GB / 237GB)
- **thin4**: 21.12% used (50GB / 237GB)
- **thin5**: 0% used (empty)
- **thin6**: 0% used (empty)

---

## Cleanup Script

Run this to clean up journal logs:

```bash
# On Proxmox host (192.168.11.12)
pct exec 5000 -- journalctl --vacuum-size=100M

# Or keep last 3 days only
pct exec 5000 -- journalctl --vacuum-time=3d

# Configure journal limits
pct exec 5000 -- bash -c "echo 'SystemMaxUse=100M' >> /etc/systemd/journald.conf && systemctl restart systemd-journald"
```

---

## Expected Results After Cleanup

- **Before**: 7.4GB used (journal: 4.0GB)
- **After cleanup**: ~3.4GB used (journal: <100MB)
- **Freed space**: ~4GB
- **Thin2 pool**: ~84% used (down from 88.78%)

---

**Next Steps**: Run the cleanup script to free up ~4GB of space.