fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
159 lines
4.1 KiB
Markdown
159 lines
4.1 KiB
Markdown
# Storage Investigation Results - thin2 Pool
|
|
|
|
**Date**: 2026-01-18
|
|
**Storage Pool**: thin2 on r630-02 (192.168.11.12)
|
|
**Status**: 88.78% Used (210.5 GB / 237.1 GB)
|
|
**Available**: 26.6 GB
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
The thin2 storage pool is at **88.78% capacity** with only **26.6 GB available**. The main consumer is **VMID 5000 (Blockscout)** which uses **96.01% of its 200GB allocation** (~192GB actual usage).
|
|
|
|
---
|
|
|
|
## Storage Breakdown
|
|
|
|
### Containers on thin2:
|
|
|
|
| VMID | Name | Status | Size | Thin2 Usage | Notes |
|
|
|------|------|--------|------|-------------|-------|
|
|
| **5000** | **blockscout-1** | **running** | **200GB** | **96.01%** | **Main consumer (~192GB)** |
|
|
| 6200 | firefly-1 | running | 50GB | 9.77% | ~4.9GB used |
|
|
| 6201 | firefly-ali-1 | running | 50GB | 7.72% | ~3.9GB used (also on thin3) |
|
|
|
|
**Total allocated**: 300GB (200GB + 50GB + 50GB)
|
|
**Total used on thin2**: ~200.8GB
|
|
|
|
---
|
|
|
|
## Blockscout (VMID 5000) Disk Usage Analysis
|
|
|
|
### Overall Usage:
|
|
- **Total disk**: 196GB
|
|
- **Used**: 7.4GB (4% of container)
|
|
- **Available**: 179GB
|
|
|
|
**Note**: The container shows only 7.4GB used, but the LVM thin pool shows 96.01% usage of the 200GB allocation. This indicates **thin provisioning** - the LVM volume is allocated but not fully written to.
|
|
|
|
### Disk Usage Breakdown:
|
|
|
|
#### 1. **Systemd Journal Logs: 4.0GB** ⚠️ **MAJOR ISSUE**
|
|
- **Location**: `/var/log/journal`
|
|
- **Usage**: 4.0GB
|
|
- **Impact**: Largest single consumer
|
|
- **Action**: **Clean up journal logs**
|
|
|
|
#### 2. **Docker Data: 2.7GB**
|
|
- **Location**: `/var/lib/docker`
|
|
- **Breakdown**:
|
|
- PostgreSQL data volume: **1.3GB**
|
|
- Docker overlay2: **1.4GB**
|
|
- Images: 724.1MB
|
|
- Containers: 689.5KB
|
|
- Volumes: 1.375GB
|
|
- **Status**: Normal usage
|
|
|
|
#### 3. **Other Directories:**
|
|
- `/usr`: 747MB
|
|
- `/opt/blockscout`: 32KB
|
|
- `/tmp`: 40KB
|
|
|
|
---
|
|
|
|
## Root Cause
|
|
|
|
The **systemd journal logs** are consuming **4.0GB** of space, which is excessive. This is the main contributor to storage usage.
|
|
|
|
### Why is the journal so large?
|
|
|
|
Systemd journal accumulates logs over time and can grow very large if not properly managed. The Blockscout container has been running for **4+ days** without journal cleanup.
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### Immediate Actions:
|
|
|
|
1. **Clean up systemd journal logs** (will free ~4GB)
|
|
```bash
|
|
# On VMID 5000
|
|
journalctl --vacuum-size=100M
|
|
# Or keep last 3 days only
|
|
journalctl --vacuum-time=3d
|
|
```
|
|
|
|
2. **Configure journal limits** to prevent future issues
|
|
```bash
|
|
# Limit journal to 100MB
|
|
echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf
|
|
systemctl restart systemd-journald
|
|
```
|
|
|
|
3. **Monitor Docker volumes** (PostgreSQL data growing over time)
|
|
- Current: 1.3GB
|
|
- Should monitor growth
|
|
- Consider periodic cleanup of old Blockscout data
|
|
|
|
### Long-term Actions:
|
|
|
|
1. **Review LVM thin pool allocation** for Blockscout
|
|
- Currently: 200GB allocated
|
|
- Actual usage: ~7.4GB (4%)
|
|
- Could reduce to 50GB or 100GB if needed
|
|
|
|
2. **Implement log rotation** for all containers
|
|
- Configure journald limits
|
|
- Set up log rotation for application logs
|
|
|
|
3. **Monitor thin2 pool usage**
|
|
- Set up alerts at 80%, 90%, 95%
|
|
- Regular cleanup of journal logs
|
|
|
|
---
|
|
|
|
## Storage Pool Status
|
|
|
|
### thin2 Pool:
|
|
- **Total**: 237.1 GB
|
|
- **Used**: 210.5 GB (88.78%)
|
|
- **Available**: 26.6 GB
|
|
- **Status**: ⚠️ **WARNING** - Low space
|
|
|
|
### Other Pools on r630-02:
|
|
- **thin1-r630-02**: 0% used (empty)
|
|
- **thin3**: 1.72% used (4GB / 237GB)
|
|
- **thin4**: 21.12% used (50GB / 237GB)
|
|
- **thin5**: 0% used (empty)
|
|
- **thin6**: 0% used (empty)
|
|
|
|
---
|
|
|
|
## Cleanup Script
|
|
|
|
Run this to clean up journal logs:
|
|
|
|
```bash
|
|
# On Proxmox host (192.168.11.12)
|
|
pct exec 5000 -- journalctl --vacuum-size=100M
|
|
|
|
# Or keep last 3 days only
|
|
pct exec 5000 -- journalctl --vacuum-time=3d
|
|
|
|
# Configure journal limits
|
|
pct exec 5000 -- bash -c "echo 'SystemMaxUse=100M' >> /etc/systemd/journald.conf && systemctl restart systemd-journald"
|
|
```
|
|
|
|
---
|
|
|
|
## Expected Results After Cleanup
|
|
|
|
- **Before**: 7.4GB used (journal: 4.0GB)
|
|
- **After cleanup**: ~3.4GB used (journal: <100MB)
|
|
- **Freed space**: ~4GB
|
|
- **Thin2 pool**: ~84% used (down from 88.78%)
|
|
|
|
---
|
|
|
|
**Next Steps**: Run the cleanup script to free up ~4GB of space. |