Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
3.4 KiB
3.4 KiB
r630-02 Fixes Applied
Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation
Date: 2026-01-15
Host: r630-02 (192.168.11.12)
Status: ✅ FIXES APPLIED
Actions Taken
1. Container Memory Limits Adjusted ✅
Issue: Multiple containers experiencing OOM (Out of Memory) kills
Actions:
- ✅ Container 7811 (mim-api-1): Increased from 2048MB to 4096MB
- ✅ Container 5000 (blockscout-1): Already at 8192MB (adequate)
- ✅ Container 6200 (firefly-1): Already at 4096MB (adequate)
- ✅ Container 6201 (firefly-ali-1): 2048MB (stopped, not an issue)
Result:
- Container 7811 restarted with new memory limit
- Memory configuration updated successfully
2. Container Status Verified ✅
Current Container Status:
- ✅ 5000 - blockscout-1: Running (Memory: 8192MB)
- ✅ 6200 - firefly-1: Running (Memory: 4096MB)
- ⏸️ 6201 - firefly-ali-1: Stopped (Memory: 2048MB)
- ✅ 7811 - mim-api-1: Running (Memory: 4096MB - Updated)
3. Storage Analysis ✅
Storage Status:
- ⚠️ thin1-r630-02: 88.51% used (26GB free)
- ⚠️ thin2: 88.33% used (27GB free)
Largest Volumes:
- vm-5000-disk-0 (thin2): 200GB, 95.78% used
- vm-5000-disk-0 (thin1): 200GB
- vm-6200-disk-0: 50GB
- vm-6201-disk-0: 50GB
Recommendation:
- Monitor storage usage
- Consider cleanup of unused volumes
- Plan for storage expansion if needed
4. System Health Monitoring ✅
Load Average:
- Current: 9.77, 11.28, 11.46
- Previous: 12.03, 11.65, 11.48
- Trend: Slightly improved
Memory:
- Total: 251GB
- Used: 9.0GB
- Available: 242GB
- Status: ✅ Healthy
Recent OOM Events
Last 2 Hours:
- 1 OOM event detected (container 7811 at 17:30:48)
- This occurred before memory limit increase
- Container restarted with new 4GB limit
Monitoring:
- Continue monitoring for OOM events
- If OOM kills persist, consider further memory increases
Summary of Fixes
✅ Completed
-
Memory Limits:
- ✅ Container 7811 increased to 4GB
- ✅ All containers have adequate memory limits
- ✅ Containers restarted to apply changes
-
Status Verification:
- ✅ All containers operational
- ✅ System resources healthy
- ✅ Load average improving
-
Storage Analysis:
- ✅ Storage usage identified
- ✅ Cleanup opportunities documented
⏳ Ongoing Monitoring
-
OOM Events:
- Monitor for new OOM kills
- Adjust memory if needed
-
Storage:
- Monitor thin pool usage
- Plan cleanup or expansion
-
Load Average:
- Continue monitoring trends
- May be normal for workload
Current Status
System: ✅ HEALTHY
Containers:
- ✅ All running containers operational
- ✅ Memory limits adequate
- ✅ No immediate issues
Resources:
- ✅ Memory: 242GB available
- ✅ Load: Improving (9.77 vs 12.03)
- ⚠️ Storage: 88% used (monitor)
Overall Assessment:
- ✅ System is stable
- ✅ Fixes applied successfully
- ✅ Ready for continued operations
Next Steps
-
Monitor:
- Watch for new OOM events
- Monitor storage usage trends
- Track load average
-
Storage:
- Review cleanup opportunities
- Plan expansion if needed
-
Optimization:
- Continue monitoring container resource usage
- Adjust as needed based on workload
Last Updated: 2026-01-15