proxmox/reports/VMID5000_CRITICAL_ISSUES_FOUND.md

VMID 5000 Critical Issues Found - 192.168.11.12

Date: 2026-01-04
Host: 192.168.11.12
Status: 🔴 CRITICAL ISSUES - IMMEDIATE ACTION REQUIRED

---

📊 Executive Summary

VMID 5000 (Blockscout Explorer) was found on 192.168.11.12 (not 192.168.11.10). The container is running but has CRITICAL ISSUES that need immediate attention.

---

✅ Working Components

Component	Status	Details
Container	✅ Running	VMID 5000, Name: blockscout-1
Blockscout Service	✅ Active	systemd service active
Nginx Service	✅ Active	Web server running
Docker Containers	✅ Running	Blockscout and PostgreSQL containers up

---

🔴 CRITICAL ISSUES

1. Disk Full - NO SPACE LEFT ON DEVICE ❌

Status: 🔴 CRITICAL
Impact: Blockscout crashed, PostgreSQL cannot write, indexing stopped

Error Messages:

ERROR 53100 (disk_full) could not write to file "base/pgsql_tmp/pgsql_tmp128.3": No space left on device
Application indexer exited: shutdown
GenServer Indexer.Block.Catchup.MissingRangesCollector terminating

Symptoms:

• Blockscout API not responding on port 4000
• PostgreSQL not accepting connections
• Blockscout application crashed
• Indexer process terminated

Immediate Action Required:

# Check disk usage
ssh root@192.168.11.12 'pct exec 5000 -- df -h'

# Clean up disk space
ssh root@192.168.11.12 'pct exec 5000 -- docker system prune -a --volumes'
ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml down'
ssh root@192.168.11.12 'pct exec 5000 -- journalctl --vacuum-time=1d'

# Restart services after cleanup
ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml up -d'

2. Cloudflare Tunnel Inactive ❌

Status: 🟠 HIGH PRIORITY
Impact: Public access to explorer.d-bis.org not available

Current Status: Inactive

Fix:

ssh root@192.168.11.12 'pct exec 5000 -- systemctl start cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl enable cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl status cloudflared'

3. Blockscout API Not Responding ❌

Status: 🔴 CRITICAL (caused by disk full)
Impact: Explorer not accessible

Current Status: Not responding on port 4000

Root Cause: Disk full causing Blockscout to crash

Fix: Resolve disk space issue first, then restart Blockscout

---

⚠️ Warnings (Non-Critical)

1. RPC Method Not Enabled Warnings ⚠️

Status: ⚠️ WARNING (Non-critical)
Impact: Some optional RPC methods not enabled (internal transactions, block rewards)

Error Messages:

Method not enabled (-32604)
failed to fetch internal transactions
failed to fetch block_reward

Note: This is expected if the RPC node doesn't have these methods enabled. Not critical for basic functionality.

---

📋 Diagnostic Results

Container Status

• VMID: 5000
• Name: blockscout-1
• Status: running
• Host: 192.168.11.12

Service Status

• Blockscout Service: ✅ active
• Nginx Service: ✅ active
• Cloudflare Tunnel: ❌ inactive

Docker Containers

• Blockscout Container: ✅ Running (df2ab8851e83)
• PostgreSQL Container: ✅ Running (a613b93eefbb, healthy)

Network Connectivity

• Blockscout API (port 4000): ❌ Not responding (disk full)
• PostgreSQL (port 5432): ❌ Not accepting connections (disk full)

---

🚨 Immediate Action Plan

Priority 1: Fix Disk Space (CRITICAL)

1. Check disk usage:

   ssh root@192.168.11.12 'pct exec 5000 -- df -h'
   

2. Free up disk space:

   # Clean Docker resources
   ssh root@192.168.11.12 'pct exec 5000 -- docker system prune -a --volumes -f'
   
   # Clean logs
   ssh root@192.168.11.12 'pct exec 5000 -- journalctl --vacuum-time=1d'
   
   # Clean old Docker images
   ssh root@192.168.11.12 'pct exec 5000 -- docker image prune -a -f'
   

3. Restart Blockscout:

   ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml restart'
   

Priority 2: Start Cloudflare Tunnel

ssh root@192.168.11.12 'pct exec 5000 -- systemctl start cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl enable cloudflared'

Priority 3: Verify Services

# Check disk space
ssh root@192.168.11.12 'pct exec 5000 -- df -h'

# Check Blockscout API
ssh root@192.168.11.12 'pct exec 5000 -- curl -s http://localhost:4000/api/v2/status'

# Check Cloudflare tunnel
ssh root@192.168.11.12 'pct exec 5000 -- systemctl status cloudflared'

---

🔧 Fix Scripts Available

Use the fix script with correct host:

cd /home/intlc/projects/proxmox
PROXMOX_HOST=192.168.11.12 ./scripts/fix-vmid5000-blockscout.sh

Note: The fix script will attempt to start services, but the disk full issue must be resolved first.

---

📊 Summary

Current Status: 🔴 CRITICAL ISSUES

Critical Issues:

1. ❌ Disk full - No space left on device (CRITICAL)
2. ❌ Blockscout API not responding (caused by disk full)
3. ❌ PostgreSQL cannot write (caused by disk full)
4. ❌ Cloudflare tunnel inactive

Working Components:

• ✅ Container running
• ✅ Blockscout service active
• ✅ Nginx service active
• ✅ Docker containers running

Next Steps:

1. IMMEDIATE: Free up disk space
2. HIGH PRIORITY: Start Cloudflare tunnel
3. VERIFY: Check all services after disk cleanup

---

Last Updated: 2026-01-04
Host: 192.168.11.12
Status: 🔴 REQUIRES IMMEDIATE ATTENTION