proxmox/reports/VMID5000_CRITICAL_ISSUES_FOUND.md

# VMID 5000 Critical Issues Found - 192.168.11.12

**Date**: 2026-01-04  
**Host**: 192.168.11.12  
**Status**: 🔴 **CRITICAL ISSUES - IMMEDIATE ACTION REQUIRED**

---

## 📊 Executive Summary

VMID 5000 (Blockscout Explorer) was found on **192.168.11.12** (not 192.168.11.10). The container is running but has **CRITICAL ISSUES** that need immediate attention.

---

## ✅ Working Components

| Component | Status | Details |
|-----------|--------|---------|
| **Container** | ✅ Running | VMID 5000, Name: blockscout-1 |
| **Blockscout Service** | ✅ Active | systemd service active |
| **Nginx Service** | ✅ Active | Web server running |
| **Docker Containers** | ✅ Running | Blockscout and PostgreSQL containers up |

---

## 🔴 CRITICAL ISSUES

### 1. Disk Full - NO SPACE LEFT ON DEVICE ❌

**Status**: 🔴 **CRITICAL**  
**Impact**: Blockscout crashed, PostgreSQL cannot write, indexing stopped

**Error Messages**:
```
ERROR 53100 (disk_full) could not write to file "base/pgsql_tmp/pgsql_tmp128.3": No space left on device
Application indexer exited: shutdown
GenServer Indexer.Block.Catchup.MissingRangesCollector terminating
```

**Symptoms**:
- Blockscout API not responding on port 4000
- PostgreSQL not accepting connections
- Blockscout application crashed
- Indexer process terminated

**Immediate Action Required**:
```bash
# Check disk usage
ssh root@192.168.11.12 'pct exec 5000 -- df -h'

# Clean up disk space
ssh root@192.168.11.12 'pct exec 5000 -- docker system prune -a --volumes'
ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml down'
ssh root@192.168.11.12 'pct exec 5000 -- journalctl --vacuum-time=1d'

# Restart services after cleanup
ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml up -d'
```

### 2. Cloudflare Tunnel Inactive ❌

**Status**: 🟠 **HIGH PRIORITY**  
**Impact**: Public access to explorer.d-bis.org not available

**Current Status**: Inactive

**Fix**:
```bash
ssh root@192.168.11.12 'pct exec 5000 -- systemctl start cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl enable cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl status cloudflared'
```

### 3. Blockscout API Not Responding ❌

**Status**: 🔴 **CRITICAL** (caused by disk full)  
**Impact**: Explorer not accessible

**Current Status**: Not responding on port 4000

**Root Cause**: Disk full causing Blockscout to crash

**Fix**: Resolve disk space issue first, then restart Blockscout

---

## ⚠️ Warnings (Non-Critical)

### 1. RPC Method Not Enabled Warnings ⚠️

**Status**: ⚠️ **WARNING** (Non-critical)  
**Impact**: Some optional RPC methods not enabled (internal transactions, block rewards)

**Error Messages**:
```
Method not enabled (-32604)
failed to fetch internal transactions
failed to fetch block_reward
```

**Note**: This is expected if the RPC node doesn't have these methods enabled. Not critical for basic functionality.

---

## 📋 Diagnostic Results

### Container Status
- **VMID**: 5000
- **Name**: blockscout-1
- **Status**: running
- **Host**: 192.168.11.12

### Service Status
- **Blockscout Service**: ✅ active
- **Nginx Service**: ✅ active
- **Cloudflare Tunnel**: ❌ inactive

### Docker Containers
- **Blockscout Container**: ✅ Running (df2ab8851e83)
- **PostgreSQL Container**: ✅ Running (a613b93eefbb, healthy)

### Network Connectivity
- **Blockscout API (port 4000)**: ❌ Not responding (disk full)
- **PostgreSQL (port 5432)**: ❌ Not accepting connections (disk full)

---

## 🚨 Immediate Action Plan

### Priority 1: Fix Disk Space (CRITICAL)

1. **Check disk usage**:
   ```bash
   ssh root@192.168.11.12 'pct exec 5000 -- df -h'
   ```

2. **Free up disk space**:
   ```bash
   # Clean Docker resources
   ssh root@192.168.11.12 'pct exec 5000 -- docker system prune -a --volumes -f'
   
   # Clean logs
   ssh root@192.168.11.12 'pct exec 5000 -- journalctl --vacuum-time=1d'
   
   # Clean old Docker images
   ssh root@192.168.11.12 'pct exec 5000 -- docker image prune -a -f'
   ```

3. **Restart Blockscout**:
   ```bash
   ssh root@192.168.11.12 'pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml restart'
   ```

### Priority 2: Start Cloudflare Tunnel

```bash
ssh root@192.168.11.12 'pct exec 5000 -- systemctl start cloudflared'
ssh root@192.168.11.12 'pct exec 5000 -- systemctl enable cloudflared'
```

### Priority 3: Verify Services

```bash
# Check disk space
ssh root@192.168.11.12 'pct exec 5000 -- df -h'

# Check Blockscout API
ssh root@192.168.11.12 'pct exec 5000 -- curl -s http://localhost:4000/api/v2/status'

# Check Cloudflare tunnel
ssh root@192.168.11.12 'pct exec 5000 -- systemctl status cloudflared'
```

---

## 🔧 Fix Scripts Available

Use the fix script with correct host:
```bash
cd /home/intlc/projects/proxmox
PROXMOX_HOST=192.168.11.12 ./scripts/fix-vmid5000-blockscout.sh
```

**Note**: The fix script will attempt to start services, but the disk full issue must be resolved first.

---

## 📊 Summary

**Current Status**: 🔴 **CRITICAL ISSUES**

**Critical Issues**:
1. ❌ Disk full - No space left on device (CRITICAL)
2. ❌ Blockscout API not responding (caused by disk full)
3. ❌ PostgreSQL cannot write (caused by disk full)
4. ❌ Cloudflare tunnel inactive

**Working Components**:
- ✅ Container running
- ✅ Blockscout service active
- ✅ Nginx service active
- ✅ Docker containers running

**Next Steps**:
1. **IMMEDIATE**: Free up disk space
2. **HIGH PRIORITY**: Start Cloudflare tunnel
3. **VERIFY**: Check all services after disk cleanup

---

**Last Updated**: 2026-01-04  
**Host**: 192.168.11.12  
**Status**: 🔴 **REQUIRES IMMEDIATE ATTENTION**