proxmox/docs/03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md

Blockscout Fix Runbook (VMID 5000)

Last Updated: 2026-02-02
Status: Active
Container: blockscout-1 @ 192.168.11.140 (VMID 5000 on r630-02)

---

Symptoms

• 502 Bad Gateway when accessing http://192.168.11.140/api or https://explorer.d-bis.org/api
• Blockscout logs: postgres:5432: non-existing domain - :nxdomain (DB unreachable)
• Docker: no space left on device when pulling/creating containers

---

Root Cause

1. Thin pool full: thin1-r630-02 is at 100% capacity. VM 5000 resides on thin1.
2. postgres nxdomain: Blockscout container cannot resolve hostname postgres (Docker network/DNS).
3. Docker cannot create overlay layers when the thin pool has no free space.

---

Fix: SSL + Migrations (migrations_status, blocks tables missing)

Symptom: Blockscout crashes with ssl not available, migrations_status does not exist, blocks does not exist. Migrations fail because Blockscout defaults to ECTO_USE_SSL=TRUE but Docker Postgres has no SSL.

Run on Proxmox host r630-02 (192.168.11.12):

# From project root, copy and run:
./scripts/fix-blockscout-ssl-and-migrations.sh

# Or via SSH:
ssh root@192.168.11.12 'bash -s' < scripts/fix-blockscout-ssl-and-migrations.sh

The script:

1. Stops Blockscout
2. Runs migrations with DATABASE_URL=...?sslmode=disable and ECTO_USE_SSL=false
3. Updates docker-compose/.env to persist SSL-disabled DB URL
4. Starts Blockscout

Manual alternative:

pct exec 5000 -- docker run --rm --network blockscout_blockscout-network \
  -e DATABASE_URL='postgresql://blockscout:blockscout@postgres:5432/blockscout?sslmode=disable' \
  -e ECTO_USE_SSL=false \
  -e ETHEREUM_JSONRPC_HTTP_URL=http://192.168.11.221:8545 \
  -e CHAIN_ID=138 \
  blockscout/blockscout:latest \
  sh -c 'bin/blockscout eval "Elixir.Explorer.ReleaseTasks.create_and_migrate()"'

# Then update /opt/blockscout/docker-compose.yml or .env: add ?sslmode=disable to DATABASE_URL
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d blockscout'

---

Fix: Migrate VM 5000 to thin5 (has free space)

Run on Proxmox host r630-02 (192.168.11.12):

# 1. Stop container
pct stop 5000

# 2. Backup to local storage (VMID 5000 is ~180G used)
vzdump 5000 --storage local --mode stop --compress 0

# 3. Remove old container (frees thin1 space)
pct destroy 5000

# 4. Restore to thin5
pct restore 5000 /var/lib/vz/dump/vzdump-lxc-5000-*.tar.gz --storage thin5

# 5. Start container
pct start 5000

# 6. Start Blockscout stack (wait ~30s for postgres)
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d'

# 7. Wait ~2 min for Blockscout to boot, then verify
curl -s "http://192.168.11.140/api?module=stats&action=eth_price" | head -c 200

---

Alternative: Free Space in thin1

If migration is not possible, free space in thin1 by migrating other VMs off thin1:

# Check what's on thin1
lvs | grep thin1
pvesm status | grep thin1-r630-02

VMs on thin1 (r630-02): 10234, 2201, 2303, 2401, 5000, 6200. Consider migrating smaller VMs to thin5/thin6.

---

After Fix: Verify Contract Verification

source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh

---

Forge Verification Compatibility

Forge verify-contract --verifier blockscout may fail with "Params 'module' and 'action' are required". Blockscout expects module/action in the query; Forge sends JSON only.

Primary: Orchestrated Script (recommended)

Starts proxy if needed; uses config from load-project-env; 600s timeout (set FORGE_VERIFY_TIMEOUT=0 for none):

source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh

Manual: Proxy + Verify

# 1. Start proxy (separate terminal)
BLOCKSCOUT_URL=http://192.168.11.140:4000 node forge-verification-proxy/server.js

# 2. Run verification
./scripts/verify-contracts-blockscout.sh

See: forge-verification-proxy/README.md, BLOCKSCOUT_FORGE_VERIFICATION_EVALUATION.md

Fallbacks

• Nginx fix: ./scripts/fix-blockscout-forge-verification.sh then retry (may still fail due to API format)
• Manual verification: https://explorer.d-bis.org/address/<CONTRACT_ADDRESS>#verify-contract

---

E2E completion (Blockscout and other sites)

• Public routing E2E: bash scripts/verify/verify-end-to-end-routing.sh tests explorer.d-bis.org (DNS, SSL, HTTPS) and an optional Blockscout API check (/api/v2/stats). The API check does not fail the run if unreachable; use SKIP_BLOCKSCOUT_API=1 to skip it. See E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md.
• Full explorer E2E (on LAN): From a host that can reach 192.168.11.140, run explorer-monorepo/scripts/e2e-test-explorer.sh for frontend, API, and service checks.
• Daily checks: scripts/maintenance/daily-weekly-checks.sh daily checks explorer indexer via /api/v2/stats (and fallback legacy API).

---

Proactive: When changing RPC or decommissioning nodes

Explorer (VMID 5000) depends on: RPC at ETHEREUM_JSONRPC_HTTP_URL (canonical: 192.168.11.221:8545, VMID 2201).

When you decommission or change IP of an RPC node that Blockscout might use:

1. Check Blockscout env on VM 5000: pct exec 5000 -- bash -c 'grep -E "ETHEREUM_JSONRPC|RPC" /opt/blockscout/.env 2>/dev/null || docker inspect blockscout 2>/dev/null | grep -A5 Env'
2. If it points to the affected node, update to a live RPC (e.g. 192.168.11.221:8545) and restart Blockscout.
3. See SOLACESCANSCOUT_DEEP_DIVE_FIXES_AND_TIMING.md for full proactive timing.

---

Related

• CONTRACT_DEPLOYMENT_RUNBOOK.md — Contract verification
• scripts/fix-blockscout-1.sh — Diagnostic script
• scripts/fix-blockscout-forge-verification.sh — Forge verification compatibility