proxmox/docs/03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md

# Blockscout Fix Runbook (VMID 5000)

**Last Updated:** 2026-02-02  
**Status:** Active  
**Container:** blockscout-1 @ 192.168.11.140 (VMID 5000 on r630-02)

---

## Symptoms

- **502 Bad Gateway** when accessing `http://192.168.11.140/api` or `https://explorer.d-bis.org/api`
- **Blockscout logs:** `postgres:5432: non-existing domain - :nxdomain` (DB unreachable)
- **Docker:** `no space left on device` when pulling/creating containers

---

## Root Cause

1. **Thin pool full:** `thin1-r630-02` is at **100%** capacity. VM 5000 resides on thin1.
2. **postgres nxdomain:** Blockscout container cannot resolve hostname `postgres` (Docker network/DNS).
3. Docker cannot create overlay layers when the thin pool has no free space.

---

## Fix: SSL + Migrations (migrations_status, blocks tables missing)

**Symptom:** Blockscout crashes with `ssl not available`, `migrations_status does not exist`, `blocks does not exist`. Migrations fail because Blockscout defaults to `ECTO_USE_SSL=TRUE` but Docker Postgres has no SSL.

**Run on Proxmox host r630-02 (192.168.11.12):**

```bash
# From project root, copy and run:
./scripts/fix-blockscout-ssl-and-migrations.sh

# Or via SSH:
ssh root@192.168.11.12 'bash -s' < scripts/fix-blockscout-ssl-and-migrations.sh
```

The script:
1. Stops Blockscout
2. Runs migrations with `DATABASE_URL=...?sslmode=disable` and `ECTO_USE_SSL=false`
3. Updates docker-compose/.env to persist SSL-disabled DB URL
4. Starts Blockscout

**Manual alternative:**

```bash
pct exec 5000 -- docker run --rm --network blockscout_blockscout-network \
  -e DATABASE_URL='postgresql://blockscout:blockscout@postgres:5432/blockscout?sslmode=disable' \
  -e ECTO_USE_SSL=false \
  -e ETHEREUM_JSONRPC_HTTP_URL=http://192.168.11.221:8545 \
  -e CHAIN_ID=138 \
  blockscout/blockscout:latest \
  sh -c 'bin/blockscout eval "Elixir.Explorer.ReleaseTasks.create_and_migrate()"'

# Then update /opt/blockscout/docker-compose.yml or .env: add ?sslmode=disable to DATABASE_URL
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d blockscout'
```

---

## Fix: Migrate VM 5000 to thin5 (has free space)

**Run on Proxmox host r630-02 (192.168.11.12):**

```bash
# 1. Stop container
pct stop 5000

# 2. Backup to local storage (VMID 5000 is ~180G used)
vzdump 5000 --storage local --mode stop --compress 0

# 3. Remove old container (frees thin1 space)
pct destroy 5000

# 4. Restore to thin5
pct restore 5000 /var/lib/vz/dump/vzdump-lxc-5000-*.tar.gz --storage thin5

# 5. Start container
pct start 5000

# 6. Start Blockscout stack (wait ~30s for postgres)
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d'

# 7. Wait ~2 min for Blockscout to boot, then verify
curl -s "http://192.168.11.140/api?module=stats&action=eth_price" | head -c 200
```

---

## Alternative: Free Space in thin1

If migration is not possible, free space in thin1 by migrating *other* VMs off thin1:

```bash
# Check what's on thin1
lvs | grep thin1
pvesm status | grep thin1-r630-02
```

VMs on thin1 (r630-02): 10234, 2201, 2303, 2401, 5000, 6200. Consider migrating smaller VMs to thin5/thin6.

---

## After Fix: Verify Contract Verification

```bash
source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh
```

---

## Forge Verification Compatibility

Forge `verify-contract --verifier blockscout` may fail with "Params 'module' and 'action' are required". Blockscout expects `module`/`action` in the query; Forge sends JSON only.

### Primary: Orchestrated Script (recommended)

Starts proxy if needed; uses config from load-project-env; 600s timeout (set `FORGE_VERIFY_TIMEOUT=0` for none):

```bash
source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh
```

### Manual: Proxy + Verify

```bash
# 1. Start proxy (separate terminal)
BLOCKSCOUT_URL=http://192.168.11.140:4000 node forge-verification-proxy/server.js

# 2. Run verification
./scripts/verify-contracts-blockscout.sh
```

**See:** [forge-verification-proxy/README.md](../../forge-verification-proxy/README.md), [BLOCKSCOUT_FORGE_VERIFICATION_EVALUATION.md](BLOCKSCOUT_FORGE_VERIFICATION_EVALUATION.md)

### Fallbacks

- **Nginx fix:** `./scripts/fix-blockscout-forge-verification.sh` then retry (may still fail due to API format)
- **Manual verification:** https://explorer.d-bis.org/address/<CONTRACT_ADDRESS>#verify-contract

---

## E2E completion (Blockscout and other sites)

- **Public routing E2E**: `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` tests explorer.d-bis.org (DNS, SSL, HTTPS) and an optional Blockscout API check (`/api/v2/stats`). The API check does not fail the run if unreachable; use `SKIP_BLOCKSCOUT_API=1` to skip it. See [E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md](../05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md).
- **Full explorer E2E (on LAN)**: From a host that can reach 192.168.11.140, run `explorer-monorepo/scripts/e2e-test-explorer.sh` for frontend, API, and service checks.
- **Daily checks**: `scripts/maintenance/daily-weekly-checks.sh daily` checks explorer indexer via `/api/v2/stats` (and fallback legacy API).

---

## Proactive: When changing RPC or decommissioning nodes

**Explorer (VMID 5000) depends on:** RPC at `ETHEREUM_JSONRPC_HTTP_URL` (canonical: 192.168.11.221:8545, VMID 2201).

When you **decommission or change IP of an RPC node** that Blockscout might use:

1. Check Blockscout env on VM 5000: `pct exec 5000 -- bash -c 'grep -E "ETHEREUM_JSONRPC|RPC" /opt/blockscout/.env 2>/dev/null || docker inspect blockscout 2>/dev/null | grep -A5 Env'`
2. If it points to the affected node, update to a live RPC (e.g. 192.168.11.221:8545) and restart Blockscout.
3. See [SOLACESCANSCOUT_DEEP_DIVE_FIXES_AND_TIMING.md](../04-configuration/verification-evidence/SOLACESCANSCOUT_DEEP_DIVE_FIXES_AND_TIMING.md) for full proactive timing.

---

## Related

- [CONTRACT_DEPLOYMENT_RUNBOOK.md](CONTRACT_DEPLOYMENT_RUNBOOK.md) — Contract verification
- [scripts/fix-blockscout-1.sh](../../scripts/fix-blockscout-1.sh) — Diagnostic script
- [scripts/fix-blockscout-forge-verification.sh](../../scripts/fix-blockscout-forge-verification.sh) — Forge verification compatibility