171 lines
6.1 KiB
Markdown
171 lines
6.1 KiB
Markdown
# Blockscout Fix Runbook (VMID 5000)
|
|
|
|
**Last Updated:** 2026-02-02
|
|
**Status:** Active
|
|
**Container:** blockscout-1 @ 192.168.11.140 (VMID 5000 on r630-02)
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- **502 Bad Gateway** when accessing `http://192.168.11.140/api` or `https://explorer.d-bis.org/api`
|
|
- **Blockscout logs:** `postgres:5432: non-existing domain - :nxdomain` (DB unreachable)
|
|
- **Docker:** `no space left on device` when pulling/creating containers
|
|
|
|
---
|
|
|
|
## Root Cause
|
|
|
|
1. **Thin pool full:** `thin1-r630-02` is at **100%** capacity. VM 5000 resides on thin1.
|
|
2. **postgres nxdomain:** Blockscout container cannot resolve hostname `postgres` (Docker network/DNS).
|
|
3. Docker cannot create overlay layers when the thin pool has no free space.
|
|
|
|
---
|
|
|
|
## Fix: SSL + Migrations (migrations_status, blocks tables missing)
|
|
|
|
**Symptom:** Blockscout crashes with `ssl not available`, `migrations_status does not exist`, `blocks does not exist`. Migrations fail because Blockscout defaults to `ECTO_USE_SSL=TRUE` but Docker Postgres has no SSL.
|
|
|
|
**Run on Proxmox host r630-02 (192.168.11.12):**
|
|
|
|
```bash
|
|
# From project root, copy and run:
|
|
./scripts/fix-blockscout-ssl-and-migrations.sh
|
|
|
|
# Or via SSH:
|
|
ssh root@192.168.11.12 'bash -s' < scripts/fix-blockscout-ssl-and-migrations.sh
|
|
```
|
|
|
|
The script:
|
|
1. Stops Blockscout
|
|
2. Runs migrations with `DATABASE_URL=...?sslmode=disable` and `ECTO_USE_SSL=false`
|
|
3. Updates docker-compose/.env to persist SSL-disabled DB URL
|
|
4. Starts Blockscout
|
|
|
|
**Manual alternative:**
|
|
|
|
```bash
|
|
pct exec 5000 -- docker run --rm --network blockscout_blockscout-network \
|
|
-e DATABASE_URL='postgresql://blockscout:blockscout@postgres:5432/blockscout?sslmode=disable' \
|
|
-e ECTO_USE_SSL=false \
|
|
-e ETHEREUM_JSONRPC_HTTP_URL=http://192.168.11.221:8545 \
|
|
-e CHAIN_ID=138 \
|
|
blockscout/blockscout:latest \
|
|
sh -c 'bin/blockscout eval "Elixir.Explorer.ReleaseTasks.create_and_migrate()"'
|
|
|
|
# Then update /opt/blockscout/docker-compose.yml or .env: add ?sslmode=disable to DATABASE_URL
|
|
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d blockscout'
|
|
```
|
|
|
|
---
|
|
|
|
## Fix: Migrate VM 5000 to thin5 (has free space)
|
|
|
|
**Run on Proxmox host r630-02 (192.168.11.12):**
|
|
|
|
```bash
|
|
# 1. Stop container
|
|
pct stop 5000
|
|
|
|
# 2. Backup to local storage (VMID 5000 is ~180G used)
|
|
vzdump 5000 --storage local --mode stop --compress 0
|
|
|
|
# 3. Remove old container (frees thin1 space)
|
|
pct destroy 5000
|
|
|
|
# 4. Restore to thin5
|
|
pct restore 5000 /var/lib/vz/dump/vzdump-lxc-5000-*.tar.gz --storage thin5
|
|
|
|
# 5. Start container
|
|
pct start 5000
|
|
|
|
# 6. Start Blockscout stack (wait ~30s for postgres)
|
|
pct exec 5000 -- bash -c 'cd /opt/blockscout && docker-compose up -d'
|
|
|
|
# 7. Wait ~2 min for Blockscout to boot, then verify
|
|
curl -s "http://192.168.11.140/api?module=stats&action=eth_price" | head -c 200
|
|
```
|
|
|
|
---
|
|
|
|
## Alternative: Free Space in thin1
|
|
|
|
If migration is not possible, free space in thin1 by migrating *other* VMs off thin1:
|
|
|
|
```bash
|
|
# Check what's on thin1
|
|
lvs | grep thin1
|
|
pvesm status | grep thin1-r630-02
|
|
```
|
|
|
|
VMs on thin1 (r630-02): 10234, 2201, 2303, 2401, 5000, 6200. Consider migrating smaller VMs to thin5/thin6.
|
|
|
|
---
|
|
|
|
## After Fix: Verify Contract Verification
|
|
|
|
```bash
|
|
source smom-dbis-138/.env 2>/dev/null
|
|
./scripts/verify/run-contract-verification-with-proxy.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Forge Verification Compatibility
|
|
|
|
Forge `verify-contract --verifier blockscout` may fail with "Params 'module' and 'action' are required". Blockscout expects `module`/`action` in the query; Forge sends JSON only.
|
|
|
|
### Primary: Orchestrated Script (recommended)
|
|
|
|
Starts proxy if needed; uses config from load-project-env; 600s timeout (set `FORGE_VERIFY_TIMEOUT=0` for none):
|
|
|
|
```bash
|
|
source smom-dbis-138/.env 2>/dev/null
|
|
./scripts/verify/run-contract-verification-with-proxy.sh
|
|
```
|
|
|
|
### Manual: Proxy + Verify
|
|
|
|
```bash
|
|
# 1. Start proxy (separate terminal)
|
|
BLOCKSCOUT_URL=http://192.168.11.140:4000 node forge-verification-proxy/server.js
|
|
|
|
# 2. Run verification
|
|
./scripts/verify-contracts-blockscout.sh
|
|
```
|
|
|
|
**See:** [forge-verification-proxy/README.md](../../forge-verification-proxy/README.md), [BLOCKSCOUT_FORGE_VERIFICATION_EVALUATION.md](BLOCKSCOUT_FORGE_VERIFICATION_EVALUATION.md)
|
|
|
|
### Fallbacks
|
|
|
|
- **Nginx fix:** `./scripts/fix-blockscout-forge-verification.sh` then retry (may still fail due to API format)
|
|
- **Manual verification:** https://explorer.d-bis.org/address/<CONTRACT_ADDRESS>#verify-contract
|
|
|
|
---
|
|
|
|
## E2E completion (Blockscout and other sites)
|
|
|
|
- **Public routing E2E**: `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` tests explorer.d-bis.org (DNS, SSL, HTTPS) and an optional Blockscout API check (`/api/v2/stats`). The API check does not fail the run if unreachable; use `SKIP_BLOCKSCOUT_API=1` to skip it. See [E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md](../05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md).
|
|
- **Full explorer E2E (on LAN)**: From a host that can reach 192.168.11.140, run `explorer-monorepo/scripts/e2e-test-explorer.sh` for frontend, API, and service checks.
|
|
- **Daily checks**: `scripts/maintenance/daily-weekly-checks.sh daily` checks explorer indexer via `/api/v2/stats` (and fallback legacy API).
|
|
|
|
---
|
|
|
|
## Proactive: When changing RPC or decommissioning nodes
|
|
|
|
**Explorer (VMID 5000) depends on:** RPC at `ETHEREUM_JSONRPC_HTTP_URL` (canonical: 192.168.11.221:8545, VMID 2201).
|
|
|
|
When you **decommission or change IP of an RPC node** that Blockscout might use:
|
|
|
|
1. Check Blockscout env on VM 5000: `pct exec 5000 -- bash -c 'grep -E "ETHEREUM_JSONRPC|RPC" /opt/blockscout/.env 2>/dev/null || docker inspect blockscout 2>/dev/null | grep -A5 Env'`
|
|
2. If it points to the affected node, update to a live RPC (e.g. 192.168.11.221:8545) and restart Blockscout.
|
|
3. See [SOLACESCANSCOUT_DEEP_DIVE_FIXES_AND_TIMING.md](../04-configuration/verification-evidence/SOLACESCANSCOUT_DEEP_DIVE_FIXES_AND_TIMING.md) for full proactive timing.
|
|
|
|
---
|
|
|
|
## Related
|
|
|
|
- [CONTRACT_DEPLOYMENT_RUNBOOK.md](CONTRACT_DEPLOYMENT_RUNBOOK.md) — Contract verification
|
|
- [scripts/fix-blockscout-1.sh](../../scripts/fix-blockscout-1.sh) — Diagnostic script
|
|
- [scripts/fix-blockscout-forge-verification.sh](../../scripts/fix-blockscout-forge-verification.sh) — Forge verification compatibility
|