explorer-monorepo/docs/BLOCKSCOUT_INITIALIZATION_FIX.md

# Blockscout Initialization Fix for VMID 5000

## Problem Summary

Blockscout container is crashing on startup due to:

1. **Uninitialized Database**: Migrations were never run, so critical tables don't exist
2. **Missing Static Assets**: `cache_manifest.json` not found (assets never built/digested)
3. **Incorrect Startup Command**: Docker image defaults to shell instead of starting Blockscout

## Root Cause

- Database migrations (`mix ecto.migrate`) were never executed
- Static assets (`mix phx.digest`) were never built
- Docker container needs explicit `bin/blockscout start` command

---

## Quick Fix Commands

### For Root User in VMID 5000

Run these commands from Proxmox host or inside VMID 5000:

```bash
# ============================================================
# STEP 1: Access Blockscout Container
# ============================================================

# Find Blockscout container
BLOCKSCOUT_CONTAINER=$(docker ps -a | grep blockscout | grep -v postgres | awk '{print $1}' | head -1)

# ============================================================
# STEP 2: Run Database Migrations
# ============================================================

# Option A: Using Release.migrate (recommended)
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout eval "Explorer.Release.migrate()"

# Option B: Using mix ecto.migrate
docker exec -it $BLOCKSCOUT_CONTAINER mix ecto.migrate

# Option C: Using blockscout migrate command
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout migrate

# ============================================================
# STEP 3: Build Static Assets
# ============================================================

# Build and digest assets
docker exec -it $BLOCKSCOUT_CONTAINER mix phx.digest

# Or if that fails, try:
docker exec -it $BLOCKSCOUT_CONTAINER npm run deploy

# ============================================================
# STEP 4: Restart with Correct Command
# ============================================================

# Stop current container
docker stop $BLOCKSCOUT_CONTAINER
docker rm $BLOCKSCOUT_CONTAINER

# Restart with proper command (update docker-compose.yml first)
cd /opt/blockscout
docker compose up -d blockscout
```

---

## Detailed Fix Procedure

### Step 1: Verify Current Status

```bash
# Check container status
docker ps -a | grep blockscout

# Check recent logs
docker logs blockscout 2>&1 | tail -50

# Check for crash dumps
ls -la /tmp/erl_crash.dump 2>/dev/null || echo "No crash dump found"
```

### Step 2: Run Database Migrations

The database user is `blockscout` (not `postgres`). Migrations will create all required tables:

```bash
BLOCKSCOUT_CONTAINER=$(docker ps -a | grep blockscout | grep -v postgres | awk '{print $1}' | head -1)

# Run migrations
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout eval "Explorer.Release.migrate()"
```

**Expected Output:**
```
[info] Running migrations...
[info] == Running Explorer.Repo.Migrations.CreateBlocks.up/0 forward
[info] create table blocks
[info] == Running Explorer.Repo.Migrations.CreateTransactions.up/0 forward
...
```

**Verify Tables Created:**
```bash
# Check critical tables exist
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout eval "
  tables = [\"blocks\", \"transactions\", \"migrations_status\", \"addresses\", \"smart_contracts\"]
  for table <- tables do
    case Explorer.Repo.query(\"SELECT 1 FROM information_schema.tables WHERE table_name = '\#{table}'\") do
      {:ok, %{rows: []}} -> IO.puts(\"❌ Table '\#{table}' MISSING\")
      {:ok, %{rows: [_]}} -> IO.puts(\"✅ Table '\#{table}' exists\")
    end
  end
"
```

### Step 3: Build Static Assets

```bash
# Build and digest assets
docker exec -it $BLOCKSCOUT_CONTAINER mix phx.digest
```

**Verify Assets:**
```bash
# Check for manifest
docker exec -it $BLOCKSCOUT_CONTAINER ls -la priv/static/cache_manifest.json

# Should show:
# -rw-r--r-- 1 root root XXXX ... cache_manifest.json
```

### Step 4: Update Docker Compose Configuration

Edit `/opt/blockscout/docker-compose.yml` to ensure Blockscout starts correctly:

```yaml
services:
  blockscout:
    image: blockscout/blockscout:latest
    command: bin/blockscout start  # Add this line
    environment:
      - DATABASE_URL=postgresql://blockscout:${DB_PASSWORD}@postgres:5432/blockscout
      # ... other environment variables
```

Or add the command via sed:

```bash
cd /opt/blockscout
sed -i '/blockscout:/a\    command: bin/blockscout start' docker-compose.yml
```

### Step 5: Restart Blockscout

```bash
cd /opt/blockscout

# Stop and remove old container
docker compose down blockscout

# Start with new configuration
docker compose up -d blockscout

# Monitor startup
docker logs -f blockscout
```

---

## Complete One-Line Fix (From Proxmox Host)

```bash
pct exec 5000 -- bash -c '
BLOCKSCOUT_CONTAINER=$(docker ps -a | grep blockscout | grep -v postgres | awk "{print \$1}" | head -1)
echo "Running migrations..."
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout eval "Explorer.Release.migrate()"
echo "Building assets..."
docker exec -it $BLOCKSCOUT_CONTAINER mix phx.digest
echo "Restarting Blockscout..."
cd /opt/blockscout && docker compose restart blockscout
'
```

---

## Verification

After running the fix, verify everything is working:

```bash
# 1. Check container is running
docker ps | grep blockscout

# 2. Check logs for errors
docker logs blockscout 2>&1 | tail -30

# 3. Verify database tables
docker exec -it blockscout bin/blockscout eval "
  case Explorer.Repo.query(\"SELECT COUNT(*) FROM blocks LIMIT 1\") do
    {:ok, _} -> IO.puts(\"✅ Database accessible\")
    error -> IO.puts(\"❌ Database error: #{inspect(error)}\")
  end
"

# 4. Check assets
docker exec -it blockscout test -f priv/static/cache_manifest.json && \
  echo "✅ Assets built" || echo "❌ Assets missing"

# 5. Test HTTP endpoint
curl -s http://localhost:4000/api/v2/stats | head -20
```

---

## Troubleshooting

### Migrations Fail

**Error**: `relation "schema_migrations" does not exist`

**Fix**: Create schema_migrations table manually:
```bash
docker exec -it $BLOCKSCOUT_CONTAINER bin/blockscout eval "
  Explorer.Repo.query(\"CREATE TABLE IF NOT EXISTS schema_migrations (version bigint PRIMARY KEY, inserted_at timestamp)\")
"
```

### Assets Build Fails

**Error**: `npm: command not found` or `mix phx.digest` fails

**Fix**: Install dependencies first:
```bash
docker exec -it $BLOCKSCOUT_CONTAINER mix deps.get
docker exec -it $BLOCKSCOUT_CONTAINER npm install --prefix apps/block_scout_web/assets
docker exec -it $BLOCKSCOUT_CONTAINER mix phx.digest
```

### Container Still Crashes

**Check logs**:
```bash
docker logs blockscout 2>&1 | grep -i error | tail -20
```

**Common issues**:
- Database connection failed → Check `DATABASE_URL` environment variable
- Missing environment variables → Check `.env` file
- Port conflict → Check if port 4000 is already in use

---

## Prevention

To prevent this issue in the future:

1. **Always run migrations on first startup**:
   ```yaml
   command: sh -c "bin/blockscout eval 'Explorer.Release.migrate()' && bin/blockscout start"
   ```

2. **Build assets in Dockerfile** or use init container

3. **Use health checks** to verify Blockscout is ready:
   ```yaml
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:4000/api/v2/stats"]
     interval: 30s
     timeout: 10s
     retries: 3
   ```

---

## Database Credentials

**Blockscout Database** (used by Blockscout application):
- User: `blockscout`
- Password: `blockscout`
- Database: `blockscout`

These credentials are set in the `blockscout-postgres` Docker container environment variables.

**Note**: The explorer backend API uses a **separate database** (`explorer`/`explorer`/`changeme`).

## References

- Blockscout Release Migration: `Explorer.Release.migrate()`
- Phoenix Asset Digest: `mix phx.digest`
- Blockscout Startup: `bin/blockscout start`
- Database User: `blockscout` (not `postgres`)
- Database Credentials: See `docs/BLOCKSCOUT_DATABASE_CREDENTIALS.md`