Block Production Monitoring

Date: $(date)
Status: ⏳ MONITORING FOR BLOCK PRODUCTION

Monitoring Plan

After applying the validator key fix, we need to monitor:

Block Numbers - Should increment from 0
QBFT Consensus Activity - Logs should show block proposal/production
Peer Connections - Nodes should maintain connections
Validator Key Usage - Confirm validators are using correct keys
Errors/Warnings - Check for any issues preventing block production

Expected Behavior

Block Production

✅ Blocks should be produced every 2 seconds (per genesis blockperiodseconds: 2)
✅ Block numbers should increment: 0 → 1 → 2 → 3 ...
✅ All nodes should see the same block numbers (consensus)

QBFT Consensus

✅ Validators should participate in consensus
✅ Logs should show block proposal/production activity
✅ At least 4 out of 5 validators must be online (2/3 quorum)

Network Status

✅ All validators should be connected (5 peers visible)
✅ Sentries should connect to validators
✅ No sync errors or connection issues

Monitoring Commands

Check Block Numbers

for vmid in 1500 1501 1502; do
  block=$(pct exec $vmid -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    -H 'Content-Type: application/json' http://localhost:8545 2>/dev/null | \
    grep -oP '"result":"\K[0-9a-f]+' | head -1)
  block_dec=$(printf '%d' 0x$block 2>/dev/null)
  echo "Sentry $vmid: Block $block_dec"
done

Check QBFT Activity

pct exec 1000 -- journalctl -u besu-validator.service --since '5 minutes ago' --no-pager | \
  grep -iE 'qbft|consensus|propose|producing|block.*produced|imported.*block'

Check Peer Connections

pct exec 1500 -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}' \
  -H 'Content-Type: application/json' http://localhost:8545 | \
  python3 -c "import json, sys; data=json.load(sys.stdin); print(f'Peers: {len(data.get(\"result\", []))}')"

Troubleshooting

If Blocks Are Not Producing

Verify Validator Keys
- Check that /data/besu/key contains validator keys (not node keys)
- Verify addresses match genesis extraData
Check Consensus Status
- Look for QBFT messages in logs
- Verify at least 4/5 validators are online
- Check for consensus errors
Verify Network Connectivity
- All validators should have peer connections
- Check that enode URLs are correct in static-nodes.json
Check Genesis Configuration
- Verify QBFT config in genesis.json
- Confirm validator addresses in extraData match actual keys

Success Criteria

✅ Block Production Working:

Block numbers increment from 0
Blocks produced approximately every 2 seconds
All nodes see same block numbers

✅ QBFT Consensus Active:

Logs show block proposal/production messages
Validators participating in consensus
No consensus errors

✅ Network Stable:

All validators connected
No connection errors
Enode URLs correct

Last Updated: $(date)
Next Check: Monitor block numbers and logs for production activity

3.2 KiB Raw Blame History