# Block Production Monitoring

**Date**: $(date)  
**Status**: ⏳ **MONITORING FOR BLOCK PRODUCTION**

---

## Monitoring Plan

After applying the validator key fix, we need to monitor:

1. **Block Numbers** - Should increment from 0
2. **QBFT Consensus Activity** - Logs should show block proposal/production
3. **Peer Connections** - Nodes should maintain connections
4. **Validator Key Usage** - Confirm validators are using correct keys
5. **Errors/Warnings** - Check for any issues preventing block production

---

## Expected Behavior

### Block Production
- ✅ Blocks should be produced every **2 seconds** (per genesis `blockperiodseconds: 2`)
- ✅ Block numbers should increment: 0 → 1 → 2 → 3 ...
- ✅ All nodes should see the same block numbers (consensus)

### QBFT Consensus
- ✅ Validators should participate in consensus
- ✅ Logs should show block proposal/production activity
- ✅ At least 4 out of 5 validators must be online (2/3 quorum)

### Network Status
- ✅ All validators should be connected (5 peers visible)
- ✅ Sentries should connect to validators
- ✅ No sync errors or connection issues

---

## Monitoring Commands

### Check Block Numbers
```bash
for vmid in 1500 1501 1502; do
  block=$(pct exec $vmid -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    -H 'Content-Type: application/json' http://localhost:8545 2>/dev/null | \
    grep -oP '"result":"\K[0-9a-f]+' | head -1)
  block_dec=$(printf '%d' 0x$block 2>/dev/null)
  echo "Sentry $vmid: Block $block_dec"
done
```

### Check QBFT Activity
```bash
pct exec 1000 -- journalctl -u besu-validator.service --since '5 minutes ago' --no-pager | \
  grep -iE 'qbft|consensus|propose|producing|block.*produced|imported.*block'
```

### Check Peer Connections
```bash
pct exec 1500 -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}' \
  -H 'Content-Type: application/json' http://localhost:8545 | \
  python3 -c "import json, sys; data=json.load(sys.stdin); print(f'Peers: {len(data.get(\"result\", []))}')"
```

---

## Troubleshooting

### If Blocks Are Not Producing

1. **Verify Validator Keys**
   - Check that `/data/besu/key` contains validator keys (not node keys)
   - Verify addresses match genesis extraData

2. **Check Consensus Status**
   - Look for QBFT messages in logs
   - Verify at least 4/5 validators are online
   - Check for consensus errors

3. **Verify Network Connectivity**
   - All validators should have peer connections
   - Check that enode URLs are correct in static-nodes.json

4. **Check Genesis Configuration**
   - Verify QBFT config in genesis.json
   - Confirm validator addresses in extraData match actual keys

---

## Success Criteria

✅ **Block Production Working:**
- Block numbers increment from 0
- Blocks produced approximately every 2 seconds
- All nodes see same block numbers

✅ **QBFT Consensus Active:**
- Logs show block proposal/production messages
- Validators participating in consensus
- No consensus errors

✅ **Network Stable:**
- All validators connected
- No connection errors
- Enode URLs correct

---

**Last Updated**: $(date)  
**Next Check**: Monitor block numbers and logs for production activity

