proxmox/docs/09-troubleshooting/QBFT_TROUBLESHOOTING.md

QBFT Consensus Troubleshooting

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-12-20
Issue: Blocks not being produced despite validators being connected

Current Status

✅ What's Working

• All validator keys deployed correctly
• Validator addresses match genesis extraData
• Network connectivity is good (10 peers connected)
• Services are running
• Genesis extraData is correct (5 validator addresses in QBFT format)
• QBFT configuration present in genesis (blockperiodseconds: 2, epochlength: 30000)
• RPC now enabled on validators (with QBFT API)

❌ What's Not Working

• No blocks being produced (still at block 0)
• No QBFT consensus activity in logs
• Validators are looking for "sync targets" instead of producing blocks
• No QBFT-specific log messages (no "proposing block", "QBFT consensus", etc.)

Root Cause Analysis

The critical observation: Validators are trying to sync from peers instead of producing blocks.

In QBFT:

• Validators should produce blocks (not sync from others)
• Non-validators sync from validators
• If validators are looking for sync targets, they don't recognize themselves as validators

Configuration Verified

Genesis Configuration ✅

{
  "config": {
    "qbft": {
      "blockperiodseconds": 2,
      "epochlength": 30000,
      "requesttimeoutseconds": 10
    }
  },
  "extraData": "0xf88fa00000000000000000000000000000000000000000000000000000000000000000f869941c25c54bf177ecf9365445706d8b9209e8f1c39b94c4c1aeeb5ab86c6179fc98220b51844b749354469422f37f6faaa353e652a0840f485e71a7e5a8937394573ff6d00d2bdc0d9c0c08615dc052db75f825749411563e26a70ed3605b80a03081be52aca9e0f141c080c0"
}

Contains 5 validator addresses:

1. 0x1c25c54bf177ecf9365445706d8b9209e8f1c39b
2. 0xc4c1aeeb5ab86c6179fc98220b51844b74935446
3. 0x22f37f6faaa353e652a0840f485e71a7e5a89373
4. 0x573ff6d00d2bdc0d9c0c08615dc052db75f82574
5. 0x11563e26a70ed3605b80a03081be52aca9e0f141

Validator Configuration ✅

• miner-enabled=false (correct for QBFT)
• sync-mode="FULL" (correct)
• Validator keys present at /keys/validators/validator-*/
• Node key at /data/besu/key matches validator key
• RPC enabled with QBFT API

Possible Issues

1. Besu Not Recognizing QBFT Consensus

• Symptom: No QBFT log messages, trying to sync instead of produce
• Possible cause: Besu may not be detecting QBFT from genesis
• Check: Look for consensus engine initialization in logs

2. Validator Address Mismatch

• Status: ✅ Verified - addresses match
• All validator addresses in logs match extraData

3. Missing Validator Key Configuration

• Status: ⚠️ Unknown
• Besu should auto-detect validators from genesis extraData
• But config file has no explicit validator key path

4. Network Synchronization Issue

• Status: ✅ Verified - peers connected
• All validators can see each other (10 peers each)

Next Steps

1. Check QBFT Validator Set: Query qbft_getValidatorsByBlockNumber via RPC to see if validators are recognized
2. Check Consensus Engine: Verify Besu is actually using QBFT consensus engine
3. Review Besu Documentation: Check if there's a required configuration option for QBFT validators
4. Check Logs for Errors: Look for any silent failures in consensus initialization

Applied Fixes

1. ✅ Enabled RPC on validators with QBFT API
2. ✅ Verified all validator keys and addresses
3. ✅ Confirmed genesis extraData is correct
4. ✅ Verified network connectivity

Status

Still investigating - Validators are connected but not producing blocks. The lack of QBFT consensus activity in logs suggests Besu may not be recognizing this as a QBFT network or the nodes as validators.