Files
proxmox/docs/08-monitoring/MONITORING_SUMMARY.md

3.5 KiB

Block Production Monitoring Summary

Date: $(date)
Status: MONITORING IN PROGRESS - Validators Still Looking for Sync Targets


Current Status

Completed

  • Validator Keys: All 5 validators using correct validator keys
  • Addresses Match: All validator addresses match genesis.json extraData
  • Services Running: All 5 validator services active
  • Configuration Updated: static-nodes.json and permissions-nodes.toml updated

⚠️ Current Issue

  • Still at Block 0: No blocks being produced
  • Looking for Sync Targets: All validators showing "Unable to find sync target. Currently checking 4 peers for usefulness"
  • No QBFT Activity: No consensus/block production messages in logs

Observations

Key Finding

Even after replacing node keys with validator keys, validators are still:

  1. Looking for sync targets (trying to sync from other nodes)
  2. Not recognizing themselves as validators that should produce blocks
  3. No QBFT consensus activity in logs

Validator Status

  • All 5 validators running
  • All using validator keys (verified addresses match)
  • All checking 4 peers (network connectivity working)
  • None producing blocks
  • None showing QBFT consensus activity

Network Status

  • Services active but RPC not fully responsive yet
  • Peer connections established (4 peers visible)
  • No sync targets found (validators trying to sync instead of produce)

Potential Issues

1. Besu Not Recognizing Validators

For QBFT with dynamic validators, Besu may need additional configuration to recognize nodes as validators. The fact that they're looking for "sync targets" suggests they think they need to sync, not produce.

2. Genesis Configuration

The genesis file uses dynamic validators (no static validators array). Initial validators come from extraData. But Besu may need explicit configuration to use these validators.

3. Sync Mode

Current config has sync-mode="FULL". For QBFT validators, this may need to be different, or validators shouldn't be trying to sync at all.


Next Steps to Investigate

  1. Verify Genesis Configuration

    • Check if QBFT needs validators explicitly listed (even for dynamic validators)
    • Verify extraData format is correct for QBFT
  2. Research QBFT Dynamic Validator Setup

    • Check if Besu needs additional configuration for dynamic validators
    • Verify if validators need special configuration to enable block production
  3. Check Sync Mode Configuration

    • For QBFT validators, sync mode may need adjustment
    • Validators shouldn't be looking for sync targets
  4. Monitor Longer

    • Allow more time for network to stabilize
    • Continue monitoring logs for QBFT activity

Monitoring Results

Block Numbers

  • All nodes still at block 0
  • No block production detected

QBFT Activity

  • No consensus messages in logs
  • No block proposal/production activity
  • Validators stuck in "looking for sync target" state

Peer Connections

  • 4 peers visible to each validator
  • Network connectivity working
  • But no useful sync targets found

Conclusion

The validator key fix was correct and necessary, but there appears to be an additional configuration issue preventing Besu from recognizing these nodes as validators that should produce blocks.

The network is connected and validators have the correct keys, but they're still operating in "sync" mode rather than "produce" mode.


Last Updated: $(date)
Next Action: Investigate QBFT dynamic validator configuration requirements