proxmox/docs/archive/BLOCK_PRODUCTION_STATUS.md

Block Production and Sync Status Investigation

Date: $(date)
Status: ⚠️ INVESTIGATING - Blocks not being produced, nodes not connecting

---

Current Status

Block Production

• ❌ Not producing blocks - All nodes at block 0 (genesis)
• ❌ No block activity - No blocks have been produced since genesis

Sync Status

• ❌ Not synced - All nodes show 0 peers
• ❌ Not connecting - Nodes cannot find each other

Issues Identified

1. ✅ FIXED: Validator Keys Missing

   • Problem: All validators were missing validator keys
   • Solution: Copied keys from /opt/smom-dbis-138/keys/validators/ to /keys/validators/validator-{N}/ on all validators
   • Status: ✅ Keys now present on all 5 validators

2. ⚠️ IN PROGRESS: p2p-host Configuration

   • Problem: All nodes have p2p-host="0.0.0.0" which causes enode URLs to show @0.0.0.0:30303
   • Impact: Nodes cannot connect because enode URLs are invalid
   • Solution: Update p2p-host to use actual IP addresses (192.168.11.100-104 for validators, etc.)
   • Status: ⏳ Fixing now

3. ⏳ TO INVESTIGATE: Peer Discovery

   • Nodes have 0 peers even after fixes
   • Need to verify static-nodes.json matches actual enode URLs
   • May need to regenerate static-nodes.json with correct IPs

---

Actions Taken

1. Copied Validator Keys

• ✅ Copied validator keys from /opt/smom-dbis-138/keys/validators/ to all validators
• ✅ Keys now in /keys/validators/validator-{1-5}/ on VMIDs 1000-1004
• ✅ Ownership set to besu:besu
• ✅ Permissions set correctly (600 for private keys)

2. Updated p2p-host Configuration

• ⏳ Updating p2p-host from 0.0.0.0 to actual IP addresses
• ⏳ Validators: 192.168.11.100-104
• ⏳ Sentries: 192.168.11.150-153
• ⏳ RPC Nodes: 192.168.11.250-252

3. Restarted Services

• ⏳ All services restarted with new configuration

---

Next Steps

1. Verify p2p-host Fix

   • Check enode URLs show correct IP addresses (not 0.0.0.0)
   • Verify nodes can now connect to each other

2. Check Peer Connections

   • Monitor peer counts - should increase from 0
   • Check logs for successful peer connections

3. Verify Block Production

   • With QBFT, validators should start producing blocks once connected
   • Check for block production in logs
   • Verify block numbers increase from 0

4. If Still Not Working

   • Regenerate static-nodes.json with correct enode URLs
   • Verify permissions-nodes.toml matches static-nodes.json
   • Check firewall rules for port 30303
   • Verify QBFT validator key configuration

---

Key Findings

Validator Keys

• ✅ All 5 validators now have keys
• ✅ Keys in correct location: /keys/validators/validator-{N}/
• ✅ Key files present: key.priv, key.pem, pubkey.pem, address.txt

Network Configuration

• ✅ Network connectivity works (ping successful between nodes)
• ❌ P2P port 30303 binding issue (using 0.0.0.0 instead of actual IP)
• ⏳ Fixing p2p-host configuration

Consensus Configuration

• ✅ QBFT configured in genesis.json
• ✅ Block period: 2 seconds
• ✅ Epoch length: 30000 blocks
• ⏳ Need to verify validators can produce blocks

---

Last Updated: $(date)