proxmox/docs/06-besu/VALIDATOR_KEY_DETAILS.md

Validator Key Count Mismatch - Detailed Analysis

Date: $(date)
Issue: Validator key count mismatch between source and proxmox projects

Current State

Source Project (/home/intlc/projects/smom-dbis-138)

• Validator Keys Found: 4
• Location: keys/validators/
• Key Directories:
  1. validator-1/ (or similar naming)
  2. validator-2/ (or similar naming)
  3. validator-3/ (or similar naming)
  4. validator-4/ (or similar naming)

Proxmox Project (/home/intlc/projects/proxmox/smom-dbis-138-proxmox)

• Validators Expected: 5
• VMID Range: 1000-1004
• Configuration: VALIDATOR_COUNT=5 in config/proxmox.conf
• Inventory Mapping:
  • VMID 1000 → besu-validator-1
  • VMID 1001 → besu-validator-2
  • VMID 1002 → besu-validator-3
  • VMID 1003 → besu-validator-4
  • VMID 1004 → besu-validator-5 ⚠️ MISSING KEY

Impact Analysis

What This Means

1. Deployment Impact:

   • Cannot deploy 5 validators without 5 validator keys
   • Only 4 validators can be deployed if keys are missing
   • Deployment scripts expect 5 validators (VMID 1000-1004)

2. Network Impact:

   • QBFT consensus requires sufficient validators for quorum
   • 5 validators provide better fault tolerance than 4
   • With 5 validators: can tolerate 2 failures (f = (N-1)/3)
   • With 4 validators: can tolerate 1 failure (f = (N-1)/3)

3. Script Impact:

   • scripts/copy-besu-config.sh expects keys for all 5 validators
   • Deployment scripts will fail or skip validator-5 if key is missing
   • Validation scripts may report errors for missing validator-5

Options to Resolve

Option 1: Generate 5th Validator Key (RECOMMENDED)

Pros:

• Better fault tolerance (can tolerate 2 failures vs 1)
• Matches planned deployment architecture
• No configuration changes needed
• Industry standard for production networks

Cons:

• Requires key generation process
• Additional key to manage and secure

Steps:

1. Generate 5th validator key using Besu-compatible method (see Besu Key Management)
2. Store in keys/validators/validator-5/ directory
3. Add validator-5 address to genesis.json alloc if needed
4. Update any key-related scripts if necessary

Key Generation Reference: Hyperledger Besu GitHub | Besu Documentation

Option 2: Reduce Validator Count to 4

Pros:

• No key generation needed
• Uses existing keys
• Faster to deploy

Cons:

• Reduced fault tolerance (1 failure vs 2)
• Requires updating proxmox configuration
• Changes deployment architecture
• Not ideal for production

Steps:

1. Update config/proxmox.conf: VALIDATOR_COUNT=4
2. Update VMID range documentation: 1000-1003 (instead of 1000-1004)
3. Update deployment scripts to exclude VMID 1004
4. Update inventory.example to remove validator-5
5. Update all documentation references

Detailed Configuration References

Proxmox Configuration

File: config/proxmox.conf

VALIDATOR_COUNT=5  # Validators: 1000-1004

File: config/inventory.example

VALIDATOR_besu-validator-1_VMID=1000
VALIDATOR_besu-validator-1_IP=192.168.11.100
VALIDATOR_besu-validator-2_VMID=1001
VALIDATOR_besu-validator-2_IP=192.168.11.101
VALIDATOR_besu-validator-3_VMID=1002
VALIDATOR_besu-validator-3_IP=192.168.11.102
VALIDATOR_besu-validator-4_VMID=1003
VALIDATOR_besu-validator-4_IP=192.168.11.103
VALIDATOR_besu-validator-5_VMID=1004  # ⚠️ KEY MISSING
VALIDATOR_besu-validator-5_IP=192.168.11.104

Script References

Files that expect 5 validators:

• scripts/copy-besu-config.sh: VALIDATORS=(1000 1001 1002 1003 1004)
• scripts/fix-besu-services.sh: VALIDATORS=(1000 1001 1002 1003 1004)
• scripts/validate-besu-config.sh: VALIDATORS=(1000 1001 1002 1003 1004)
• scripts/fix-container-ips.sh: Includes all 5 VMIDs
• scripts/deployment/deploy-besu-nodes.sh: Uses VALIDATOR_COUNT=5

Recommended Solution

Generate 5th Validator Key

Rationale:

1. Production Best Practice: 5 validators is a common production configuration
2. Fault Tolerance: Better resilience (tolerate 2 failures vs 1)
3. Architecture Alignment: Matches planned deployment architecture
4. No Breaking Changes: No need to update existing configuration

Key Generation Process:

1. Using Besu CLI:

   cd /home/intlc/projects/smom-dbis-138
   mkdir -p keys/validators/validator-5
   
   # Generate node key pair
   docker run --rm -v "$(pwd)/keys/validators/validator-5:/keys" \
     hyperledger/besu:latest \
     besu operator generate-blockchain-config \
     --config-file=/keys/config.toml \
     --to=/keys/genesis.json \
     --private-key-file-name=key
   

2. Or using OpenSSL:

   # Generate private key
   openssl ecparam -name secp256k1 -genkey -noout \
     -out keys/validators/validator-5/key.priv
   
   # Extract public key
   openssl ec -in keys/validators/validator-5/key.priv \
     -pubout -out keys/validators/validator-5/key.pub
   

3. Verify Key Structure:

   # Check key files exist
   ls -la keys/validators/validator-5/
   
   # Verify key format (should be hex-encoded)
   head -1 keys/validators/validator-5/key.priv
   

4. Update Genesis.json (if validator address needs pre-allocation):

   • Extract validator address from key
   • Add to alloc section in config/genesis.json

Files That Need Updates (If Generating 5th Key)

• None required if key structure matches existing keys
• Scripts should auto-detect validator-5 directory

Files That Need Updates (If Reducing to 4 Validators)

If choosing Option 2 (reduce to 4 validators), update:

1. config/proxmox.conf: VALIDATOR_COUNT=4
2. config/inventory.example: Remove validator-5 entries
3. All scripts with VALIDATORS=(1000 1001 1002 1003 1004) arrays
4. Documentation referencing 5 validators

Verification

After resolution, verify:

# Check key count matches configuration
KEY_COUNT=$(find keys/validators -mindepth 1 -maxdepth 1 -type d | wc -l)
CONFIG_COUNT=$(grep "^VALIDATOR_COUNT=" config/proxmox.conf | cut -d= -f2)

if [ "$KEY_COUNT" -eq "$CONFIG_COUNT" ]; then
    echo "✅ Validator key count matches configuration: $KEY_COUNT"
else
    echo "⚠️  Mismatch: $KEY_COUNT keys found, $CONFIG_COUNT expected"
fi

Next Steps

1. Decision: Choose Option 1 (generate key) or Option 2 (reduce count)
2. Execute: Perform chosen option
3. Verify: Run verification checks
4. Update: Update documentation if reducing count
5. Deploy: Proceed with deployment