proxmox/docs/06-besu/VALIDATOR_MISSING_ISSUE.md

Validator Missing Issue - Complete Analysis

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Status: 🔍 ISSUE IDENTIFIED

---

Problem Summary

Validators 1000-1002 are not found on Proxmox host 192.168.11.10, but documentation indicates they should exist.

---

Investigation Results

Validators Found

• Validator-1003 (VMID 1003): ✅ Running, Service active
• Validator-1004 (VMID 1004): ✅ Running, Service active

Validators NOT Found

• Validator-1000 (VMID 1000): ❌ Not found on Proxmox host
• Validator-1001 (VMID 1001): ❌ Not found on Proxmox host
• Validator-1002 (VMID 1002): ❌ Not found on Proxmox host

Error Message

Configuration file 'nodes/ml110/lxc/1000.conf' does not exist

This suggests Proxmox is looking for config files in a specific path that doesn't exist.

---

Possible Explanations

1. Validators on Different Proxmox Host

• Validators 1000-1002 may be on a different Proxmox node
• Documentation shows they should have IPs 192.168.11.100-102
• Need to check other Proxmox hosts in the cluster

2. Validators Were Removed

• Validators may have been deleted/migrated
• VMIDs may have been reused for other purposes
• Network may have been reconfigured

3. Validators Not Yet Deployed

• Validators may not have been created yet
• Only validators 1003-1004 were deployed
• Network may have been running with fewer validators

4. Proxmox Configuration Path Issue

• Config files may be in different location
• Proxmox may use different storage backend
• Path nodes/ml110/lxc/ may not be correct

---

Impact on Network

QBFT Consensus Requirements

• Typical Requirement: Majority of validators (N/2 + 1)
• For 5 Validators: Need at least 3 validators running
• Current Status: Only 2/5 validators available
• Result: Block production stopped

Current Network State

• Block Production: ❌ Stopped (stuck at 1145367)
• Consensus: ⚠️ Cannot reach majority
• Transactions: ⏳ Pending, cannot confirm
• Deployments: ⏳ Blocked by lack of block production

---

Solutions

Option 1: Locate Validators on Different Host

If validators exist elsewhere:

1. Identify which Proxmox host has validators 1000-1002
2. Start validators on that host
3. Verify they join the network
4. Confirm block production resumes

Option 2: Deploy Missing Validators

If validators don't exist:

1. Create validators 1000-1002 on Proxmox host 192.168.11.10
2. Configure them with appropriate IPs (192.168.11.100-102)
3. Set up Besu validator services
4. Join them to the existing network
5. Wait for sync and consensus

Option 3: Modify QBFT Configuration

If network must function with 2 validators:

1. Check QBFT genesis configuration
2. Modify fault tolerance settings (if possible)
3. Update validator list to only 2 validators
4. Restart network with new configuration
5. Note: This may require network restart/regenesis

Option 4: Check IP Connectivity

Test if validators are reachable by IP:

1. Ping validator IPs (192.168.11.100-102)
2. Check if RPC ports respond
3. Verify if validators are running but not visible in Proxmox
4. May indicate validators are on different infrastructure

---

Recommended Approach

Immediate Actions

1. ✅ Check validator IP addresses for connectivity
2. ✅ Verify validators 1003-1004 are fully operational
3. ⏳ Check if validators 1000-1002 exist on different Proxmox host
4. ⏳ Determine if validators need to be deployed

If Validators Don't Exist

1. Deploy validators 1000-1002
2. Configure with IPs 192.168.11.100-102
3. Join to existing network
4. Wait for sync
5. Verify block production resumes

If Validators Exist Elsewhere

1. Access correct Proxmox host
2. Start validators
3. Verify network connectivity
4. Confirm they join consensus
5. Monitor block production

---

Verification Steps

Check Validator IPs

# Ping validator IPs
ping -c 1 192.168.11.100
ping -c 1 192.168.11.101
ping -c 1 192.168.11.102

# Check if RPC responds
cast block-number --rpc-url http://192.168.11.100:8545

Check Proxmox Hosts

# List all containers on current host
ssh root@192.168.11.10 "pct list"

# Check other Proxmox hosts if known
# ssh root@<other-host> "pct list | grep validator"

Check Network Status

# Verify block production
cast block-number --rpc-url http://192.168.11.211:8545

# Monitor for advancement
watch -n 5 "cast block-number --rpc-url http://192.168.11.211:8545"

---

Next Steps

1. ✅ Check validator IP connectivity
2. ✅ Verify validators 1003-1004 status
3. ⏳ Determine location of validators 1000-1002
4. ⏳ Start/deploy missing validators
5. ⏳ Verify block production resumes
6. ⏳ Complete transaction confirmations
7. ⏳ Finish remaining deployments

---

Status: Validators 1000-1002 missing, investigation in progress
Priority: Locate or deploy missing validators to resume block production
Impact: Block production stopped, transactions cannot confirm