proxmox/docs/04-configuration/RPC_TESTING_FINAL_REPORT.md

RPC Node Testing - Final Report

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-17
Test Type: Comprehensive Full Testing

---

Executive Summary

Current Status

• Containers Running: 11/12 (2301 has startup error)
• RPC Responding: 2/12 nodes (2101, 2201)
• Block Synchronization: ✅ Perfect (0 block difference between responding nodes)
• Peer Connections: 2 nodes with peers
• Chain ID: ✅ All nodes on Chain 138 (0x8a)

Issues Identified

1. Most nodes not responding to RPC - Services may still be starting or have configuration issues
2. Container 2301 startup failure - Requires investigation
3. Services restarting repeatedly - Some nodes showing high restart counts (49, 58, 63, 66785!)
4. Permissioning errors - VMID 2305 showed "not in nodes-allowlist" error (fixed by redeployment)

---

Detailed Test Results

Test 1: Container Status

VMID	Status	Name
2101	✅ Running	besu-rpc-core-1
2201	✅ Running	besu-rpc-public-1
2301	❌ Stopped	besu-rpc-private-1 (startup error)
2303	✅ Running	besu-rpc-ali-0x8a
2304	✅ Running	besu-rpc-ali-0x1
2305	✅ Running	besu-rpc-luis-0x8a
2306	✅ Running	besu-rpc-luis-0x1
2307	✅ Running	besu-rpc-putu-0x8a
2308	✅ Running	besu-rpc-putu-0x1
2401	✅ Running	besu-rpc-thirdweb-0x8a-1
2402	✅ Running	besu-rpc-thirdweb-0x8a-2
2403	✅ Running	besu-rpc-thirdweb-0x8a-3

Result: 11/12 containers running

---

Test 2: RPC Connectivity

Responding Nodes (2/12)

• ✅ 2101 (192.168.11.211): Block 1135876
• ✅ 2201 (192.168.11.221): Block 1135876

Non-Responding Nodes (10/12)

• ❌ 2301 (192.168.11.232) - Container stopped
• ❌ 2303 (192.168.11.233) - Service starting
• ❌ 2304 (192.168.11.234) - Service starting
• ❌ 2305 (192.168.11.235) - Service restarting (permissioning error)
• ❌ 2306 (192.168.11.236) - Service starting
• ❌ 2307 (192.168.11.237) - Service restarting
• ❌ 2308 (192.168.11.238) - Service restarting
• ❌ 2401 (192.168.11.241) - Service restarting (66785 restarts!)
• ❌ 2402 (192.168.11.242) - Service starting
• ❌ 2403 (192.168.11.243) - Service starting

Result: 2/12 nodes responding

---

Test 3: Block Synchronization

Responding Nodes Block Numbers:

• 2101: Block 1135876
• 2201: Block 1135876

Synchronization Status: ✅ Perfect

• Block difference: 0
• Both nodes are at the same block height

Note: Once other nodes come online, they will need to sync to the current block height.

---

Test 4: Peer Connections

• 2101: 1 peer connected ✅
• 2201: 1 peer connected ✅
• Other nodes: 0 peers (not responding to RPC)

Result: 2 nodes with peer connections

---

Test 5: Sync Status

• 2101: ✅ Synced
• 2201: ✅ Synced
• Other nodes: Unknown (not responding)

Result: 2/12 nodes synced

---

Test 6: Chain ID Verification

• 2101: Chain ID 0x8a (138) ✅
• 2201: Chain ID 0x8a (138) ✅
• Other nodes: Could not verify (not responding)

Result: All responding nodes on correct chain

---

Critical Issues Found

1. Container 2301 Startup Failure

Error: lxc_init: 845 Failed to run lxc.hook.pre-start

Impact: Container cannot start

Action Required:

• Investigate pre-start hook
• Check container configuration
• May need to fix or recreate container

---

2. Services Restarting Repeatedly

Affected Nodes: 2304, 2305, 2307, 2308, 2401, 2402, 2403

Restart Counts Observed:

• 2304: 49 restarts
• 2305: Permissioning error (fixed)
• 2307: 63 restarts
• 2308: 58 restarts
• 2401: 66,785 restarts! ⚠️ CRITICAL

Possible Causes:

• Configuration errors
• Permissioning issues (partially fixed)
• Missing files
• Port conflicts

Action Required:

• Investigate VMID 2401 (extremely high restart count)
• Check Besu logs for all restarting services
• Verify all configuration files are correct

---

3. RPC Services Not Responding

Status: 10/12 nodes not responding to RPC calls

Possible Causes:

• Services still starting (Besu can take 5-10 minutes)
• Services crashing and restarting
• Configuration issues preventing startup
• Port binding issues

Action Required:

• Wait longer for services to fully start
• Check logs for startup errors
• Verify network configuration

---

Configuration Files Status

static-nodes.json

• ✅ Status: Valid with 15 unique enodes
• ✅ Deployed: To all 11 running nodes
• ✅ Uniqueness: All enode IDs verified unique

permissions-nodes.toml

• ✅ Status: Valid with 15 enodes (matches static-nodes.json)
• ✅ Deployed: To all 11 running nodes
• ✅ Match: All enodes from static-nodes.json are in permissions-nodes.toml

---

Working Nodes

Fully Operational (2/12)

1. 2101 (192.168.11.211) - besu-rpc-core-1

   • ✅ Container running
   • ✅ RPC responding
   • ✅ Block: 1135876
   • ✅ 1 peer connected
   • ✅ Synced
   • ✅ Chain ID: 0x8a

2. 2201 (192.168.11.221) - besu-rpc-public-1

   • ✅ Container running
   • ✅ RPC responding
   • ✅ Block: 1135876
   • ✅ 1 peer connected
   • ✅ Synced
   • ✅ Chain ID: 0x8a

---

Recommendations

Immediate Actions

1. Wait for Services to Start

   • Besu services can take 5-10 minutes to fully start
   • Monitor services and retest after 10-15 minutes

2. Investigate VMID 2401

   • Extremely high restart count (66,785) indicates critical issue
   • Check logs for root cause
   • May need to stop service and investigate configuration

3. Fix Container 2301

   • Investigate pre-start hook error
   • Check container configuration
   • May need to recreate container

4. Monitor Service Logs

   • Check logs for all restarting services
   • Look for configuration errors
   • Verify all required files exist

Follow-up Testing

1. Wait 10-15 minutes for all services to fully start
2. Re-run comprehensive test: bash scripts/test-rpc-nodes-complete.sh 192.168.11.10
3. Verify block synchronization across all responding nodes
4. Check peer connections improve as nodes come online

---

Test Scripts

• Comprehensive Test: /home/intlc/projects/proxmox/scripts/test-rpc-nodes-complete.sh
• Enode Verification: /home/intlc/projects/proxmox/scripts/verify-unique-enodes.sh

---

Conclusion

Current State: 2/12 nodes fully operational and synchronized

Next Steps:

1. Wait for services to fully start (10-15 minutes)
2. Investigate and fix restarting services
3. Fix container 2301 startup issue
4. Retest after fixes

Expected Outcome: Once all services are running, all 12 nodes should be responding and synchronized.

---

Last Updated: 2026-01-17