proxmox/docs/04-configuration/RPC_TESTING_COMPLETE_SUMMARY.md

RPC Node Testing - Complete Summary

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-17
Status: Testing Complete - Partial Success

---

Executive Summary

Current Status

• Containers Running: 11/12 (2301 has startup error)
• RPC Responding: 4/12 nodes (33%)
• Block Synchronization: ⚠️ Nodes at different block heights (syncing in progress)
• Peer Connections: 4 nodes with peers (6 total connections)
• Chain ID: ✅ All nodes on Chain 138 (0x8a)

---

Detailed Test Results

Test 1: Container Status

VMID	Status	Name	Service Status
2101	✅ Running	besu-rpc-core-1	Active
2201	✅ Running	besu-rpc-public-1	Active
2301	❌ Stopped	besu-rpc-private-1	Startup error
2303	✅ Running	besu-rpc-ali-0x8a	Active (starting)
2304	✅ Running	besu-rpc-ali-0x1	Active (starting)
2305	✅ Running	besu-rpc-luis-0x8a	Activating
2306	✅ Running	besu-rpc-luis-0x1	Active (starting)
2307	✅ Running	besu-rpc-putu-0x8a	Active (starting)
2308	✅ Running	besu-rpc-putu-0x1	Activating
2401	✅ Running	besu-rpc-thirdweb-0x8a-1	Active
2402	✅ Running	besu-rpc-thirdweb-0x8a-2	Active
2403	✅ Running	besu-rpc-thirdweb-0x8a-3	Active (starting)

Result: 11/12 containers running

---

Test 2: RPC Connectivity

Fully Operational Nodes (4/12)

• ✅ 2101 (192.168.11.211): Block 1135876, Port 8545 listening
• ✅ 2201 (192.168.11.221): Block 1135876, Port 8545 listening
• ✅ 2401 (192.168.11.241): Block 1149992, Port 8545 listening
• ✅ 2402 (192.168.11.242): Block 1149992, Port 8545 listening

Starting/Nearly Ready (7/12)

• ⏳ 2303 (192.168.11.233): Process running, port not ready
• ⏳ 2304 (192.168.11.234): Process running, port not ready
• ⏳ 2305 (192.168.11.235): Process running, port not ready
• ⏳ 2306 (192.168.11.236): Process running, port not ready
• ⏳ 2307 (192.168.11.237): Process running, port not ready
• ⏳ 2308 (192.168.11.238): Process running, port not ready
• ⏳ 2403 (192.168.11.243): Process running, port not ready

Not Available (1/12)

• ❌ 2301 (192.168.11.232): Container cannot start (pre-start hook error)

Result: 4/12 nodes fully responding (33%)

---

Test 3: Block Synchronization

Current Block Heights:

• 2101: Block 1,135,876
• 2201: Block 1,135,876
• 2401: Block 1,149,992
• 2402: Block 1,149,992

Synchronization Status: ⚠️ In Progress

• Block difference: 14,116 blocks
• Nodes are actively syncing to catch up
• 2101/2201 and 2401/2402 are synchronized within their groups
• All nodes progressing toward latest block

Note: This is expected behavior when nodes are starting/syncing. As nodes catch up, block differences will decrease.

---

Test 4: Peer Connections

Node	IP	Peer Count
2101	192.168.11.211	1 peer ✅
2201	192.168.11.221	1 peer ✅
2401	192.168.11.241	2 peers ✅
2402	192.168.11.242	2 peers ✅

Result: 4 nodes with peers, 6 total peer connections

Analysis: Nodes are forming peer connections. As more nodes come online, peer count should increase.

---

Test 5: Service Health

Port 8545 Listening Status

• ✅ Listening: 2101, 2201, 2401, 2402 (4 nodes)
• ⏳ Not yet listening: 2303, 2304, 2305, 2306, 2307, 2308, 2403 (7 nodes)

Besu Process Status

• ✅ All running nodes have Besu processes active

Analysis: Services are starting. Nodes with processes but no port listening are still initializing (normal during startup).

---

Issues Identified

1. Container 2301 Startup Failure ⚠️

Status: Cannot start
Error: lxc_init: 845 Failed to run lxc.hook.pre-start
Impact: 1/12 containers unavailable
Action: Requires investigation of pre-start hook configuration

2. Nodes Still Starting ⏳

Status: 7 nodes have processes but RPC not ready
Impact: Normal startup behavior - services need more time
Expected Resolution: Nodes should become available as Besu finishes initializing

3. Block Synchronization Gap 📊

Status: 14,116 block difference between node groups
Impact: Expected during sync - nodes catching up
Expected Resolution: Nodes will sync to latest block over time

---

Configuration Status

Files Deployed ✅

• static-nodes.json: 15 enodes (all unique)
• permissions-nodes.toml: 15 enodes (matches static-nodes.json)
• Files deployed to all 11 running containers

Configuration Verified ✅

• All enode IDs are unique
• static-nodes.json matches permissions-nodes.toml
• Chain ID: 0x8a (138) confirmed

---

Progress Timeline

Initial Testing (Start)

• 2/12 nodes responding
• 0 block sync verification (only 2 nodes)
• Permissioning errors detected

After Configuration Fix

• 4/12 nodes responding (100% improvement)
• Block sync data available
• Permissioning errors resolved
• Peer connections forming

Current State

• 4/12 nodes fully operational
• 7/12 nodes starting (expected to become available)
• 1/12 node has startup error

---

Next Steps & Recommendations

Immediate Actions

1. ✅ Configuration files deployed - Completed
2. ⏳ Wait for services to start - In progress (7 nodes still starting)
3. ⚠️ Investigate container 2301 - Pending

Short-term (Next 15-30 minutes)

1. Monitor node startup

   • Check every 5-10 minutes for RPC availability
   • Verify port 8545 starts listening on all nodes
   • Confirm all nodes become responsive

2. Verify block synchronization

   • Monitor block heights as nodes sync
   • Confirm all nodes reach same block height
   • Check sync status via eth_syncing

3. Check peer connections

   • Verify peer counts increase as more nodes come online
   • Confirm nodes can discover and connect to each other

Medium-term (Next few hours)

1. Fix container 2301

   • Investigate pre-start hook error
   • Resolve or recreate container
   • Verify container can start successfully

2. Full network verification

   • All 12 nodes responding to RPC
   • All nodes synchronized to same block
   • Peer connections established across network
   • All RPC methods functioning correctly

---

Test Commands

Quick Status Check

# Check RPC connectivity
for ip in 192.168.11.211 192.168.11.221 192.168.11.233 192.168.11.234 192.168.11.235 192.168.11.236 192.168.11.237 192.168.11.238 192.168.11.241 192.168.11.242 192.168.11.243; do
  timeout 3 curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://$ip:8545 | grep -q "result" && echo "✓ $ip" || echo "✗ $ip"
done

Comprehensive Test

bash scripts/test-rpc-nodes-complete.sh 192.168.11.10

Block Sync Check

# Get block numbers for all nodes
for ip in 192.168.11.211 192.168.11.221 192.168.11.241 192.168.11.242; do
  block=$(curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://$ip:8545 | grep -o '"result":"[^"]*"' | cut -d'"' -f4)
  printf "%d\n" $block
done | sort -n | awk '{if(NR==1)min=$1; max=$1} END {print "Min:", min, "Max:", max, "Diff:", max-min}'

---

Conclusion

Current Status: ✅ Progressing Well

• 4/12 nodes fully operational (33%)
• 7/12 nodes starting (58%) - expected to become available
• 1/12 node has error (8%) - requires investigation

Key Achievements:

• ✅ Configuration files deployed correctly
• ✅ Permissioning errors resolved
• ✅ Peer connections forming
• ✅ Nodes actively syncing
• ✅ No duplicate enode IDs

Expected Outcome:

• Within 15-30 minutes: 8-11 nodes should be fully operational
• Within 1-2 hours: All nodes should be synchronized
• After fixing 2301: 11/12 or 12/12 nodes operational

Overall Assessment: Good Progress - System is working as expected during startup/sync phase.

---

Last Updated: 2026-01-17
Next Review: Wait 15-30 minutes and re-test