proxmox/docs/archive/completion/RPC_TROUBLESHOOTING_COMPLETE.md

RPC Troubleshooting - Complete Summary

Date: $(date)
Issue: RPC-01 (VMID 2500) troubleshooting and resolution

---

🔍 Issue Identified

RPC-01 (VMID 2500) was experiencing multiple issues preventing proper operation:

1. Missing Configuration File: Service expected /etc/besu/config-rpc.toml but only config-rpc-public.toml existed
2. Service File Mismatch: Service file referenced wrong config file name
3. Database Corruption: Corrupted DATABASE_METADATA.json file preventing startup
4. Missing Required Files: Genesis, static-nodes, and permissions files in wrong locations
5. Database Directory Missing: /data/besu/database/ directory did not exist

---

✅ Resolution Steps Taken

1. Configuration File Fix

Problem: Service expected config-rpc.toml but only config-rpc-public.toml existed

Solution:

pct exec 2500 -- cp /etc/besu/config-rpc-public.toml /etc/besu/config-rpc.toml
pct exec 2500 -- chown besu:besu /etc/besu/config-rpc.toml

2. Service File Update

Problem: Service file referenced config-rpc-public.toml instead of config-rpc.toml

Solution:

pct exec 2500 -- sed -i 's|config-rpc-public.toml|config-rpc.toml|g' /etc/systemd/system/besu-rpc.service
pct exec 2500 -- systemctl daemon-reload

3. Database Corruption Fix

Problem: Corrupted DATABASE_METADATA.json causing startup failures

Solution:

pct exec 2500 -- systemctl stop besu-rpc.service
pct exec 2500 -- rm -f /data/besu/DATABASE_METADATA.json
pct exec 2500 -- rm -rf /data/besu/database/*

4. Required Files Setup

Problem: Genesis, static-nodes, and permissions files in /etc/besu/ but config expects /genesis/ and /permissions/

Solution:

# Create directories
pct exec 2500 -- mkdir -p /genesis /permissions

# Copy files
pct exec 2500 -- cp /etc/besu/genesis.json /genesis/
pct exec 2500 -- cp /etc/besu/static-nodes.json /genesis/
pct exec 2500 -- cp /etc/besu/permissions-nodes.toml /permissions/

# Set ownership
pct exec 2500 -- chown -R besu:besu /genesis /permissions

5. Database Directory Creation

Problem: Database directory missing

Solution:

pct exec 2500 -- mkdir -p /data/besu/database
pct exec 2500 -- chown -R besu:besu /data/besu

6. Service Restart

Solution:

pct exec 2500 -- systemctl start besu-rpc.service

---

✅ Verification Results

Service Status

• Status: ✅ Active (running)
• Process: ✅ Besu process running (PID 327821)
• Uptime: Stable since restart

Network Ports

• 8545 (HTTP RPC): ✅ Listening
• 8546 (WebSocket RPC): ✅ Listening
• 30303 (P2P): ✅ Listening
• 9545 (Metrics): ✅ Listening

Network Connectivity

• RPC Endpoint: ✅ Responding
• Chain ID: ✅ 138 (correct)
• Block Production: ✅ Active (syncing blocks)
• Peers: ✅ Connected to 5 peers

Block Sync Status

• Current Block: > 11,200 (at time of fix)
• Sync Status: ✅ Actively syncing
• Import Rate: Processing blocks successfully

---

🛠️ Tools Created

1. Troubleshooting Script

File: scripts/troubleshoot-rpc-2500.sh

Features:

• Container status check
• Network configuration verification
• Service status check
• Configuration file validation
• Required files check
• Port listening check
• RPC endpoint test
• Process verification
• Error log analysis

2. Fix Script

File: scripts/fix-rpc-2500.sh

Features:

• Automated configuration file creation
• Deprecated option removal
• Service file update
• Required files setup
• Service restart
• Verification

3. Documentation

Files Created:

• docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING.md - Complete guide
• docs/09-troubleshooting/RPC_2500_QUICK_FIX.md - Quick reference
• docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING_SUMMARY.md - Summary

---

📋 Configuration Fixes Applied

Template Updates

File: smom-dbis-138-proxmox/templates/besu-configs/config-rpc.toml

Changes:

• ✅ Removed log-destination (deprecated)
• ✅ Removed max-remote-initiated-connections (deprecated)
• ✅ Removed trie-logs-enabled (deprecated)
• ✅ Removed accounts-enabled (deprecated)
• ✅ Removed database-path (deprecated)
• ✅ Removed rpc-http-host-allowlist (deprecated)

Installation Script Updates

File: smom-dbis-138-proxmox/install/besu-rpc-install.sh

Changes:

• ✅ Changed service to use config-rpc.toml (not config-rpc-public.toml)
• ✅ Updated template file name
• ✅ Removed deprecated options from template
• ✅ Fixed file paths (/genesis/ instead of /etc/besu/)

---

✅ Current Status

RPC-01 (VMID 2500): ✅ FULLY OPERATIONAL

• Service: Active and stable
• Network: Connected and syncing
• RPC: Accessible and responding
• All ports: Listening correctly

---

🔄 Next Steps

Immediate

1. ✅ RPC-01 fixed and operational
2. ⏳ Verify RPC-02 (VMID 2501) status
3. ⏳ Verify RPC-03 (VMID 2502) status

Short-term

1. Apply same fixes to RPC-02 and RPC-03 if needed
2. Verify all RPC nodes are in sync
3. Test load balancing across RPC nodes

---

📚 Related Documentation

• RPC 2500 Troubleshooting Guide
• RPC 2500 Quick Fix
• Deployment Readiness

---

Resolution Date: $(date)
Status: ✅ RESOLVED