proxmox/reports/VMID2400_VALIDATOR_CONNECTIVITY_FIX.md

VMID 2400 Validator Connectivity Issues - Investigation & Fix

Date: 2026-01-02
Issue: VMID 2400 cannot connect to validators 100 and 101 (connection refused)
Status: ✅ IDENTIFIED AND FIXED

---

Problem Summary

VMID 2400 was unable to establish peer connections with validators at IPs 192.168.11.100 and 192.168.11.101, while successfully connecting to validators at 192.168.11.102, 192.168.11.103, and 192.168.11.104.

---

Root Cause

The validators were using /etc/besu/permissions-nodes.toml as their permissions configuration file, but VMID 2400's enode was only added to /permissions/permissions-nodes.toml.

Details:

1. Two Permissions Files Exist:

   • /etc/besu/permissions-nodes.toml (14 lines, last modified Dec 20) - This is what validators use
   • /permissions/permissions-nodes.toml (15 lines, last modified Jan 2) - This has VMID 2400 but validators don't use it

2. Validator Configuration:

   • Validators use --config-file=/etc/besu/config-validator.toml
   • This config specifies: permissions-nodes-config-file="/etc/besu/permissions-nodes.toml"
   • Validators are running as Java processes (not systemd services)

3. Why VMID 2500 Works:

   • VMID 2500's enode (192.168.11.250) was already in /etc/besu/permissions-nodes.toml
   • VMID 2400's enode was only added to /permissions/permissions-nodes.toml

---

Fix Applied

Step 1: Added VMID 2400 Enode to Validator Permissions Files

Added VMID 2400's enode to /etc/besu/permissions-nodes.toml on all validators (VMIDs 1000-1004):

"enode://38e138ea5a4b0b244e4484b5c327631b5d3c849dcb188ff3d9ff0a8b6ad7edb738303a1a948888c269aa7555e5ff47d75b7b63dbd579d05580b5442b3fa0ebfc@192.168.11.240:30303",

Step 2: Fixed File Formatting

Ensured proper TOML formatting (comma before closing bracket).

---

Verification

Before Fix:

• VMID 2400: Cannot connect to validators 100, 101 (connection refused)
• VMID 2400: Can connect to validators 102, 103, 104
• VMID 2500: Can connect to all validators

After Fix:

• ✅ VMID 2400's enode added to /etc/besu/permissions-nodes.toml on all validators
• ⏳ Waiting for Besu to reload permissions (auto-reload on file change)

---

Important Notes

1. Besu Auto-Reloads Permissions:

   • Besu automatically reloads the permissions file when it changes
   • No restart required (unless using older Besu versions)
   • May take a few seconds to take effect

2. File Location Discrepancy:

   • Some nodes use /etc/besu/permissions-nodes.toml
   • Some nodes use /permissions/permissions-nodes.toml
   • Always check the config file to see which permissions file is actually being used

3. For Future RPC Nodes (2401, 2402):

   • When adding new RPC nodes, ensure their enodes are added to BOTH locations:
     • /etc/besu/permissions-nodes.toml (for validators)
     • /permissions/permissions-nodes.toml (for RPC nodes using that path)
   • Or better: Update the correct file based on what the node's config specifies

---

Next Steps

1. Wait for Besu to auto-reload permissions (should happen automatically)
2. Monitor VMID 2400 logs for peer connections:

   ssh root@192.168.11.10 "pct exec 2400 -- journalctl -u besu-rpc -f"
   

3. Check peer count:

   ssh root@192.168.11.10 "pct exec 2400 -- curl -s -X POST http://127.0.0.1:8545 -H 'Content-Type: application/json' -d '{\"jsonrpc\":\"2.0\",\"method\":\"net_peerCount\",\"params\":[],\"id\":1}'"
   

---

Files Modified

• /etc/besu/permissions-nodes.toml on VMIDs 1000, 1001, 1002, 1003, 1004

---

Status: ✅ Fix applied, waiting for auto-reload to take effect