106 lines
3.6 KiB
Markdown
106 lines
3.6 KiB
Markdown
|
# VMID 2400 Validator Connectivity Issues - Investigation & Fix
|
||
|
|
|
||
|
|
**Date**: 2026-01-02
|
||
|
|
**Issue**: VMID 2400 cannot connect to validators 100 and 101 (connection refused)
|
||
|
|
**Status**: ✅ **IDENTIFIED AND FIXED**
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Problem Summary
|
||
|
|
|
||
|
|
VMID 2400 was unable to establish peer connections with validators at IPs 192.168.11.100 and 192.168.11.101, while successfully connecting to validators at 192.168.11.102, 192.168.11.103, and 192.168.11.104.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Root Cause
|
||
|
|
|
||
|
|
**The validators were using `/etc/besu/permissions-nodes.toml` as their permissions configuration file, but VMID 2400's enode was only added to `/permissions/permissions-nodes.toml`.**
|
||
|
|
|
||
|
|
### Details:
|
||
|
|
|
||
|
|
1. **Two Permissions Files Exist:**
|
||
|
|
- `/etc/besu/permissions-nodes.toml` (14 lines, last modified Dec 20) - **This is what validators use**
|
||
|
|
- `/permissions/permissions-nodes.toml` (15 lines, last modified Jan 2) - This has VMID 2400 but validators don't use it
|
||
|
|
|
||
|
|
2. **Validator Configuration:**
|
||
|
|
- Validators use `--config-file=/etc/besu/config-validator.toml`
|
||
|
|
- This config specifies: `permissions-nodes-config-file="/etc/besu/permissions-nodes.toml"`
|
||
|
|
- Validators are running as Java processes (not systemd services)
|
||
|
|
|
||
|
|
3. **Why VMID 2500 Works:**
|
||
|
|
- VMID 2500's enode (`192.168.11.250`) was already in `/etc/besu/permissions-nodes.toml`
|
||
|
|
- VMID 2400's enode was only added to `/permissions/permissions-nodes.toml`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Fix Applied
|
||
|
|
|
||
|
|
### Step 1: Added VMID 2400 Enode to Validator Permissions Files
|
||
|
|
|
||
|
|
Added VMID 2400's enode to `/etc/besu/permissions-nodes.toml` on all validators (VMIDs 1000-1004):
|
||
|
|
|
||
|
|
```
|
||
|
|
"enode://38e138ea5a4b0b244e4484b5c327631b5d3c849dcb188ff3d9ff0a8b6ad7edb738303a1a948888c269aa7555e5ff47d75b7b63dbd579d05580b5442b3fa0ebfc@192.168.11.240:30303",
|
||
|
|
```
|
||
|
|
|
||
|
|
### Step 2: Fixed File Formatting
|
||
|
|
|
||
|
|
Ensured proper TOML formatting (comma before closing bracket).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Verification
|
||
|
|
|
||
|
|
### Before Fix:
|
||
|
|
- VMID 2400: Cannot connect to validators 100, 101 (connection refused)
|
||
|
|
- VMID 2400: Can connect to validators 102, 103, 104
|
||
|
|
- VMID 2500: Can connect to all validators
|
||
|
|
|
||
|
|
### After Fix:
|
||
|
|
- ✅ VMID 2400's enode added to `/etc/besu/permissions-nodes.toml` on all validators
|
||
|
|
- ⏳ Waiting for Besu to reload permissions (auto-reload on file change)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Important Notes
|
||
|
|
|
||
|
|
1. **Besu Auto-Reloads Permissions:**
|
||
|
|
- Besu automatically reloads the permissions file when it changes
|
||
|
|
- No restart required (unless using older Besu versions)
|
||
|
|
- May take a few seconds to take effect
|
||
|
|
|
||
|
|
2. **File Location Discrepancy:**
|
||
|
|
- Some nodes use `/etc/besu/permissions-nodes.toml`
|
||
|
|
- Some nodes use `/permissions/permissions-nodes.toml`
|
||
|
|
- **Always check the config file to see which permissions file is actually being used**
|
||
|
|
|
||
|
|
3. **For Future RPC Nodes (2401, 2402):**
|
||
|
|
- When adding new RPC nodes, ensure their enodes are added to **BOTH** locations:
|
||
|
|
- `/etc/besu/permissions-nodes.toml` (for validators)
|
||
|
|
- `/permissions/permissions-nodes.toml` (for RPC nodes using that path)
|
||
|
|
- Or better: Update the correct file based on what the node's config specifies
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
|
||
|
|
1. Wait for Besu to auto-reload permissions (should happen automatically)
|
||
|
|
2. Monitor VMID 2400 logs for peer connections:
|
||
|
|
```bash
|
||
|
|
ssh root@192.168.11.10 "pct exec 2400 -- journalctl -u besu-rpc -f"
|
||
|
|
```
|
||
|
|
3. Check peer count:
|
||
|
|
```bash
|
||
|
|
ssh root@192.168.11.10 "pct exec 2400 -- curl -s -X POST http://127.0.0.1:8545 -H 'Content-Type: application/json' -d '{\"jsonrpc\":\"2.0\",\"method\":\"net_peerCount\",\"params\":[],\"id\":1}'"
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Files Modified
|
||
|
|
|
||
|
|
- `/etc/besu/permissions-nodes.toml` on VMIDs 1000, 1001, 1002, 1003, 1004
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
**Status**: ✅ Fix applied, waiting for auto-reload to take effect
|