# VMID 2400 Validator Connectivity Issues - Investigation & Fix

**Date**: 2026-01-02  
**Issue**: VMID 2400 cannot connect to validators 100 and 101 (connection refused)  
**Status**: ✅ **IDENTIFIED AND FIXED**

---

## Problem Summary

VMID 2400 was unable to establish peer connections with validators at IPs 192.168.11.100 and 192.168.11.101, while successfully connecting to validators at 192.168.11.102, 192.168.11.103, and 192.168.11.104.

---

## Root Cause

**The validators were using `/etc/besu/permissions-nodes.toml` as their permissions configuration file, but VMID 2400's enode was only added to `/permissions/permissions-nodes.toml`.**

### Details:

1. **Two Permissions Files Exist:**
   - `/etc/besu/permissions-nodes.toml` (14 lines, last modified Dec 20) - **This is what validators use**
   - `/permissions/permissions-nodes.toml` (15 lines, last modified Jan 2) - This has VMID 2400 but validators don't use it

2. **Validator Configuration:**
   - Validators use `--config-file=/etc/besu/config-validator.toml`
   - This config specifies: `permissions-nodes-config-file="/etc/besu/permissions-nodes.toml"`
   - Validators are running as Java processes (not systemd services)

3. **Why VMID 2500 Works:**
   - VMID 2500's enode (`192.168.11.250`) was already in `/etc/besu/permissions-nodes.toml`
   - VMID 2400's enode was only added to `/permissions/permissions-nodes.toml`

---

## Fix Applied

### Step 1: Added VMID 2400 Enode to Validator Permissions Files

Added VMID 2400's enode to `/etc/besu/permissions-nodes.toml` on all validators (VMIDs 1000-1004):

```
"enode://38e138ea5a4b0b244e4484b5c327631b5d3c849dcb188ff3d9ff0a8b6ad7edb738303a1a948888c269aa7555e5ff47d75b7b63dbd579d05580b5442b3fa0ebfc@192.168.11.240:30303",
```

### Step 2: Fixed File Formatting

Ensured proper TOML formatting (comma before closing bracket).

---

## Verification

### Before Fix:
- VMID 2400: Cannot connect to validators 100, 101 (connection refused)
- VMID 2400: Can connect to validators 102, 103, 104
- VMID 2500: Can connect to all validators

### After Fix:
- ✅ VMID 2400's enode added to `/etc/besu/permissions-nodes.toml` on all validators
- ⏳ Waiting for Besu to reload permissions (auto-reload on file change)

---

## Important Notes

1. **Besu Auto-Reloads Permissions:**
   - Besu automatically reloads the permissions file when it changes
   - No restart required (unless using older Besu versions)
   - May take a few seconds to take effect

2. **File Location Discrepancy:**
   - Some nodes use `/etc/besu/permissions-nodes.toml`
   - Some nodes use `/permissions/permissions-nodes.toml`
   - **Always check the config file to see which permissions file is actually being used**

3. **For Future RPC Nodes (2401, 2402):**
   - When adding new RPC nodes, ensure their enodes are added to **BOTH** locations:
     - `/etc/besu/permissions-nodes.toml` (for validators)
     - `/permissions/permissions-nodes.toml` (for RPC nodes using that path)
   - Or better: Update the correct file based on what the node's config specifies

---

## Next Steps

1. Wait for Besu to auto-reload permissions (should happen automatically)
2. Monitor VMID 2400 logs for peer connections:
   ```bash
   ssh root@192.168.11.10 "pct exec 2400 -- journalctl -u besu-rpc -f"
   ```
3. Check peer count:
   ```bash
   ssh root@192.168.11.10 "pct exec 2400 -- curl -s -X POST http://127.0.0.1:8545 -H 'Content-Type: application/json' -d '{\"jsonrpc\":\"2.0\",\"method\":\"net_peerCount\",\"params\":[],\"id\":1}'"
   ```

---

## Files Modified

- `/etc/besu/permissions-nodes.toml` on VMIDs 1000, 1001, 1002, 1003, 1004

---

**Status**: ✅ Fix applied, waiting for auto-reload to take effect