proxmox/docs/06-besu/BESU_ALLOWLIST_RUNBOOK.md

Hyperledger Besu Node Allowlist Generation Runbook

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Purpose: Generate, validate, and deploy correct Besu node allowlists using only Besu-native commands.

Scope: Private LAN network (192.168.11.0/24), QBFT consensus, strict permissions.

---

Prerequisites

• Besu installed on all nodes (/opt/besu/bin/besu or in PATH)
• JSON-RPC enabled with ADMIN API (for verification) OR nodekey files accessible
• SSH access to all nodes OR local execution on each node
• jq and python3 for JSON processing (optional but recommended)

---

1. Node-Level Enode Extraction (Execute on EACH Node)

1.1 Method A: Using Besu CLI (Nodekey File)

When to use: Node is not running, or RPC is disabled.

#!/bin/bash
# Execute on each Besu node
# File: extract-enode-from-nodekey.sh

set -euo pipefail

# Configuration
DATA_PATH="${DATA_PATH:-/data/besu}"
BESU_BIN="${BESU_BIN:-/opt/besu/bin/besu}"
NODE_IP="${NODE_IP:-}"  # Set to actual node IP (e.g., 192.168.11.13)
P2P_PORT="${P2P_PORT:-30303}"

# Find nodekey file
NODEKEY_FILE=""
for path in "${DATA_PATH}/key" "${DATA_PATH}/nodekey" "/keys/besu/nodekey"; do
    if [[ -f "$path" ]]; then
        NODEKEY_FILE="$path"
        break
    fi
done

if [[ -z "$NODEKEY_FILE" ]]; then
    echo "ERROR: Nodekey file not found in ${DATA_PATH}/key, ${DATA_PATH}/nodekey, or /keys/besu/nodekey"
    exit 1
fi

echo "Found nodekey: $NODEKEY_FILE"

# Generate enode using Besu CLI
if [[ -n "$NODE_IP" ]]; then
    # If IP is provided, generate enode and replace IP
    ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="${NODEKEY_FILE}" --format=enode 2>/dev/null | sed "s/@[0-9.]*:/@${NODE_IP}:/")
else
    # Use IP from generated enode (may be 0.0.0.0 or localhost)
    ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="${NODEKEY_FILE}" --format=enode 2>/dev/null)
fi

if [[ -z "$ENODE" ]]; then
    echo "ERROR: Failed to generate enode from nodekey"
    exit 1
fi

# Extract and validate node ID length
NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
NODE_ID_LEN=${#NODE_ID}

if [[ "$NODE_ID_LEN" -ne 128 ]]; then
    echo "ERROR: Invalid node ID length: $NODE_ID_LEN (expected 128)"
    echo "Node ID: ${NODE_ID:0:32}...${NODE_ID: -32}"
    exit 1
fi

# Validate hex format
if ! echo "$NODE_ID" | grep -qE '^[0-9a-f]{128}$'; then
    echo "ERROR: Node ID contains invalid hex characters"
    exit 1
fi

echo "✓ Valid enode extracted:"
echo "$ENODE"
echo ""
echo "Node ID length: $NODE_ID_LEN (✓)"
echo "Node ID (first 32 chars): ${NODE_ID:0:32}..."
echo "Node ID (last 32 chars): ...${NODE_ID: -32}"

Usage on each node:

# On node 192.168.11.13
export NODE_IP="192.168.11.13"
export DATA_PATH="/data/besu"
bash extract-enode-from-nodekey.sh > enode-192.168.11.13.txt

---

1.2 Method B: Using JSON-RPC (Running Node)

When to use: Node is running and RPC is enabled with ADMIN API.

#!/bin/bash
# Execute on each Besu node OR from central management host
# File: extract-enode-from-rpc.sh

set -euo pipefail

# Configuration
RPC_URL="${RPC_URL:-http://localhost:8545}"
NODE_IP="${NODE_IP:-}"  # Optional: verify IP matches

# Get node info via JSON-RPC
RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
    "${RPC_URL}")

# Check for errors
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
    ERROR_MSG=$(echo "$RESPONSE" | jq -r '.error.message')
    echo "ERROR: JSON-RPC error: $ERROR_MSG"
    echo "NOTE: Ensure RPC is enabled with --rpc-http-api=ADMIN,NET"
    exit 1
fi

# Extract enode
ENODE=$(echo "$RESPONSE" | jq -r '.result.enode // empty')

if [[ -z "$ENODE" ]] || [[ "$ENODE" == "null" ]]; then
    echo "ERROR: Could not extract enode from admin_nodeInfo"
    echo "Response: $RESPONSE"
    exit 1
fi

# Extract and validate node ID
NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
NODE_ID_LEN=${#NODE_ID}

if [[ "$NODE_ID_LEN" -ne 128 ]]; then
    echo "ERROR: Invalid node ID length: $NODE_ID_LEN (expected 128)"
    echo "Enode: $ENODE"
    exit 1
fi

# Extract IP and port
ENODE_IP=$(echo "$ENODE" | sed 's|.*@||' | cut -d':' -f1)
ENODE_PORT=$(echo "$ENODE" | sed 's|.*:||')

# Verify IP if provided
if [[ -n "$NODE_IP" ]] && [[ "$ENODE_IP" != "$NODE_IP" ]]; then
    echo "WARNING: Enode IP ($ENODE_IP) does not match expected IP ($NODE_IP)"
    echo "NOTE: Check --p2p-host and --nat-method configuration"
fi

echo "✓ Valid enode extracted via RPC:"
echo "$ENODE"
echo ""
echo "Node ID length: $NODE_ID_LEN (✓)"
echo "Advertised IP: $ENODE_IP"
echo "Advertised Port: $ENODE_PORT"

Usage:

# From management host
export RPC_URL="http://192.168.11.13:8545"
export NODE_IP="192.168.11.13"
bash extract-enode-from-rpc.sh > enode-192.168.11.13.txt

---

2. Automated Collection Script (All Nodes)

2.1 Complete Collection Script

#!/bin/bash
# Collect enodes from all nodes and generate allowlist
# File: collect-all-enodes.sh

set -euo pipefail

# Configuration
WORK_DIR="${WORK_DIR:-./besu-enodes-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$WORK_DIR"

# Node inventory (IP -> RPC_PORT mapping)
# Format: IP:RPC_PORT:USE_RPC (use_rpc=1 if RPC available, 0 to use nodekey method)
declare -A NODES=(
    ["192.168.11.13"]="8545:1"  # validator-1 (RPC may be disabled)
    ["192.168.11.14"]="8545:1"  # validator-2
    ["192.168.11.15"]="8545:1"  # validator-3
    ["192.168.11.16"]="8545:1"  # validator-4
    ["192.168.11.18"]="8545:1"  # validator-5
    ["192.168.11.19"]="8545:1"  # sentry-2
    ["192.168.11.20"]="8545:1"  # sentry-3
    ["192.168.11.21"]="8545:1"  # sentry-4
    ["192.168.11.22"]="8545:1"  # sentry-5
    ["192.168.11.23"]="8545:1"  # rpc-1 (RPC definitely enabled)
    ["192.168.11.24"]="8545:1"  # rpc-2
    ["192.168.11.25"]="8545:1"  # rpc-3
)

# SSH configuration (if accessing remote nodes)
SSH_USER="${SSH_USER:-root}"
SSH_OPTS="${SSH_OPTS:--o StrictHostKeyChecking=accept-new}"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log_info() { echo -e "${BLUE}[INFO]${NC} $1"; }
log_success() { echo -e "${GREEN}[✓]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARNING]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }

# Validate enode format
validate_enode() {
    local enode="$1"
    local node_id
    
    node_id=$(echo "$enode" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
    
    if [[ ${#node_id} -ne 128 ]]; then
        log_error "Invalid node ID length: ${#node_id} (expected 128)"
        return 1
    fi
    
    if ! echo "$node_id" | grep -qE '^[0-9a-f]{128}$'; then
        log_error "Node ID contains invalid hex characters"
        return 1
    fi
    
    return 0
}

# Extract enode via RPC
extract_via_rpc() {
    local ip="$1"
    local rpc_port="$2"
    local rpc_url="http://${ip}:${rpc_port}"
    
    log_info "Attempting RPC extraction from $rpc_url..."
    
    local response
    response=$(curl -s -m 5 -X POST \
        -H "Content-Type: application/json" \
        --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
        "${rpc_url}" 2>/dev/null || echo "")
    
    if [[ -z "$response" ]]; then
        return 1
    fi
    
    # Check for JSON-RPC error
    if echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); sys.exit(0 if 'error' not in data else 1)" 2>/dev/null; then
        local error_msg
        error_msg=$(echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); print(data.get('error', {}).get('message', 'Unknown error'))" 2>/dev/null)
        if [[ -n "$error_msg" ]]; then
            log_warn "RPC error: $error_msg"
            return 1
        fi
    fi
    
    local enode
    enode=$(echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); print(data.get('result', {}).get('enode', ''))" 2>/dev/null)
    
    if [[ -z "$enode" ]] || [[ "$enode" == "None" ]] || [[ "$enode" == "null" ]]; then
        return 1
    fi
    
    if validate_enode "$enode"; then
        echo "$enode"
        return 0
    fi
    
    return 1
}

# Extract enode via SSH + nodekey (requires SSH access)
extract_via_ssh_nodekey() {
    local ip="$1"
    local ssh_target="${SSH_USER}@${ip}"
    
    log_info "Attempting SSH+nodekey extraction from $ssh_target..."
    
    # Try to find nodekey and generate enode
    local enode
    enode=$(ssh $SSH_OPTS "$ssh_target" bash << 'REMOTE_SCRIPT'
DATA_PATH="/data/besu"
BESU_BIN="/opt/besu/bin/besu"

# Find nodekey
for path in "${DATA_PATH}/key" "${DATA_PATH}/nodekey" "/keys/besu/nodekey"; do
    if [[ -f "$path" ]]; then
        ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="$path" --format=enode 2>/dev/null | sed "s/@[0-9.]*:/@${HOST_IP}:/")
        if [[ -n "$ENODE" ]]; then
            echo "$ENODE"
            exit 0
        fi
    fi
done
exit 1
REMOTE_SCRIPT
    )
    
    if [[ -n "$enode" ]] && validate_enode "$enode"; then
        echo "$enode"
        return 0
    fi
    
    return 1
}

# Main collection loop
log_info "Starting enode collection..."
echo ""

COLLECTED_ENODES="$WORK_DIR/collected-enodes.txt"
DUPLICATES="$WORK_DIR/duplicates.txt"
INVALIDS="$WORK_DIR/invalid-enodes.txt"

> "$COLLECTED_ENODES"
> "$DUPLICATES"
> "$INVALIDS"

declare -A ENODE_BY_IP
declare -A NODE_ID_SET

for ip in "${!NODES[@]}"; do
    IFS=':' read -r rpc_port use_rpc <<< "${NODES[$ip]}"
    
    log_info "Processing node: $ip"
    
    ENODE=""
    
    # Try RPC first if enabled
    if [[ "$use_rpc" == "1" ]]; then
        ENODE=$(extract_via_rpc "$ip" "$rpc_port" || echo "")
    fi
    
    # Fallback to SSH+nodekey if RPC failed
    if [[ -z "$ENODE" ]]; then
        ENODE=$(extract_via_ssh_nodekey "$ip" || echo "")
    fi
    
    if [[ -z "$ENODE" ]]; then
        log_error "Failed to extract enode from $ip"
        echo "$ip|FAILED" >> "$INVALIDS"
        continue
    fi
    
    # Validate
    if ! validate_enode "$ENODE"; then
        log_error "Invalid enode format from $ip"
        echo "$ip|$ENODE" >> "$INVALIDS"
        continue
    fi
    
    # Extract node ID and IP:PORT
    NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1)
    ENDPOINT=$(echo "$ENODE" | sed 's|.*@||')
    
    # Check for duplicate node IDs
    if [[ -n "${NODE_ID_SET[$NODE_ID]:-}" ]]; then
        log_warn "Duplicate node ID detected: ${NODE_ID:0:32}..."
        echo "$ip|$ENODE|DUPLICATE_NODE_ID|${NODE_ID_SET[$NODE_ID]}" >> "$DUPLICATES"
        continue
    fi
    
    # Check for duplicate endpoints (IP:PORT)
    if [[ -n "${ENODE_BY_IP[$ENDPOINT]:-}" ]]; then
        log_warn "Duplicate endpoint detected: $ENDPOINT"
        echo "$ip|$ENODE|DUPLICATE_ENDPOINT|${ENODE_BY_IP[$ENDPOINT]}" >> "$DUPLICATES"
        continue
    fi
    
    # Store valid enode
    NODE_ID_SET[$NODE_ID]="$ip"
    ENODE_BY_IP[$ENDPOINT]="$ip"
    
    echo "$ip|$ENODE" >> "$COLLECTED_ENODES"
    log_success "Collected enode from $ip"
done

# Summary
VALID_COUNT=$(wc -l < "$COLLECTED_ENODES")
DUP_COUNT=$(wc -l < "$DUPLICATES" 2>/dev/null || echo "0")
INVALID_COUNT=$(wc -l < "$INVALIDS" 2>/dev/null || echo "0")

echo ""
log_info "Collection Summary:"
log_success "Valid enodes: $VALID_COUNT"
if [[ "$DUP_COUNT" -gt 0 ]]; then
    log_warn "Duplicates: $DUP_COUNT (see $DUPLICATES)"
fi
if [[ "$INVALID_COUNT" -gt 0 ]]; then
    log_error "Invalid: $INVALID_COUNT (see $INVALIDS)"
fi

# Generate output files (next section)

---

2.2 Generate Allowlist Files

#!/bin/bash
# Generate Besu allowlist files from collected enodes
# File: generate-allowlist-files.sh

set -euo pipefail

COLLECTED_FILE="${1:-${WORK_DIR}/collected-enodes.txt}"
OUTPUT_DIR="${OUTPUT_DIR:-${WORK_DIR}}"

if [[ ! -f "$COLLECTED_FILE" ]]; then
    echo "ERROR: Collected enodes file not found: $COLLECTED_FILE"
    exit 1
fi

log_info "Generating allowlist files..."

# Generate static-nodes.json (validators only, if specified)
# For this example, we'll include all nodes, but you can filter to validators
VALIDATOR_IPS=("192.168.11.13" "192.168.11.14" "192.168.11.15" "192.168.11.16" "192.168.11.18")

python3 << PYEOF
import json
import sys

# Read collected enodes
enodes_all = []
enodes_validators = []

with open('$COLLECTED_FILE', 'r') as f:
    for line in f:
        line = line.strip()
        if not line or '|' not in line:
            continue
        parts = line.split('|')
        if len(parts) >= 2:
            ip = parts[0]
            enode = parts[1]
            enodes_all.append(enode)
            if ip in ${VALIDATOR_IPS[@]}:
                enodes_validators.append(enode)

# Sort for determinism
enodes_all.sort()
enodes_validators.sort()

# Generate static-nodes.json (validators only)
with open('${OUTPUT_DIR}/static-nodes.json', 'w') as f:
    json.dump(enodes_validators, f, indent=2)

print(f"Generated static-nodes.json with {len(enodes_validators)} validators")

# Generate permissions-nodes.toml (all nodes)
toml_content = """# Node Permissioning Configuration
# Lists nodes that are allowed to connect to this node
# Generated: $(date)
# Total nodes: {len(enodes_all)}

nodes-allowlist=[
"""

for enode in enodes_all:
    toml_content += f'  "{enode}",\n'

toml_content = toml_content.rstrip(',\n') + '\n]'

with open('${OUTPUT_DIR}/permissions-nodes.toml', 'w') as f:
    f.write(toml_content)

print(f"Generated permissions-nodes.toml with {len(enodes_all)} nodes")
PYEOF

log_success "Files generated:"
log_info "  ${OUTPUT_DIR}/static-nodes.json"
log_info "  ${OUTPUT_DIR}/permissions-nodes.toml"

---

3. Besu Configuration for Node Permissions

3.1 Configuration Method A: Permissions-Based (Recommended for Dynamic Networks)

Use case: Network where nodes may join/leave, but you want strict allowlisting.

# Besu startup flags
besu \
  --data-path=/data/besu \
  --genesis-file=/etc/besu/genesis.json \
  --config-file=/etc/besu/config.toml \
  \
  # Enable node permissions
  --permissions-nodes-config-file-enabled=true \
  --permissions-nodes-config-file=/etc/besu/permissions-nodes.toml \
  \
  # Network configuration
  --p2p-host=0.0.0.0 \
  --p2p-port=30303 \
  --nat-method=UPNP \
  \
  # Discovery (can be enabled with permissions)
  --discovery-enabled=true \
  \
  # RPC (enable ADMIN for verification)
  --rpc-http-enabled=true \
  --rpc-http-host=0.0.0.0 \
  --rpc-http-port=8545 \
  --rpc-http-api=ETH,NET,ADMIN,QBFT \
  \
  # Static nodes (optional, for faster initial connection)
  --static-nodes-file=/etc/besu/static-nodes.json

Key flags:

• --permissions-nodes-config-file-enabled=true: Enables allowlist checking
• --permissions-nodes-config-file: Path to TOML allowlist
• --discovery-enabled=true: Can be enabled; Besu will still enforce allowlist
• --static-nodes-file: Optional, but recommended for faster peer discovery

---

3.2 Configuration Method B: Static-Only (Recommended for Strict Private Networks)

Use case: Completely private network, no dynamic discovery needed.

# Besu startup flags
besu \
  --data-path=/data/besu \
  --genesis-file=/etc/besu/genesis.json \
  --config-file=/etc/besu/config.toml \
  \
  # Static nodes only (discovery disabled)
  --static-nodes-file=/etc/besu/static-nodes.json \
  --discovery-enabled=false \
  \
  # Network configuration
  --p2p-host=0.0.0.0 \
  --p2p-port=30303 \
  --nat-method=NONE \
  \
  # RPC
  --rpc-http-enabled=true \
  --rpc-http-host=0.0.0.0 \
  --rpc-http-port=8545 \
  --rpc-http-api=ETH,NET,ADMIN,QBFT \
  \
  # Permissions still recommended for extra security
  --permissions-nodes-config-file-enabled=true \
  --permissions-nodes-config-file=/etc/besu/permissions-nodes.toml

Key differences:

• --discovery-enabled=false: No peer discovery, only static connections
• --nat-method=NONE: No NAT traversal needed
• Still use permissions-nodes.toml for defense-in-depth

---

3.3 TOML Configuration File Example

# /etc/besu/config.toml
data-path="/data/besu"
genesis-file="/etc/besu/genesis.json"

# Network
network-id=138
p2p-host="0.0.0.0"
p2p-port=30303

# Permissions
permissions-nodes-config-file-enabled=true
permissions-nodes-config-file="/etc/besu/permissions-nodes.toml"

# Static nodes
static-nodes-file="/etc/besu/static-nodes.json"

# Discovery
discovery-enabled=true

# RPC
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","ADMIN","QBFT"]

# Consensus (QBFT)
miner-enabled=false

---

4. Validation & Verification

4.1 Validate Enode Format (Pre-Deployment)

#!/bin/bash
# Validate all enodes in generated files
# File: validate-allowlist.sh

set -euo pipefail

STATIC_NODES="${1:-static-nodes.json}"
PERMISSIONS_TOML="${2:-permissions-nodes.toml}"

ERRORS=0

validate_enode_file() {
    local file="$1"
    local file_type="$2"
    
    echo "Validating $file_type: $file"
    
    if [[ "$file_type" == "json" ]]; then
        # JSON format (static-nodes.json)
        python3 << PYEOF
import json
import re
import sys

with open('$file', 'r') as f:
    enodes = json.load(f)

errors = 0
node_ids_seen = set()
endpoints_seen = set()

for i, enode in enumerate(enodes):
    # Extract node ID
    match = re.match(r'enode://([0-9a-fA-F]+)@([0-9.]+):(\d+)', enode)
    if not match:
        print(f"ERROR: Invalid enode format at index {i}: {enode}")
        errors += 1
        continue
    
    node_id = match.group(1).lower()
    endpoint = f"{match.group(2)}:{match.group(3)}"
    
    # Check length
    if len(node_id) != 128:
        print(f"ERROR: Node ID length {len(node_id)} at index {i} (expected 128): {node_id[:32]}...")
        errors += 1
        continue
    
    # Check hex
    if not re.match(r'^[0-9a-f]{128}$', node_id):
        print(f"ERROR: Invalid hex in node ID at index {i}: {node_id[:32]}...")
        errors += 1
        continue
    
    # Check duplicates
    if node_id in node_ids_seen:
        print(f"WARNING: Duplicate node ID at index {i}: {node_id[:32]}...")
    node_ids_seen.add(node_id)
    
    if endpoint in endpoints_seen:
        print(f"WARNING: Duplicate endpoint at index {i}: {endpoint}")
    endpoints_seen.add(endpoint)

sys.exit(errors)
PYEOF
        ERRORS=$((ERRORS + $?))
    else
        # TOML format (permissions-nodes.toml)
        python3 << PYEOF
import re
import sys

with open('$file', 'r') as f:
    content = f.read()

# Extract all enodes from TOML
enodes = re.findall(r'"enode://([0-9a-fA-F]+)@([0-9.]+):(\d+)"', content)

errors = 0
node_ids_seen = set()
endpoints_seen = set()

for i, (node_id_hex, ip, port) in enumerate(enodes):
    node_id = node_id_hex.lower()
    endpoint = f"{ip}:{port}"
    
    if len(node_id) != 128:
        print(f"ERROR: Node ID length {len(node_id)} at entry {i+1} (expected 128): {node_id[:32]}...")
        errors += 1
        continue
    
    if not re.match(r'^[0-9a-f]{128}$', node_id):
        print(f"ERROR: Invalid hex in node ID at entry {i+1}: {node_id[:32]}...")
        errors += 1
        continue
    
    if node_id in node_ids_seen:
        print(f"WARNING: Duplicate node ID at entry {i+1}: {node_id[:32]}...")
    node_ids_seen.add(node_id)
    
    if endpoint in endpoints_seen:
        print(f"WARNING: Duplicate endpoint at entry {i+1}: {endpoint}")
    endpoints_seen.add(endpoint)

sys.exit(errors)
PYEOF
        ERRORS=$((ERRORS + $?))
    fi
}

validate_enode_file "$STATIC_NODES" "json"
validate_enode_file "$PERMISSIONS_TOML" "toml"

if [[ $ERRORS -eq 0 ]]; then
    echo "✓ All enodes validated successfully"
    exit 0
else
    echo "✗ Validation failed with $ERRORS errors"
    exit 1
fi

---

4.2 Verify Peers Connected (Runtime)

#!/bin/bash
# Check peer connections on all nodes
# File: verify-peers.sh

set -euo pipefail

RPC_URL="${1:-http://localhost:8545}"

echo "Checking peers on: $RPC_URL"
echo ""

# Get node info
NODE_INFO=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
    "${RPC_URL}")

ENODE=$(echo "$NODE_INFO" | python3 -c "import sys, json; print(json.load(sys.stdin).get('result', {}).get('enode', 'ERROR'))" 2>/dev/null)

if [[ "$ENODE" == "ERROR" ]] || [[ -z "$ENODE" ]]; then
    echo "ERROR: Could not get node info. Is RPC enabled with ADMIN API?"
    exit 1
fi

echo "This node's enode:"
echo "$ENODE"
echo ""

# Get peers
PEERS_RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":2}' \
    "${RPC_URL}")

PEERS=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; peers=json.load(sys.stdin).get('result', []); print(len(peers))" 2>/dev/null)
PEERS_LIST=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; peers=json.load(sys.stdin).get('result', []); [print(f\"  - {p.get('enode', 'unknown')}\") for p in peers]" 2>/dev/null)

echo "Connected peers: $PEERS"
echo ""

if [[ "$PEERS" == "0" ]]; then
    echo "⚠️  NO PEERS CONNECTED"
    echo ""
    echo "Possible causes:"
    echo "1. Other nodes not running"
    echo "2. Firewall blocking port 30303"
    echo "3. Malformed enodes in allowlist"
    echo "4. Discovery disabled and static-nodes.json incorrect"
    echo "5. Permissions enabled but allowlist missing this node"
    echo "6. Network connectivity issues"
else
    echo "Peer list:"
    echo "$PEERS_LIST"
fi

# Check peer details
if [[ "$PEERS" != "0" ]] && command -v jq >/dev/null 2>&1; then
    echo ""
    echo "Peer details:"
    echo "$PEERS_RESPONSE" | jq -r '.result[] | "  - \(.id): \(.name) @ \(.network.remoteAddress)"'
fi

Usage:

# Check local node
bash verify-peers.sh

# Check remote node
bash verify-peers.sh http://192.168.11.13:8545

# Check all nodes
for ip in 192.168.11.{13,14,15,16,18,19,20,21,22,23,24,25}; do
    echo "=== Node $ip ==="
    bash verify-peers.sh "http://${ip}:8545"
    echo ""
done

---

4.3 Troubleshooting Decision Tree

#!/bin/bash
# Comprehensive troubleshooting script
# File: troubleshoot-besu-peers.sh

set -euo pipefail

NODE_IP="${1:-192.168.11.13}"
RPC_PORT="${2:-8545}"
RPC_URL="http://${NODE_IP}:${RPC_PORT}"

echo "╔════════════════════════════════════════════════════════════════╗"
echo "║         BESU PEER CONNECTION TROUBLESHOOTING                    ║"
echo "╚════════════════════════════════════════════════════════════════╝"
echo ""
echo "Node: $NODE_IP"
echo ""

# 1. Check if node is running
echo "1. Checking if Besu is running..."
if ! curl -s -m 2 "${RPC_URL}" > /dev/null 2>&1; then
    echo "   ✗ Node not responding to RPC"
    echo "   → Check: systemctl status besu-validator (or besu-sentry/besu-rpc)"
    exit 1
fi
echo "   ✓ Node is running"

# 2. Check RPC API access
echo ""
echo "2. Checking RPC API access..."
RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":1}' \
    "${RPC_URL}")

if echo "$RESPONSE" | grep -q "error"; then
    echo "   ✗ RPC error (check --rpc-http-api includes NET,ADMIN)"
else
    echo "   ✓ RPC accessible"
fi

# 3. Get node info
echo ""
echo "3. Getting node information..."
NODE_INFO=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":2}' \
    "${RPC_URL}")

ENODE=$(echo "$NODE_INFO" | python3 -c "import sys, json; print(json.load(sys.stdin).get('result', {}).get('enode', ''))" 2>/dev/null)

if [[ -z "$ENODE" ]]; then
    echo "   ✗ Could not get enode (ADMIN API may not be enabled)"
    echo "   → Fix: Add ADMIN to --rpc-http-api"
else
    echo "   ✓ Enode: $ENODE"
    
    # Validate enode
    NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1)
    if [[ ${#NODE_ID} -ne 128 ]]; then
        echo "   ⚠️  WARNING: Node ID length is ${#NODE_ID}, not 128!"
    fi
fi

# 4. Check peers
echo ""
echo "4. Checking peer connections..."
PEERS_RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":3}' \
    "${RPC_URL}")

PEER_COUNT=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; print(len(json.load(sys.stdin).get('result', [])))" 2>/dev/null)

echo "   Connected peers: $PEER_COUNT"

if [[ "$PEER_COUNT" == "0" ]]; then
    echo ""
    echo "   ═══════════════════════════════════════════════════════"
    echo "   TROUBLESHOOTING: No peers connected"
    echo "   ═══════════════════════════════════════════════════════"
    echo ""
    
    # Check static-nodes.json
    echo "   4a. Checking static-nodes.json..."
    if ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" "test -f /etc/besu/static-nodes.json" 2>/dev/null; then
        STATIC_COUNT=$(ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" \
            "python3 -c \"import json; print(len(json.load(open('/etc/besu/static-nodes.json'))))\" 2>/dev/null" || echo "0")
        echo "      → Found static-nodes.json with $STATIC_COUNT entries"
    else
        echo "      ✗ static-nodes.json not found"
    fi
    
    # Check permissions-nodes.toml
    echo ""
    echo "   4b. Checking permissions-nodes.toml..."
    if ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" "test -f /etc/besu/permissions-nodes.toml" 2>/dev/null; then
        PERM_COUNT=$(ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" \
            "grep -c 'enode://' /etc/besu/permissions-nodes.toml 2>/dev/null" || echo "0")
        echo "      → Found permissions-nodes.toml with $PERM_COUNT entries"
    else
        echo "      ✗ permissions-nodes.toml not found"
    fi
    
    # Check discovery
    echo ""
    echo "   4c. Checking discovery configuration..."
    # This would require parsing config file or checking logs
    echo "      → Check: --discovery-enabled flag in config"
    echo "      → If false, static-nodes.json must be correct"
    echo "      → If true, ensure permissions allowlist includes all nodes"
    
    # Check firewall
    echo ""
    echo "   4d. Checking network connectivity..."
    echo "      → Test: nc -zv <peer-ip> 30303"
    echo "      → Check: iptables/firewalld rules"
    echo "      → Verify: Port 30303 is open and accessible"
    
    # Check advertised host
    echo ""
    echo "   4e. Checking advertised host/IP..."
    ADVERTISED_IP=$(echo "$ENODE" | sed 's|.*@||' | cut -d':' -f1)
    echo "      → Advertised IP: $ADVERTISED_IP"
    if [[ "$ADVERTISED_IP" != "$NODE_IP" ]] && [[ "$ADVERTISED_IP" != "0.0.0.0" ]]; then
        echo "      ⚠️  WARNING: Advertised IP ($ADVERTISED_IP) differs from actual IP ($NODE_IP)"
        echo "      → Fix: Set --p2p-host=$NODE_IP or --nat-method correctly"
    fi
fi

echo ""
echo "═══════════════════════════════════════════════════════════════"
echo "Troubleshooting complete"
echo "═══════════════════════════════════════════════════════════════"

---

5. Deployment Script

#!/bin/bash
# Deploy corrected allowlist files to all containers
# File: deploy-allowlist.sh

set -euo pipefail

PROXMOX_HOST="${PROXMOX_HOST:-192.168.11.10}"
STATIC_NODES_FILE="${1:-static-nodes.json}"
PERMISSIONS_TOML_FILE="${2:-permissions-nodes.toml}"

if [[ ! -f "$STATIC_NODES_FILE" ]] || [[ ! -f "$PERMISSIONS_TOML_FILE" ]]; then
    echo "ERROR: Files not found:"
    [[ ! -f "$STATIC_NODES_FILE" ]] && echo "  - $STATIC_NODES_FILE"
    [[ ! -f "$PERMISSIONS_TOML_FILE" ]] && echo "  - $PERMISSIONS_TOML_FILE"
    exit 1
fi

# Validate files first
echo "Validating files before deployment..."
bash validate-allowlist.sh "$STATIC_NODES_FILE" "$PERMISSIONS_TOML_FILE" || {
    echo "ERROR: Validation failed. Fix files before deploying."
    exit 1
}

echo ""
echo "Deploying to Proxmox host: $PROXMOX_HOST"

# Copy files to host
scp -o StrictHostKeyChecking=accept-new \
    "$STATIC_NODES_FILE" \
    "$PERMISSIONS_TOML_FILE" \
    "root@${PROXMOX_HOST}:/tmp/"

# Deploy to all containers
for vmid in 106 107 108 109 110 111 112 113 114 115 116 117; do
    echo "Deploying to container $vmid..."
    
    ssh -o StrictHostKeyChecking=accept-new "root@${PROXMOX_HOST}" << DEPLOY_SCRIPT
if ! pct status $vmid 2>/dev/null | grep -q running; then
    echo "  Container $vmid not running, skipping"
    exit 0
fi

# Copy files
pct push $vmid /tmp/$(basename $STATIC_NODES_FILE) /etc/besu/static-nodes.json
pct push $vmid /tmp/$(basename $PERMISSIONS_TOML_FILE) /etc/besu/permissions-nodes.toml

# Set ownership
pct exec $vmid -- chown besu:besu /etc/besu/static-nodes.json /etc/besu/permissions-nodes.toml

# Verify
if pct exec $vmid -- test -f /etc/besu/static-nodes.json && \
   pct exec $vmid -- test -f /etc/besu/permissions-nodes.toml; then
    echo "  ✓ Container $vmid: Files deployed"
else
    echo "  ✗ Container $vmid: Deployment failed"
fi
DEPLOY_SCRIPT

done

# Cleanup
ssh -o StrictHostKeyChecking=accept-new "root@${PROXMOX_HOST}" \
    "rm -f /tmp/$(basename $STATIC_NODES_FILE) /tmp/$(basename $PERMISSIONS_TOML_FILE)"

echo ""
echo "✓ Deployment complete"
echo ""
echo "Next steps:"
echo "1. Restart Besu services on all containers"
echo "2. Run verification: bash verify-peers.sh <rpc-url>"

---

Runbook Summary

Execute in this order:

1. Extract enodes from each node:

   # On each node, or via SSH/management host
   bash extract-enode-from-nodekey.sh > enode-<ip>.txt
   # OR
   export RPC_URL="http://<ip>:8545"
   bash extract-enode-from-rpc.sh > enode-<ip>.txt
   

2. Collect all enodes:

   # Update NODES array in script with your IPs
   bash collect-all-enodes.sh
   

3. Generate allowlist files:

   bash generate-allowlist-files.sh ${WORK_DIR}/collected-enodes.txt
   

4. Validate generated files:

   bash validate-allowlist.sh static-nodes.json permissions-nodes.toml
   

5. Deploy to all containers:

   bash deploy-allowlist.sh static-nodes.json permissions-nodes.toml
   

6. Restart Besu services:

   # On Proxmox host
   for vmid in 106 107 108 109 110 111 112 113 114 115 116 117; do
       pct exec $vmid -- systemctl restart besu-validator 2>/dev/null || \
       pct exec $vmid -- systemctl restart besu-sentry 2>/dev/null || \
       pct exec $vmid -- systemctl restart besu-rpc 2>/dev/null
   done
   

7. Verify peer connections:

   # Check all nodes
   for ip in 192.168.11.{13,14,15,16,18,19,20,21,22,23,24,25}; do
       echo "=== $ip ==="
       bash verify-peers.sh "http://${ip}:8545"
   done
   

8. Troubleshoot if needed:

   bash troubleshoot-besu-peers.sh <node-ip>
   

---

Common Pitfalls & Warnings

⚠️ Padding Zeros: Never pad node IDs with trailing zeros. Besu will reject them.

⚠️ Multiple Enodes per IP:PORT: One node = one enode. Keep only the enode returned by that node's admin_nodeInfo.

⚠️ P2P Host Configuration: Set --p2p-host to the actual IP or use --nat-method=UPNP for correct advertisement.

⚠️ RPC API: Must include ADMIN in --rpc-http-api to use admin_nodeInfo and admin_peers.

⚠️ Discovery vs Permissions: If discovery-enabled=false, static-nodes.json must be correct. If permissions-nodes-enabled=true, all peers must be in allowlist.

⚠️ File Ownership: Ensure files are owned by besu:besu user.

⚠️ Service Restart: Always restart Besu services after updating allowlist files.