d-bis/proxmox

Fork 0

Files

defiQUG b45c2006be Refactor code for improved readability and performance

2025-12-21 22:32:09 -08:00

32 KiB

Raw Blame History

Hyperledger Besu Node Allowlist Generation Runbook

Purpose: Generate, validate, and deploy correct Besu node allowlists using only Besu-native commands.

Scope: Private LAN network (192.168.11.0/24), QBFT consensus, strict permissions.

Prerequisites

Besu installed on all nodes (/opt/besu/bin/besu or in PATH)
JSON-RPC enabled with ADMIN API (for verification) OR nodekey files accessible
SSH access to all nodes OR local execution on each node
jq and python3 for JSON processing (optional but recommended)

1. Node-Level Enode Extraction (Execute on EACH Node)

1.1 Method A: Using Besu CLI (Nodekey File)

When to use: Node is not running, or RPC is disabled.

#!/bin/bash
# Execute on each Besu node
# File: extract-enode-from-nodekey.sh

set -euo pipefail

# Configuration
DATA_PATH="${DATA_PATH:-/data/besu}"
BESU_BIN="${BESU_BIN:-/opt/besu/bin/besu}"
NODE_IP="${NODE_IP:-}"  # Set to actual node IP (e.g., 192.168.11.13)
P2P_PORT="${P2P_PORT:-30303}"

# Find nodekey file
NODEKEY_FILE=""
for path in "${DATA_PATH}/key" "${DATA_PATH}/nodekey" "/keys/besu/nodekey"; do
    if [[ -f "$path" ]]; then
        NODEKEY_FILE="$path"
        break
    fi
done

if [[ -z "$NODEKEY_FILE" ]]; then
    echo "ERROR: Nodekey file not found in ${DATA_PATH}/key, ${DATA_PATH}/nodekey, or /keys/besu/nodekey"
    exit 1
fi

echo "Found nodekey: $NODEKEY_FILE"

# Generate enode using Besu CLI
if [[ -n "$NODE_IP" ]]; then
    # If IP is provided, generate enode and replace IP
    ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="${NODEKEY_FILE}" --format=enode 2>/dev/null | sed "s/@[0-9.]*:/@${NODE_IP}:/")
else
    # Use IP from generated enode (may be 0.0.0.0 or localhost)
    ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="${NODEKEY_FILE}" --format=enode 2>/dev/null)
fi

if [[ -z "$ENODE" ]]; then
    echo "ERROR: Failed to generate enode from nodekey"
    exit 1
fi

# Extract and validate node ID length
NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
NODE_ID_LEN=${#NODE_ID}

if [[ "$NODE_ID_LEN" -ne 128 ]]; then
    echo "ERROR: Invalid node ID length: $NODE_ID_LEN (expected 128)"
    echo "Node ID: ${NODE_ID:0:32}...${NODE_ID: -32}"
    exit 1
fi

# Validate hex format
if ! echo "$NODE_ID" | grep -qE '^[0-9a-f]{128}$'; then
    echo "ERROR: Node ID contains invalid hex characters"
    exit 1
fi

echo "✓ Valid enode extracted:"
echo "$ENODE"
echo ""
echo "Node ID length: $NODE_ID_LEN (✓)"
echo "Node ID (first 32 chars): ${NODE_ID:0:32}..."
echo "Node ID (last 32 chars): ...${NODE_ID: -32}"

Usage on each node:

# On node 192.168.11.13
export NODE_IP="192.168.11.13"
export DATA_PATH="/data/besu"
bash extract-enode-from-nodekey.sh > enode-192.168.11.13.txt

1.2 Method B: Using JSON-RPC (Running Node)

When to use: Node is running and RPC is enabled with ADMIN API.

#!/bin/bash
# Execute on each Besu node OR from central management host
# File: extract-enode-from-rpc.sh

set -euo pipefail

# Configuration
RPC_URL="${RPC_URL:-http://localhost:8545}"
NODE_IP="${NODE_IP:-}"  # Optional: verify IP matches

# Get node info via JSON-RPC
RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
    "${RPC_URL}")

# Check for errors
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
    ERROR_MSG=$(echo "$RESPONSE" | jq -r '.error.message')
    echo "ERROR: JSON-RPC error: $ERROR_MSG"
    echo "NOTE: Ensure RPC is enabled with --rpc-http-api=ADMIN,NET"
    exit 1
fi

# Extract enode
ENODE=$(echo "$RESPONSE" | jq -r '.result.enode // empty')

if [[ -z "$ENODE" ]] || [[ "$ENODE" == "null" ]]; then
    echo "ERROR: Could not extract enode from admin_nodeInfo"
    echo "Response: $RESPONSE"
    exit 1
fi

# Extract and validate node ID
NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
NODE_ID_LEN=${#NODE_ID}

if [[ "$NODE_ID_LEN" -ne 128 ]]; then
    echo "ERROR: Invalid node ID length: $NODE_ID_LEN (expected 128)"
    echo "Enode: $ENODE"
    exit 1
fi

# Extract IP and port
ENODE_IP=$(echo "$ENODE" | sed 's|.*@||' | cut -d':' -f1)
ENODE_PORT=$(echo "$ENODE" | sed 's|.*:||')

# Verify IP if provided
if [[ -n "$NODE_IP" ]] && [[ "$ENODE_IP" != "$NODE_IP" ]]; then
    echo "WARNING: Enode IP ($ENODE_IP) does not match expected IP ($NODE_IP)"
    echo "NOTE: Check --p2p-host and --nat-method configuration"
fi

echo "✓ Valid enode extracted via RPC:"
echo "$ENODE"
echo ""
echo "Node ID length: $NODE_ID_LEN (✓)"
echo "Advertised IP: $ENODE_IP"
echo "Advertised Port: $ENODE_PORT"

Usage:

# From management host
export RPC_URL="http://192.168.11.13:8545"
export NODE_IP="192.168.11.13"
bash extract-enode-from-rpc.sh > enode-192.168.11.13.txt

2. Automated Collection Script (All Nodes)

2.1 Complete Collection Script

#!/bin/bash
# Collect enodes from all nodes and generate allowlist
# File: collect-all-enodes.sh

set -euo pipefail

# Configuration
WORK_DIR="${WORK_DIR:-./besu-enodes-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$WORK_DIR"

# Node inventory (IP -> RPC_PORT mapping)
# Format: IP:RPC_PORT:USE_RPC (use_rpc=1 if RPC available, 0 to use nodekey method)
declare -A NODES=(
    ["192.168.11.13"]="8545:1"  # validator-1 (RPC may be disabled)
    ["192.168.11.14"]="8545:1"  # validator-2
    ["192.168.11.15"]="8545:1"  # validator-3
    ["192.168.11.16"]="8545:1"  # validator-4
    ["192.168.11.18"]="8545:1"  # validator-5
    ["192.168.11.19"]="8545:1"  # sentry-2
    ["192.168.11.20"]="8545:1"  # sentry-3
    ["192.168.11.21"]="8545:1"  # sentry-4
    ["192.168.11.22"]="8545:1"  # sentry-5
    ["192.168.11.23"]="8545:1"  # rpc-1 (RPC definitely enabled)
    ["192.168.11.24"]="8545:1"  # rpc-2
    ["192.168.11.25"]="8545:1"  # rpc-3
)

# SSH configuration (if accessing remote nodes)
SSH_USER="${SSH_USER:-root}"
SSH_OPTS="${SSH_OPTS:--o StrictHostKeyChecking=accept-new}"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log_info() { echo -e "${BLUE}[INFO]${NC} $1"; }
log_success() { echo -e "${GREEN}[✓]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARNING]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }

# Validate enode format
validate_enode() {
    local enode="$1"
    local node_id
    
    node_id=$(echo "$enode" | sed 's|^enode://||' | cut -d'@' -f1 | tr '[:upper:]' '[:lower:]')
    
    if [[ ${#node_id} -ne 128 ]]; then
        log_error "Invalid node ID length: ${#node_id} (expected 128)"
        return 1
    fi
    
    if ! echo "$node_id" | grep -qE '^[0-9a-f]{128}$'; then
        log_error "Node ID contains invalid hex characters"
        return 1
    fi
    
    return 0
}

# Extract enode via RPC
extract_via_rpc() {
    local ip="$1"
    local rpc_port="$2"
    local rpc_url="http://${ip}:${rpc_port}"
    
    log_info "Attempting RPC extraction from $rpc_url..."
    
    local response
    response=$(curl -s -m 5 -X POST \
        -H "Content-Type: application/json" \
        --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
        "${rpc_url}" 2>/dev/null || echo "")
    
    if [[ -z "$response" ]]; then
        return 1
    fi
    
    # Check for JSON-RPC error
    if echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); sys.exit(0 if 'error' not in data else 1)" 2>/dev/null; then
        local error_msg
        error_msg=$(echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); print(data.get('error', {}).get('message', 'Unknown error'))" 2>/dev/null)
        if [[ -n "$error_msg" ]]; then
            log_warn "RPC error: $error_msg"
            return 1
        fi
    fi
    
    local enode
    enode=$(echo "$response" | python3 -c "import sys, json; data=json.load(sys.stdin); print(data.get('result', {}).get('enode', ''))" 2>/dev/null)
    
    if [[ -z "$enode" ]] || [[ "$enode" == "None" ]] || [[ "$enode" == "null" ]]; then
        return 1
    fi
    
    if validate_enode "$enode"; then
        echo "$enode"
        return 0
    fi
    
    return 1
}

# Extract enode via SSH + nodekey (requires SSH access)
extract_via_ssh_nodekey() {
    local ip="$1"
    local ssh_target="${SSH_USER}@${ip}"
    
    log_info "Attempting SSH+nodekey extraction from $ssh_target..."
    
    # Try to find nodekey and generate enode
    local enode
    enode=$(ssh $SSH_OPTS "$ssh_target" bash << 'REMOTE_SCRIPT'
DATA_PATH="/data/besu"
BESU_BIN="/opt/besu/bin/besu"

# Find nodekey
for path in "${DATA_PATH}/key" "${DATA_PATH}/nodekey" "/keys/besu/nodekey"; do
    if [[ -f "$path" ]]; then
        ENODE=$("${BESU_BIN}" public-key export --node-private-key-file="$path" --format=enode 2>/dev/null | sed "s/@[0-9.]*:/@${HOST_IP}:/")
        if [[ -n "$ENODE" ]]; then
            echo "$ENODE"
            exit 0
        fi
    fi
done
exit 1
REMOTE_SCRIPT
    )
    
    if [[ -n "$enode" ]] && validate_enode "$enode"; then
        echo "$enode"
        return 0
    fi
    
    return 1
}

# Main collection loop
log_info "Starting enode collection..."
echo ""

COLLECTED_ENODES="$WORK_DIR/collected-enodes.txt"
DUPLICATES="$WORK_DIR/duplicates.txt"
INVALIDS="$WORK_DIR/invalid-enodes.txt"

> "$COLLECTED_ENODES"
> "$DUPLICATES"
> "$INVALIDS"

declare -A ENODE_BY_IP
declare -A NODE_ID_SET

for ip in "${!NODES[@]}"; do
    IFS=':' read -r rpc_port use_rpc <<< "${NODES[$ip]}"
    
    log_info "Processing node: $ip"
    
    ENODE=""
    
    # Try RPC first if enabled
    if [[ "$use_rpc" == "1" ]]; then
        ENODE=$(extract_via_rpc "$ip" "$rpc_port" || echo "")
    fi
    
    # Fallback to SSH+nodekey if RPC failed
    if [[ -z "$ENODE" ]]; then
        ENODE=$(extract_via_ssh_nodekey "$ip" || echo "")
    fi
    
    if [[ -z "$ENODE" ]]; then
        log_error "Failed to extract enode from $ip"
        echo "$ip|FAILED" >> "$INVALIDS"
        continue
    fi
    
    # Validate
    if ! validate_enode "$ENODE"; then
        log_error "Invalid enode format from $ip"
        echo "$ip|$ENODE" >> "$INVALIDS"
        continue
    fi
    
    # Extract node ID and IP:PORT
    NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1)
    ENDPOINT=$(echo "$ENODE" | sed 's|.*@||')
    
    # Check for duplicate node IDs
    if [[ -n "${NODE_ID_SET[$NODE_ID]:-}" ]]; then
        log_warn "Duplicate node ID detected: ${NODE_ID:0:32}..."
        echo "$ip|$ENODE|DUPLICATE_NODE_ID|${NODE_ID_SET[$NODE_ID]}" >> "$DUPLICATES"
        continue
    fi
    
    # Check for duplicate endpoints (IP:PORT)
    if [[ -n "${ENODE_BY_IP[$ENDPOINT]:-}" ]]; then
        log_warn "Duplicate endpoint detected: $ENDPOINT"
        echo "$ip|$ENODE|DUPLICATE_ENDPOINT|${ENODE_BY_IP[$ENDPOINT]}" >> "$DUPLICATES"
        continue
    fi
    
    # Store valid enode
    NODE_ID_SET[$NODE_ID]="$ip"
    ENODE_BY_IP[$ENDPOINT]="$ip"
    
    echo "$ip|$ENODE" >> "$COLLECTED_ENODES"
    log_success "Collected enode from $ip"
done

# Summary
VALID_COUNT=$(wc -l < "$COLLECTED_ENODES")
DUP_COUNT=$(wc -l < "$DUPLICATES" 2>/dev/null || echo "0")
INVALID_COUNT=$(wc -l < "$INVALIDS" 2>/dev/null || echo "0")

echo ""
log_info "Collection Summary:"
log_success "Valid enodes: $VALID_COUNT"
if [[ "$DUP_COUNT" -gt 0 ]]; then
    log_warn "Duplicates: $DUP_COUNT (see $DUPLICATES)"
fi
if [[ "$INVALID_COUNT" -gt 0 ]]; then
    log_error "Invalid: $INVALID_COUNT (see $INVALIDS)"
fi

# Generate output files (next section)

2.2 Generate Allowlist Files

#!/bin/bash
# Generate Besu allowlist files from collected enodes
# File: generate-allowlist-files.sh

set -euo pipefail

COLLECTED_FILE="${1:-${WORK_DIR}/collected-enodes.txt}"
OUTPUT_DIR="${OUTPUT_DIR:-${WORK_DIR}}"

if [[ ! -f "$COLLECTED_FILE" ]]; then
    echo "ERROR: Collected enodes file not found: $COLLECTED_FILE"
    exit 1
fi

log_info "Generating allowlist files..."

# Generate static-nodes.json (validators only, if specified)
# For this example, we'll include all nodes, but you can filter to validators
VALIDATOR_IPS=("192.168.11.13" "192.168.11.14" "192.168.11.15" "192.168.11.16" "192.168.11.18")

python3 << PYEOF
import json
import sys

# Read collected enodes
enodes_all = []
enodes_validators = []

with open('$COLLECTED_FILE', 'r') as f:
    for line in f:
        line = line.strip()
        if not line or '|' not in line:
            continue
        parts = line.split('|')
        if len(parts) >= 2:
            ip = parts[0]
            enode = parts[1]
            enodes_all.append(enode)
            if ip in ${VALIDATOR_IPS[@]}:
                enodes_validators.append(enode)

# Sort for determinism
enodes_all.sort()
enodes_validators.sort()

# Generate static-nodes.json (validators only)
with open('${OUTPUT_DIR}/static-nodes.json', 'w') as f:
    json.dump(enodes_validators, f, indent=2)

print(f"Generated static-nodes.json with {len(enodes_validators)} validators")

# Generate permissions-nodes.toml (all nodes)
toml_content = """# Node Permissioning Configuration
# Lists nodes that are allowed to connect to this node
# Generated: $(date)
# Total nodes: {len(enodes_all)}

nodes-allowlist=[
"""

for enode in enodes_all:
    toml_content += f'  "{enode}",\n'

toml_content = toml_content.rstrip(',\n') + '\n]'

with open('${OUTPUT_DIR}/permissions-nodes.toml', 'w') as f:
    f.write(toml_content)

print(f"Generated permissions-nodes.toml with {len(enodes_all)} nodes")
PYEOF

log_success "Files generated:"
log_info "  ${OUTPUT_DIR}/static-nodes.json"
log_info "  ${OUTPUT_DIR}/permissions-nodes.toml"

3. Besu Configuration for Node Permissions

3.1 Configuration Method A: Permissions-Based (Recommended for Dynamic Networks)

Use case: Network where nodes may join/leave, but you want strict allowlisting.

# Besu startup flags
besu \
  --data-path=/data/besu \
  --genesis-file=/etc/besu/genesis.json \
  --config-file=/etc/besu/config.toml \
  \
  # Enable node permissions
  --permissions-nodes-config-file-enabled=true \
  --permissions-nodes-config-file=/etc/besu/permissions-nodes.toml \
  \
  # Network configuration
  --p2p-host=0.0.0.0 \
  --p2p-port=30303 \
  --nat-method=UPNP \
  \
  # Discovery (can be enabled with permissions)
  --discovery-enabled=true \
  \
  # RPC (enable ADMIN for verification)
  --rpc-http-enabled=true \
  --rpc-http-host=0.0.0.0 \
  --rpc-http-port=8545 \
  --rpc-http-api=ETH,NET,ADMIN,QBFT \
  \
  # Static nodes (optional, for faster initial connection)
  --static-nodes-file=/etc/besu/static-nodes.json

Key flags:

--permissions-nodes-config-file-enabled=true: Enables allowlist checking
--permissions-nodes-config-file: Path to TOML allowlist
--discovery-enabled=true: Can be enabled; Besu will still enforce allowlist
--static-nodes-file: Optional, but recommended for faster peer discovery

3.2 Configuration Method B: Static-Only (Recommended for Strict Private Networks)

Use case: Completely private network, no dynamic discovery needed.

# Besu startup flags
besu \
  --data-path=/data/besu \
  --genesis-file=/etc/besu/genesis.json \
  --config-file=/etc/besu/config.toml \
  \
  # Static nodes only (discovery disabled)
  --static-nodes-file=/etc/besu/static-nodes.json \
  --discovery-enabled=false \
  \
  # Network configuration
  --p2p-host=0.0.0.0 \
  --p2p-port=30303 \
  --nat-method=NONE \
  \
  # RPC
  --rpc-http-enabled=true \
  --rpc-http-host=0.0.0.0 \
  --rpc-http-port=8545 \
  --rpc-http-api=ETH,NET,ADMIN,QBFT \
  \
  # Permissions still recommended for extra security
  --permissions-nodes-config-file-enabled=true \
  --permissions-nodes-config-file=/etc/besu/permissions-nodes.toml

Key differences:

--discovery-enabled=false: No peer discovery, only static connections
--nat-method=NONE: No NAT traversal needed
Still use permissions-nodes.toml for defense-in-depth

3.3 TOML Configuration File Example

# /etc/besu/config.toml
data-path="/data/besu"
genesis-file="/etc/besu/genesis.json"

# Network
network-id=138
p2p-host="0.0.0.0"
p2p-port=30303

# Permissions
permissions-nodes-config-file-enabled=true
permissions-nodes-config-file="/etc/besu/permissions-nodes.toml"

# Static nodes
static-nodes-file="/etc/besu/static-nodes.json"

# Discovery
discovery-enabled=true

# RPC
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","ADMIN","QBFT"]

# Consensus (QBFT)
miner-enabled=false

4. Validation & Verification

4.1 Validate Enode Format (Pre-Deployment)

#!/bin/bash
# Validate all enodes in generated files
# File: validate-allowlist.sh

set -euo pipefail

STATIC_NODES="${1:-static-nodes.json}"
PERMISSIONS_TOML="${2:-permissions-nodes.toml}"

ERRORS=0

validate_enode_file() {
    local file="$1"
    local file_type="$2"
    
    echo "Validating $file_type: $file"
    
    if [[ "$file_type" == "json" ]]; then
        # JSON format (static-nodes.json)
        python3 << PYEOF
import json
import re
import sys

with open('$file', 'r') as f:
    enodes = json.load(f)

errors = 0
node_ids_seen = set()
endpoints_seen = set()

for i, enode in enumerate(enodes):
    # Extract node ID
    match = re.match(r'enode://([0-9a-fA-F]+)@([0-9.]+):(\d+)', enode)
    if not match:
        print(f"ERROR: Invalid enode format at index {i}: {enode}")
        errors += 1
        continue
    
    node_id = match.group(1).lower()
    endpoint = f"{match.group(2)}:{match.group(3)}"
    
    # Check length
    if len(node_id) != 128:
        print(f"ERROR: Node ID length {len(node_id)} at index {i} (expected 128): {node_id[:32]}...")
        errors += 1
        continue
    
    # Check hex
    if not re.match(r'^[0-9a-f]{128}$', node_id):
        print(f"ERROR: Invalid hex in node ID at index {i}: {node_id[:32]}...")
        errors += 1
        continue
    
    # Check duplicates
    if node_id in node_ids_seen:
        print(f"WARNING: Duplicate node ID at index {i}: {node_id[:32]}...")
    node_ids_seen.add(node_id)
    
    if endpoint in endpoints_seen:
        print(f"WARNING: Duplicate endpoint at index {i}: {endpoint}")
    endpoints_seen.add(endpoint)

sys.exit(errors)
PYEOF
        ERRORS=$((ERRORS + $?))
    else
        # TOML format (permissions-nodes.toml)
        python3 << PYEOF
import re
import sys

with open('$file', 'r') as f:
    content = f.read()

# Extract all enodes from TOML
enodes = re.findall(r'"enode://([0-9a-fA-F]+)@([0-9.]+):(\d+)"', content)

errors = 0
node_ids_seen = set()
endpoints_seen = set()

for i, (node_id_hex, ip, port) in enumerate(enodes):
    node_id = node_id_hex.lower()
    endpoint = f"{ip}:{port}"
    
    if len(node_id) != 128:
        print(f"ERROR: Node ID length {len(node_id)} at entry {i+1} (expected 128): {node_id[:32]}...")
        errors += 1
        continue
    
    if not re.match(r'^[0-9a-f]{128}$', node_id):
        print(f"ERROR: Invalid hex in node ID at entry {i+1}: {node_id[:32]}...")
        errors += 1
        continue
    
    if node_id in node_ids_seen:
        print(f"WARNING: Duplicate node ID at entry {i+1}: {node_id[:32]}...")
    node_ids_seen.add(node_id)
    
    if endpoint in endpoints_seen:
        print(f"WARNING: Duplicate endpoint at entry {i+1}: {endpoint}")
    endpoints_seen.add(endpoint)

sys.exit(errors)
PYEOF
        ERRORS=$((ERRORS + $?))
    fi
}

validate_enode_file "$STATIC_NODES" "json"
validate_enode_file "$PERMISSIONS_TOML" "toml"

if [[ $ERRORS -eq 0 ]]; then
    echo "✓ All enodes validated successfully"
    exit 0
else
    echo "✗ Validation failed with $ERRORS errors"
    exit 1
fi

4.2 Verify Peers Connected (Runtime)

#!/bin/bash
# Check peer connections on all nodes
# File: verify-peers.sh

set -euo pipefail

RPC_URL="${1:-http://localhost:8545}"

echo "Checking peers on: $RPC_URL"
echo ""

# Get node info
NODE_INFO=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
    "${RPC_URL}")

ENODE=$(echo "$NODE_INFO" | python3 -c "import sys, json; print(json.load(sys.stdin).get('result', {}).get('enode', 'ERROR'))" 2>/dev/null)

if [[ "$ENODE" == "ERROR" ]] || [[ -z "$ENODE" ]]; then
    echo "ERROR: Could not get node info. Is RPC enabled with ADMIN API?"
    exit 1
fi

echo "This node's enode:"
echo "$ENODE"
echo ""

# Get peers
PEERS_RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":2}' \
    "${RPC_URL}")

PEERS=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; peers=json.load(sys.stdin).get('result', []); print(len(peers))" 2>/dev/null)
PEERS_LIST=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; peers=json.load(sys.stdin).get('result', []); [print(f\"  - {p.get('enode', 'unknown')}\") for p in peers]" 2>/dev/null)

echo "Connected peers: $PEERS"
echo ""

if [[ "$PEERS" == "0" ]]; then
    echo "⚠️  NO PEERS CONNECTED"
    echo ""
    echo "Possible causes:"
    echo "1. Other nodes not running"
    echo "2. Firewall blocking port 30303"
    echo "3. Malformed enodes in allowlist"
    echo "4. Discovery disabled and static-nodes.json incorrect"
    echo "5. Permissions enabled but allowlist missing this node"
    echo "6. Network connectivity issues"
else
    echo "Peer list:"
    echo "$PEERS_LIST"
fi

# Check peer details
if [[ "$PEERS" != "0" ]] && command -v jq >/dev/null 2>&1; then
    echo ""
    echo "Peer details:"
    echo "$PEERS_RESPONSE" | jq -r '.result[] | "  - \(.id): \(.name) @ \(.network.remoteAddress)"'
fi

Usage:

# Check local node
bash verify-peers.sh

# Check remote node
bash verify-peers.sh http://192.168.11.13:8545

# Check all nodes
for ip in 192.168.11.{13,14,15,16,18,19,20,21,22,23,24,25}; do
    echo "=== Node $ip ==="
    bash verify-peers.sh "http://${ip}:8545"
    echo ""
done

4.3 Troubleshooting Decision Tree

#!/bin/bash
# Comprehensive troubleshooting script
# File: troubleshoot-besu-peers.sh

set -euo pipefail

NODE_IP="${1:-192.168.11.13}"
RPC_PORT="${2:-8545}"
RPC_URL="http://${NODE_IP}:${RPC_PORT}"

echo "╔════════════════════════════════════════════════════════════════╗"
echo "║         BESU PEER CONNECTION TROUBLESHOOTING                    ║"
echo "╚════════════════════════════════════════════════════════════════╝"
echo ""
echo "Node: $NODE_IP"
echo ""

# 1. Check if node is running
echo "1. Checking if Besu is running..."
if ! curl -s -m 2 "${RPC_URL}" > /dev/null 2>&1; then
    echo "   ✗ Node not responding to RPC"
    echo "   → Check: systemctl status besu-validator (or besu-sentry/besu-rpc)"
    exit 1
fi
echo "   ✓ Node is running"

# 2. Check RPC API access
echo ""
echo "2. Checking RPC API access..."
RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":1}' \
    "${RPC_URL}")

if echo "$RESPONSE" | grep -q "error"; then
    echo "   ✗ RPC error (check --rpc-http-api includes NET,ADMIN)"
else
    echo "   ✓ RPC accessible"
fi

# 3. Get node info
echo ""
echo "3. Getting node information..."
NODE_INFO=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":2}' \
    "${RPC_URL}")

ENODE=$(echo "$NODE_INFO" | python3 -c "import sys, json; print(json.load(sys.stdin).get('result', {}).get('enode', ''))" 2>/dev/null)

if [[ -z "$ENODE" ]]; then
    echo "   ✗ Could not get enode (ADMIN API may not be enabled)"
    echo "   → Fix: Add ADMIN to --rpc-http-api"
else
    echo "   ✓ Enode: $ENODE"
    
    # Validate enode
    NODE_ID=$(echo "$ENODE" | sed 's|^enode://||' | cut -d'@' -f1)
    if [[ ${#NODE_ID} -ne 128 ]]; then
        echo "   ⚠️  WARNING: Node ID length is ${#NODE_ID}, not 128!"
    fi
fi

# 4. Check peers
echo ""
echo "4. Checking peer connections..."
PEERS_RESPONSE=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":3}' \
    "${RPC_URL}")

PEER_COUNT=$(echo "$PEERS_RESPONSE" | python3 -c "import sys, json; print(len(json.load(sys.stdin).get('result', [])))" 2>/dev/null)

echo "   Connected peers: $PEER_COUNT"

if [[ "$PEER_COUNT" == "0" ]]; then
    echo ""
    echo "   ═══════════════════════════════════════════════════════"
    echo "   TROUBLESHOOTING: No peers connected"
    echo "   ═══════════════════════════════════════════════════════"
    echo ""
    
    # Check static-nodes.json
    echo "   4a. Checking static-nodes.json..."
    if ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" "test -f /etc/besu/static-nodes.json" 2>/dev/null; then
        STATIC_COUNT=$(ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" \
            "python3 -c \"import json; print(len(json.load(open('/etc/besu/static-nodes.json'))))\" 2>/dev/null" || echo "0")
        echo "      → Found static-nodes.json with $STATIC_COUNT entries"
    else
        echo "      ✗ static-nodes.json not found"
    fi
    
    # Check permissions-nodes.toml
    echo ""
    echo "   4b. Checking permissions-nodes.toml..."
    if ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" "test -f /etc/besu/permissions-nodes.toml" 2>/dev/null; then
        PERM_COUNT=$(ssh -o StrictHostKeyChecking=accept-new "root@${NODE_IP}" \
            "grep -c 'enode://' /etc/besu/permissions-nodes.toml 2>/dev/null" || echo "0")
        echo "      → Found permissions-nodes.toml with $PERM_COUNT entries"
    else
        echo "      ✗ permissions-nodes.toml not found"
    fi
    
    # Check discovery
    echo ""
    echo "   4c. Checking discovery configuration..."
    # This would require parsing config file or checking logs
    echo "      → Check: --discovery-enabled flag in config"
    echo "      → If false, static-nodes.json must be correct"
    echo "      → If true, ensure permissions allowlist includes all nodes"
    
    # Check firewall
    echo ""
    echo "   4d. Checking network connectivity..."
    echo "      → Test: nc -zv <peer-ip> 30303"
    echo "      → Check: iptables/firewalld rules"
    echo "      → Verify: Port 30303 is open and accessible"
    
    # Check advertised host
    echo ""
    echo "   4e. Checking advertised host/IP..."
    ADVERTISED_IP=$(echo "$ENODE" | sed 's|.*@||' | cut -d':' -f1)
    echo "      → Advertised IP: $ADVERTISED_IP"
    if [[ "$ADVERTISED_IP" != "$NODE_IP" ]] && [[ "$ADVERTISED_IP" != "0.0.0.0" ]]; then
        echo "      ⚠️  WARNING: Advertised IP ($ADVERTISED_IP) differs from actual IP ($NODE_IP)"
        echo "      → Fix: Set --p2p-host=$NODE_IP or --nat-method correctly"
    fi
fi

echo ""
echo "═══════════════════════════════════════════════════════════════"
echo "Troubleshooting complete"
echo "═══════════════════════════════════════════════════════════════"

5. Deployment Script

#!/bin/bash
# Deploy corrected allowlist files to all containers
# File: deploy-allowlist.sh

set -euo pipefail

PROXMOX_HOST="${PROXMOX_HOST:-192.168.11.10}"
STATIC_NODES_FILE="${1:-static-nodes.json}"
PERMISSIONS_TOML_FILE="${2:-permissions-nodes.toml}"

if [[ ! -f "$STATIC_NODES_FILE" ]] || [[ ! -f "$PERMISSIONS_TOML_FILE" ]]; then
    echo "ERROR: Files not found:"
    [[ ! -f "$STATIC_NODES_FILE" ]] && echo "  - $STATIC_NODES_FILE"
    [[ ! -f "$PERMISSIONS_TOML_FILE" ]] && echo "  - $PERMISSIONS_TOML_FILE"
    exit 1
fi

# Validate files first
echo "Validating files before deployment..."
bash validate-allowlist.sh "$STATIC_NODES_FILE" "$PERMISSIONS_TOML_FILE" || {
    echo "ERROR: Validation failed. Fix files before deploying."
    exit 1
}

echo ""
echo "Deploying to Proxmox host: $PROXMOX_HOST"

# Copy files to host
scp -o StrictHostKeyChecking=accept-new \
    "$STATIC_NODES_FILE" \
    "$PERMISSIONS_TOML_FILE" \
    "root@${PROXMOX_HOST}:/tmp/"

# Deploy to all containers
for vmid in 106 107 108 109 110 111 112 113 114 115 116 117; do
    echo "Deploying to container $vmid..."
    
    ssh -o StrictHostKeyChecking=accept-new "root@${PROXMOX_HOST}" << DEPLOY_SCRIPT
if ! pct status $vmid 2>/dev/null | grep -q running; then
    echo "  Container $vmid not running, skipping"
    exit 0
fi

# Copy files
pct push $vmid /tmp/$(basename $STATIC_NODES_FILE) /etc/besu/static-nodes.json
pct push $vmid /tmp/$(basename $PERMISSIONS_TOML_FILE) /etc/besu/permissions-nodes.toml

# Set ownership
pct exec $vmid -- chown besu:besu /etc/besu/static-nodes.json /etc/besu/permissions-nodes.toml

# Verify
if pct exec $vmid -- test -f /etc/besu/static-nodes.json && \
   pct exec $vmid -- test -f /etc/besu/permissions-nodes.toml; then
    echo "  ✓ Container $vmid: Files deployed"
else
    echo "  ✗ Container $vmid: Deployment failed"
fi
DEPLOY_SCRIPT

done

# Cleanup
ssh -o StrictHostKeyChecking=accept-new "root@${PROXMOX_HOST}" \
    "rm -f /tmp/$(basename $STATIC_NODES_FILE) /tmp/$(basename $PERMISSIONS_TOML_FILE)"

echo ""
echo "✓ Deployment complete"
echo ""
echo "Next steps:"
echo "1. Restart Besu services on all containers"
echo "2. Run verification: bash verify-peers.sh <rpc-url>"

Runbook Summary

Execute in this order:

Extract enodes from each node:

# On each node, or via SSH/management host
bash extract-enode-from-nodekey.sh > enode-<ip>.txt
# OR
export RPC_URL="http://<ip>:8545"
bash extract-enode-from-rpc.sh > enode-<ip>.txt

Collect all enodes:

# Update NODES array in script with your IPs
bash collect-all-enodes.sh

Generate allowlist files:

bash generate-allowlist-files.sh ${WORK_DIR}/collected-enodes.txt

Validate generated files:

bash validate-allowlist.sh static-nodes.json permissions-nodes.toml

Deploy to all containers:

bash deploy-allowlist.sh static-nodes.json permissions-nodes.toml

Restart Besu services:

# On Proxmox host
for vmid in 106 107 108 109 110 111 112 113 114 115 116 117; do
    pct exec $vmid -- systemctl restart besu-validator 2>/dev/null || \
    pct exec $vmid -- systemctl restart besu-sentry 2>/dev/null || \
    pct exec $vmid -- systemctl restart besu-rpc 2>/dev/null
done

Verify peer connections:

# Check all nodes
for ip in 192.168.11.{13,14,15,16,18,19,20,21,22,23,24,25}; do
    echo "=== $ip ==="
    bash verify-peers.sh "http://${ip}:8545"
done

Troubleshoot if needed:

bash troubleshoot-besu-peers.sh <node-ip>

Common Pitfalls & Warnings

⚠️ Padding Zeros: Never pad node IDs with trailing zeros. Besu will reject them.

⚠️ Multiple Enodes per IP:PORT: One node = one enode. Keep only the enode returned by that node's admin_nodeInfo.

⚠️ P2P Host Configuration: Set --p2p-host to the actual IP or use --nat-method=UPNP for correct advertisement.

⚠️ RPC API: Must include ADMIN in --rpc-http-api to use admin_nodeInfo and admin_peers.

⚠️ Discovery vs Permissions: If discovery-enabled=false, static-nodes.json must be correct. If permissions-nodes-enabled=true, all peers must be in allowlist.

⚠️ File Ownership: Ensure files are owned by besu:besu user.

⚠️ Service Restart: Always restart Besu services after updating allowlist files.

32 KiB Raw Blame History