proxmox/docs/archive/BESU_CONFIGURATION_ISSUE.md

Critical Issue: Missing Besu Configuration Files

Date: $(date)
Severity: 🔴 CRITICAL
Impact: All Besu services failing in restart loop

---

Issue Summary

All Besu services across all LXC containers are failing with the error:

Unable to read TOML configuration, file not found.

Services Affected:

• ✅ Validators (1000-1004): All failing
• ✅ Sentries (1500-1502): All failing
• ✅ RPC Nodes (2500-2502): All failing
• ⚠️ Sentry 1503: Service file missing

---

Root Cause

The systemd services are configured to use:

• Expected Path: /etc/besu/config-validator.toml (validators)
• Expected Path: /etc/besu/config-sentry.toml (sentries)
• Expected Path: /etc/besu/config-rpc.toml (RPC nodes)

Actual Status: Only template files exist:

• /etc/besu/config-validator.toml.template ✅ (exists)
• /etc/besu/config-validator.toml ❌ (missing)

---

Service Status

All services are in a restart loop:

Node Type	VMID Range	Restart Count	Status
Validators	1000-1004	47-54 restarts	🔴 Failing
Sentries	1500-1502	47-53 restarts	🔴 Failing
RPC Nodes	2500-2502	45-52 restarts	🔴 Failing

Error Pattern: Service starts → fails immediately (config file not found) → systemd restarts → repeat

---

Verification

What's Missing

# Service expects:
/etc/besu/config-validator.toml  ❌ NOT FOUND

# What exists:
/etc/besu/config-validator.toml.template  ✅ EXISTS

Service Configuration

The systemd service files reference:

ExecStart=/opt/besu/bin/besu \
    --config-file=/etc/besu/config-validator.toml

---

Solution Options

Option 1: Copy Template to Config File (Quick Fix)

Copy the template files to the actual config files:

# For Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- cp /etc/besu/config-validator.toml.template /etc/besu/config-validator.toml
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
done

# For Sentries
for vmid in 1500 1501 1502 1503; do
    pct exec $vmid -- cp /etc/besu/config-sentry.toml.template /etc/besu/config-sentry.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-sentry.toml 2>/dev/null
done

# For RPC Nodes
for vmid in 2500 2501 2502; do
    pct exec $vmid -- cp /etc/besu/config-rpc.toml.template /etc/besu/config-rpc.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-rpc.toml 2>/dev/null
done

Note: This uses template configuration which may need customization.

Option 2: Copy from Source Project (Recommended)

Copy actual configuration files from the source project:

# Assuming source project is at /opt/smom-dbis-138 on Proxmox host

# Validators
for vmid in 1000 1001 1002 1003 1004; do
    # Determine validator number (1-5)
    validator_num=$((vmid - 999))
    
    # Copy from source project (adjust path as needed)
    # Option A: If node-specific configs exist
    pct push $vmid /opt/smom-dbis-138/config/nodes/validator-${validator_num}/config-validator.toml \
        /etc/besu/config-validator.toml
    
    # Option B: If single template exists
    pct push $vmid /opt/smom-dbis-138/config/config-validator.toml \
        /etc/besu/config-validator.toml
    
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
done

# Similar for sentries and RPC nodes

Option 3: Run Configuration Deployment Script

Use the deployment scripts to properly copy and configure files:

cd /opt/smom-dbis-138-proxmox
# Check for config copy scripts
./scripts/deployment/copy-configs-to-containers.sh /opt/smom-dbis-138

---

Additional Required Files

Even after fixing the main config files, ensure these files exist:

Required for All Nodes

• ✅ /etc/besu/genesis.json - Network genesis block
• ✅ /etc/besu/static-nodes.json - Static peer list
• ✅ /etc/besu/permissions-nodes.toml - Node permissions

Required for Validators

• ✅ /keys/validators/validator-*/ - Validator signing keys

---

Verification After Fix

After copying configuration files, verify:

# Check if config files exist
for vmid in 1000 1001 1002 1003 1004; do
    echo "VMID $vmid:"
    pct exec $vmid -- ls -la /etc/besu/config-validator.toml
done

# Restart services
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- systemctl restart besu-validator.service
    sleep 2
    pct exec $vmid -- systemctl status besu-validator.service --no-pager | head -10
done

# Check logs for errors
for vmid in 1000 1001 1002 1003 1004; do
    echo "=== VMID $vmid ==="
    pct exec $vmid -- journalctl -u besu-validator.service --since "1 minute ago" --no-pager | tail -10
done

---

Current Logs Summary

All services showing identical error pattern:

Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: Unable to read TOML configuration, file not found.
Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: To display full help:
Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: besu [COMMAND] --help
Dec 20 15:51:XX besu-validator-X systemd[1]: besu-validator.service: Deactivated successfully.

Restart Counter: Services have restarted 45-54 times each, indicating this has been failing for an extended period.

---

Priority Actions

1. 🔴 URGENT: Copy configuration files to all containers
2. 🔴 URGENT: Restart services after fixing config files
3. ⚠️ HIGH: Verify all required files (genesis.json, static-nodes.json, etc.)
4. ⚠️ HIGH: Check service logs after restart to ensure proper startup
5. 📋 MEDIUM: Verify validator keys are in place (for validators only)

---

Related Documentation

• Files Copy Checklist
• Path Reference
• Current Deployment Status

---

Issue Identified: $(date)
Status: 🔴 NEEDS IMMEDIATE ATTENTION