proxmox/docs/archive/BESU_LOGS_SUMMARY.md

Besu Logs Analysis Summary

Date: $(date)
Analysis: Complete log review of all Besu services in LXC containers

---

Executive Summary

🔴 CRITICAL ISSUE: All Besu services are failing due to missing configuration files.

Status:

• All services in restart loops (45-54 restarts each)
• Services start → fail immediately → systemd restarts → repeat
• Root Cause: Configuration files not found

---

Service Status Overview

Category	VMID Range	Service Status	Restart Count	Config Status
Validators	1000-1004	🔴 Failing	47-54	❌ Missing
Sentries	1500-1502	🔴 Failing	47-53	❌ Missing
Sentry	1503	⚠️ Inactive	N/A	❌ Service file missing
RPC Nodes	2500-2502	🔴 Failing	45-52	❌ Missing

---

Error Pattern (All Services)

Common Error Message

Unable to read TOML configuration, file not found.
To display full help:
besu [COMMAND] --help

Service Restart Loop

1. Service starts (systemd)
2. Besu process begins
3. Besu fails to read config file
4. Process exits immediately
5. Systemd restarts service (after 10 seconds)
6. Loop repeats

Restart Counter: Services have restarted 45-54 times, indicating this issue has persisted for an extended period.

---

Detailed Logs by Service Type

Validators (VMID 1000-1004)

Sample Log (VMID 1000):

Dec 20 15:51:07 besu-validator-1 systemd[1]: Started Hyperledger Besu Validator Node.
Dec 20 15:51:10 besu-validator-1 besu-validator[2160]: Unable to read TOML configuration, file not found.
Dec 20 15:51:10 besu-validator-1 systemd[1]: besu-validator.service: Deactivated successfully.
Dec 20 15:51:21 systemd[1]: besu-validator.service: Scheduled restart job, restart counter is at 54.

All Validators: Showing identical pattern with restart counters 47-54.

Sentries (VMID 1500-1502)

Sample Log (VMID 1500):

Dec 20 15:51:12 besu-sentry-1 systemd[1]: Started Hyperledger Besu Sentry Node.
Dec 20 15:51:18 besu-sentry-1 besu-sentry[16206]: Unable to read TOML configuration, file not found.
Dec 20 15:51:18 besu-sentry-1 systemd[1]: besu-sentry.service: Deactivated successfully.
Dec 20 15:51:29 systemd[1]: besu-sentry.service: Scheduled restart job, restart counter is at 48.

All Sentries: Showing identical pattern with restart counters 47-53.

Note: VMID 1503 has no service file, so no logs to review.

RPC Nodes (VMID 2500-2502)

Sample Log (VMID 2500):

Dec 20 15:51:22 besu-rpc-1 systemd[1]: Started Hyperledger Besu RPC Node.
Dec 20 15:51:25 besu-rpc-1 besu-rpc[16213]: Unable to read TOML configuration, file not found.
Dec 20 15:51:25 besu-rpc-1 systemd[1]: besu-rpc.service: Deactivated successfully.
Dec 20 15:51:35 systemd[1]: besu-rpc.service: Scheduled restart job, restart counter is at 52.

All RPC Nodes: Showing identical pattern with restart counters 45-52.

---

Root Cause Analysis

Expected Configuration Files

Service Type	Expected Path	Status
Validators	/etc/besu/config-validator.toml	❌ Missing
Sentries	/etc/besu/config-sentry.toml	❌ Missing
RPC Nodes	/etc/besu/config-rpc.toml	❌ Missing

Actual Files Found

File	Status	Location
config-validator.toml.template	✅ Exists	/etc/besu/config-validator.toml.template
config-validator.toml	❌ Missing	Should be at /etc/besu/config-validator.toml

Service Configuration

Service files are correctly configured to use:

ExecStart=/opt/besu/bin/besu \
    --config-file=/etc/besu/config-validator.toml

Issue: The file /etc/besu/config-validator.toml does not exist.

---

Impact Assessment

Immediate Impact

• ❌ No Besu nodes are running - All services failing
• ❌ Network not operational - No consensus, no block production
• ❌ Resources wasted - Containers running but services restarting constantly
• ⚠️ High restart counts - Systemd has attempted 45-54 restarts per service

Service Health

Metric	Value	Status
Services Running	0 / 12	🔴 Critical
Services in Restart Loop	11 / 12	🔴 Critical
Average Restart Count	~50	🔴 Critical
Configuration Files Present	0 / 11	🔴 Critical

---

Required Actions

Immediate (Priority 1)

1. Copy Configuration Files

   • Option A: Copy template to config file (quick fix)
   • Option B: Copy from source project (recommended)

2. Verify File Permissions

   • Ensure files owned by besu:besu
   • Ensure readable permissions

3. Restart Services

   • Restart services after copying config files
   • Monitor logs to verify successful startup

High Priority (Priority 2)

4. Verify Additional Required Files

   • /etc/besu/genesis.json
   • /etc/besu/static-nodes.json
   • /etc/besu/permissions-nodes.toml

5. Check Validator Keys (for validators only)

   • Verify keys exist in /keys/validators/

Medium Priority (Priority 3)

6. Fix VMID 1503 Service File

   • Create service file if missing
   • Or verify if this container was intentionally excluded

7. Monitor Services

   • Watch logs after restart
   • Verify services stay running
   • Check for additional errors

---

Quick Fix Command

Copy Template to Config (Quick Solution)

# Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- cp /etc/besu/config-validator.toml.template /etc/besu/config-validator.toml
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
    pct exec $vmid -- systemctl restart besu-validator.service
done

# Sentries (if templates exist)
for vmid in 1500 1501 1502; do
    pct exec $vmid -- cp /etc/besu/config-sentry.toml.template /etc/besu/config-sentry.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-sentry.toml 2>/dev/null
    pct exec $vmid -- systemctl restart besu-sentry.service
done

# RPC Nodes (if templates exist)
for vmid in 2500 2501 2502; do
    pct exec $vmid -- cp /etc/besu/config-rpc.toml.template /etc/besu/config-rpc.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-rpc.toml 2>/dev/null
    pct exec $vmid -- systemctl restart besu-rpc.service
done

Note: This uses template configurations which may need customization for production use.

---

Verification Commands

After applying fixes:

# Check if config files exist
for vmid in 1000 1001 1002 1003 1004 1500 1501 1502 2500 2501 2502; do
    if [[ $vmid -lt 1500 ]]; then
        config_file="config-validator.toml"
        service="besu-validator"
    elif [[ $vmid -lt 2500 ]]; then
        config_file="config-sentry.toml"
        service="besu-sentry"
    else
        config_file="config-rpc.toml"
        service="besu-rpc"
    fi
    
    echo "=== VMID $vmid ==="
    pct exec $vmid -- ls -la /etc/besu/$config_file 2>/dev/null && echo "✅ Config exists" || echo "❌ Config missing"
    pct exec $vmid -- systemctl is-active $service.service 2>/dev/null && echo "✅ Service active" || echo "❌ Service inactive"
    echo ""
done

---

Related Documentation

• Configuration Issue Details
• Current Deployment Status
• Files Copy Checklist

---

Analysis Completed: $(date)
Status: 🔴 CRITICAL - IMMEDIATE ACTION REQUIRED