proxmox/docs/04-configuration/DEPLOYMENT_CHECKLIST.md

Besu Configuration Deployment Checklist

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-17
Purpose: Step-by-step checklist for deploying cleaned Besu configurations to running nodes

---

Pre-Deployment Verification

✅ Step 1: Verify Configuration Readiness

# Run automated readiness check
./scripts/verify-deployment-readiness.sh

Expected: All checks pass ✅

Manual verification:

• All 16 configuration files exist
• All configs validated (0 errors)
• No deprecated options detected
• Deployment scripts executable

---

✅ Step 2: Validate Configurations

# Validate all configs
./scripts/validate-besu-config.sh

# Expected output:
# Total files validated: 16
# Passed: 16
# Failed: 0

Verify:

• All configs pass validation
• No syntax errors
• No deprecated options
• All required options present

---

✅ Step 3: Audit Configurations

# Compare configs to templates
./scripts/audit-besu-configs.sh

# Expected output:
# Total configs audited: 11
# Matching templates: 10+
# Drift detected: 0

Verify:

• All configs match templates (or expected differences documented)
• No unexpected drift detected

---

Deployment Process

📋 Step 4: Review Deployment Plan

Deployment Order (rolling deployment):

1. Validators (1000-1004) - First
2. Sentries (1500-1503) - Second
3. RPC Nodes (2500+) - Last

Process per Node:

1. Backup existing config
2. Copy cleaned config to node
3. Validate config on node
4. Restart service gracefully
5. Verify service started successfully
6. Wait 30 seconds before next node

---

🧪 Step 5: Dry-Run Deployment

# Preview deployment (no actual changes)
./scripts/deploy-besu-configs.sh --dry-run

Review:

• All nodes identified correctly
• Config mappings correct (VMID → config file)
• Service names correct
• Deployment order correct (validators → sentries → RPC)

Fix any issues before proceeding.

---

🚀 Step 6: Deploy to Validators

# Deploy to validators (1000-1004)
./scripts/deploy-besu-configs.sh

Monitor during deployment:

• Each validator config backed up
• Config copied successfully
• Service restarts successfully
• No errors in deployment output

Wait 30 seconds between validators.

After validators complete:

• Check validator services: pct exec 1000 -- systemctl status besu-validator.service
• Verify logs: pct exec 1000 -- journalctl -u besu-validator.service -n 20
• Check logging level (should be WARN, minimal logs)
• No configuration errors in logs

---

🚀 Step 7: Deploy to Sentries

After validators stable (wait 5-10 minutes):

# Deploy to sentries (1500-1503)
./scripts/deploy-besu-configs.sh

Monitor during deployment:

• Each sentry config backed up
• Config copied successfully
• Service restarts successfully
• No errors in deployment output

Wait 30 seconds between sentries.

After sentries complete:

• Check sentry services: pct exec 1500 -- systemctl status besu-sentry.service
• Verify logs: pct exec 1500 -- journalctl -u besu-sentry.service -n 20
• Check logging level (should be INFO, detailed logs)
• No configuration errors in logs

---

🚀 Step 8: Deploy to RPC Nodes

After sentries stable (wait 5-10 minutes):

# Deploy to RPC nodes (2500+)
./scripts/deploy-besu-configs.sh

Monitor during deployment:

• Each RPC config backed up
• Config copied successfully
• Service restarts successfully
• No errors in deployment output

Wait 30 seconds between RPC nodes.

After RPC nodes complete:

• Check RPC services: pct exec 2500 -- systemctl status besu-rpc.service
• Verify logs: pct exec 2500 -- journalctl -u besu-rpc.service -n 20
• Check logging level (should be WARN, minimal logs)
• No configuration errors in logs

---

Post-Deployment Verification

✅ Step 9: Immediate Verification (0-1 hour)

Service Status:

# Check all validator services
for vmid in 1000 1001 1002 1003 1004; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-validator.service
done

# Check all sentry services
for vmid in 1500 1501 1502 1503; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-sentry.service
done

# Check all RPC services
for vmid in 2500 2501 2502 2503 2505 2506 2507 2508; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-rpc.service
done

Expected: All services active ✅

Configuration Verification:

# Verify logging levels on sample nodes
# Validator (should be WARN)
pct exec 1000 -- grep "^logging" /etc/besu/config-validator.toml

# Sentry (should be INFO)
pct exec 1500 -- grep "^logging" /etc/besu/config-sentry.toml

# RPC (should be WARN)
pct exec 2500 -- grep "^logging" /etc/besu/config-rpc-core.toml

Expected:

• Validators: logging="WARN"
• Sentries: logging="INFO"
• RPC: logging="WARN"

Error Checking:

# Check for configuration errors
for vmid in 1000 1500 2500; do
    echo "VMID $vmid:"
    pct exec $vmid -- journalctl -u besu-*.service --since "10 minutes ago" | grep -i "error\|unknown option" | head -5
done

Expected: No configuration errors ✅

---

✅ Step 10: Functional Verification (1-6 hours)

Validator Verification:

• Validators participating in consensus
• No validator errors in logs
• Validators synced with network

Sentry Verification (Archive Nodes):

• Sentries serving archive queries
• Test historical query: curl -X POST http://192.168.11.150:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x0000000000000000000000000000000000000000","0x100"],"id":1}'
• Sentries synced with network

RPC Verification:

• RPC endpoints responding
• Test RPC: curl -X POST http://192.168.11.250:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
• CORS working (if applicable)

Network Connectivity:

• Validators connected to sentries
• Sentries connected to validators and external peers
• RPC nodes connected to internal peers

---

✅ Step 11: Performance Monitoring (6-48 hours)

Resource Usage:

• Memory usage within expected ranges
• CPU usage normal
• Disk I/O acceptable

Service Stability:

• No restart loops
• Services stable for 24+ hours
• No configuration drift detected

Performance Metrics:

• Block propagation normal
• RPC response times acceptable
• Archive queries performing well

---

Troubleshooting

Issue: Service Fails to Start

Check:

pct exec <VMID> -- systemctl status besu-*.service
pct exec <VMID> -- journalctl -u besu-*.service -n 50

Common Causes:

1. Configuration syntax error
2. Deprecated options still present
3. Invalid option values

Fix:

1. Restore from backup: /etc/besu/config-*.toml.backup.*
2. Validate config: ./scripts/validate-besu-config.sh
3. Review error logs for specific issues

---

Issue: Logging Level Not Applied

Check:

# Verify config file
pct exec <VMID> -- grep "^logging" /etc/besu/config-*.toml

# Check actual logs
pct exec <VMID> -- journalctl -u besu-*.service -n 20

Fix:

1. Verify config file was updated
2. Ensure service was restarted
3. Check file permissions

---

Issue: Configuration Drift

Check:

./scripts/audit-besu-configs.sh

Fix:

1. Re-deploy from templates
2. Verify no manual changes were made
3. Document any intentional differences

---

Rollback Procedure

If deployment fails or issues occur:

Immediate Rollback

# For each node, restore from backup
pct exec <VMID> -- cp /etc/besu/config-*.toml.backup.<timestamp> /etc/besu/config-*.toml
pct exec <VMID> -- systemctl restart besu-*.service

Full Rollback

1. Stop deployment immediately
2. Restore all nodes from backups (created by deploy script)
3. Restart all services
4. Verify services running correctly
5. Investigate issues before retrying

---

Deployment Sign-Off

Pre-Deployment Sign-Off

• All configurations validated
• Dry-run successful
• Deployment plan reviewed
• Rollback procedure understood
• Monitoring plan ready

Sign-off: _________________ Date: ___________

---

Post-Deployment Sign-Off

After 48 hours of stable operation:

• All services running correctly
• No configuration errors
• Logging levels correct
• Network connectivity normal
• Performance metrics acceptable
• No configuration drift detected

Sign-off: _________________ Date: ___________

---

Related Documentation

• BESU_DEPLOYMENT_MONITORING.md - Detailed monitoring guide
• BESU_CONFIGURATION_GUIDE.md - Configuration reference
• BESU_IMPLEMENTATION_COMPLETE.md - Implementation summary

---

Last Updated: 2026-01-17
Status: Ready for Deployment