proxmox/docs/archive/NEXT_STEPS_BOOT_VALIDATED_SET.md

Next Steps: Script-Based Deployment & Boot Node for Validated Set

This document outlines the complete set of next steps needed to build out a working and functional deployment system using EITHER:

• Script-based approach: Automated scripts to deploy and configure the validated set
• Boot node approach: A dedicated boot node to bootstrap the network discovery

Both approaches can be used together or separately, depending on network requirements.

Overview

The goal is to create a comprehensive, production-ready deployment system that:

1. ✅ Deploys containers (already done)
2. 🔄 Properly bootstraps the network using scripts OR boot node (or both)
3. ✅ Validates and verifies the entire deployment
4. ✅ Ensures all validators are properly configured and connected
5. ✅ Provides end-to-end orchestration scripts

Two Approaches: Script vs Boot Node

Approach 1: Script-Based Deployment

Use when: Private/permissioned network with known static nodes, no external discovery needed

Characteristics:

• Uses static-nodes.json for peer discovery
• Scripts orchestrate deployment and configuration
• No dedicated boot node required
• All nodes listed statically
• Faster initial setup
• Recommended for your current setup (validators ↔ sentries topology)

Approach 2: Boot Node Deployment

Use when: Network needs dynamic peer discovery, external nodes will join later

Characteristics:

• Dedicated boot node for initial discovery
• Other nodes connect to boot node first
• Boot node helps discover additional peers
• More flexible for network expansion
• Required for public/open networks
• Can be combined with static nodes

Approach 3: Hybrid (Script + Boot Node)

Use when: Best of both worlds - script orchestration + boot node for discovery

Characteristics:

• Scripts handle deployment and configuration
• Boot node provides discovery service
• Static nodes for critical connections
• Boot node for dynamic discovery
• Most flexible approach

---

Phase 1: Script-Based Deployment (Primary Approach)

1.1 Create Validated Set Deployment Script

File: scripts/deployment/deploy-validated-set.sh

Purpose: Script-based deployment that orchestrates the entire validated set without requiring a boot node

Functionality:

• Deploy all containers (validators, sentries, RPC)
• Copy configuration files (genesis, static-nodes, permissions)
• Copy validator keys
• Start services in correct order (sentries → validators → RPC)
• Validate deployment
• Generate deployment report

Key Features:

• Uses static-nodes.json (no boot node needed)
• Sequential startup orchestration
• Comprehensive validation
• Error handling and rollback
• Detailed logging

Status: ❌ NOT CREATED (This is the PRIMARY script-based approach)

Alternative: If boot node is desired, see Phase 1A below

---

Phase 1A: Boot Node Deployment (Optional)

1A.1 Create Boot Node Deployment Script

File: scripts/deployment/deploy-boot-node.sh

Purpose: Deploy and configure a dedicated boot node (optional - only if using boot node approach)

Functionality:

• Deploy container with boot node configuration
• Configure as discovery/bootstrap node
• Expose only P2P port (30303) - no RPC
• Generate and export enode for use by other nodes
• Ensure boot node starts first before other nodes

Key Features:

• Special configuration for boot node (if separate)
• OR configure first validator (106) as boot node
• Generate boot node enode for inclusion in genesis or static-nodes
• Health checks to ensure boot node is ready before proceeding

Status: ❌ NOT CREATED (Optional - only if boot node approach is chosen)

Decision Point: Do you need a boot node, or can you use script-based static-nodes approach?

---

1.2 Create Network Bootstrap Script

File: scripts/network/bootstrap-network.sh

Purpose: Orchestrate the initial network bootstrap sequence (works with EITHER script-based or boot node approach)

Functionality (Script-Based Approach):

1. Extract enodes from all deployed containers
2. Generate static-nodes.json with all validator enodes
3. Deploy static-nodes.json to all nodes
4. Start nodes in sequence (sentries → validators → RPC)
5. Verify peer connections
6. Validate network is operational

Functionality (Boot Node Approach):

1. Start boot node first
2. Wait for boot node to be ready (P2P listening, enode available)
3. Extract boot node enode
4. Update static-nodes.json for all other nodes with boot node enode
5. Deploy static-nodes.json to all nodes
6. Start remaining nodes in sequence (sentries, then validators, then RPC)
7. Verify peer connections

Key Features:

• Supports both script-based and boot node approaches
• Sequential startup with dependencies
• Health checks between steps
• Automatic enode extraction
• Updates static-nodes.json dynamically
• Validates peer connections after startup

Status: ❌ NOT CREATED

Recommendation: Start with script-based approach (simpler for permissioned networks)

---

Phase 2: Validated Set Deployment

2.1 Create Validator Set Validation Script

File: scripts/validation/validate-validator-set.sh

Purpose: Validate that all validators are properly configured and can participate in consensus

Functionality:

• Check validator keys exist and are accessible
• Verify validator addresses match configuration
• Validate validator keys are loaded by Besu
• Check validators are in genesis (if static) or validator contract
• Verify validator services are running
• Check validators can connect to each other
• Validate consensus is active (blocks being produced)

Key Features:

• Comprehensive validator health checks
• Validator key validation
• Consensus participation verification
• Network connectivity checks
• QBFT-specific validation

Status: ❌ NOT CREATED

---

2.2 Create Validator Registration Script

File: scripts/validation/register-validators.sh

Purpose: Register validators in the validator contract (for dynamic validator management)

Functionality:

• Read validator addresses from key files
• Submit validator registration transactions
• Verify validator registration on-chain
• Wait for epoch change if needed
• Validate validators are active in consensus

Key Features:

• Smart contract interaction
• Transaction submission and verification
• Epoch management
• Validator set verification

Status: ❌ NOT CREATED (Note: Only needed if using dynamic validator management via contract)

---

2.3 Create Deployment Validation Orchestrator

File: scripts/validation/validate-deployment.sh

Purpose: Comprehensive end-to-end validation of the entire deployment

Functionality:

1. Validate container deployment (all containers running)
2. Validate network connectivity (P2P, RPC)
3. Validate configuration files (genesis, static-nodes, permissions)
4. Validate validator set (keys, addresses, consensus participation)
5. Validate sentry connectivity (can connect to validators)
6. Validate RPC endpoints (can query blockchain state)
7. Validate allowlist configuration (permissions-nodes.toml)
8. Generate validation report

Key Features:

• Multi-phase validation
• Comprehensive checks
• Detailed reporting
• Error collection and reporting
• Exit codes for CI/CD integration

Status: ⚠️ PARTIAL (Some validation exists, but not comprehensive)

---

Phase 3: End-to-End Deployment Orchestration

3.1 Create Complete Deployment Orchestrator

File: scripts/deployment/deploy-validated-set.sh

Purpose: Single script that orchestrates the entire validated set deployment

Functionality:

1. Pre-deployment Validation

   • Check prerequisites
   • Validate configuration
   • Check resources
   • Verify no conflicts

2. Deploy Containers

   • Deploy boot node (or first validator)
   • Deploy remaining validators
   • Deploy sentries
   • Deploy RPC nodes

3. Bootstrap Network

   • Start boot node
   • Extract boot node enode
   • Update static-nodes.json
   • Deploy configuration files
   • Start remaining nodes in correct order

4. Configure Validators

   • Copy validator keys
   • Register validators (if dynamic)
   • Verify validator set

5. Post-Deployment Validation

   • Run comprehensive validation
   • Verify consensus is active
   • Check all services
   • Generate deployment report

6. Rollback on Failure

   • Clean up partial deployments
   • Restore previous state if needed

Key Features:

• Single command deployment
• Error handling and rollback
• Progress reporting
• Detailed logging
• Validation at each step

Status: ❌ NOT CREATED

---

3.2 Create Quick Bootstrap Script

File: scripts/deployment/bootstrap-quick.sh

Purpose: Quick bootstrap for existing deployed containers

Functionality:

• Assume containers already deployed
• Extract boot node enode
• Update static-nodes.json
• Deploy updated configs
• Restart services in correct order
• Verify connectivity

Use Case: When containers are deployed but network needs to be bootstrapped/rebootstrapped

Status: ❌ NOT CREATED

---

Phase 4: Health Checks & Monitoring

4.1 Create Node Health Check Script

File: scripts/health/check-node-health.sh

Purpose: Check health of individual nodes

Functionality:

• Container status
• Service status (systemd)
• Process status
• P2P connectivity
• RPC availability (if enabled)
• Block sync status
• Peer count
• Consensus participation (for validators)

Key Features:

• Per-node health checks
• Detailed status output
• JSON output option (for monitoring)
• Exit codes for alerts

Status: ⚠️ PARTIAL (Some checks exist in other scripts)

---

4.2 Create Network Health Dashboard Script

File: scripts/health/network-health-dashboard.sh

Purpose: Display comprehensive network health overview

Functionality:

• All nodes status table
• Peer connectivity matrix
• Block height comparison
• Consensus status
• Validator participation
• Error summary

Key Features:

• Human-readable dashboard
• Color-coded status
• Quick problem identification
• Summary statistics

Status: ❌ NOT CREATED

---

Phase 5: Configuration Management

5.1 Create Configuration Generator

File: scripts/config/generate-configs.sh

Purpose: Generate all configuration files from templates

Functionality:

• Generate genesis.json (if needed)
• Generate static-nodes.json from live nodes
• Generate permissions-nodes.toml
• Generate node-specific config files (config-validator.toml, etc.)
• Validate generated configs

Key Features:

• Template-based generation
• Dynamic enode extraction
• Validation of generated files
• Backup of existing configs

Status: ⚠️ PARTIAL (Some config generation exists for allowlist)

---

5.2 Create Configuration Validator

File: scripts/config/validate-configs.sh

Purpose: Validate all configuration files before deployment

Functionality:

• Validate JSON/TOML syntax
• Validate genesis.json structure
• Validate static-nodes.json (enode format, node IDs)
• Validate permissions-nodes.toml
• Check for missing files
• Verify file permissions

Key Features:

• Pre-deployment validation
• Detailed error messages
• Report generation

Status: ❌ NOT CREATED

---

Phase 6: Documentation & Runbooks

6.1 Create Boot Node Runbook

File: docs/BOOT_NODE_RUNBOOK.md

Purpose: Detailed runbook for boot node setup and troubleshooting

Contents:

• Boot node concept explanation
• Setup instructions
• Configuration details
• Troubleshooting guide
• Best practices

Status: ❌ NOT CREATED

---

6.2 Create Validated Set Deployment Guide

File: docs/VALIDATED_SET_DEPLOYMENT_GUIDE.md

Purpose: Step-by-step guide for deploying a validated set

Contents:

• Prerequisites
• Deployment steps
• Validation procedures
• Troubleshooting
• Rollback procedures

Status: ❌ NOT CREATED

---

6.3 Create Network Bootstrap Guide

File: docs/NETWORK_BOOTSTRAP_GUIDE.md

Purpose: Guide for bootstrapping the network from scratch

Contents:

• Bootstrap sequence
• Node startup order
• Configuration updates
• Verification steps
• Common issues

Status: ❌ NOT CREATED

---

Phase 7: Testing & Validation

7.1 Create Integration Test Suite

File: scripts/test/test-deployment.sh

Purpose: Automated integration tests for deployment

Functionality:

• Test container deployment
• Test network bootstrap
• Test validator connectivity
• Test consensus functionality
• Test RPC endpoints
• Test rollback procedures

Key Features:

• Automated testing
• Test reports
• CI/CD integration

Status: ❌ NOT CREATED

---

7.2 Create Smoke Tests

File: scripts/test/smoke-tests.sh

Purpose: Quick smoke tests after deployment

Functionality:

• Basic connectivity checks
• Service status checks
• RPC endpoint checks
• Quick consensus check

Key Features:

• Fast execution
• Critical path validation
• Exit codes for automation

Status: ❌ NOT CREATED

---

Implementation Priority

High Priority (Critical Path) - Script-Based Approach

1. ✅ Validated Set Deployment Script (deploy-validated-set.sh) - PRIMARY
2. ✅ Network Bootstrap Script (bootstrap-network.sh) - Script-based mode
3. ✅ Deployment Validation Orchestrator (validate-deployment.sh)
4. ✅ Validator Set Validation (validate-validator-set.sh)

Optional (Boot Node Approach)

5. ⚠️ Boot Node Deployment Script (deploy-boot-node.sh) - Only if boot node needed
6. ⚠️ Network Bootstrap Script - Boot node mode (enhance existing script)

Medium Priority (Important Features)

5. ⚠️ Validator Set Validation (validate-validator-set.sh)
6. ⚠️ Node Health Checks (check-node-health.sh)
7. ⚠️ Configuration Generator (enhance existing)
8. ⚠️ Quick Bootstrap Script (bootstrap-quick.sh)

Low Priority (Nice to Have)

9. 📝 Network Health Dashboard (network-health-dashboard.sh)
10. 📝 Validator Registration (register-validators.sh) - only if using dynamic validators
11. 📝 Configuration Validator (validate-configs.sh)
12. 📝 Documentation (runbooks and guides)
13. 📝 Test Suites (integration and smoke tests)

---

Recommended Implementation Order

Week 1: Core Infrastructure (Script-Based)

1. Create deploy-validated-set.sh - Primary script-based deployment
2. Create bootstrap-network.sh - Script-based mode (uses static-nodes)
3. Enhance existing validate-deployment.sh
4. Create validate-validator-set.sh

Optional: Boot Node Support (If Needed)

5. Create deploy-boot-node.sh - Only if boot node approach is chosen
6. Enhance bootstrap-network.sh - Add boot node mode support

Week 2: Orchestration

4. Create deploy-validated-set.sh
5. Create validate-validator-set.sh
6. Create bootstrap-quick.sh

Week 3: Health & Monitoring

7. Create check-node-health.sh
8. Create network-health-dashboard.sh
9. Enhance configuration generation scripts

Week 4: Documentation & Testing

10. Create documentation (runbooks, guides)
11. Create test suites
12. Final validation and testing

---

Existing Assets to Leverage

Already Implemented

• ✅ Container deployment scripts (deploy-besu-nodes.sh, etc.)
• ✅ Configuration copying (copy-besu-config.sh)
• ✅ Allowlist management (besu-*.sh scripts)
• ✅ Network utilities (update-static-nodes.sh)
• ✅ Basic validation scripts (validate-ml110-deployment.sh)
• ✅ Deployment status checks (check-deployments.sh)

Can Be Enhanced

• ⚠️ validate-deployment.sh - needs comprehensive validator set validation
• ⚠️ deploy-all.sh - needs boot node support and sequential startup
• ⚠️ Configuration generation - needs boot node enode integration

---

Success Criteria

A successful implementation should provide:

1. Single Command Deployment

   ./scripts/deployment/deploy-validated-set.sh
   

   • Deploys all containers
   • Bootstraps network correctly
   • Validates entire deployment
   • Reports success/failure

2. Network Bootstrap

   • Boot node starts first
   • Other nodes connect successfully
   • All validators participate in consensus
   • Network is fully operational

3. Validation

   • All validators are validated and active
   • Network connectivity verified
   • Consensus is functional
   • RPC endpoints are working

4. Documentation

   • Complete runbooks for all procedures
   • Troubleshooting guides
   • Best practices documented

---

Quick Start Checklist

Script-Based Approach (Recommended for Your Setup)

• Review existing deployment scripts
• Create deploy-validated-set.sh - Main deployment orchestrator
• Create bootstrap-network.sh - Script-based mode (static-nodes)
• Create validate-validator-set.sh - Validator validation
• Enhance existing validate-deployment.sh
• Test deployment sequence on test environment
• Document procedures
• Test end-to-end on production-like environment

Boot Node Approach (Optional - Only If Needed)

• Decide if boot node is needed (probably not for permissioned network)
• Design boot node strategy (separate node vs first validator)
• Create deploy-boot-node.sh (if using dedicated boot node)
• Enhance bootstrap-network.sh with boot node mode
• Test boot node bootstrap sequence
• Document boot node procedures

---

Notes

Script-Based vs Boot Node Decision

For Your Current Setup (Permissioned Network with Validators ↔ Sentries):

• ✅ Recommend: Script-Based Approach

  • You already use static-nodes.json for peer discovery
  • All nodes are known and static
  • No external discovery needed
  • Simpler and faster deployment
  • Script orchestrates everything using static configuration

• ❌ Boot Node Not Required

  • Boot nodes are for dynamic peer discovery
  • Public/open networks that need discovery
  • Your network is permissioned with known validators
  • Static-nodes.json already serves the bootstrap purpose

When to Use Boot Node:

• Network will expand with external nodes
• Dynamic peer discovery needed
• Public network deployment
• Combining static + dynamic discovery

Other Notes

• Validator Registration: Only needed if using dynamic validator management via smart contract. If validators are statically defined in genesis, skip this step.

• Sequential Startup: Critical for network bootstrap. Nodes must start in correct order: sentries → validators → RPC nodes (for script-based) OR boot node → sentries → validators → RPC nodes (for boot node approach).

• Validation: Comprehensive validation should happen at multiple stages: pre-deployment, post-deployment, and ongoing health checks.