6.2 KiB
Parallel Execution Limits
Last Updated: 2025-01-11
Purpose: Document recommended parallel execution limits for deployment operations
Overview
Deployment operations can be parallelized to reduce total deployment time, but must respect Proxmox host resources to prevent exhaustion and instability.
Recommended Limits
Container Creation
Operation: Creating LXC containers
Recommended: 10 containers simultaneously
Rationale:
- Container creation is I/O intensive
- Proxmox can handle ~10 concurrent container creations
- More than 10 can cause resource contention
Configuration:
# In proxmox.conf
MAX_PARALLEL_CREATE=10
Package Installation
Operation: Installing packages (apt, npm, etc.)
Recommended: 8 containers simultaneously
Rationale:
- Package installation is CPU and RAM intensive
- Downloads can saturate network
- Compilation (if any) requires CPU
- 8 containers balances speed vs. resource usage
Configuration:
# In proxmox.conf
MAX_PARALLEL_PACKAGES=8
CCIP Nodes: Use 8 nodes per batch (16 nodes = 2 batches)
- CCIP nodes have complex dependencies
- Chainlink installation is resource-intensive
- Batching prevents resource exhaustion
Service Startup
Operation: Starting services (systemd, Docker, etc.)
Recommended: 15 containers simultaneously
Rationale:
- Service startup is less resource-intensive
- Mostly I/O and network operations
- Can handle more concurrent startups
- Services initialize independently
Configuration:
# In proxmox.conf
MAX_PARALLEL_START=15
OS Template Installation
Operation: Installing OS templates to containers
Recommended: 10-20 containers simultaneously
Rationale:
- Template installation is I/O intensive
- Can parallelize more than creation
- Depends on storage performance
- 10-20 is safe for most systems
Configuration:
# In proxmox.conf
MAX_PARALLEL_TEMPLATE=15
Service-Specific Recommendations
Besu Nodes
Container Creation: 5 validators + 4 sentries + 3 RPC = 12 containers
- Can create all 12 in parallel (within 10 limit)
- Or create by type: 5 validators → 4 sentries → 3 RPC
Package Installation: Can install on all 12 in parallel (within 8 limit)
Service Startup: Can start all 12 in parallel (within 15 limit)
CCIP Nodes (41-43 containers)
Container Creation: Batch by type
- CCIP-OPS (2): Parallel ✓
- CCIP-MON (2): Parallel ✓
- CCIP-COMMIT (16): 8 per batch, 2 batches
- CCIP-EXEC (16): 8 per batch, 2 batches
- CCIP-RMN (5-7): All parallel ✓
Package Installation: Same batching strategy
- Install packages in same batches as creation
- 8 nodes per batch prevents resource exhaustion
Service Startup: Can start more in parallel
- CCIP-OPS (2): Parallel
- CCIP-MON (2): Parallel
- CCIP-COMMIT (16): 8 per batch or all parallel
- CCIP-EXEC (16): 8 per batch or all parallel
- CCIP-RMN (5-7): All parallel
Other Services (14 containers)
Container Creation: Can create all in parallel (within 10 limit)
- Or create by type for organization
Package Installation: Varies by service complexity
- Simple services (Besu services): Can install in parallel
- Complex services (Blockscout, Fabric): Install sequentially or 2-3 at a time
Service Startup: Can start most in parallel
- Database services (Blockscout, Firefly) may need sequential startup
- Other services can start in parallel
Resource Considerations
CPU
Impact: Package installation, compilation, service startup
Limiting Factor: Number of CPU cores
Recommendation:
- For 8-core host: MAX_PARALLEL_PACKAGES=6
- For 16-core host: MAX_PARALLEL_PACKAGES=8
- For 32+ core host: MAX_PARALLEL_PACKAGES=12
RAM
Impact: Package installation, service startup, container overhead
Limiting Factor: Available RAM
Recommendation:
- Ensure sufficient RAM for parallel operations
- Each container uses RAM for OS + packages + services
- CCIP nodes may need more RAM during installation
Storage I/O
Impact: Container creation, template installation, package installation
Limiting Factor: Storage type and performance
Recommendation:
- Local SSD: Can handle higher parallelism
- Network storage: Lower parallelism recommended
- HDD: Even lower parallelism
Network Bandwidth
Impact: Package downloads, file transfers
Limiting Factor: Network connection speed
Recommendation:
- 1 Gbps: Can handle full parallelism
- 100 Mbps: Reduce parallelism by 50%
- Slower: Reduce further or use local package mirror
Configuration Example
# In proxmox.conf
# Parallel deployment enabled
PARALLEL_DEPLOY=true
# General limits
MAX_PARALLEL=10 # Default for operations without specific limit
# Operation-specific limits
MAX_PARALLEL_CREATE=10 # Container creation
MAX_PARALLEL_PACKAGES=8 # Package installation (CPU/RAM intensive)
MAX_PARALLEL_START=15 # Service startup (less intensive)
MAX_PARALLEL_TEMPLATE=15 # OS template installation
# Service-specific limits
MAX_PARALLEL_CCIP=8 # CCIP nodes (resource intensive)
MAX_PARALLEL_BESU=12 # Besu nodes (can handle more)
# CCIP batching
CCIP_BATCH_SIZE=8 # Process 8 CCIP nodes at a time
Monitoring During Deployment
Watch for:
- High CPU usage (>90% sustained)
- High RAM usage (>90% used)
- Storage I/O wait times
- Network saturation
- Container creation failures
If issues occur:
- Reduce MAX_PARALLEL values
- Increase delays between batches
- Deploy in smaller batches
- Check Proxmox host resource usage
Best Practices
- Start Conservative: Begin with lower limits, increase if system handles well
- Monitor Resources: Watch CPU, RAM, I/O during deployment
- Batch Large Deployments: CCIP nodes should always be batched
- Respect Service Dependencies: Some services need to start in order
- Test Limits: On test environment before production deployment
References
- Deployment Time Estimate:
docs/DEPLOYMENT_TIME_ESTIMATE.md - Optimization Recommendations:
docs/DEPLOYMENT_OPTIMIZATION_RECOMMENDATIONS.md - Configuration:
smom-dbis-138-proxmox/config/proxmox.conf