proxmox/docs/archive/DEPLOYMENT_COMPARISON.md

Deployment Comparison: LXC Containers vs Single VM

This document compares the two deployment options for SMOM-DBIS-138 Besu network nodes.

Overview

Deployment 1: Individual LXC Containers (VMID 1000-2502)

Each Besu node runs in its own LXC container on Proxmox VE. This is the production-ready deployment method.

Deployment 2: Single VM with Docker (VMID 9000)

All Besu nodes run as Docker containers within a single VM. This is the temporary/testing deployment method.

---

Deployment 1: LXC Containers (1000-2502)

Architecture

┌──────────────────────────────────────────────────────────────┐
│ Proxmox VE Host                                              │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ LXC 1000     │  │ LXC 1001     │  │ LXC 1002     │      │
│  │ validator-1  │  │ validator-2  │  │ validator-3  │      │
│  │ 8GB, 4 cores │  │ 8GB, 4 cores │  │ 8GB, 4 cores │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐                        │
│  │ LXC 1003     │  │ LXC 1004     │                        │
│  │ validator-4  │  │ validator-5  │                        │
│  │ 8GB, 4 cores │  │ 8GB, 4 cores │                        │
│  └──────────────┘  └──────────────┘                        │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ LXC 1500     │  │ LXC 1501     │  │ LXC 1502     │      │
│  │ sentry-1     │  │ sentry-2     │  │ sentry-3     │      │
│  │ 4GB, 2 cores │  │ 4GB, 2 cores │  │ 4GB, 2 cores │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                              │
│  ┌──────────────┐                                            │
│  │ LXC 1503     │                                            │
│  │ sentry-4     │                                            │
│  │ 4GB, 2 cores │                                            │
│  └──────────────┘                                            │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ LXC 2500     │  │ LXC 2501     │  │ LXC 2502     │      │
│  │ rpc-1        │  │ rpc-2        │  │ rpc-3        │      │
│  │ 16GB, 4 cores│  │ 16GB, 4 cores│  │ 16GB, 4 cores│      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└──────────────────────────────────────────────────────────────┘

VMID Allocation

Category	VMID Range	Count	Hostnames	IP Range
Validators	1000-1004	5	besu-validator-1 through besu-validator-5	192.168.11.100-104
Sentries	1500-1503	4	besu-sentry-1 through besu-sentry-4	192.168.11.150-153
RPC Nodes	2500-2502	3	besu-rpc-1 through besu-rpc-3	192.168.11.250-252

Total Containers: 12

Resource Allocation

Validators (1000-1004)

• Memory: 8GB each (40GB total)
• CPU Cores: 4 each (20 cores total)
• Disk: 100GB each (500GB total)
• IP Addresses: 192.168.11.100-104
• Purpose: QBFT consensus nodes

Sentries (1500-1503)

• Memory: 4GB each (16GB total)
• CPU Cores: 2 each (8 cores total)
• Disk: 100GB each (400GB total)
• IP Addresses: 192.168.11.150-153
• Purpose: P2P relay and protection nodes

RPC Nodes (2500-2502)

• Memory: 16GB each (48GB total)
• CPU Cores: 4 each (12 cores total)
• Disk: 200GB each (600GB total)
• IP Addresses: 192.168.11.250-252
• Purpose: Public RPC endpoints

Total Resources:

• Total Memory: 104GB
• Total CPU Cores: 40 cores
• Total Disk: 1.5TB

Deployment Script

cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-nodes.sh

Configuration Files

Each container has:

• /etc/besu/config-{type}.toml - Besu configuration
• /opt/besu/data/ - Blockchain data directory
• /opt/besu/keys/ - Validator keys (validators only)
• Systemd service: besu-validator, besu-sentry, or besu-rpc

Service Management

# Start/stop individual container services
pct exec 1000 -- systemctl start besu-validator
pct exec 1500 -- systemctl start besu-sentry
pct exec 2500 -- systemctl start besu-rpc

# Check status
pct exec 1000 -- systemctl status besu-validator

# View logs
pct exec 1000 -- journalctl -u besu-validator -f

Advantages

✅ Resource Isolation: Each node has dedicated resources
✅ Individual Scaling: Scale nodes independently
✅ Production Ready: Suitable for production deployments
✅ Security Isolation: Better security boundaries
✅ Independent Management: Start/stop nodes individually
✅ Better Monitoring: Monitor each node separately
✅ Fault Isolation: Failure of one node doesn't affect others

Disadvantages

❌ Longer Deployment Time: ~30-45 minutes
❌ More Complex Setup: Requires configuration per container
❌ Higher Resource Usage: More overhead per node
❌ More Management Overhead: Manage 12 containers

---

Deployment 2: Single VM with Docker (VMID 9000)

Architecture

┌──────────────────────────────────────────────────────────────┐
│ Proxmox VE Host                                              │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ VM 9000                                                │ │
│  │ IP: 192.168.11.90                                     │ │
│  │ Memory: 32GB, CPU: 8 cores, Disk: 500GB              │ │
│  │                                                        │ │
│  │  ┌──────────────────────────────────────────────────┐ │ │
│  │  │ Docker Network                                   │ │ │
│  │  │                                                  │ │ │
│  │  │  ┌─────────┐  ┌─────────┐  ┌─────────┐         │ │ │
│  │  │  │validator│  │validator│  │validator│         │ │ │
│  │  │  │   -1    │  │   -2    │  │   -3    │         │ │ │
│  │  │  └─────────┘  └─────────┘  └─────────┘         │ │ │
│  │  │                                                  │ │ │
│  │  │  ┌─────────┐  ┌─────────┐                      │ │ │
│  │  │  │validator│  │validator│                      │ │ │
│  │  │  │   -4    │  │   -5    │                      │ │ │
│  │  │  └─────────┘  └─────────┘                      │ │ │
│  │  │                                                  │ │ │
│  │  │  ┌─────────┐  ┌─────────┐  ┌─────────┐         │ │ │
│  │  │  │ sentry  │  │ sentry  │  │ sentry  │         │ │ │
│  │  │  │   -1    │  │   -2    │  │   -3    │         │ │ │
│  │  │  └─────────┘  └─────────┘  └─────────┘         │ │ │
│  │  │                                                  │ │ │
│  │  │  ┌─────────┐                                    │ │ │
│  │  │  │ sentry  │                                    │ │ │
│  │  │  │   -4    │                                    │ │ │
│  │  │  └─────────┘                                    │ │ │
│  │  │                                                  │ │ │
│  │  │  ┌─────────┐  ┌─────────┐  ┌─────────┐         │ │ │
│  │  │  │  rpc-1  │  │  rpc-2  │  │  rpc-3  │         │ │ │
│  │  │  └─────────┘  └─────────┘  └─────────┘         │ │ │
│  │  └──────────────────────────────────────────────────┘ │ │
│  └────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

VMID Allocation

Category	VMID	Description	IP Address
Temporary VM	9000	All 12 Besu nodes in Docker	192.168.11.90

Total VMs: 1
Total Docker Containers: 12 (5 validators + 4 sentries + 3 RPC)

Resource Allocation

• VM Memory: 32GB (shared by all containers)
• VM CPU Cores: 8 cores (shared by all containers)
• VM Disk: 500GB (shared by all containers)
• VM IP Address: 192.168.11.90
• OS: Ubuntu 22.04

Container Resource Allocation (within VM)

Validators (5 containers)

• Memory: 4GB each (20GB total)
• P2P Ports: 30303-30307
• Metrics Ports: 9545-9549

Sentries (4 containers)

• Memory: 2GB each (8GB total)
• P2P Ports: 30308-30311
• Metrics Ports: 9550-9553

RPC Nodes (3 containers)

• Memory: 8GB each (24GB total)
• HTTP RPC Ports: 8545, 8547, 8549
• WS RPC Ports: 8546, 8548, 8550
• P2P Ports: 30312-30314
• Metrics Ports: 9554-9556

Total Container Memory: 52GB (with some headroom in 32GB VM)

Deployment Script

cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-temp-vm-complete.sh /opt/smom-dbis-138

Or step-by-step:

# Step 1: Create VM
sudo ./scripts/deployment/deploy-besu-temp-vm.sh

# Step 2: SSH into VM and setup
ssh root@192.168.11.90
cd /opt/smom-dbis-138-proxmox
./scripts/deployment/setup-docker-besu.sh /opt/smom-dbis-138

# Step 3: Start containers
cd /opt/besu
docker compose up -d

Configuration Files

All containers share:

• /opt/besu/docker-compose.yml - Docker Compose configuration
• /opt/besu/validators/validator-{N}/config/config-validator.toml - Validator configs
• /opt/besu/sentries/sentry-{N}/config/config-sentry.toml - Sentry configs
• /opt/besu/rpc/rpc-{N}/config/config-rpc.toml - RPC configs
• /opt/besu/shared/genesis/ - Shared genesis.json
• /opt/besu/shared/permissions/ - Shared permissions files

Service Management

# SSH into VM
ssh root@192.168.11.90

# Start/stop all containers
cd /opt/besu
docker compose up -d
docker compose stop
docker compose restart

# Start/stop specific container
docker compose start besu-validator-1
docker compose stop besu-validator-1

# View logs
docker compose logs -f
docker compose logs -f besu-validator-1

# Check status
docker compose ps
docker stats

RPC Endpoints

• RPC-1: http://192.168.11.90:8545
• RPC-2: http://192.168.11.90:8547
• RPC-3: http://192.168.11.90:8549

Metrics Endpoints

• Validators: http://192.168.11.90:9545-9549/metrics
• Sentries: http://192.168.11.90:9550-9553/metrics
• RPC Nodes: http://192.168.11.90:9554-9556/metrics

Advantages

✅ Faster Deployment: ~15-30 minutes
✅ Easier Initial Testing: Single point of management
✅ Simplified Troubleshooting: All nodes in one place
✅ Lower Resource Overhead: Shared VM resources
✅ Easy Migration Path: Can migrate to LXC later
✅ Single Management Point: Manage all nodes together

Disadvantages

❌ Resource Sharing: All nodes share VM resources
❌ Less Isolation: No resource boundaries between nodes
❌ Not Production Ready: Temporary/testing solution
❌ Single Point of Failure: VM failure affects all nodes
❌ Resource Contention: Nodes compete for resources

---

Side-by-Side Comparison

Feature	LXC Containers (1000-2502)	Single VM (9000)
VMID Range	1000-2502 (12 containers)	9000 (1 VM)
Architecture	Individual LXC containers	Docker containers in single VM
Deployment Time	30-45 minutes	15-30 minutes
Total Memory	104GB (dedicated)	32GB (shared)
Total CPU Cores	40 cores (dedicated)	8 cores (shared)
Total Disk	1.5TB (distributed)	500GB (shared)
Resource Isolation	High (per container)	Low (shared VM)
Scalability	Individual per node	Limited (VM limits)
Management	Per container (12 instances)	Single VM (1 instance)
Production Ready	✅ Yes	❌ No (temporary)
Security Isolation	High	Medium
Fault Isolation	High (independent failures)	Low (VM is SPOF)
Monitoring	Per container	Per container (within VM)
Migration Path	N/A (final state)	✅ To LXC available

---

Network Configuration Comparison

LXC Containers

Node Type	VMID Range	IP Range	VLAN
Validators	1000-1004	192.168.11.100-104	100
Sentries	1500-1503	192.168.11.150-153	101
RPC Nodes	2500-2502	192.168.11.250-252	102

Network Isolation: Separate VLANs per node type

Single VM

Node Type	Container Count	Port Mappings	Network
Validators	5	P2P: 30303-30307, Metrics: 9545-9549	Docker bridge
Sentries	4	P2P: 30308-30311, Metrics: 9550-9553	Docker bridge
RPC Nodes	3	HTTP: 8545/8547/8549, WS: 8546/8548/8550	Docker bridge

Network Isolation: Docker bridge network, port-based separation

---

When to Use Each Deployment

Use LXC Containers (1000-2502) When:

✅ Production deployment
✅ Need resource isolation
✅ Individual node scaling required
✅ Long-term deployment
✅ Maximum security needed
✅ Need independent node management
✅ Want better fault isolation

Use Single VM (9000) When:

✅ Quick testing and validation
✅ Development environment
✅ Proof of concept
✅ Limited initial resources
✅ Planning to migrate to LXC later
✅ Need faster initial deployment

---

Migration Path

From VM (9000) to LXC Containers (1000-2502)

# 1. Deploy LXC containers
cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-nodes.sh

# 2. Migrate data from VM to LXC
sudo ./scripts/migration/migrate-vm-to-lxc.sh /opt/smom-dbis-138

# 3. Verify and start services in LXC containers
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- systemctl start besu-validator
done

for vmid in 1500 1501 1502 1503; do
    pct exec $vmid -- systemctl start besu-sentry
done

for vmid in 2500 2501 2502; do
    pct exec $vmid -- systemctl start besu-rpc
done

# 4. Validate deployment
./scripts/validation/validate-deployment-comprehensive.sh

# 5. Shut down temporary VM
qm stop 9000

---

Deployment Scripts Reference

LXC Containers Deployment

Main Script: scripts/deployment/deploy-besu-nodes.sh

Key Functions:

• Creates 12 LXC containers (5 validators, 4 sentries, 3 RPC)
• Configures network (static IPs, VLANs)
• Installs Besu in each container
• Sets up systemd services
• Generates inventory file

Single VM Deployment

Main Scripts:

• scripts/deployment/deploy-besu-temp-vm.sh - Creates VM only
• scripts/deployment/deploy-besu-temp-vm-complete.sh - Complete deployment
• scripts/deployment/setup-docker-besu.sh - Sets up Docker and configs

Key Functions:

• Creates single VM (VMID 9000)
• Installs Docker and Docker Compose
• Copies configuration files
• Sets up Docker Compose file
• Starts all 12 containers

---

Validation and Monitoring

LXC Containers

# Validate all containers
./scripts/validation/validate-deployment-comprehensive.sh

# Check individual container status
pct exec 1000 -- systemctl status besu-validator
pct exec 1500 -- systemctl status besu-sentry
pct exec 2500 -- systemctl status besu-rpc

# View logs
pct exec 1000 -- journalctl -u besu-validator -f

Single VM

# Validate VM deployment
./scripts/validation/validate-besu-temp-vm.sh

# Check container status (from within VM)
ssh root@192.168.11.90
docker compose ps
docker stats

# View logs
docker compose logs -f

---

Related Documentation

• Validated Set Deployment Guide
• Deployment Status Consolidated
• Troubleshooting FAQ

---

Last Updated: $(date)