---
title: IRU Technical Architecture - Proxmox VE LXC Deployment
version: 1.0.0
status: draft
last_updated: 2025-01-27
document_type: technical_architecture
layer: technical
provider: Sankofa_Phoenix_Cloud_Service
---

# IRU TECHNICAL ARCHITECTURE
## Proxmox VE LXC Deployment Architecture

**Service Provider**: Sankofa Phoenix Cloud Service Provider  
**Deployment Model**: Proxmox VE LXC Container-Based Infrastructure  
**Related Documentation**: [IRU Participation Agreement](./IRU_Participation_Agreement.md)

---

## 1. CONTAINER TOPOLOGY OVERVIEW

### 1.1 Host Layer

- **Proxmox VE cluster node(s)**: Primary virtualization and container orchestration platform
- **Linux kernel**: Shared across all LXC containers for resource efficiency
- **Storage**: ZFS-backed storage pools (or equivalent high-performance storage)

### 1.2 Container Layer (LXC)

The IRU infrastructure is deployed across the following LXC containers:

- **`lxc-besu-sentry`**: Besu blockchain sentry node for P2P network connectivity
- **`lxc-firefly-core`**: FireFly core service for event listening and transaction orchestration
- **`lxc-firefly-db`**: FireFly database (optional/internal) for state persistence
- **`lxc-monitoring`**: Monitoring and observability services (optional)

Each container operates in an isolated namespace with explicit resource and network constraints, ensuring security and performance isolation.

---

## 2. TEXT-BASED TOPOLOGY DIAGRAM

```
                        External P2P Network
                               |
                               |  (P2P / TLS)
                               v
                   +--------------------------+
                   |   LXC: Besu Sentry Node  |
                   |  - P2P Interface         |
                   |  - RPC (restricted)     |
                   +------------+-------------+
                                |
                                |  (JSON-RPC / mTLS)
                                v
                   +--------------------------+
                   |   LXC: FireFly Core      |
                   |  - Event Listener        |
                   |  - TX Orchestrator       |
                   +------------+-------------+
                                |
                                |  (Internal DB / MQ)
                                v
                   +--------------------------+
                   |   LXC: FireFly DB        |
                   +--------------------------+
```

**Network Flow**:
- All inter-container traffic occurs over private Proxmox bridges or SDN segments
- External P2P network connects only to Besu Sentry nodes
- FireFly Core and DB containers are not directly exposed to external networks

---

## 3. INTER-CONTAINER NETWORKING

### 3.1 Network Architecture

- **Proxmox Linux Bridge or SDN VLAN**: Private network segments for container communication
- **Private RFC1918 addressing**: Internal IP addressing (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
- **No public IPs**: FireFly and DB containers do not have public IP addresses
- **Sentry exposure**: Besu Sentry exposes only required P2P ports externally

### 3.2 Firewall Enforcement

**Host-Level**:
- nftables / iptables rules on Proxmox host
- Default deny policies with explicit allowlists

**Container-Level**:
- Container-specific firewall rules
- Network namespace isolation
- Restricted inter-container communication

---

## 4. RESOURCE SIZING BASELINES

### 4.1 Besu Sentry Node (LXC)

- **vCPU**: 4 cores (pinned for consistent performance)
- **RAM**: 8–16 GB (depending on network size and transaction volume)
- **Disk**: 200–500 GB (fast I/O, SSD recommended)
- **Network**: High-throughput, low-latency NIC (10 Gbps or higher recommended)

**Performance Characteristics**:
- Handles P2P network connectivity
- Processes blockchain synchronization
- Provides RPC interface (restricted access)

### 4.2 FireFly Core (LXC)

- **vCPU**: 2–4 cores
- **RAM**: 4–8 GB
- **Disk**: 50–100 GB (for logs and temporary data)

**Performance Characteristics**:
- Event listening and processing
- Transaction orchestration
- API service provision

### 4.3 FireFly Database (LXC)

- **vCPU**: 2 cores
- **RAM**: 4–8 GB
- **Disk**: 100–200 GB (IOPS prioritized, SSD recommended)

**Performance Characteristics**:
- State persistence
- Transaction history
- Event indexing

### 4.4 Monitoring Container (Optional)

- **vCPU**: 1–2 cores
- **RAM**: 2–4 GB
- **Disk**: 50–100 GB (for metrics and log retention)

---

## 5. DEPLOYMENT & PROVISIONING FLOW

### 5.1 Provisioning Sequence

1. **Provision Proxmox VE host(s)**
   - Install and configure Proxmox VE
   - Configure storage pools (ZFS or equivalent)
   - Set up network bridges/VLANs

2. **Create isolated LXC containers**
   - Create containers with pinned resources
   - Configure container templates
   - Set resource limits (CPU, RAM, disk)

3. **Attach containers to private bridges**
   - Assign containers to appropriate network segments
   - Configure IP addressing
   - Set up DNS resolution

4. **Mount secrets via read-only volumes**
   - Deploy node keys and certificates
   - Configure TLS certificates
   - Set up authentication credentials

5. **Deploy Besu Sentry and FireFly binaries**
   - Install version-pinned binaries
   - Configure service files
   - Set up systemd services

6. **Configure endpoints**
   - RPC endpoints (restricted access)
   - P2P endpoints (external access)
   - Internal communication endpoints

7. **Validate interconnectivity and health checks**
   - Test container-to-container communication
   - Verify external P2P connectivity
   - Confirm health check endpoints

---

## 6. SECURITY & KEY MANAGEMENT

### 6.1 Key Storage

- **Node keys and certificates**: Stored outside container images
- **Read-only mounts**: Secrets mounted as read-only volumes
- **Runtime injection**: Sensitive data injected at runtime when possible
- **No keys in images**: Container images do not contain keys or certificates

### 6.2 Security Protocols

- **mTLS enforcement**: Mutual TLS between FireFly and Besu
- **No lateral access**: Containers cannot access each other by default
- **Explicit allowlists**: Only declared flows pass firewall rules
- **Certificate rotation**: Regular rotation of TLS certificates and API credentials

### 6.3 Access Control

- **Restricted RPC access**: RPC endpoints allowlisted to specific sources
- **VPN/Admin access**: Administrative access via VPN or management VLAN
- **No public exposure**: FireFly and DB containers not exposed to public networks

---

## 7. LIFECYCLE & OPERATIONS

### 7.1 Snapshot-Based Rollback

- **ZFS snapshots**: Leverage ZFS snapshot capabilities for rollback
- **Point-in-time recovery**: Ability to restore to previous states
- **Configuration snapshots**: Capture container and network configurations

### 7.2 Rolling Restarts

- **Per-container restarts**: Restart containers individually without service disruption
- **Health check validation**: Verify service health after restart
- **Zero-downtime upgrades**: Rolling updates where possible

### 7.3 Live Migration

- **Host-level migration**: Support for live migration at Proxmox host level
- **Cluster support**: Migration between Proxmox cluster nodes
- **Resource continuity**: Maintain resource allocations during migration

### 7.4 Version Upgrades

- **Per-container upgrades**: Upgrade containers individually
- **Version pinning**: Maintain version control and rollback capability
- **Testing procedures**: Validate upgrades in staging before production

---

## 8. EXPANDABILITY

The architecture supports adding additional components without disruption:

### 8.1 Additional Besu Nodes

- **Additional sentry nodes**: Scale P2P connectivity
- **Validator nodes**: Add consensus participation
- **Quorum nodes**: Enhance network reliability

### 8.2 Additional Services

- **Indexers**: Add blockchain indexing services
- **Analytics**: Deploy analytics and reporting services
- **API gateways**: Add API gateway services for external access

### 8.3 Network Expansion

- **Additional containers**: Add containers without disrupting existing ones
- **Network paths**: Maintain existing network paths during expansion
- **Resource allocation**: Scale resources per container independently

---

## 9. HIGH AVAILABILITY (HA) & FAILOVER OPTIONS

### 9.1 Multi-Sentry Pattern (Recommended)

**Architecture**:
- Deploy **2+ Besu Sentry** containers (`lxc-besu-sentry-01`, `lxc-besu-sentry-02`) on separate Proxmox hosts
- External peer connections distributed via DNS round-robin or upstream load balancer for RPC
- P2P peers may connect to both sentry nodes for redundancy

**Benefits**:
- **Redundancy**: Multiple sentry nodes prevent single point of failure
- **Load distribution**: Distribute P2P and RPC traffic across nodes
- **Isolation**: Internal core/validator nodes never exposed; only accept traffic from trusted sentry IPs

### 9.2 FireFly HA (Active/Passive)

**Architecture**:
- Run **one active FireFly Core** (`lxc-firefly-core-01`) and one **warm standby** (`lxc-firefly-core-02`)
- Both point to the same database (or replicated DB)
- Controlled leader election handled operationally

**Failover Procedure**:
1. Freeze/stop active container
2. Promote standby container to active
3. Confirm event listener offsets and resume processing
4. Validate service health and connectivity

### 9.3 Database HA (Optional)

**Managed DB HA** (Preferred):
- Use managed database services with built-in HA if available
- Leverage cloud provider HA capabilities

**Containerized DB HA**:
- **PostgreSQL primary/replica**: Set up primary/replica configuration
- **Synchronous replication**: Where feasible, use synchronous replication for data consistency
- **Backups + PITR**: Point-in-time recovery (PITR) as baseline for data protection
- **Automatic failover**: Configure automatic failover mechanisms where possible

---

## 10. PORT & FLOW MATRIX (BASELINE)

> **Note**: Exact ports may vary based on Besu/FireFly configuration; this matrix defines **intended flows**.

### 10.1 External Flows

**Besu Sentry P2P**:
- **Direction**: External Peers → `lxc-besu-sentry-*`
- **Protocol**: P2P inbound/outbound (TLS if enabled)
- **Ports**: Standard Besu P2P ports (typically 30303)

**RPC (Optional / Restricted)**:
- **Direction**: Admin/VPN → `lxc-besu-sentry-*`
- **Protocol**: JSON-RPC over mTLS
- **Access**: Allowlisted sources only
- **Ports**: Custom RPC ports (typically 8545, 8546)

### 10.2 Internal Flows (Private Bridge / VLAN)

**FireFly → Besu**:
- **Direction**: `lxc-firefly-core` → `lxc-besu-sentry-*`
- **Protocol**: JSON-RPC
- **Security**: mTLS enforced
- **Network**: Private bridge/VLAN

**FireFly → DB**:
- **Direction**: `lxc-firefly-core` → `lxc-firefly-db`
- **Protocol**: PostgreSQL
- **Network**: Private bridge/VLAN

**Monitoring → All Containers**:
- **Direction**: `lxc-monitoring` → all containers
- **Protocol**: Metrics/log shipping
- **Network**: Private bridge/VLAN

### 10.3 Default Denies

- **No direct external access**: No direct access from external networks to FireFly or DB containers
- **No lateral access**: No container-to-container access unless explicitly required
- **Explicit allowlists**: Only declared flows pass firewall rules

---

## 11. PROXMOX VE NETWORKING IMPLEMENTATION

### 11.1 Simple Bridge (Single Host or Flat Network)

**Bridge Configuration**:
- **`vmbr0`**: Management + WAN (host)
- **`vmbr1`**: Private service network (LXC only)

**Use Case**: Single-host deployments or flat network topologies

### 11.2 VLAN Segmentation (Recommended)

**VLAN-Backed Bridges or SDN VNets**:

- **VLAN 10**: Management
  - Proxmox hosts
  - Admin endpoints
  - Monitoring access

- **VLAN 20**: Private Services (FireFly/DB)
  - FireFly Core containers
  - FireFly DB containers
  - Internal service communication
  - **Policy**: Non-routable externally

- **VLAN 30**: Sentry DMZ (Besu P2P/RPC)
  - Besu Sentry nodes
  - P2P network connectivity
  - Restricted RPC access
  - **Policy**: Controlled external exposure

**Policy Intent**:
- Only VLAN 30 has controlled external exposure
- VLAN 20 is non-routable externally
- VLAN 10 is management-only

---

## 12. CONTAINER NAMING, IP SCHEMA, AND DNS

### 12.1 Naming Convention

**Standard Naming Pattern**:
- `lxc-besu-sentry-01`, `lxc-besu-sentry-02`
- `lxc-firefly-core-01`, `lxc-firefly-core-02` (standby)
- `lxc-firefly-db-01`
- `lxc-monitoring-01`

**Naming Benefits**:
- Clear identification of container purpose
- Sequential numbering for multiple instances
- Consistent naming across deployments

### 12.2 IP Schema (Example)

**Management (VLAN 10)**: `10.10.10.0/24`
- Proxmox hosts: `10.10.10.1-10.10.10.10`
- Admin endpoints: `10.10.10.11-10.10.10.50`

**Services (VLAN 20)**: `10.20.20.0/24`
- FireFly Core: `10.20.20.10-10.20.20.20`
- FireFly DB: `10.20.20.30-10.20.20.40`
- Monitoring: `10.20.20.50-10.20.20.60`

**DMZ (VLAN 30)**: `10.30.30.0/24`
- Besu Sentry nodes: `10.30.30.10-10.30.30.30`

### 12.3 DNS / Service Discovery

**Internal DNS Records**:
- `besu-sentry.service.local` → Points to sentry nodes (round-robin)
- `firefly-core.service.local` → Active core container
- `firefly-db.service.local` → DB primary

**Service Discovery Benefits**:
- Simplified configuration management
- Automatic failover via DNS
- Load distribution via round-robin

---

## 13. HARDENING CHECKLIST

### 13.1 Proxmox Host Hardening

- [ ] **Keep Proxmox updated**: Regular security updates and patches
- [ ] **Restrict GUI/API access**: Limit to VPN or management VLAN
- [ ] **Enable host firewall**: Default deny inbound; allow only required management ports
- [ ] **Separate networks**: Separate management from service/DMZ networks
- [ ] **SSH key authentication**: Disable password authentication
- [ ] **Regular backups**: Automated backup procedures
- [ ] **Monitoring**: Host-level monitoring and alerting

### 13.2 LXC Container Hardening

- [ ] **Unprivileged containers**: Use unprivileged containers where compatible
- [ ] **Drop unnecessary capabilities**: Minimal capability set
- [ ] **Minimal base images**: Use minimal base images (Alpine, Debian minimal)
- [ ] **Read-only root FS**: Read-only root filesystem when feasible
- [ ] **Writable volumes**: Writable volumes only for data directories
- [ ] **Strict resource quotas**: CPU/RAM/disk quotas enforced
- [ ] **Disable nesting**: Disable container nesting unless required
- [ ] **Regular updates**: Keep container images and packages updated

### 13.3 Network Hardening

- [ ] **Default deny inter-VLAN routing**: Explicit routing rules only
- [ ] **Allowlist flows**: Only declared flows allowed:
  - FireFly → Besu (RPC)
  - FireFly → DB
  - Monitoring → metrics/logs
- [ ] **Enforce mTLS**: Mutual TLS for RPC where possible
- [ ] **Network segmentation**: Strict VLAN separation
- [ ] **Firewall logging**: Log all denied connections

### 13.4 Secrets & Key Material

- [ ] **No keys in images**: Keys not stored in container images
- [ ] **Read-only mounts**: Secrets mounted as read-only volumes
- [ ] **Runtime injection**: Sensitive data injected at runtime
- [ ] **Rotate TLS certs**: Regular rotation of TLS certificates
- [ ] **Rotate API credentials**: Regular rotation of API credentials
- [ ] **Restricted permissions**: Store node keys in restricted permission paths
- [ ] **Key management system**: Use key management system where available

### 13.5 Observability & Audit

- [ ] **Centralize logs**: Centralized logging with immutable retention policy
- [ ] **Export metrics**: Export metrics and set alert thresholds:
  - CPU usage
  - RAM usage
  - Disk I/O
  - Peer count
  - RPC errors
- [ ] **Change log**: Record configuration changes and deployments
- [ ] **Audit trail**: Maintain audit trail for all administrative actions
- [ ] **Monitoring dashboards**: Real-time monitoring dashboards
- [ ] **Alerting**: Automated alerting for critical events

---

## 14. DEPLOYMENT ACCEPTANCE TESTS

### 14.1 Besu Sentry Tests

**P2P Connectivity**:
- [ ] Confirms P2P peer connectivity (minimum peer count threshold met)
- [ ] Validates P2P handshake and protocol negotiation
- [ ] Verifies blockchain synchronization status

**RPC Health**:
- [ ] Confirms RPC health endpoint reachable only from allowlisted sources
- [ ] Validates RPC authentication and authorization
- [ ] Tests JSON-RPC functionality

### 14.2 FireFly Tests

**Besu Integration**:
- [ ] Confirms RPC handshake to Besu
- [ ] Validates event subscription and block listener operation
- [ ] Tests transaction submission and monitoring

**Database Integration**:
- [ ] Confirms DB connectivity
- [ ] Validates database migrations complete
- [ ] Tests data persistence and retrieval

**Service Health**:
- [ ] Validates FireFly API endpoints
- [ ] Confirms event processing pipeline
- [ ] Tests transaction orchestration

### 14.3 Network Tests

**Security Validation**:
- [ ] Confirms no external route to FireFly/DB containers
- [ ] Validates only declared flows pass firewall rules
- [ ] Tests network segmentation and isolation

**Connectivity Validation**:
- [ ] Confirms inter-container communication works
- [ ] Validates DNS resolution
- [ ] Tests service discovery functionality

### 14.4 Performance Tests

**Resource Utilization**:
- [ ] Validates resource usage within allocated limits
- [ ] Confirms no resource contention
- [ ] Tests under load conditions

**Latency Tests**:
- [ ] Measures RPC latency
- [ ] Validates P2P network latency
- [ ] Tests transaction processing time

---

## RELATED DOCUMENTATION

- [IRU Participation Agreement](./IRU_Participation_Agreement.md) - Master IRU Agreement
- [Foundational Charter IRU Excerpt](./Foundational_Charter_IRU_Excerpt.md) - Constitutional foundation
- [Regulatory Positioning Memo](./Regulatory_Positioning_Memo_CBs_DFIs.md) - Regulatory guidance
- [DBIS Architecture Atlas](../architecture-atlas-overview.md) - DBIS technical architecture

---

**END OF TECHNICAL ARCHITECTURE DOCUMENT**
