d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
b45c2006be629f228d1ae98bc6372e83372653df
proxmox/docs/05-network/NGINX_ARCHITECTURE_RPC.md

7.4 KiB
Raw Blame History

Nginx Architecture for RPC Nodes

Overview

There are two different nginx use cases in the RPC architecture:

  1. nginx-proxy-manager (VMID 105) - Centralized reverse proxy/load balancer
  2. nginx on RPC nodes (2500-2502) - Local nginx on each RPC container

Current Architecture

VMID 105: nginx-proxy-manager

  • Purpose: Centralized reverse proxy management with web UI
  • Status: Existing container (running)
  • Use Case: Route traffic to multiple services, SSL termination, load balancing
  • Advantages:
    • Centralized management via web UI
    • Easy SSL certificate management
    • Can load balance across multiple RPC nodes
    • Single point of configuration

nginx on RPC Nodes (2500-2502)

  • Purpose: Local nginx on each RPC container
  • Current Status: Installed but not necessarily configured
  • Use Case: SSL termination, local load balancing, rate limiting per node
  • Advantages:
    • Node-specific configuration
    • Redundancy (each node has its own nginx)
    • Can handle local routing needs

Recommendation: Use VMID 105 for RPC

✅ YES - VMID 105 can and should be used for RPC

Recommended Architecture:

Clients → nginx-proxy-manager (VMID 105) → Besu RPC Nodes (2500-2502:8545)

Benefits:

  1. Centralized Management: Single web UI to manage all RPC routing
  2. Type-Based Routing: Route requests to appropriate RPC node type (Public, Core, Permissioned, etc.)
  3. SSL Termination: Handle HTTPS at the proxy level
  4. Access Control: Different access rules per RPC node type
  5. Simplified RPC Nodes: Remove nginx from RPC nodes (they just run Besu)
  6. Better Monitoring: Central point to monitor RPC traffic

Note: RPC nodes 2500-2502 are different types, not redundant instances. Therefore, load balancing/failover between them is NOT appropriate. See docs/RPC_NODE_TYPES_ARCHITECTURE.md for details.

Implementation Options

Option 1: Use VMID 105 Only (Recommended)

Remove nginx from RPC nodes and use nginx-proxy-manager exclusively:

Steps:

  1. Remove nginx package from install/besu-rpc-install.sh ✅ DONE
  2. Configure nginx-proxy-manager (VMID 105) with separate proxy hosts for each RPC node type:
    • Core RPC: rpc-core.besu.local → 192.168.11.250:8545 (VMID 2500)
    • Permissioned RPC: rpc-perm.besu.local → 192.168.11.251:8545 (VMID 2501)
    • Public RPC: rpc.besu.local → 192.168.11.252:8545 (VMID 2502)
  3. Configure access control per proxy host (public vs internal)
  4. Expose appropriate endpoints based on RPC node type

Important: Do NOT set up load balancing between these nodes, as they are different types serving different purposes.

Configuration in nginx-proxy-manager (separate proxy host per type):

Public RPC Proxy:

  • Domain: rpc.besu.local (or rpc-public.chainid138.local)
  • Scheme: http
  • Forward Hostname/IP: 192.168.11.250 (Public RPC node)
  • Forward Port: 8545
  • Websockets: Enabled (for WS-RPC on port 8546)
  • Access: Public (as appropriate for public RPC)

Core RPC Proxy:

  • Domain: rpc-core.besu.local (or rpc-core.chainid138.local)
  • Scheme: http
  • Forward Hostname/IP: 192.168.11.251 (Core RPC node)
  • Forward Port: 8545
  • Websockets: Enabled
  • Access: Restricted to internal network IPs

Permissioned RPC Proxy:

  • Domain: rpc-perm.besu.local (or rpc-perm.chainid138.local)
  • Scheme: http
  • Forward Hostname/IP: 192.168.11.252 (Permissioned RPC node)
  • Forward Port: 8545
  • Websockets: Enabled
  • Access: Additional authentication/authorization as needed

Option 2: Hybrid Approach

Keep both but use them for different purposes:

  • nginx-proxy-manager (VMID 105):

    • Public-facing entry point
    • SSL termination
    • Load balancing across RPC nodes

  • nginx on RPC nodes:

    • Optional: Local rate limiting
    • Optional: Node-specific routing
    • Can be used for internal routing within the container

Use Case: If you need per-node rate limiting or complex local routing

Configuration Details

nginx-proxy-manager Configuration (VMID 105)

Proxy Host Setup:

  1. Access nginx-proxy-manager web UI (typically port 81)
  2. Add Proxy Host:
    • Domain Names: rpc.besu.local, rpc.chainid138.local (or your domain)
    • Scheme: http
    • Forward Hostname/IP: Use load balancer with:
      • 192.168.11.250:8545
      • 192.168.11.251:8545
      • 192.168.11.252:8545
    • Forward Port: 8545
    • Cache Assets: Disabled (RPC responses shouldn't be cached)
    • Websockets: Enabled
    • Block Common Exploits: Enabled
    • SSL: Configure Let's Encrypt or custom certificate

Type-Based Routing Configuration: Since RPC nodes are different types (not redundant instances), configure separate proxy hosts rather than load balancing:

  1. Core RPC Proxy: Routes to 192.168.11.250:8545 only (VMID 2500)
  2. Permissioned RPC Proxy: Routes to 192.168.11.251:8545 only (VMID 2501)
  3. Public RPC Proxy: Routes to 192.168.11.252:8545 only (VMID 2502)

Health Checks: Enable health checks for each proxy host to detect if the specific node type is down

Note: If you deploy multiple instances of the same type (e.g., 2 Public RPC nodes), THEN you can configure load balancing within that type's proxy host.

WebSocket Support:

  • Add separate proxy host for WebSocket:
    • Forward Port: 8546
    • Websockets: Enabled
    • Domain: rpc-ws.besu.local (or subdomain)

Removing nginx from RPC Nodes (Option 1)

Update install/besu-rpc-install.sh:

Remove nginx from apt packages:

apt-get install -y -qq \
    openjdk-17-jdk \
    wget \
    curl \
    jq \
    netcat-openbsd \
    iproute2 \
    iptables \
    ca-certificates \
    gnupg \
    lsb-release
    # nginx  <-- REMOVE THIS LINE

Update documentation:

  • Remove nginx from docs/APT_PACKAGES_CHECKLIST.md for RPC nodes
  • Update architecture diagrams to show nginx-proxy-manager as entry point

Network Flow

Current Flow (with nginx on RPC nodes):

Internet → nginx-proxy-manager (VMID 105) → [Optional] nginx on RPC node → Besu (8545)

Recommended Flow (nginx-proxy-manager only):

Internet → nginx-proxy-manager (VMID 105) → Besu RPC Node (2500-2502:8545)

Verification

Test RPC through nginx-proxy-manager:

# Test HTTP RPC
curl -X POST http://rpc.besu.local:8080 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Test WebSocket RPC (if configured)
wscat -c ws://rpc-ws.besu.local:8080

Verify Load Balancing:

# Check which backend is serving requests
# (nginx-proxy-manager logs will show backend selection)

Recommendation Summary

✅ Use VMID 105 (nginx-proxy-manager) for RPC

Benefits:

  • Centralized management
  • Load balancing across RPC nodes
  • SSL termination
  • High availability
  • Simplified RPC node configuration

Action Items:

  1. Remove nginx package from install/besu-rpc-install.sh (if going with Option 1)
  2. Configure nginx-proxy-manager to proxy to RPC nodes (2500-2502)
  3. Update documentation to reflect architecture
  4. Test load balancing and failover

References

Reference in New Issue View Git Blame Copy Permalink