d-bis/explorer-monorepo
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
explorer-monorepo/docs/specs/infrastructure/node-rpc-architecture.md

11 KiB
Raw Permalink Blame History

Node & RPC Architecture Specification

Overview

This document specifies the architecture for ChainID 138 blockchain nodes and RPC endpoints that power the explorer platform. The architecture ensures high availability, performance, and reliability for block data retrieval and transaction submission.

Architecture

flowchart TB
    subgraph Nodes[ChainID 138 Nodes]
        AN[Archive Node<br/>Full History]
        TN[Tracing Node<br/>Call Traces]
        VN[Validator Node<br/>Block Production]
    end
    
    subgraph LoadBalancer[Load Balancer Layer]
        LB[NGINX/HAProxy<br/>SSL Termination]
    end
    
    subgraph RPCPool[RPC Endpoint Pool]
        RPC1[RPC Endpoint 1<br/>Archive + Trace]
        RPC2[RPC Endpoint 2<br/>Archive + Trace]
        RPC3[RPC Endpoint 3<br/>Archive Only]
    end
    
    subgraph Services[Explorer Services]
        Indexer[Indexer Service]
        API[API Gateway]
        Mempool[Mempool Service]
    end
    
    AN --> RPC1
    TN --> RPC1
    AN --> RPC2
    TN --> RPC2
    AN --> RPC3
    
    LB --> RPC1
    LB --> RPC2
    LB --> RPC3
    
    Indexer --> LB
    API --> LB
    Mempool --> LB

Node Setup Requirements

Archive Node Configuration

Purpose: Store complete blockchain history for historical queries and data indexing.

Requirements:

  • Archive mode: Full historical state access (--sync-mode=full --gcmode=archive)
  • Storage: High-capacity storage with SSD for recent blocks, HDD for cold storage
  • Memory: Minimum 32GB RAM for efficient state access
  • Database: Fast state database (LevelDB, RocksDB)
  • Network: High-bandwidth connection for syncing

Configuration Example (Hyperledger Besu):

data-path="./data"
genesis-file="./genesis.json"
network-id=138
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3","ADMIN","DEBUG","TRACE","EEA","PRIV"]
rpc-http-cors-origins=["*"]
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
sync-mode="FAST"
data-storage-format="BONSAI"

Tracing Node Configuration

Purpose: Provide call trace capabilities for transaction debugging and internal transaction tracking.

Requirements:

  • Archive mode enabled for historical trace access
  • Debug and trace API endpoints enabled
  • Additional CPU for trace computation
  • Trace storage strategy (on-demand vs cached)

Trace Types Supported:

  • trace_call: Single call trace
  • trace_block: All traces in a block
  • trace_transaction: All traces in a transaction
  • trace_replayBlockTransactions: Replay with state
  • trace_replayTransaction: Replay single transaction

Node Health Monitoring

Health Check Endpoints:

  • eth_blockNumber: Returns latest block number
  • eth_syncing: Returns sync status
  • net_peerCount: Returns connected peer count

Health Check Criteria:

  • Latest block is within 5 blocks of network head
  • Peer count > 5 (configurable)
  • Response time < 500ms for simple queries
  • No critical errors in logs

Monitoring Metrics:

  • Block height lag vs network
  • Peer connection count
  • RPC request latency (p50, p95, p99)
  • Error rate by endpoint
  • CPU and memory usage
  • Disk I/O and storage usage

RPC Endpoint Architecture

Load Balancing Strategy

Distribution Method: Round-robin with health-aware routing

Failover Strategy:

  • Primary: Archive + Trace capable nodes
  • Secondary: Archive-only nodes (fallback for non-trace requests)
  • Automatic failover on health check failure
  • Circuit breaker pattern to prevent cascading failures

Session Affinity: Not required for stateless RPC requests

RPC Capabilities Matrix

Capability Archive Node Tracing Node Light Node
Historical blocks ✅ Full ✅ Full ❌ Limited
Current state ✅ Full ✅ Full ✅ Full
Call traces ❌ No ✅ Yes ❌ No
Archive state queries ✅ Yes ✅ Yes ❌ No
Transaction submission ✅ Yes ✅ Yes ✅ Yes

Endpoint Types

Public RPC Endpoint

  • Internal URL: http://192.168.11.221:8545 (VMID 2201 - direct connection, recommended for internal services)
  • Public URL: https://rpc-http-pub.d-bis.org (via proxy - for external access)
  • Authentication: Optional API keys for rate limiting
  • Rate Limits:
    • Unauthenticated: 10 req/s
    • Authenticated: 100 req/s (tiered)
  • CORS: Enabled for web applications

Private RPC Endpoint

  • URL: https://rpc-http-prv.d-bis.org
  • Authentication: Required (authorized access)
  • Rate Limits: Higher limits for authenticated users
  • CORS: Enabled for authorized domains

Internal RPC Endpoint

  • URL: http://192.168.11.211:8545 (internal network - VMID 2101)
  • Authentication: Network-based (firewall rules)
  • Rate Limits: Higher limits for internal services
  • CORS: Disabled

WebSocket Endpoint

Configuration:

  • Public WS (Internal): ws://192.168.11.221:8546 (VMID 2201 - direct connection, recommended for internal services)
  • Public WS (Public): wss://rpc-ws-pub.d-bis.org (via proxy - for external access)
  • Private WS (Internal): ws://192.168.11.211:8546 (VMID 2101 - direct connection)
  • Private WS (Public): wss://rpc-ws-prv.d-bis.org (via proxy - for external access)
  • Subscriptions: New blocks, pending transactions, logs
  • Connection Limits: 100 concurrent connections per IP
  • Heartbeat: 30-second ping/pong

Subscription Types:

  • newHeads: New block headers
  • logs: Event logs matching filter
  • pendingTransactions: Pending transactions
  • syncing: Sync status updates

Rate Limiting and Access Control

Rate Limiting Strategy

Tiers:

  1. Anonymous: 10 requests/second, 100 requests/minute
  2. API Key (Free): 100 requests/second, 1000 requests/minute
  3. API Key (Pro): 500 requests/second, 5000 requests/minute
  4. Internal Services: 1000 requests/second, unlimited

Implementation:

  • Token bucket algorithm
  • Per-IP and per-API-key limits
  • Separate limits for different endpoint categories:
    • Simple queries (eth_blockNumber, eth_getBalance): Higher limits
    • Complex queries (trace_block, eth_call with large state): Lower limits
    • Write operations (eth_sendTransaction): Strict limits

Access Control

API Key Management:

  • Key generation with secure random tokens
  • Key metadata: name, tier, creation date, last used
  • Key rotation support
  • Revocation capability

IP Whitelisting:

  • Support for IP whitelisting per API key
  • CIDR notation support
  • Geographic restrictions (optional)

Rate Limit Headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200

Failover Strategies

Automatic Failover

Detection:

  • Health checks every 10 seconds
  • Consecutive failures threshold: 3
  • Recovery threshold: 2 successful checks

Failover Actions:

  1. Mark node as unhealthy
  2. Route traffic to healthy nodes
  3. Continue health checking for recovery
  4. Log failover events

Manual Failover

Scenarios:

  • Planned maintenance
  • Node performance degradation
  • Security incidents

Procedure:

  1. Drain connections (wait for active requests)
  2. Remove from load balancer pool
  3. Perform maintenance
  4. Verify health
  5. Re-add to pool

Performance Requirements

Latency Targets

Operation Type p50 Target p95 Target p99 Target
Simple queries (blockNumber, balance) < 50ms < 100ms < 200ms
Block queries < 100ms < 200ms < 500ms
Transaction queries < 150ms < 300ms < 1000ms
Trace queries < 500ms < 2000ms < 5000ms
Historical state queries < 1000ms < 5000ms < 10000ms

Throughput Targets

  • Read operations: 1000 req/s per node
  • Write operations: 100 req/s per node
  • WebSocket subscriptions: 1000 concurrent per node

Availability Targets

  • Uptime: 99.9% (8.76 hours downtime/year)
  • Failover time: < 30 seconds
  • Data consistency: Zero data loss

Security Considerations

Network Security

  • TLS/SSL: Required for public endpoints (TLS 1.2+)
  • Firewall Rules: Restrict access to internal endpoints
  • DDoS Protection: WAF/CDN integration for public endpoints
  • IP Filtering: Support for IP allowlists/blocklists

Authentication

  • API Keys: HMAC-based authentication
  • Key Storage: Encrypted storage with KMS
  • Key Rotation: Support for periodic rotation

Audit Logging

  • All RPC requests logged (with PII sanitization)
  • Log retention: 90 days
  • Log fields: timestamp, IP, API key (hash), endpoint, response code, latency

Integration Points

Indexer Service Integration

  • Primary connection to archive nodes
  • Batch requests for efficiency
  • Connection pooling
  • Retry logic with exponential backoff

API Gateway Integration

  • Proxy requests to RPC pool
  • Add authentication/authorization
  • Apply rate limiting
  • Cache common queries

Mempool Service Integration

  • WebSocket subscription to pending transactions
  • Direct connection for transaction submission
  • Priority queuing for indexer submissions

Implementation Guidelines

Node Deployment

Infrastructure:

  • Use container orchestration (Kubernetes) for scalability
  • Separate deployments for archive and tracing nodes
  • Horizontal scaling based on load

Configuration Management:

  • Use configuration files (TOML/YAML)
  • Environment-specific overrides
  • Secrets management via KMS/Vault

Monitoring Integration

Metrics to Export:

  • Prometheus metrics for node health
  • Custom metrics for RPC performance
  • Integration with Grafana dashboards

Alerting Rules:

  • Node down alert
  • High latency alert
  • High error rate alert
  • Low peer count alert

Testing Strategy

Unit Tests

  • RPC endpoint implementations
  • Rate limiting logic
  • Health check logic

Integration Tests

  • Load balancer failover
  • Rate limiting enforcement
  • Authentication/authorization

Load Tests

  • Simulate production load
  • Test failover scenarios
  • Validate performance targets

Migration from Existing Setup

Current Setup (Blockscout Integration)

  • Public RPC (Internal): http://192.168.11.221:8545 (VMID 2201 - recommended for internal services)
  • Public RPC (Public): https://rpc-http-pub.d-bis.org (via proxy - for external access)
  • Private RPC (Internal): http://192.168.11.211:8545 (VMID 2101)
  • Private RPC (Public): https://rpc-http-prv.d-bis.org (via proxy)
  • Deprecated: https://rpc-core.d-bis.org (no longer public, internal only)
  • Internal RPC: http://192.168.11.250:8545
  • WebSocket: ws://192.168.11.250:8546

Migration Steps

  1. Deploy new load balancer layer
  2. Configure health checks
  3. Gradually migrate traffic
  4. Monitor performance
  5. Decommission old direct connections

References

  • Existing Blockscout deployment: docs/BLOCKSCOUT_COMPLETE_SUMMARY.md
  • Network architecture: See network-topology.md
  • API Gateway: See ../api/api-gateway.md
Reference in New Issue View Git Blame Copy Permalink