180 lines
3.9 KiB
Markdown
180 lines
3.9 KiB
Markdown
# Multi-Chain Indexing Strategy
|
|
|
|
## Overview
|
|
|
|
This document specifies the strategy for indexing multiple blockchain networks with shared schema and per-chain workers.
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Chains[Blockchain Networks]
|
|
C1[Chain 138]
|
|
C2[Chain 1]
|
|
C3[Chain 137]
|
|
end
|
|
|
|
subgraph Workers[Indexer Workers]
|
|
W1[Worker: Chain 138]
|
|
W2[Worker: Chain 1]
|
|
W3[Worker: Chain 137]
|
|
end
|
|
|
|
subgraph DB[(Shared Database)]
|
|
PG[(PostgreSQL<br/>chain_id partitioned)]
|
|
ES[(Elasticsearch<br/>per-chain indices)]
|
|
end
|
|
|
|
C1 --> W1
|
|
C2 --> W2
|
|
C3 --> W3
|
|
|
|
W1 --> PG
|
|
W2 --> PG
|
|
W3 --> PG
|
|
|
|
W1 --> ES
|
|
W2 --> ES
|
|
W3 --> ES
|
|
```
|
|
|
|
## Per-Chain Indexer Workers
|
|
|
|
### Worker Architecture
|
|
|
|
**Design**: One indexer worker process per chain
|
|
|
|
**Isolation**:
|
|
- Separate processes/containers per chain
|
|
- Independent scaling per chain
|
|
- Chain-specific configuration
|
|
|
|
**Shared Components**:
|
|
- Database connection pool
|
|
- Message queue consumers
|
|
- Common indexing logic
|
|
|
|
### Worker Configuration
|
|
|
|
```yaml
|
|
workers:
|
|
- chain_id: 138
|
|
name: "indexer-chain-138"
|
|
rpc_url: "http://192.168.11.221:8545" # Internal IP for direct connection
|
|
adapter: "evm"
|
|
enabled: true
|
|
|
|
- chain_id: 1
|
|
name: "indexer-chain-1"
|
|
rpc_url: "https://eth-mainnet.alchemyapi.io/..."
|
|
adapter: "evm"
|
|
enabled: true
|
|
```
|
|
|
|
### Worker Scaling
|
|
|
|
**Horizontal Scaling**:
|
|
- Multiple workers per chain for high-throughput chains
|
|
- Partition blocks across workers (by block number range)
|
|
- Coordinate via database locks or message queue
|
|
|
|
**Scaling Strategy**:
|
|
- Scale workers based on chain activity
|
|
- Monitor worker lag per chain
|
|
- Auto-scale based on metrics
|
|
|
|
## Shared Schema with Chain ID Partitioning
|
|
|
|
### Database Partitioning
|
|
|
|
**Strategy**: Partition all tables by `chain_id`
|
|
|
|
**Benefits**:
|
|
- Efficient queries (partition pruning)
|
|
- Independent maintenance per chain
|
|
- Easy data isolation
|
|
|
|
**Implementation**: See `../database/postgres-schema.md`
|
|
|
|
### Index Strategy
|
|
|
|
**Global Indexes**: Across all chains for unified queries
|
|
**Chain-Specific Indexes**: Within partitions for chain-specific queries
|
|
|
|
## Cross-Chain Data Consistency
|
|
|
|
### Consistency Model
|
|
|
|
**Per-Chain Consistency**: Strong consistency within each chain
|
|
**Cross-Chain Consistency**: Eventual consistency (independent chains)
|
|
|
|
### Transaction Ordering
|
|
|
|
**Within Chain**: Maintain strict block/transaction ordering
|
|
**Across Chains**: No ordering guarantees (independent chains)
|
|
|
|
## Worker Coordination
|
|
|
|
### Block Processing Coordination
|
|
|
|
**Strategy**: Use database locks or message queue for coordination
|
|
|
|
**Approach 1: Database Locks**
|
|
- Acquire lock before processing block
|
|
- Release lock after completion
|
|
- Prevent duplicate processing
|
|
|
|
**Approach 2: Message Queue**
|
|
- Single consumer per chain partition
|
|
- Queue ensures ordering
|
|
- Automatic retry on failure
|
|
|
|
### Checkpointing
|
|
|
|
**Per-Chain Checkpoints**:
|
|
- Track last processed block per chain
|
|
- Stored in database
|
|
- Enable recovery and resume
|
|
|
|
## Performance Optimization
|
|
|
|
### Chain-Specific Optimizations
|
|
|
|
**High-Activity Chains**:
|
|
- More workers
|
|
- Larger batch sizes
|
|
- Optimized processing
|
|
|
|
**Low-Activity Chains**:
|
|
- Fewer workers
|
|
- Longer polling intervals
|
|
- Resource-efficient processing
|
|
|
|
### Resource Allocation
|
|
|
|
**CPU/Memory**: Allocate based on chain activity
|
|
**Database Connections**: Pool per chain
|
|
**Network Bandwidth**: Prioritize high-activity chains
|
|
|
|
## Monitoring
|
|
|
|
### Per-Chain Metrics
|
|
|
|
- Block lag (current block - last indexed)
|
|
- Processing rate (blocks/minute)
|
|
- Error rate
|
|
- Worker health
|
|
|
|
### Cross-Chain Metrics
|
|
|
|
- Total chains indexed
|
|
- Total blocks indexed across all chains
|
|
- System-wide throughput
|
|
|
|
## References
|
|
|
|
- Chain Adapter Interface: See `chain-adapter-interface.md`
|
|
- Database Schema: See `../database/postgres-schema.md`
|
|
- Indexer Architecture: See `../indexing/indexer-architecture.md`
|
|
|