371 lines
6.0 KiB
Markdown
371 lines
6.0 KiB
Markdown
# CCIP Operations Runbook
|
|
|
|
**Date**: 2025-01-12
|
|
**Network**: ChainID 138
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This runbook provides step-by-step procedures for common CCIP operations and troubleshooting.
|
|
|
|
---
|
|
|
|
## Common Operations
|
|
|
|
### Starting CCIP Monitor Service
|
|
|
|
**Task**: Task 83
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Start container
|
|
pct start 3501
|
|
|
|
# Start service
|
|
pct exec 3501 -- systemctl start ccip-monitor
|
|
|
|
# Verify status
|
|
pct exec 3501 -- systemctl status ccip-monitor
|
|
|
|
# Check health
|
|
./scripts/check-ccip-monitor-health.sh
|
|
```
|
|
|
|
**Verification**:
|
|
- Container running
|
|
- Service active
|
|
- Metrics endpoint accessible
|
|
|
|
---
|
|
|
|
### Checking Bridge Configuration
|
|
|
|
**Task**: Task 55
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Check all bridge destinations
|
|
./scripts/check-bridge-config.sh
|
|
|
|
# Check specific chain
|
|
./scripts/verify-destination-chain-config.sh Ethereum
|
|
```
|
|
|
|
**Expected Output**:
|
|
- List of configured destinations
|
|
- Status for each destination
|
|
- Missing destinations highlighted
|
|
|
|
---
|
|
|
|
### Verifying Router and Sender
|
|
|
|
**Tasks**: Tasks 1, 2
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Verify Router
|
|
./scripts/verify-ccip-router.sh
|
|
|
|
# Verify Sender
|
|
./scripts/verify-ccip-sender.sh
|
|
```
|
|
|
|
**Expected Output**:
|
|
- Contract deployment confirmed
|
|
- Functions accessible
|
|
- Configuration verified
|
|
|
|
---
|
|
|
|
### Configuring Bridge Destination
|
|
|
|
**Tasks**: Tasks 3, 50, 51
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Configure all destinations
|
|
./scripts/configure-all-bridge-destinations.sh
|
|
|
|
# Configure specific destination
|
|
./scripts/fix-bridge-errors.sh <private_key> <bridge_address>
|
|
```
|
|
|
|
**Requirements**:
|
|
- Private key for bridge owner
|
|
- Destination bridge address
|
|
- Sufficient gas
|
|
|
|
**Verification**:
|
|
```bash
|
|
./scripts/check-bridge-config.sh
|
|
```
|
|
|
|
---
|
|
|
|
### Verifying Token Mechanisms
|
|
|
|
**Tasks**: Tasks 36, 37
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Verify WETH9
|
|
./scripts/inspect-weth9-contract.sh
|
|
|
|
# Verify WETH10
|
|
./scripts/inspect-weth10-contract.sh
|
|
```
|
|
|
|
**Expected Output**:
|
|
- 1:1 ratio verified
|
|
- Contract balance = total supply
|
|
- Functions accessible
|
|
|
|
---
|
|
|
|
### Comprehensive System Verification
|
|
|
|
**Task**: Task 139
|
|
|
|
**Procedure**:
|
|
```bash
|
|
# Run comprehensive verification
|
|
./scripts/verify-complete-ccip-setup.sh
|
|
```
|
|
|
|
**Expected Output**:
|
|
- All components checked
|
|
- Status for each component
|
|
- Summary report
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Bridge Configuration Issues
|
|
|
|
**Symptoms**:
|
|
- "destination not enabled" error
|
|
- Destinations not configured
|
|
|
|
**Solutions**:
|
|
1. Check current configuration:
|
|
```bash
|
|
./scripts/check-bridge-config.sh
|
|
```
|
|
|
|
2. Configure missing destinations:
|
|
```bash
|
|
./scripts/configure-all-bridge-destinations.sh
|
|
```
|
|
|
|
3. Verify configuration:
|
|
```bash
|
|
./scripts/verify-destination-chain-config.sh
|
|
```
|
|
|
|
---
|
|
|
|
### Fee Calculation Issues
|
|
|
|
**Symptoms**:
|
|
- Fee calculation returns 0
|
|
- Fee calculation fails
|
|
|
|
**Solutions**:
|
|
1. Verify Router configuration:
|
|
```bash
|
|
./scripts/verify-ccip-router.sh
|
|
```
|
|
|
|
2. Check fee calculation:
|
|
```bash
|
|
./scripts/verify-fee-calculation.sh 0.001 5009297550715157269
|
|
```
|
|
|
|
3. Verify LINK token:
|
|
```bash
|
|
cast call 0x514910771AF9Ca656af840dff83E8264EcF986CA \
|
|
"balanceOf(address)" \
|
|
<bridge_address> \
|
|
--rpc-url <rpc_url>
|
|
```
|
|
|
|
---
|
|
|
|
### Stuck Transactions
|
|
|
|
**Symptoms**:
|
|
- Transaction pending in mempool
|
|
- Transaction not confirming
|
|
|
|
**Solutions**:
|
|
1. Check transaction status:
|
|
```bash
|
|
cast tx <tx_hash> --rpc-url <rpc_url>
|
|
```
|
|
|
|
2. Check nonce:
|
|
```bash
|
|
cast nonce <address> --rpc-url <rpc_url>
|
|
```
|
|
|
|
3. Replace transaction (if needed):
|
|
```bash
|
|
cast send <contract> <function> \
|
|
--rpc-url <rpc_url> \
|
|
--private-key <key> \
|
|
--gas-price <higher_gas_price> \
|
|
--nonce <next_nonce>
|
|
```
|
|
|
|
---
|
|
|
|
### CCIP Monitor Issues
|
|
|
|
**Symptoms**:
|
|
- Monitor service not running
|
|
- Metrics not accessible
|
|
- Errors in logs
|
|
|
|
**Solutions**:
|
|
1. Check service status:
|
|
```bash
|
|
./scripts/check-ccip-monitor-health.sh
|
|
```
|
|
|
|
2. Check logs:
|
|
```bash
|
|
pct exec 3501 -- journalctl -u ccip-monitor -n 50
|
|
```
|
|
|
|
3. Restart service:
|
|
```bash
|
|
pct exec 3501 -- systemctl restart ccip-monitor
|
|
```
|
|
|
|
---
|
|
|
|
### Token Pool Issues
|
|
|
|
**Symptoms**:
|
|
- Pool addresses unknown
|
|
- Pool configuration unclear
|
|
|
|
**Solutions**:
|
|
1. Find TokenAdminRegistry:
|
|
```bash
|
|
./scripts/verify-token-admin-registry.sh
|
|
```
|
|
|
|
2. Verify pool configuration:
|
|
```bash
|
|
./scripts/verify-token-pool-config.sh <pool_address>
|
|
```
|
|
|
|
---
|
|
|
|
## Emergency Procedures
|
|
|
|
### Pause Bridge Operations
|
|
|
|
**Procedure**:
|
|
1. Identify bridge contracts
|
|
2. Check for pause function
|
|
3. Call pause function (if available)
|
|
4. Verify pause status
|
|
|
|
**Note**: Not all bridge contracts may have pause functionality.
|
|
|
|
### Emergency Configuration Changes
|
|
|
|
**Procedure**:
|
|
1. Assess situation
|
|
2. Determine required changes
|
|
3. Verify changes are safe
|
|
4. Execute changes
|
|
5. Verify changes
|
|
6. Monitor system
|
|
|
|
---
|
|
|
|
## Maintenance
|
|
|
|
### Regular Maintenance Tasks
|
|
|
|
1. **Daily**:
|
|
- Check system health
|
|
- Review monitoring alerts
|
|
- Verify service status
|
|
|
|
2. **Weekly**:
|
|
- Review configuration
|
|
- Check rate limit usage
|
|
- Review logs
|
|
|
|
3. **Monthly**:
|
|
- Comprehensive verification
|
|
- Security review
|
|
- Documentation updates
|
|
|
|
### Backup Procedures
|
|
|
|
1. **Configuration Backup**:
|
|
```bash
|
|
# Backup bridge configuration
|
|
./scripts/backup-ccip-config.sh
|
|
```
|
|
|
|
2. **Restore Configuration**:
|
|
```bash
|
|
# Restore bridge configuration
|
|
./scripts/restore-ccip-config.sh <backup_file>
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
### Key Metrics
|
|
|
|
1. **System Health**:
|
|
- Service status
|
|
- Component health
|
|
- Error rates
|
|
|
|
2. **Bridge Operations**:
|
|
- Transaction success rate
|
|
- Message delivery rate
|
|
- Fee usage
|
|
|
|
3. **Token Operations**:
|
|
- Token balances
|
|
- Pool balances
|
|
- Transfer volumes
|
|
|
|
### Alerting
|
|
|
|
1. **Critical Alerts**:
|
|
- Service failures
|
|
- Configuration errors
|
|
- Security incidents
|
|
|
|
2. **Warning Alerts**:
|
|
- High error rates
|
|
- Approaching limits
|
|
- Performance degradation
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [CCIP Verification Checklist](./CCIP_VERIFICATION_CHECKLIST.md) (Task 120)
|
|
- [CCIP Configuration Status](./CCIP_CONFIGURATION_STATUS.md)
|
|
- [CCIP Best Practices](./CCIP_BEST_PRACTICES.md) (Task 136)
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-01-12
|
|
|