explorer-monorepo/docs/CCIP_OPERATIONS_RUNBOOK.md

CCIP Operations Runbook

Date: 2025-01-12
Network: ChainID 138

---

Overview

This runbook provides step-by-step procedures for common CCIP operations and troubleshooting.

---

Common Operations

Starting CCIP Monitor Service

Task: Task 83

Procedure:

# Start container
pct start 3501

# Start service
pct exec 3501 -- systemctl start ccip-monitor

# Verify status
pct exec 3501 -- systemctl status ccip-monitor

# Check health
./scripts/check-ccip-monitor-health.sh

Verification:

• Container running
• Service active
• Metrics endpoint accessible

---

Checking Bridge Configuration

Task: Task 55

Procedure:

# Check all bridge destinations
./scripts/check-bridge-config.sh

# Check specific chain
./scripts/verify-destination-chain-config.sh Ethereum

Expected Output:

• List of configured destinations
• Status for each destination
• Missing destinations highlighted

---

Verifying Router and Sender

Tasks: Tasks 1, 2

Procedure:

# Verify Router
./scripts/verify-ccip-router.sh

# Verify Sender
./scripts/verify-ccip-sender.sh

Expected Output:

• Contract deployment confirmed
• Functions accessible
• Configuration verified

---

Configuring Bridge Destination

Tasks: Tasks 3, 50, 51

Procedure:

# Configure all destinations
./scripts/configure-all-bridge-destinations.sh

# Configure specific destination
./scripts/fix-bridge-errors.sh <private_key> <bridge_address>

Requirements:

• Private key for bridge owner
• Destination bridge address
• Sufficient gas

Verification:

./scripts/check-bridge-config.sh

---

Verifying Token Mechanisms

Tasks: Tasks 36, 37

Procedure:

# Verify WETH9
./scripts/inspect-weth9-contract.sh

# Verify WETH10
./scripts/inspect-weth10-contract.sh

Expected Output:

• 1:1 ratio verified
• Contract balance = total supply
• Functions accessible

---

Comprehensive System Verification

Task: Task 139

Procedure:

# Run comprehensive verification
./scripts/verify-complete-ccip-setup.sh

Expected Output:

• All components checked
• Status for each component
• Summary report

---

Troubleshooting

Bridge Configuration Issues

Symptoms:

• "destination not enabled" error
• Destinations not configured

Solutions:

1. Check current configuration:

   ./scripts/check-bridge-config.sh
   

2. Configure missing destinations:

   ./scripts/configure-all-bridge-destinations.sh
   

3. Verify configuration:

   ./scripts/verify-destination-chain-config.sh
   

---

Fee Calculation Issues

Symptoms:

• Fee calculation returns 0
• Fee calculation fails

Solutions:

1. Verify Router configuration:

   ./scripts/verify-ccip-router.sh
   

2. Check fee calculation:

   ./scripts/verify-fee-calculation.sh 0.001 5009297550715157269
   

3. Verify LINK token:

   cast call 0x514910771AF9Ca656af840dff83E8264EcF986CA \
     "balanceOf(address)" \
     <bridge_address> \
     --rpc-url <rpc_url>
   

---

Stuck Transactions

Symptoms:

• Transaction pending in mempool
• Transaction not confirming

Solutions:

1. Check transaction status:

   cast tx <tx_hash> --rpc-url <rpc_url>
   

2. Check nonce:

   cast nonce <address> --rpc-url <rpc_url>
   

3. Replace transaction (if needed):

   cast send <contract> <function> \
     --rpc-url <rpc_url> \
     --private-key <key> \
     --gas-price <higher_gas_price> \
     --nonce <next_nonce>
   

---

CCIP Monitor Issues

Symptoms:

• Monitor service not running
• Metrics not accessible
• Errors in logs

Solutions:

1. Check service status:

   ./scripts/check-ccip-monitor-health.sh
   

2. Check logs:

   pct exec 3501 -- journalctl -u ccip-monitor -n 50
   

3. Restart service:

   pct exec 3501 -- systemctl restart ccip-monitor
   

---

Token Pool Issues

Symptoms:

• Pool addresses unknown
• Pool configuration unclear

Solutions:

1. Find TokenAdminRegistry:

   ./scripts/verify-token-admin-registry.sh
   

2. Verify pool configuration:

   ./scripts/verify-token-pool-config.sh <pool_address>
   

---

Emergency Procedures

Pause Bridge Operations

Procedure:

1. Identify bridge contracts
2. Check for pause function
3. Call pause function (if available)
4. Verify pause status

Note: Not all bridge contracts may have pause functionality.

Emergency Configuration Changes

Procedure:

1. Assess situation
2. Determine required changes
3. Verify changes are safe
4. Execute changes
5. Verify changes
6. Monitor system

---

Maintenance

Regular Maintenance Tasks

1. Daily:

   • Check system health
   • Review monitoring alerts
   • Verify service status

2. Weekly:

   • Review configuration
   • Check rate limit usage
   • Review logs

3. Monthly:

   • Comprehensive verification
   • Security review
   • Documentation updates

Backup Procedures

1. Configuration Backup:

   # Backup bridge configuration
   ./scripts/backup-ccip-config.sh
   

2. Restore Configuration:

   # Restore bridge configuration
   ./scripts/restore-ccip-config.sh <backup_file>
   

---

Monitoring

Key Metrics

1. System Health:

   • Service status
   • Component health
   • Error rates

2. Bridge Operations:

   • Transaction success rate
   • Message delivery rate
   • Fee usage

3. Token Operations:

   • Token balances
   • Pool balances
   • Transfer volumes

Alerting

1. Critical Alerts:

   • Service failures
   • Configuration errors
   • Security incidents

2. Warning Alerts:

   • High error rates
   • Approaching limits
   • Performance degradation

---

Related Documentation

• CCIP Verification Checklist (Task 120)
• CCIP Configuration Status
• CCIP Best Practices (Task 136)

---

Last Updated: 2025-01-12