explorer-monorepo/docs/CCIP_OPERATIONS_RUNBOOK.md

# CCIP Operations Runbook

**Date**: 2025-01-12  
**Network**: ChainID 138

---

## Overview

This runbook provides step-by-step procedures for common CCIP operations and troubleshooting.

---

## Common Operations

### Starting CCIP Monitor Service

**Task**: Task 83

**Procedure**:
```bash
# Start container
pct start 3501

# Start service
pct exec 3501 -- systemctl start ccip-monitor

# Verify status
pct exec 3501 -- systemctl status ccip-monitor

# Check health
./scripts/check-ccip-monitor-health.sh
```

**Verification**:
- Container running
- Service active
- Metrics endpoint accessible

---

### Checking Bridge Configuration

**Task**: Task 55

**Procedure**:
```bash
# Check all bridge destinations
./scripts/check-bridge-config.sh

# Check specific chain
./scripts/verify-destination-chain-config.sh Ethereum
```

**Expected Output**:
- List of configured destinations
- Status for each destination
- Missing destinations highlighted

---

### Verifying Router and Sender

**Tasks**: Tasks 1, 2

**Procedure**:
```bash
# Verify Router
./scripts/verify-ccip-router.sh

# Verify Sender
./scripts/verify-ccip-sender.sh
```

**Expected Output**:
- Contract deployment confirmed
- Functions accessible
- Configuration verified

---

### Configuring Bridge Destination

**Tasks**: Tasks 3, 50, 51

**Procedure**:
```bash
# Configure all destinations
./scripts/configure-all-bridge-destinations.sh

# Configure specific destination
./scripts/fix-bridge-errors.sh <private_key> <bridge_address>
```

**Requirements**:
- Private key for bridge owner
- Destination bridge address
- Sufficient gas

**Verification**:
```bash
./scripts/check-bridge-config.sh
```

---

### Verifying Token Mechanisms

**Tasks**: Tasks 36, 37

**Procedure**:
```bash
# Verify WETH9
./scripts/inspect-weth9-contract.sh

# Verify WETH10
./scripts/inspect-weth10-contract.sh
```

**Expected Output**:
- 1:1 ratio verified
- Contract balance = total supply
- Functions accessible

---

### Comprehensive System Verification

**Task**: Task 139

**Procedure**:
```bash
# Run comprehensive verification
./scripts/verify-complete-ccip-setup.sh
```

**Expected Output**:
- All components checked
- Status for each component
- Summary report

---

## Troubleshooting

### Bridge Configuration Issues

**Symptoms**:
- "destination not enabled" error
- Destinations not configured

**Solutions**:
1. Check current configuration:
   ```bash
   ./scripts/check-bridge-config.sh
   ```

2. Configure missing destinations:
   ```bash
   ./scripts/configure-all-bridge-destinations.sh
   ```

3. Verify configuration:
   ```bash
   ./scripts/verify-destination-chain-config.sh
   ```

---

### Fee Calculation Issues

**Symptoms**:
- Fee calculation returns 0
- Fee calculation fails

**Solutions**:
1. Verify Router configuration:
   ```bash
   ./scripts/verify-ccip-router.sh
   ```

2. Check fee calculation:
   ```bash
   ./scripts/verify-fee-calculation.sh 0.001 5009297550715157269
   ```

3. Verify LINK token:
   ```bash
   cast call 0x514910771AF9Ca656af840dff83E8264EcF986CA \
     "balanceOf(address)" \
     <bridge_address> \
     --rpc-url <rpc_url>
   ```

---

### Stuck Transactions

**Symptoms**:
- Transaction pending in mempool
- Transaction not confirming

**Solutions**:
1. Check transaction status:
   ```bash
   cast tx <tx_hash> --rpc-url <rpc_url>
   ```

2. Check nonce:
   ```bash
   cast nonce <address> --rpc-url <rpc_url>
   ```

3. Replace transaction (if needed):
   ```bash
   cast send <contract> <function> \
     --rpc-url <rpc_url> \
     --private-key <key> \
     --gas-price <higher_gas_price> \
     --nonce <next_nonce>
   ```

---

### CCIP Monitor Issues

**Symptoms**:
- Monitor service not running
- Metrics not accessible
- Errors in logs

**Solutions**:
1. Check service status:
   ```bash
   ./scripts/check-ccip-monitor-health.sh
   ```

2. Check logs:
   ```bash
   pct exec 3501 -- journalctl -u ccip-monitor -n 50
   ```

3. Restart service:
   ```bash
   pct exec 3501 -- systemctl restart ccip-monitor
   ```

---

### Token Pool Issues

**Symptoms**:
- Pool addresses unknown
- Pool configuration unclear

**Solutions**:
1. Find TokenAdminRegistry:
   ```bash
   ./scripts/verify-token-admin-registry.sh
   ```

2. Verify pool configuration:
   ```bash
   ./scripts/verify-token-pool-config.sh <pool_address>
   ```

---

## Emergency Procedures

### Pause Bridge Operations

**Procedure**:
1. Identify bridge contracts
2. Check for pause function
3. Call pause function (if available)
4. Verify pause status

**Note**: Not all bridge contracts may have pause functionality.

### Emergency Configuration Changes

**Procedure**:
1. Assess situation
2. Determine required changes
3. Verify changes are safe
4. Execute changes
5. Verify changes
6. Monitor system

---

## Maintenance

### Regular Maintenance Tasks

1. **Daily**:
   - Check system health
   - Review monitoring alerts
   - Verify service status

2. **Weekly**:
   - Review configuration
   - Check rate limit usage
   - Review logs

3. **Monthly**:
   - Comprehensive verification
   - Security review
   - Documentation updates

### Backup Procedures

1. **Configuration Backup**:
   ```bash
   # Backup bridge configuration
   ./scripts/backup-ccip-config.sh
   ```

2. **Restore Configuration**:
   ```bash
   # Restore bridge configuration
   ./scripts/restore-ccip-config.sh <backup_file>
   ```

---

## Monitoring

### Key Metrics

1. **System Health**:
   - Service status
   - Component health
   - Error rates

2. **Bridge Operations**:
   - Transaction success rate
   - Message delivery rate
   - Fee usage

3. **Token Operations**:
   - Token balances
   - Pool balances
   - Transfer volumes

### Alerting

1. **Critical Alerts**:
   - Service failures
   - Configuration errors
   - Security incidents

2. **Warning Alerts**:
   - High error rates
   - Approaching limits
   - Performance degradation

---

## Related Documentation

- [CCIP Verification Checklist](./CCIP_VERIFICATION_CHECKLIST.md) (Task 120)
- [CCIP Configuration Status](./CCIP_CONFIGURATION_STATUS.md)
- [CCIP Best Practices](./CCIP_BEST_PRACTICES.md) (Task 136)

---

**Last Updated**: 2025-01-12