smom-dbis-138/runbooks/ccip-operations.md

CCIP Operations Runbook

Overview

This runbook provides operational procedures for managing CCIP (Chainlink Cross-Chain Interoperability Protocol) on the DeFi Oracle Meta Mainnet.

Daily Operations

Health Checks

1. Check CCIP Router Status

   kubectl get pods -n besu-network -l app=ccip-router
   kubectl logs -n besu-network -l app=ccip-router --tail=100
   

2. Check Message Processing

   # Check recent messages
   cast logs --from-block latest-100 --address $CCIP_SENDER --rpc-url $RPC_URL | grep MessageSent
   

3. Monitor Metrics

   • Message send success rate
   • Message delivery latency
   • Fee consumption
   • Error rates

LINK Balance Monitoring

1. Check Balance

   cast call $LINK_TOKEN "balanceOf(address)" $CCIP_SENDER --rpc-url $RPC_URL
   

2. Set Alert Threshold

   • Alert when balance < 10 LINK
   • Alert when balance < 5 LINK (critical)

3. Refill Balance

   cast send $LINK_TOKEN "transfer(address,uint256)" $CCIP_SENDER $AMOUNT \
     --rpc-url $RPC_URL --private-key $PRIVATE_KEY
   

Weekly Operations

Review Message Statistics

1. Message Volume

   • Total messages sent
   • Success rate
   • Average latency

2. Fee Analysis

   • Total fees spent
   • Average fee per message
   • Fee trends

3. Error Analysis

   • Error types
   • Error frequency
   • Root causes

Performance Review

1. Latency Analysis

   • Average delivery time
   • P95/P99 latency
   • Outliers

2. Throughput

   • Messages per hour/day
   • Peak load times
   • Capacity planning

Monthly Operations

Security Review

1. Access Control Audit

   • Review authorized senders/receivers
   • Check for unauthorized access
   • Verify role assignments

2. Message Validation

   • Review message format compliance
   • Check for anomalies
   • Verify replay protection

Cost Optimization

1. Fee Optimization

   • Review fee trends
   • Identify optimization opportunities
   • Implement improvements

2. Message Optimization

   • Reduce message size
   • Batch updates when possible
   • Optimize encoding

Incident Response

Message Delivery Failure

1. Identify Issue

   • Check message status
   • Verify target chain status
   • Check router logs

2. Diagnose

   • Check LINK balance
   • Verify router configuration
   • Check target chain connectivity

3. Resolve

   • Fix underlying issue
   • Resend message if needed
   • Update configuration if required

High Error Rate

1. Investigate

   • Check error logs
   • Identify error patterns
   • Review recent changes

2. Mitigate

   • Pause sending if critical
   • Fix root cause
   • Resume operations

Router Unavailable

1. Check Status

   • Verify router deployment
   • Check service health
   • Review logs

2. Recovery

   • Restart router if needed
   • Verify connectivity
   • Test message sending

Maintenance

Contract Upgrades

1. Plan Upgrade

   • Review upgrade proposal
   • Test in staging
   • Schedule maintenance window

2. Execute Upgrade

   • Pause operations
   • Deploy new contracts
   • Update configurations
   • Resume operations

3. Verify

   • Test message sending
   • Verify message receiving
   • Monitor for issues

Configuration Changes

1. Review Impact

   • Assess change impact
   • Test in staging
   • Plan rollback

2. Apply Changes

   • Update configuration
   • Monitor closely
   • Verify functionality

Monitoring

Key Metrics

• ccip_messages_sent_total: Total messages sent
• ccip_messages_received_total: Total messages received
• ccip_message_latency_seconds: Message delivery latency
• ccip_fees_total: Total LINK spent on fees
• ccip_errors_total: Total errors

Alerts

• High error rate (> 5%)
• Low success rate (< 95%)
• High latency (> 5 minutes)
• Low LINK balance (< 10 LINK)
• Router unavailable

References

• CCIP Integration Guide
• CCIP Router Setup
• CCIP Troubleshooting
• CCIP Incident Response