smom-dbis-138/docs/bridge/RUNBOOK.md

Bridge Operations Runbook

Table of Contents

1. Incident Response
2. Common Operations
3. Troubleshooting
4. Emergency Procedures

Incident Response

High Failure Rate

Symptoms:

• Success rate drops below 95%
• Multiple failed transfers in short time

Actions:

1. Check Prometheus metrics: bridge_success_rate < 95
2. Review recent transfer logs for error patterns
3. Check destination chain status (RPC availability, finality issues)
4. Verify thirdweb API status
5. Check XRPL connection if XRPL routes affected
6. If issue persists > 10 minutes, pause affected route:

   forge script script/bridge/interop/PauseDestination.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address,uint256)" $REGISTRY_ADDRESS $CHAIN_ID
   

Liquidity Failure

Symptoms:

• Transfers failing with "insufficient liquidity" errors
• XRPL hot wallet balance low

Actions:

1. Check XRPL hot wallet balance:

   curl -X POST $XRPL_SERVER \
     -d '{"method":"account_info","params":[{"account":"$XRPL_ACCOUNT"}]}'
   

2. Replenish hot wallet if balance < threshold
3. Check EVM destination liquidity pools
4. If critical, pause affected token:

   forge script script/bridge/interop/PauseToken.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address,address)" $REGISTRY_ADDRESS $TOKEN_ADDRESS
   

High Settlement Time

Symptoms:

• Average settlement time > 10 minutes
• Users reporting slow transfers

Actions:

1. Check destination chain finality requirements
2. Verify FireFly workflow engine is processing transfers
3. Check Cacti connector status
4. Review route health scores
5. Consider switching to alternative route if available

Bridge Pause

Symptoms:

• All transfers failing
• Bridge status shows "PAUSED"

Actions:

1. Identify reason for pause (check admin logs)
2. Resolve underlying issue
3. Unpause bridge:

   forge script script/bridge/interop/UnpauseBridge.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address)" $VAULT_ADDRESS
   

Common Operations

Add New Destination

1. Register destination in registry:

   forge script script/bridge/interop/RegisterDestination.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address,uint256,string,uint256,uint256,uint256,address)" \
       $REGISTRY_ADDRESS \
       $CHAIN_ID \
       "Chain Name" \
       $MIN_FINALITY_BLOCKS \
       $TIMEOUT_SECONDS \
       $BASE_FEE_BPS \
       $FEE_RECIPIENT
   

2. Update FireFly configuration

3. Configure Cacti connector if needed

4. Test with small amount transfer

Add New Token

1. Register token in registry:

   forge script script/bridge/interop/RegisterToken.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address,address,uint256,uint256,uint256[],uint8,uint256)" \
       $REGISTRY_ADDRESS \
       $TOKEN_ADDRESS \
       $MIN_AMOUNT \
       $MAX_AMOUNT \
       "[137,10,8453]" \
       $RISK_LEVEL \
       $BRIDGE_FEE_BPS
   

2. Verify token contract is valid

3. Test with small amount transfer

Process Refund

1. Verify transfer is eligible for refund:

   cast call $VAULT_ADDRESS \
       "isRefundable(bytes32)" \
       $TRANSFER_ID \
       --rpc-url $RPC_URL
   

2. Initiate refund (requires HSM signature):

   # Generate HSM signature first
   # Then call initiateRefund with signature
   

3. Execute refund:

   forge script script/bridge/interop/ExecuteRefund.s.sol \
       --rpc-url $RPC_URL \
       --private-key $REFUND_OPERATOR_KEY \
       --broadcast \
       --sig "run(address,bytes32)" $VAULT_ADDRESS $TRANSFER_ID
   

Update Route Health

After successful/failed transfer, update route health:

curl -X POST $API_URL/api/admin/update-route-health \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "chainId": 137,
    "token": "0x...",
    "success": true,
    "settlementTime": 300
  }'

Troubleshooting

Transfer Stuck in EXECUTING

Check:

1. FireFly workflow status
2. Cacti connector logs
3. Destination chain transaction status

Resolution:

• If destination tx confirmed, update status manually
• If destination tx failed, mark transfer as FAILED and initiate refund

HSM Signing Fails

Check:

1. HSM service health: curl $HSM_ENDPOINT/health
2. HSM API key validity
3. Key ID exists and is accessible

Resolution:

• Restart HSM service if needed
• Verify HSM key configuration
• Check HSM logs for errors

XRPL Connection Issues

Check:

1. XRPL server connectivity: ping xrpl-server
2. XRPL account balance
3. XRPL network status

Resolution:

• Switch to backup XRPL server if available
• Verify XRPL account credentials
• Check XRPL network status page

FireFly Not Processing

Check:

1. FireFly service status: kubectl get pods -n firefly
2. FireFly logs: kubectl logs -f firefly-core -n firefly
3. Database connectivity

Resolution:

• Restart FireFly service if needed
• Check database connection
• Verify FireFly configuration

Emergency Procedures

Global Pause

If critical security issue detected:

# Pause all contracts
forge script script/bridge/interop/EmergencyPause.s.sol \
    --rpc-url $RPC_URL \
    --private-key $ADMIN_KEY \
    --broadcast

Key Rotation

If HSM key compromised:

1. Generate new HSM key
2. Update HSM signer address in contracts:

   forge script script/bridge/interop/UpdateHSMSigner.s.sol \
       --rpc-url $RPC_URL \
       --private-key $ADMIN_KEY \
       --broadcast \
       --sig "run(address,address)" $CONTROLLER_ADDRESS $NEW_HSM_SIGNER
   

3. Revoke old key access
4. Test with small operation

Disaster Recovery

If bridge infrastructure fails:

1. Immediate Actions:

   • Pause all bridge operations
   • Notify users via status page
   • Assess damage scope

2. Recovery Steps:

   • Restore from backups
   • Redeploy infrastructure
   • Verify contract states
   • Test with small transfers
   • Gradually resume operations

3. Post-Incident:

   • Document incident
   • Review logs and metrics
   • Update runbooks
   • Conduct post-mortem

Monitoring Checklist

Daily:

• Review success rate metrics
• Check for failed transfers
• Verify XRPL hot wallet balance
• Review alert notifications

Weekly:

• Review route health scores
• Analyze settlement time trends
• Check HSM service health
• Review proof-of-reserves

Monthly:

• Security audit review
• Update documentation
• Review and update runbooks
• Capacity planning review

Contact Information

On-Call Engineer: oncall@chain138.example.com
Security Team: security@chain138.example.com
DevOps: devops@chain138.example.com

Emergency Escalation:

1. Page on-call engineer
2. If no response in 15 minutes, escalate to team lead
3. For security incidents, immediately contact security team