CurrenciCombo/docs/Error_Handling_Rollback_Spec.md

Error Handling & Rollback Specification

Overview

This document specifies comprehensive error handling and rollback procedures for the ISO-20022 Combo Flow system, including failure modes (DLT fail after bank prepare, bank fail after DLT commit, liquidity denial), recovery mechanisms, partial execution prevention, and audit trail requirements for aborted plans.

---

1. Failure Modes

Mode A: DLT Execution Fails After Bank Prepare

Scenario: Bank has accepted a provisional ISO message (pacs.008 in prepared state), but DLT execution fails during commit phase.

Detection:

// DLT execution fails
const dltResult = await dltHandler.executeSteps(plan.steps);
if (!dltResult.success) {
  // Trigger rollback
  await rollbackBankPrepare(planId);
}

Recovery Mechanism:

1. Abort Bank Instruction: Send abort/cancel message to bank (camt.056)
2. Release DLT Reservations: Release any reserved collateral/amounts
3. Update Notary: Record abort in notary registry
4. Audit Trail: Log failure and recovery actions

Implementation:

async function handleDLTFailureAfterBankPrepare(planId: string) {
  // 1. Abort bank instruction
  await bankConnector.abortPayment(planId, {
    reason: 'DLT execution failed',
    messageType: 'camt.056'
  });

  // 2. Release DLT reservations
  await dltHandler.releaseReservations(planId);

  // 3. Update notary
  await notaryRegistry.finalizePlan(planId, false);

  // 4. Log audit trail
  await auditLogger.log({
    planId,
    event: 'ROLLBACK_DLT_FAILURE',
    timestamp: new Date(),
    actions: ['bank_aborted', 'dlt_released', 'notary_updated']
  });

  // 5. Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_FAILED',
    message: 'DLT execution failed. Bank instruction has been cancelled.'
  });
}

---

Mode B: Bank Fails After DLT Commit

Scenario: DLT execution completes successfully, but bank rejects or fails to process the ISO message.

Detection:

// Bank rejects ISO message
const bankResult = await bankConnector.sendPayment(isoMessage);
if (bankResult.status === 'REJECTED' || bankResult.status === 'FAILED') {
  // Trigger rollback
  await rollbackDLTCommit(planId);
}

Recovery Mechanism:

1. Reverse DLT Transaction: Execute reverse DLT operations if possible
2. Contingency Hold: If DLT commit is irreversible, hold funds in pending state
3. Escalation: Notify Notary for remedial measures
4. Audit Trail: Log bank failure and recovery attempts

Implementation:

async function handleBankFailureAfterDLTCommit(planId: string) {
  // 1. Attempt DLT rollback (if reversible)
  const dltRollback = await dltHandler.attemptRollback(planId);
  
  if (dltRollback.reversible) {
    // Successfully rolled back
    await dltHandler.executeRollback(planId);
    await notaryRegistry.finalizePlan(planId, false);
  } else {
    // DLT commit is irreversible, use contingency
    await contingencyManager.holdFunds(planId, {
      reason: 'Bank failure after DLT commit',
      holdDuration: 24 * 60 * 60 * 1000 // 24 hours
    });

    // Escalate to Notary
    await notaryRegistry.escalate(planId, {
      issue: 'Bank failure after DLT commit',
      dltIrreversible: true,
      requiresManualIntervention: true
    });
  }

  // Log audit trail
  await auditLogger.log({
    planId,
    event: 'ROLLBACK_BANK_FAILURE',
    timestamp: new Date(),
    dltReversible: dltRollback.reversible,
    actions: dltRollback.reversible 
      ? ['dlt_rolled_back', 'notary_updated']
      : ['funds_held', 'notary_escalated']
  });

  // Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_FAILED',
    message: 'Bank processing failed. Funds are being held pending resolution.',
    requiresAction: !dltRollback.reversible
  });
}

---

Mode C: Liquidity Hub Denies Flash Credit Mid-Plan

Scenario: Liquidity Hub rejects provisional intra-day credit request during plan execution.

Detection:

// Liquidity Hub denies credit
const creditRequest = await liquidityHub.requestCredit(plan);
if (!creditRequest.approved) {
  // Trigger abort
  await abortPlan(planId);
}

Recovery Mechanism:

1. Abort Plan: Immediately abort all steps
2. Release Collateral: Unlock any reserved collateral
3. Cleanup Reservations: Clear all prepared states
4. Audit Trail: Log denial and abort

Implementation:

async function handleLiquidityDenial(planId: string) {
  // 1. Abort plan execution
  await planHandler.abort(planId);

  // 2. Release collateral
  await dltHandler.releaseCollateral(planId);

  // 3. Cleanup bank reservations
  await bankConnector.cancelProvisional(planId);

  // 4. Update notary
  await notaryRegistry.finalizePlan(planId, false);

  // 5. Log audit trail
  await auditLogger.log({
    planId,
    event: 'ABORT_LIQUIDITY_DENIAL',
    timestamp: new Date(),
    reason: 'Liquidity Hub credit denied',
    actions: ['plan_aborted', 'collateral_released', 'reservations_cleared']
  });

  // 6. Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_ABORTED',
    message: 'Liquidity credit was denied. Plan execution aborted.',
    reason: 'Insufficient liquidity facility'
  });
}

---

2. Partial Execution Prevention

Atomicity Guarantees

All-or-Nothing Execution:

class AtomicExecutor {
  async executePlan(plan: Plan): Promise<ExecutionResult> {
    // Phase 1: Prepare all steps
    const prepareResults = await this.prepareAllSteps(plan);
    if (!prepareResults.allPrepared) {
      await this.abortAll(plan.planId);
      return { success: false, reason: 'Prepare phase failed' };
    }

    // Phase 2: Execute all steps
    try {
      const executeResults = await this.executeAllSteps(plan);
      if (!executeResults.allSucceeded) {
        await this.rollbackAll(plan.planId);
        return { success: false, reason: 'Execute phase failed' };
      }
    } catch (error) {
      await this.rollbackAll(plan.planId);
      throw error;
    }

    // Phase 3: Commit all steps
    const commitResults = await this.commitAllSteps(plan);
    if (!commitResults.allCommitted) {
      await this.rollbackAll(plan.planId);
      return { success: false, reason: 'Commit phase failed' };
    }

    return { success: true };
  }

  async prepareAllSteps(plan: Plan): Promise<PrepareResult> {
    const results = await Promise.all(
      plan.steps.map(step => this.prepareStep(step))
    );
    return {
      allPrepared: results.every(r => r.prepared),
      results
    };
  }

  async rollbackAll(planId: string): Promise<void> {
    // Rollback in reverse order
    const plan = await this.getPlan(planId);
    for (let i = plan.steps.length - 1; i >= 0; i--) {
      await this.rollbackStep(plan.steps[i], i);
    }
  }
}

Two-Phase Commit (2PC) Pattern

class TwoPhaseCommit {
  async execute(plan: Plan): Promise<boolean> {
    // Phase 1: Prepare
    const prepared = await this.prepare(plan);
    if (!prepared) {
      await this.abort(plan.planId);
      return false;
    }

    // Phase 2: Commit
    const committed = await this.commit(plan);
    if (!committed) {
      await this.abort(plan.planId);
      return false;
    }

    return true;
  }

  async prepare(plan: Plan): Promise<boolean> {
    // Prepare DLT steps
    const dltPrepared = await this.dltHandler.prepare(plan.dltSteps);
    
    // Prepare bank steps (provisional ISO messages)
    const bankPrepared = await this.bankConnector.prepare(plan.bankSteps);

    return dltPrepared && bankPrepared;
  }

  async commit(plan: Plan): Promise<boolean> {
    // Commit DLT steps
    const dltCommitted = await this.dltHandler.commit(plan.planId);
    
    // Commit bank steps (finalize ISO messages)
    const bankCommitted = await this.bankConnector.commit(plan.planId);

    return dltCommitted && bankCommitted;
  }

  async abort(planId: string): Promise<void> {
    // Abort DLT reservations
    await this.dltHandler.abort(planId);
    
    // Abort bank provisional messages
    await this.bankConnector.abort(planId);
  }
}

---

3. Recovery Mechanisms

Automatic Recovery

class RecoveryManager {
  async recover(planId: string): Promise<RecoveryResult> {
    const plan = await this.getPlan(planId);
    const executionState = await this.getExecutionState(planId);

    switch (executionState.phase) {
      case 'PREPARE':
        return await this.recoverFromPrepare(planId);
      case 'EXECUTE_DLT':
        return await this.recoverFromDLT(planId);
      case 'BANK_INSTRUCTION':
        return await this.recoverFromBank(planId);
      case 'COMMIT':
        return await this.recoverFromCommit(planId);
      default:
        return { recovered: false, reason: 'Unknown phase' };
    }
  }

  async recoverFromPrepare(planId: string): Promise<RecoveryResult> {
    // Simple: Just abort
    await this.abortPlan(planId);
    return { recovered: true, action: 'aborted' };
  }

  async recoverFromDLT(planId: string): Promise<RecoveryResult> {
    // Check if DLT execution can be rolled back
    const dltState = await this.dltHandler.getState(planId);
    
    if (dltState.reversible) {
      await this.dltHandler.rollback(planId);
      await this.bankConnector.abort(planId);
      return { recovered: true, action: 'rolled_back' };
    } else {
      // DLT is committed, need manual intervention
      await this.escalate(planId, 'DLT_COMMITTED_BUT_BANK_FAILED');
      return { recovered: false, requiresManualIntervention: true };
    }
  }
}

Manual Recovery Escalation

class EscalationManager {
  async escalate(planId: string, issue: string): Promise<void> {
    // Create escalation ticket
    const ticket = await this.createTicket({
      planId,
      issue,
      severity: 'HIGH',
      requiresManualIntervention: true,
      assignedTo: 'operations-team'
    });

    // Notify operations team
    await this.notificationService.notify({
      to: 'operations@example.com',
      subject: `Escalation Required: Plan ${planId}`,
      body: `Plan ${planId} requires manual intervention: ${issue}`
    });

    // Update notary
    await this.notaryRegistry.escalate(planId, {
      ticketId: ticket.id,
      issue,
      timestamp: new Date()
    });
  }
}

---

4. Audit Trail for Aborted Plans

Abort Audit Log Structure

interface AbortAuditLog {
  planId: string;
  timestamp: string;
  abortReason: string;
  phase: 'PREPARE' | 'EXECUTE_DLT' | 'BANK_INSTRUCTION' | 'COMMIT';
  stepsCompleted: number;
  stepsTotal: number;
  rollbackActions: RollbackAction[];
  recoveryAttempted: boolean;
  recoveryResult?: RecoveryResult;
  notaryProof: string;
}

interface RollbackAction {
  stepIndex: number;
  actionType: 'DLT_ROLLBACK' | 'BANK_ABORT' | 'COLLATERAL_RELEASE';
  timestamp: string;
  success: boolean;
  error?: string;
}

Audit Log Generation

class AuditLogger {
  async logAbort(planId: string, reason: string): Promise<AbortAuditLog> {
    const plan = await this.getPlan(planId);
    const executionState = await this.getExecutionState(planId);
    const rollbackActions = await this.getRollbackActions(planId);

    const auditLog: AbortAuditLog = {
      planId,
      timestamp: new Date().toISOString(),
      abortReason: reason,
      phase: executionState.currentPhase,
      stepsCompleted: executionState.completedSteps,
      stepsTotal: plan.steps.length,
      rollbackActions,
      recoveryAttempted: executionState.recoveryAttempted,
      recoveryResult: executionState.recoveryResult,
      notaryProof: await this.notaryRegistry.getProof(planId)
    };

    // Store in database
    await this.db.auditLogs.insert(auditLog);

    // Store in immutable storage (IPFS/Arweave)
    const ipfsHash = await this.ipfs.add(JSON.stringify(auditLog));
    await this.notaryRegistry.recordAuditHash(planId, ipfsHash);

    return auditLog;
  }
}

---

5. Error Response Format

Standardized Error Response

interface ErrorResponse {
  error: {
    code: string;
    message: string;
    planId?: string;
    phase?: string;
    stepIndex?: number;
    details?: {
      dltError?: string;
      bankError?: string;
      liquidityError?: string;
    };
    recovery?: {
      attempted: boolean;
      success: boolean;
      action?: string;
    };
    auditLogId?: string;
  };
}

Error Codes

Code	Description	Recovery Action
DLT_PREPARE_FAILED	DLT prepare phase failed	Abort all, release reservations
DLT_EXECUTE_FAILED	DLT execution failed	Rollback DLT, abort bank
DLT_COMMIT_FAILED	DLT commit failed	Rollback if possible, else escalate
BANK_PREPARE_FAILED	Bank prepare phase failed	Abort DLT, release collateral
BANK_EXECUTE_FAILED	Bank execution failed	Reverse DLT if possible, else hold funds
BANK_COMMIT_FAILED	Bank commit failed	Escalate, manual intervention
LIQUIDITY_DENIED	Liquidity credit denied	Abort plan, release all
STEP_DEPENDENCY_FAILED	Step dependency check failed	Abort before execution
COMPLIANCE_FAILED	Compliance check failed	Abort, log compliance issue

---

6. User Notification

Notification Types

interface Notification {
  planId: string;
  type: 'EXECUTION_FAILED' | 'EXECUTION_ABORTED' | 'RECOVERY_IN_PROGRESS' | 'MANUAL_INTERVENTION_REQUIRED';
  message: string;
  timestamp: string;
  actions?: {
    label: string;
    url: string;
  }[];
  requiresAction: boolean;
}

// Example notifications
const notifications = {
  DLT_FAILURE: {
    type: 'EXECUTION_FAILED',
    message: 'DLT execution failed. Bank instruction has been cancelled. No funds were transferred.',
    requiresAction: false
  },
  BANK_FAILURE: {
    type: 'EXECUTION_FAILED',
    message: 'Bank processing failed after DLT execution. Funds are being held pending resolution.',
    requiresAction: true,
    actions: [
      { label: 'View Details', url: `/plans/${planId}` },
      { label: 'Contact Support', url: '/support' }
    ]
  },
  LIQUIDITY_DENIAL: {
    type: 'EXECUTION_ABORTED',
    message: 'Liquidity credit was denied. Plan execution aborted. No funds were transferred.',
    requiresAction: false
  }
};

---

7. Testing Requirements

Unit Tests

describe('Error Handling', () => {
  it('should abort plan on DLT failure', async () => {
    const plan = createTestPlan();
    mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
    
    await executor.executePlan(plan);
    
    expect(mockBank.abort).toHaveBeenCalled();
    expect(mockDLT.releaseReservations).toHaveBeenCalled();
  });

  it('should rollback DLT on bank failure', async () => {
    const plan = createTestPlan();
    mockDLT.execute.mockResolvedValue({ success: true });
    mockBank.sendPayment.mockRejectedValue(new Error('Bank failed'));
    
    await executor.executePlan(plan);
    
    expect(mockDLT.rollback).toHaveBeenCalled();
  });
});

Integration Tests

describe('End-to-End Error Handling', () => {
  it('should handle DLT failure after bank prepare', async () => {
    // Setup: Bank prepare succeeds
    mockBank.prepare.mockResolvedValue({ prepared: true });
    
    // Trigger: DLT execution fails
    mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
    
    // Execute
    const result = await executor.executePlan(plan);
    
    // Verify: Bank aborted, DLT released, audit logged
    expect(result.success).toBe(false);
    expect(mockBank.abort).toHaveBeenCalled();
    expect(mockAuditLogger.log).toHaveBeenCalled();
  });
});

---

Document Version: 1.0
Last Updated: 2025-01-15
Author: Engineering Team