CurrenciCombo/docs/Error_Handling_Rollback_Spec.md

# Error Handling & Rollback Specification

## Overview
This document specifies comprehensive error handling and rollback procedures for the ISO-20022 Combo Flow system, including failure modes (DLT fail after bank prepare, bank fail after DLT commit, liquidity denial), recovery mechanisms, partial execution prevention, and audit trail requirements for aborted plans.

---

## 1. Failure Modes

### Mode A: DLT Execution Fails After Bank Prepare

**Scenario**: Bank has accepted a provisional ISO message (pacs.008 in prepared state), but DLT execution fails during commit phase.

**Detection**:
```typescript
// DLT execution fails
const dltResult = await dltHandler.executeSteps(plan.steps);
if (!dltResult.success) {
  // Trigger rollback
  await rollbackBankPrepare(planId);
}
```

**Recovery Mechanism**:
1. **Abort Bank Instruction**: Send abort/cancel message to bank (camt.056)
2. **Release DLT Reservations**: Release any reserved collateral/amounts
3. **Update Notary**: Record abort in notary registry
4. **Audit Trail**: Log failure and recovery actions

**Implementation**:
```typescript
async function handleDLTFailureAfterBankPrepare(planId: string) {
  // 1. Abort bank instruction
  await bankConnector.abortPayment(planId, {
    reason: 'DLT execution failed',
    messageType: 'camt.056'
  });

  // 2. Release DLT reservations
  await dltHandler.releaseReservations(planId);

  // 3. Update notary
  await notaryRegistry.finalizePlan(planId, false);

  // 4. Log audit trail
  await auditLogger.log({
    planId,
    event: 'ROLLBACK_DLT_FAILURE',
    timestamp: new Date(),
    actions: ['bank_aborted', 'dlt_released', 'notary_updated']
  });

  // 5. Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_FAILED',
    message: 'DLT execution failed. Bank instruction has been cancelled.'
  });
}
```

---

### Mode B: Bank Fails After DLT Commit

**Scenario**: DLT execution completes successfully, but bank rejects or fails to process the ISO message.

**Detection**:
```typescript
// Bank rejects ISO message
const bankResult = await bankConnector.sendPayment(isoMessage);
if (bankResult.status === 'REJECTED' || bankResult.status === 'FAILED') {
  // Trigger rollback
  await rollbackDLTCommit(planId);
}
```

**Recovery Mechanism**:
1. **Reverse DLT Transaction**: Execute reverse DLT operations if possible
2. **Contingency Hold**: If DLT commit is irreversible, hold funds in pending state
3. **Escalation**: Notify Notary for remedial measures
4. **Audit Trail**: Log bank failure and recovery attempts

**Implementation**:
```typescript
async function handleBankFailureAfterDLTCommit(planId: string) {
  // 1. Attempt DLT rollback (if reversible)
  const dltRollback = await dltHandler.attemptRollback(planId);
  
  if (dltRollback.reversible) {
    // Successfully rolled back
    await dltHandler.executeRollback(planId);
    await notaryRegistry.finalizePlan(planId, false);
  } else {
    // DLT commit is irreversible, use contingency
    await contingencyManager.holdFunds(planId, {
      reason: 'Bank failure after DLT commit',
      holdDuration: 24 * 60 * 60 * 1000 // 24 hours
    });

    // Escalate to Notary
    await notaryRegistry.escalate(planId, {
      issue: 'Bank failure after DLT commit',
      dltIrreversible: true,
      requiresManualIntervention: true
    });
  }

  // Log audit trail
  await auditLogger.log({
    planId,
    event: 'ROLLBACK_BANK_FAILURE',
    timestamp: new Date(),
    dltReversible: dltRollback.reversible,
    actions: dltRollback.reversible 
      ? ['dlt_rolled_back', 'notary_updated']
      : ['funds_held', 'notary_escalated']
  });

  // Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_FAILED',
    message: 'Bank processing failed. Funds are being held pending resolution.',
    requiresAction: !dltRollback.reversible
  });
}
```

---

### Mode C: Liquidity Hub Denies Flash Credit Mid-Plan

**Scenario**: Liquidity Hub rejects provisional intra-day credit request during plan execution.

**Detection**:
```typescript
// Liquidity Hub denies credit
const creditRequest = await liquidityHub.requestCredit(plan);
if (!creditRequest.approved) {
  // Trigger abort
  await abortPlan(planId);
}
```

**Recovery Mechanism**:
1. **Abort Plan**: Immediately abort all steps
2. **Release Collateral**: Unlock any reserved collateral
3. **Cleanup Reservations**: Clear all prepared states
4. **Audit Trail**: Log denial and abort

**Implementation**:
```typescript
async function handleLiquidityDenial(planId: string) {
  // 1. Abort plan execution
  await planHandler.abort(planId);

  // 2. Release collateral
  await dltHandler.releaseCollateral(planId);

  // 3. Cleanup bank reservations
  await bankConnector.cancelProvisional(planId);

  // 4. Update notary
  await notaryRegistry.finalizePlan(planId, false);

  // 5. Log audit trail
  await auditLogger.log({
    planId,
    event: 'ABORT_LIQUIDITY_DENIAL',
    timestamp: new Date(),
    reason: 'Liquidity Hub credit denied',
    actions: ['plan_aborted', 'collateral_released', 'reservations_cleared']
  });

  // 6. Notify user
  await notificationService.notifyUser(planId, {
    type: 'EXECUTION_ABORTED',
    message: 'Liquidity credit was denied. Plan execution aborted.',
    reason: 'Insufficient liquidity facility'
  });
}
```

---

## 2. Partial Execution Prevention

### Atomicity Guarantees

**All-or-Nothing Execution**:
```typescript
class AtomicExecutor {
  async executePlan(plan: Plan): Promise<ExecutionResult> {
    // Phase 1: Prepare all steps
    const prepareResults = await this.prepareAllSteps(plan);
    if (!prepareResults.allPrepared) {
      await this.abortAll(plan.planId);
      return { success: false, reason: 'Prepare phase failed' };
    }

    // Phase 2: Execute all steps
    try {
      const executeResults = await this.executeAllSteps(plan);
      if (!executeResults.allSucceeded) {
        await this.rollbackAll(plan.planId);
        return { success: false, reason: 'Execute phase failed' };
      }
    } catch (error) {
      await this.rollbackAll(plan.planId);
      throw error;
    }

    // Phase 3: Commit all steps
    const commitResults = await this.commitAllSteps(plan);
    if (!commitResults.allCommitted) {
      await this.rollbackAll(plan.planId);
      return { success: false, reason: 'Commit phase failed' };
    }

    return { success: true };
  }

  async prepareAllSteps(plan: Plan): Promise<PrepareResult> {
    const results = await Promise.all(
      plan.steps.map(step => this.prepareStep(step))
    );
    return {
      allPrepared: results.every(r => r.prepared),
      results
    };
  }

  async rollbackAll(planId: string): Promise<void> {
    // Rollback in reverse order
    const plan = await this.getPlan(planId);
    for (let i = plan.steps.length - 1; i >= 0; i--) {
      await this.rollbackStep(plan.steps[i], i);
    }
  }
}
```

### Two-Phase Commit (2PC) Pattern

```typescript
class TwoPhaseCommit {
  async execute(plan: Plan): Promise<boolean> {
    // Phase 1: Prepare
    const prepared = await this.prepare(plan);
    if (!prepared) {
      await this.abort(plan.planId);
      return false;
    }

    // Phase 2: Commit
    const committed = await this.commit(plan);
    if (!committed) {
      await this.abort(plan.planId);
      return false;
    }

    return true;
  }

  async prepare(plan: Plan): Promise<boolean> {
    // Prepare DLT steps
    const dltPrepared = await this.dltHandler.prepare(plan.dltSteps);
    
    // Prepare bank steps (provisional ISO messages)
    const bankPrepared = await this.bankConnector.prepare(plan.bankSteps);

    return dltPrepared && bankPrepared;
  }

  async commit(plan: Plan): Promise<boolean> {
    // Commit DLT steps
    const dltCommitted = await this.dltHandler.commit(plan.planId);
    
    // Commit bank steps (finalize ISO messages)
    const bankCommitted = await this.bankConnector.commit(plan.planId);

    return dltCommitted && bankCommitted;
  }

  async abort(planId: string): Promise<void> {
    // Abort DLT reservations
    await this.dltHandler.abort(planId);
    
    // Abort bank provisional messages
    await this.bankConnector.abort(planId);
  }
}
```

---

## 3. Recovery Mechanisms

### Automatic Recovery

```typescript
class RecoveryManager {
  async recover(planId: string): Promise<RecoveryResult> {
    const plan = await this.getPlan(planId);
    const executionState = await this.getExecutionState(planId);

    switch (executionState.phase) {
      case 'PREPARE':
        return await this.recoverFromPrepare(planId);
      case 'EXECUTE_DLT':
        return await this.recoverFromDLT(planId);
      case 'BANK_INSTRUCTION':
        return await this.recoverFromBank(planId);
      case 'COMMIT':
        return await this.recoverFromCommit(planId);
      default:
        return { recovered: false, reason: 'Unknown phase' };
    }
  }

  async recoverFromPrepare(planId: string): Promise<RecoveryResult> {
    // Simple: Just abort
    await this.abortPlan(planId);
    return { recovered: true, action: 'aborted' };
  }

  async recoverFromDLT(planId: string): Promise<RecoveryResult> {
    // Check if DLT execution can be rolled back
    const dltState = await this.dltHandler.getState(planId);
    
    if (dltState.reversible) {
      await this.dltHandler.rollback(planId);
      await this.bankConnector.abort(planId);
      return { recovered: true, action: 'rolled_back' };
    } else {
      // DLT is committed, need manual intervention
      await this.escalate(planId, 'DLT_COMMITTED_BUT_BANK_FAILED');
      return { recovered: false, requiresManualIntervention: true };
    }
  }
}
```

### Manual Recovery Escalation

```typescript
class EscalationManager {
  async escalate(planId: string, issue: string): Promise<void> {
    // Create escalation ticket
    const ticket = await this.createTicket({
      planId,
      issue,
      severity: 'HIGH',
      requiresManualIntervention: true,
      assignedTo: 'operations-team'
    });

    // Notify operations team
    await this.notificationService.notify({
      to: 'operations@example.com',
      subject: `Escalation Required: Plan ${planId}`,
      body: `Plan ${planId} requires manual intervention: ${issue}`
    });

    // Update notary
    await this.notaryRegistry.escalate(planId, {
      ticketId: ticket.id,
      issue,
      timestamp: new Date()
    });
  }
}
```

---

## 4. Audit Trail for Aborted Plans

### Abort Audit Log Structure

```typescript
interface AbortAuditLog {
  planId: string;
  timestamp: string;
  abortReason: string;
  phase: 'PREPARE' | 'EXECUTE_DLT' | 'BANK_INSTRUCTION' | 'COMMIT';
  stepsCompleted: number;
  stepsTotal: number;
  rollbackActions: RollbackAction[];
  recoveryAttempted: boolean;
  recoveryResult?: RecoveryResult;
  notaryProof: string;
}

interface RollbackAction {
  stepIndex: number;
  actionType: 'DLT_ROLLBACK' | 'BANK_ABORT' | 'COLLATERAL_RELEASE';
  timestamp: string;
  success: boolean;
  error?: string;
}
```

### Audit Log Generation

```typescript
class AuditLogger {
  async logAbort(planId: string, reason: string): Promise<AbortAuditLog> {
    const plan = await this.getPlan(planId);
    const executionState = await this.getExecutionState(planId);
    const rollbackActions = await this.getRollbackActions(planId);

    const auditLog: AbortAuditLog = {
      planId,
      timestamp: new Date().toISOString(),
      abortReason: reason,
      phase: executionState.currentPhase,
      stepsCompleted: executionState.completedSteps,
      stepsTotal: plan.steps.length,
      rollbackActions,
      recoveryAttempted: executionState.recoveryAttempted,
      recoveryResult: executionState.recoveryResult,
      notaryProof: await this.notaryRegistry.getProof(planId)
    };

    // Store in database
    await this.db.auditLogs.insert(auditLog);

    // Store in immutable storage (IPFS/Arweave)
    const ipfsHash = await this.ipfs.add(JSON.stringify(auditLog));
    await this.notaryRegistry.recordAuditHash(planId, ipfsHash);

    return auditLog;
  }
}
```

---

## 5. Error Response Format

### Standardized Error Response

```typescript
interface ErrorResponse {
  error: {
    code: string;
    message: string;
    planId?: string;
    phase?: string;
    stepIndex?: number;
    details?: {
      dltError?: string;
      bankError?: string;
      liquidityError?: string;
    };
    recovery?: {
      attempted: boolean;
      success: boolean;
      action?: string;
    };
    auditLogId?: string;
  };
}
```

### Error Codes

| Code | Description | Recovery Action |
|------|-------------|-----------------|
| `DLT_PREPARE_FAILED` | DLT prepare phase failed | Abort all, release reservations |
| `DLT_EXECUTE_FAILED` | DLT execution failed | Rollback DLT, abort bank |
| `DLT_COMMIT_FAILED` | DLT commit failed | Rollback if possible, else escalate |
| `BANK_PREPARE_FAILED` | Bank prepare phase failed | Abort DLT, release collateral |
| `BANK_EXECUTE_FAILED` | Bank execution failed | Reverse DLT if possible, else hold funds |
| `BANK_COMMIT_FAILED` | Bank commit failed | Escalate, manual intervention |
| `LIQUIDITY_DENIED` | Liquidity credit denied | Abort plan, release all |
| `STEP_DEPENDENCY_FAILED` | Step dependency check failed | Abort before execution |
| `COMPLIANCE_FAILED` | Compliance check failed | Abort, log compliance issue |

---

## 6. User Notification

### Notification Types

```typescript
interface Notification {
  planId: string;
  type: 'EXECUTION_FAILED' | 'EXECUTION_ABORTED' | 'RECOVERY_IN_PROGRESS' | 'MANUAL_INTERVENTION_REQUIRED';
  message: string;
  timestamp: string;
  actions?: {
    label: string;
    url: string;
  }[];
  requiresAction: boolean;
}

// Example notifications
const notifications = {
  DLT_FAILURE: {
    type: 'EXECUTION_FAILED',
    message: 'DLT execution failed. Bank instruction has been cancelled. No funds were transferred.',
    requiresAction: false
  },
  BANK_FAILURE: {
    type: 'EXECUTION_FAILED',
    message: 'Bank processing failed after DLT execution. Funds are being held pending resolution.',
    requiresAction: true,
    actions: [
      { label: 'View Details', url: `/plans/${planId}` },
      { label: 'Contact Support', url: '/support' }
    ]
  },
  LIQUIDITY_DENIAL: {
    type: 'EXECUTION_ABORTED',
    message: 'Liquidity credit was denied. Plan execution aborted. No funds were transferred.',
    requiresAction: false
  }
};
```

---

## 7. Testing Requirements

### Unit Tests

```typescript
describe('Error Handling', () => {
  it('should abort plan on DLT failure', async () => {
    const plan = createTestPlan();
    mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
    
    await executor.executePlan(plan);
    
    expect(mockBank.abort).toHaveBeenCalled();
    expect(mockDLT.releaseReservations).toHaveBeenCalled();
  });

  it('should rollback DLT on bank failure', async () => {
    const plan = createTestPlan();
    mockDLT.execute.mockResolvedValue({ success: true });
    mockBank.sendPayment.mockRejectedValue(new Error('Bank failed'));
    
    await executor.executePlan(plan);
    
    expect(mockDLT.rollback).toHaveBeenCalled();
  });
});
```

### Integration Tests

```typescript
describe('End-to-End Error Handling', () => {
  it('should handle DLT failure after bank prepare', async () => {
    // Setup: Bank prepare succeeds
    mockBank.prepare.mockResolvedValue({ prepared: true });
    
    // Trigger: DLT execution fails
    mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
    
    // Execute
    const result = await executor.executePlan(plan);
    
    // Verify: Bank aborted, DLT released, audit logged
    expect(result.success).toBe(false);
    expect(mockBank.abort).toHaveBeenCalled();
    expect(mockAuditLogger.log).toHaveBeenCalled();
  });
});
```

---

**Document Version**: 1.0  
**Last Updated**: 2025-01-15  
**Author**: Engineering Team