581 lines
16 KiB
Markdown
581 lines
16 KiB
Markdown
# Error Handling & Rollback Specification
|
|
|
|
## Overview
|
|
This document specifies comprehensive error handling and rollback procedures for the ISO-20022 Combo Flow system, including failure modes (DLT fail after bank prepare, bank fail after DLT commit, liquidity denial), recovery mechanisms, partial execution prevention, and audit trail requirements for aborted plans.
|
|
|
|
---
|
|
|
|
## 1. Failure Modes
|
|
|
|
### Mode A: DLT Execution Fails After Bank Prepare
|
|
|
|
**Scenario**: Bank has accepted a provisional ISO message (pacs.008 in prepared state), but DLT execution fails during commit phase.
|
|
|
|
**Detection**:
|
|
```typescript
|
|
// DLT execution fails
|
|
const dltResult = await dltHandler.executeSteps(plan.steps);
|
|
if (!dltResult.success) {
|
|
// Trigger rollback
|
|
await rollbackBankPrepare(planId);
|
|
}
|
|
```
|
|
|
|
**Recovery Mechanism**:
|
|
1. **Abort Bank Instruction**: Send abort/cancel message to bank (camt.056)
|
|
2. **Release DLT Reservations**: Release any reserved collateral/amounts
|
|
3. **Update Notary**: Record abort in notary registry
|
|
4. **Audit Trail**: Log failure and recovery actions
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
async function handleDLTFailureAfterBankPrepare(planId: string) {
|
|
// 1. Abort bank instruction
|
|
await bankConnector.abortPayment(planId, {
|
|
reason: 'DLT execution failed',
|
|
messageType: 'camt.056'
|
|
});
|
|
|
|
// 2. Release DLT reservations
|
|
await dltHandler.releaseReservations(planId);
|
|
|
|
// 3. Update notary
|
|
await notaryRegistry.finalizePlan(planId, false);
|
|
|
|
// 4. Log audit trail
|
|
await auditLogger.log({
|
|
planId,
|
|
event: 'ROLLBACK_DLT_FAILURE',
|
|
timestamp: new Date(),
|
|
actions: ['bank_aborted', 'dlt_released', 'notary_updated']
|
|
});
|
|
|
|
// 5. Notify user
|
|
await notificationService.notifyUser(planId, {
|
|
type: 'EXECUTION_FAILED',
|
|
message: 'DLT execution failed. Bank instruction has been cancelled.'
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Mode B: Bank Fails After DLT Commit
|
|
|
|
**Scenario**: DLT execution completes successfully, but bank rejects or fails to process the ISO message.
|
|
|
|
**Detection**:
|
|
```typescript
|
|
// Bank rejects ISO message
|
|
const bankResult = await bankConnector.sendPayment(isoMessage);
|
|
if (bankResult.status === 'REJECTED' || bankResult.status === 'FAILED') {
|
|
// Trigger rollback
|
|
await rollbackDLTCommit(planId);
|
|
}
|
|
```
|
|
|
|
**Recovery Mechanism**:
|
|
1. **Reverse DLT Transaction**: Execute reverse DLT operations if possible
|
|
2. **Contingency Hold**: If DLT commit is irreversible, hold funds in pending state
|
|
3. **Escalation**: Notify Notary for remedial measures
|
|
4. **Audit Trail**: Log bank failure and recovery attempts
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
async function handleBankFailureAfterDLTCommit(planId: string) {
|
|
// 1. Attempt DLT rollback (if reversible)
|
|
const dltRollback = await dltHandler.attemptRollback(planId);
|
|
|
|
if (dltRollback.reversible) {
|
|
// Successfully rolled back
|
|
await dltHandler.executeRollback(planId);
|
|
await notaryRegistry.finalizePlan(planId, false);
|
|
} else {
|
|
// DLT commit is irreversible, use contingency
|
|
await contingencyManager.holdFunds(planId, {
|
|
reason: 'Bank failure after DLT commit',
|
|
holdDuration: 24 * 60 * 60 * 1000 // 24 hours
|
|
});
|
|
|
|
// Escalate to Notary
|
|
await notaryRegistry.escalate(planId, {
|
|
issue: 'Bank failure after DLT commit',
|
|
dltIrreversible: true,
|
|
requiresManualIntervention: true
|
|
});
|
|
}
|
|
|
|
// Log audit trail
|
|
await auditLogger.log({
|
|
planId,
|
|
event: 'ROLLBACK_BANK_FAILURE',
|
|
timestamp: new Date(),
|
|
dltReversible: dltRollback.reversible,
|
|
actions: dltRollback.reversible
|
|
? ['dlt_rolled_back', 'notary_updated']
|
|
: ['funds_held', 'notary_escalated']
|
|
});
|
|
|
|
// Notify user
|
|
await notificationService.notifyUser(planId, {
|
|
type: 'EXECUTION_FAILED',
|
|
message: 'Bank processing failed. Funds are being held pending resolution.',
|
|
requiresAction: !dltRollback.reversible
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Mode C: Liquidity Hub Denies Flash Credit Mid-Plan
|
|
|
|
**Scenario**: Liquidity Hub rejects provisional intra-day credit request during plan execution.
|
|
|
|
**Detection**:
|
|
```typescript
|
|
// Liquidity Hub denies credit
|
|
const creditRequest = await liquidityHub.requestCredit(plan);
|
|
if (!creditRequest.approved) {
|
|
// Trigger abort
|
|
await abortPlan(planId);
|
|
}
|
|
```
|
|
|
|
**Recovery Mechanism**:
|
|
1. **Abort Plan**: Immediately abort all steps
|
|
2. **Release Collateral**: Unlock any reserved collateral
|
|
3. **Cleanup Reservations**: Clear all prepared states
|
|
4. **Audit Trail**: Log denial and abort
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
async function handleLiquidityDenial(planId: string) {
|
|
// 1. Abort plan execution
|
|
await planHandler.abort(planId);
|
|
|
|
// 2. Release collateral
|
|
await dltHandler.releaseCollateral(planId);
|
|
|
|
// 3. Cleanup bank reservations
|
|
await bankConnector.cancelProvisional(planId);
|
|
|
|
// 4. Update notary
|
|
await notaryRegistry.finalizePlan(planId, false);
|
|
|
|
// 5. Log audit trail
|
|
await auditLogger.log({
|
|
planId,
|
|
event: 'ABORT_LIQUIDITY_DENIAL',
|
|
timestamp: new Date(),
|
|
reason: 'Liquidity Hub credit denied',
|
|
actions: ['plan_aborted', 'collateral_released', 'reservations_cleared']
|
|
});
|
|
|
|
// 6. Notify user
|
|
await notificationService.notifyUser(planId, {
|
|
type: 'EXECUTION_ABORTED',
|
|
message: 'Liquidity credit was denied. Plan execution aborted.',
|
|
reason: 'Insufficient liquidity facility'
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Partial Execution Prevention
|
|
|
|
### Atomicity Guarantees
|
|
|
|
**All-or-Nothing Execution**:
|
|
```typescript
|
|
class AtomicExecutor {
|
|
async executePlan(plan: Plan): Promise<ExecutionResult> {
|
|
// Phase 1: Prepare all steps
|
|
const prepareResults = await this.prepareAllSteps(plan);
|
|
if (!prepareResults.allPrepared) {
|
|
await this.abortAll(plan.planId);
|
|
return { success: false, reason: 'Prepare phase failed' };
|
|
}
|
|
|
|
// Phase 2: Execute all steps
|
|
try {
|
|
const executeResults = await this.executeAllSteps(plan);
|
|
if (!executeResults.allSucceeded) {
|
|
await this.rollbackAll(plan.planId);
|
|
return { success: false, reason: 'Execute phase failed' };
|
|
}
|
|
} catch (error) {
|
|
await this.rollbackAll(plan.planId);
|
|
throw error;
|
|
}
|
|
|
|
// Phase 3: Commit all steps
|
|
const commitResults = await this.commitAllSteps(plan);
|
|
if (!commitResults.allCommitted) {
|
|
await this.rollbackAll(plan.planId);
|
|
return { success: false, reason: 'Commit phase failed' };
|
|
}
|
|
|
|
return { success: true };
|
|
}
|
|
|
|
async prepareAllSteps(plan: Plan): Promise<PrepareResult> {
|
|
const results = await Promise.all(
|
|
plan.steps.map(step => this.prepareStep(step))
|
|
);
|
|
return {
|
|
allPrepared: results.every(r => r.prepared),
|
|
results
|
|
};
|
|
}
|
|
|
|
async rollbackAll(planId: string): Promise<void> {
|
|
// Rollback in reverse order
|
|
const plan = await this.getPlan(planId);
|
|
for (let i = plan.steps.length - 1; i >= 0; i--) {
|
|
await this.rollbackStep(plan.steps[i], i);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Two-Phase Commit (2PC) Pattern
|
|
|
|
```typescript
|
|
class TwoPhaseCommit {
|
|
async execute(plan: Plan): Promise<boolean> {
|
|
// Phase 1: Prepare
|
|
const prepared = await this.prepare(plan);
|
|
if (!prepared) {
|
|
await this.abort(plan.planId);
|
|
return false;
|
|
}
|
|
|
|
// Phase 2: Commit
|
|
const committed = await this.commit(plan);
|
|
if (!committed) {
|
|
await this.abort(plan.planId);
|
|
return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async prepare(plan: Plan): Promise<boolean> {
|
|
// Prepare DLT steps
|
|
const dltPrepared = await this.dltHandler.prepare(plan.dltSteps);
|
|
|
|
// Prepare bank steps (provisional ISO messages)
|
|
const bankPrepared = await this.bankConnector.prepare(plan.bankSteps);
|
|
|
|
return dltPrepared && bankPrepared;
|
|
}
|
|
|
|
async commit(plan: Plan): Promise<boolean> {
|
|
// Commit DLT steps
|
|
const dltCommitted = await this.dltHandler.commit(plan.planId);
|
|
|
|
// Commit bank steps (finalize ISO messages)
|
|
const bankCommitted = await this.bankConnector.commit(plan.planId);
|
|
|
|
return dltCommitted && bankCommitted;
|
|
}
|
|
|
|
async abort(planId: string): Promise<void> {
|
|
// Abort DLT reservations
|
|
await this.dltHandler.abort(planId);
|
|
|
|
// Abort bank provisional messages
|
|
await this.bankConnector.abort(planId);
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Recovery Mechanisms
|
|
|
|
### Automatic Recovery
|
|
|
|
```typescript
|
|
class RecoveryManager {
|
|
async recover(planId: string): Promise<RecoveryResult> {
|
|
const plan = await this.getPlan(planId);
|
|
const executionState = await this.getExecutionState(planId);
|
|
|
|
switch (executionState.phase) {
|
|
case 'PREPARE':
|
|
return await this.recoverFromPrepare(planId);
|
|
case 'EXECUTE_DLT':
|
|
return await this.recoverFromDLT(planId);
|
|
case 'BANK_INSTRUCTION':
|
|
return await this.recoverFromBank(planId);
|
|
case 'COMMIT':
|
|
return await this.recoverFromCommit(planId);
|
|
default:
|
|
return { recovered: false, reason: 'Unknown phase' };
|
|
}
|
|
}
|
|
|
|
async recoverFromPrepare(planId: string): Promise<RecoveryResult> {
|
|
// Simple: Just abort
|
|
await this.abortPlan(planId);
|
|
return { recovered: true, action: 'aborted' };
|
|
}
|
|
|
|
async recoverFromDLT(planId: string): Promise<RecoveryResult> {
|
|
// Check if DLT execution can be rolled back
|
|
const dltState = await this.dltHandler.getState(planId);
|
|
|
|
if (dltState.reversible) {
|
|
await this.dltHandler.rollback(planId);
|
|
await this.bankConnector.abort(planId);
|
|
return { recovered: true, action: 'rolled_back' };
|
|
} else {
|
|
// DLT is committed, need manual intervention
|
|
await this.escalate(planId, 'DLT_COMMITTED_BUT_BANK_FAILED');
|
|
return { recovered: false, requiresManualIntervention: true };
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Manual Recovery Escalation
|
|
|
|
```typescript
|
|
class EscalationManager {
|
|
async escalate(planId: string, issue: string): Promise<void> {
|
|
// Create escalation ticket
|
|
const ticket = await this.createTicket({
|
|
planId,
|
|
issue,
|
|
severity: 'HIGH',
|
|
requiresManualIntervention: true,
|
|
assignedTo: 'operations-team'
|
|
});
|
|
|
|
// Notify operations team
|
|
await this.notificationService.notify({
|
|
to: 'operations@example.com',
|
|
subject: `Escalation Required: Plan ${planId}`,
|
|
body: `Plan ${planId} requires manual intervention: ${issue}`
|
|
});
|
|
|
|
// Update notary
|
|
await this.notaryRegistry.escalate(planId, {
|
|
ticketId: ticket.id,
|
|
issue,
|
|
timestamp: new Date()
|
|
});
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Audit Trail for Aborted Plans
|
|
|
|
### Abort Audit Log Structure
|
|
|
|
```typescript
|
|
interface AbortAuditLog {
|
|
planId: string;
|
|
timestamp: string;
|
|
abortReason: string;
|
|
phase: 'PREPARE' | 'EXECUTE_DLT' | 'BANK_INSTRUCTION' | 'COMMIT';
|
|
stepsCompleted: number;
|
|
stepsTotal: number;
|
|
rollbackActions: RollbackAction[];
|
|
recoveryAttempted: boolean;
|
|
recoveryResult?: RecoveryResult;
|
|
notaryProof: string;
|
|
}
|
|
|
|
interface RollbackAction {
|
|
stepIndex: number;
|
|
actionType: 'DLT_ROLLBACK' | 'BANK_ABORT' | 'COLLATERAL_RELEASE';
|
|
timestamp: string;
|
|
success: boolean;
|
|
error?: string;
|
|
}
|
|
```
|
|
|
|
### Audit Log Generation
|
|
|
|
```typescript
|
|
class AuditLogger {
|
|
async logAbort(planId: string, reason: string): Promise<AbortAuditLog> {
|
|
const plan = await this.getPlan(planId);
|
|
const executionState = await this.getExecutionState(planId);
|
|
const rollbackActions = await this.getRollbackActions(planId);
|
|
|
|
const auditLog: AbortAuditLog = {
|
|
planId,
|
|
timestamp: new Date().toISOString(),
|
|
abortReason: reason,
|
|
phase: executionState.currentPhase,
|
|
stepsCompleted: executionState.completedSteps,
|
|
stepsTotal: plan.steps.length,
|
|
rollbackActions,
|
|
recoveryAttempted: executionState.recoveryAttempted,
|
|
recoveryResult: executionState.recoveryResult,
|
|
notaryProof: await this.notaryRegistry.getProof(planId)
|
|
};
|
|
|
|
// Store in database
|
|
await this.db.auditLogs.insert(auditLog);
|
|
|
|
// Store in immutable storage (IPFS/Arweave)
|
|
const ipfsHash = await this.ipfs.add(JSON.stringify(auditLog));
|
|
await this.notaryRegistry.recordAuditHash(planId, ipfsHash);
|
|
|
|
return auditLog;
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Error Response Format
|
|
|
|
### Standardized Error Response
|
|
|
|
```typescript
|
|
interface ErrorResponse {
|
|
error: {
|
|
code: string;
|
|
message: string;
|
|
planId?: string;
|
|
phase?: string;
|
|
stepIndex?: number;
|
|
details?: {
|
|
dltError?: string;
|
|
bankError?: string;
|
|
liquidityError?: string;
|
|
};
|
|
recovery?: {
|
|
attempted: boolean;
|
|
success: boolean;
|
|
action?: string;
|
|
};
|
|
auditLogId?: string;
|
|
};
|
|
}
|
|
```
|
|
|
|
### Error Codes
|
|
|
|
| Code | Description | Recovery Action |
|
|
|------|-------------|-----------------|
|
|
| `DLT_PREPARE_FAILED` | DLT prepare phase failed | Abort all, release reservations |
|
|
| `DLT_EXECUTE_FAILED` | DLT execution failed | Rollback DLT, abort bank |
|
|
| `DLT_COMMIT_FAILED` | DLT commit failed | Rollback if possible, else escalate |
|
|
| `BANK_PREPARE_FAILED` | Bank prepare phase failed | Abort DLT, release collateral |
|
|
| `BANK_EXECUTE_FAILED` | Bank execution failed | Reverse DLT if possible, else hold funds |
|
|
| `BANK_COMMIT_FAILED` | Bank commit failed | Escalate, manual intervention |
|
|
| `LIQUIDITY_DENIED` | Liquidity credit denied | Abort plan, release all |
|
|
| `STEP_DEPENDENCY_FAILED` | Step dependency check failed | Abort before execution |
|
|
| `COMPLIANCE_FAILED` | Compliance check failed | Abort, log compliance issue |
|
|
|
|
---
|
|
|
|
## 6. User Notification
|
|
|
|
### Notification Types
|
|
|
|
```typescript
|
|
interface Notification {
|
|
planId: string;
|
|
type: 'EXECUTION_FAILED' | 'EXECUTION_ABORTED' | 'RECOVERY_IN_PROGRESS' | 'MANUAL_INTERVENTION_REQUIRED';
|
|
message: string;
|
|
timestamp: string;
|
|
actions?: {
|
|
label: string;
|
|
url: string;
|
|
}[];
|
|
requiresAction: boolean;
|
|
}
|
|
|
|
// Example notifications
|
|
const notifications = {
|
|
DLT_FAILURE: {
|
|
type: 'EXECUTION_FAILED',
|
|
message: 'DLT execution failed. Bank instruction has been cancelled. No funds were transferred.',
|
|
requiresAction: false
|
|
},
|
|
BANK_FAILURE: {
|
|
type: 'EXECUTION_FAILED',
|
|
message: 'Bank processing failed after DLT execution. Funds are being held pending resolution.',
|
|
requiresAction: true,
|
|
actions: [
|
|
{ label: 'View Details', url: `/plans/${planId}` },
|
|
{ label: 'Contact Support', url: '/support' }
|
|
]
|
|
},
|
|
LIQUIDITY_DENIAL: {
|
|
type: 'EXECUTION_ABORTED',
|
|
message: 'Liquidity credit was denied. Plan execution aborted. No funds were transferred.',
|
|
requiresAction: false
|
|
}
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Testing Requirements
|
|
|
|
### Unit Tests
|
|
|
|
```typescript
|
|
describe('Error Handling', () => {
|
|
it('should abort plan on DLT failure', async () => {
|
|
const plan = createTestPlan();
|
|
mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
|
|
|
|
await executor.executePlan(plan);
|
|
|
|
expect(mockBank.abort).toHaveBeenCalled();
|
|
expect(mockDLT.releaseReservations).toHaveBeenCalled();
|
|
});
|
|
|
|
it('should rollback DLT on bank failure', async () => {
|
|
const plan = createTestPlan();
|
|
mockDLT.execute.mockResolvedValue({ success: true });
|
|
mockBank.sendPayment.mockRejectedValue(new Error('Bank failed'));
|
|
|
|
await executor.executePlan(plan);
|
|
|
|
expect(mockDLT.rollback).toHaveBeenCalled();
|
|
});
|
|
});
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
```typescript
|
|
describe('End-to-End Error Handling', () => {
|
|
it('should handle DLT failure after bank prepare', async () => {
|
|
// Setup: Bank prepare succeeds
|
|
mockBank.prepare.mockResolvedValue({ prepared: true });
|
|
|
|
// Trigger: DLT execution fails
|
|
mockDLT.execute.mockRejectedValue(new Error('DLT failed'));
|
|
|
|
// Execute
|
|
const result = await executor.executePlan(plan);
|
|
|
|
// Verify: Bank aborted, DLT released, audit logged
|
|
expect(result.success).toBe(false);
|
|
expect(mockBank.abort).toHaveBeenCalled();
|
|
expect(mockAuditLogger.log).toHaveBeenCalled();
|
|
});
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
**Document Version**: 1.0
|
|
**Last Updated**: 2025-01-15
|
|
**Author**: Engineering Team
|
|
|