143 lines
2.7 KiB
Markdown
143 lines
2.7 KiB
Markdown
|
# AS4 Settlement Operational Runbooks
|
||
|
|
|
||
|
|
**Date**: 2026-01-19
|
||
|
|
**Version**: 1.0.0
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 1. Daily Operations
|
||
|
|
|
||
|
|
### 1.1 Health Checks
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Check AS4 Gateway health: `GET /api/v1/as4/gateway/health`
|
||
|
|
2. Check Member Directory: `GET /api/v1/as4/directory/members?status=active`
|
||
|
|
3. Check certificate expiration: `GET /api/v1/as4/directory/certificates/expiration-warnings`
|
||
|
|
4. Review error logs for anomalies
|
||
|
|
|
||
|
|
**Frequency**: Every 4 hours
|
||
|
|
|
||
|
|
### 1.2 Certificate Expiration Monitoring
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Query expiration warnings (30-day threshold)
|
||
|
|
2. Notify members of expiring certificates
|
||
|
|
3. Schedule certificate rotation
|
||
|
|
|
||
|
|
**Frequency**: Daily
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. Incident Response
|
||
|
|
|
||
|
|
### 2.1 Service Outage
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Identify affected services
|
||
|
|
2. Check system logs
|
||
|
|
3. Notify affected members
|
||
|
|
4. Escalate to engineering team
|
||
|
|
5. Document incident
|
||
|
|
|
||
|
|
**SLA**: 15-minute response time
|
||
|
|
|
||
|
|
### 2.2 Message Processing Failure
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Identify failed instruction
|
||
|
|
2. Check error logs
|
||
|
|
3. Verify member status
|
||
|
|
4. Retry if appropriate
|
||
|
|
5. Notify member if manual intervention required
|
||
|
|
|
||
|
|
**SLA**: 1-hour resolution
|
||
|
|
|
||
|
|
### 2.3 Certificate Compromise
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Immediately revoke compromised certificate
|
||
|
|
2. Notify affected member
|
||
|
|
3. Issue new certificate
|
||
|
|
4. Update Member Directory
|
||
|
|
5. Audit all transactions using compromised certificate
|
||
|
|
|
||
|
|
**SLA**: Immediate action
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. Maintenance Windows
|
||
|
|
|
||
|
|
### 3.1 Scheduled Maintenance
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Notify members 7 days in advance
|
||
|
|
2. Schedule during low-traffic period
|
||
|
|
3. Perform maintenance
|
||
|
|
4. Verify service health
|
||
|
|
5. Notify members of completion
|
||
|
|
|
||
|
|
**Frequency**: Monthly
|
||
|
|
|
||
|
|
### 3.2 Emergency Maintenance
|
||
|
|
|
||
|
|
**Procedure**:
|
||
|
|
1. Notify members immediately
|
||
|
|
2. Perform maintenance
|
||
|
|
3. Verify service health
|
||
|
|
4. Post-incident report
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 4. Monitoring and Alerts
|
||
|
|
|
||
|
|
### 4.1 Key Metrics
|
||
|
|
|
||
|
|
- Message processing latency (P99 < 5 seconds)
|
||
|
|
- System availability (99.9% target)
|
||
|
|
- Certificate expiration warnings
|
||
|
|
- Failed instruction rate
|
||
|
|
- Posting success rate
|
||
|
|
|
||
|
|
### 4.2 Alert Thresholds
|
||
|
|
|
||
|
|
- Availability < 99.9%: CRITICAL
|
||
|
|
- P99 latency > 5 seconds: WARNING
|
||
|
|
- Failed instruction rate > 1%: WARNING
|
||
|
|
- Certificate expiring < 7 days: WARNING
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 5. Backup and Recovery
|
||
|
|
|
||
|
|
### 5.1 Database Backups
|
||
|
|
|
||
|
|
**Frequency**: Daily full backup, hourly incremental
|
||
|
|
|
||
|
|
**Retention**: 30 days
|
||
|
|
|
||
|
|
### 5.2 Payload Vault Backups
|
||
|
|
|
||
|
|
**Frequency**: Real-time replication
|
||
|
|
|
||
|
|
**Retention**: 7 years (regulatory requirement)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 6. Security Procedures
|
||
|
|
|
||
|
|
### 6.1 Access Control
|
||
|
|
|
||
|
|
- Multi-factor authentication required
|
||
|
|
- Role-based access control
|
||
|
|
- Audit logging for all access
|
||
|
|
|
||
|
|
### 6.2 Key Rotation
|
||
|
|
|
||
|
|
- Certificate rotation: 30 days before expiration
|
||
|
|
- HSM key rotation: Per security policy
|
||
|
|
- Member notification: 7 days in advance
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
**End of Runbooks**
|