Sankofa/docs/runbooks/DATA_RETENTION_POLICY.md

# Data Retention Policy

## Overview

This document defines data retention policies for the Sankofa Phoenix platform to ensure compliance with regulatory requirements and optimize storage costs.

## Retention Periods

### Application Data

#### User Data
- **Active Users**: Retained indefinitely while account is active
- **Inactive Users**: Retained for 7 years after last login
- **Deleted Users**: Soft delete for 90 days, then permanent deletion
- **User Activity Logs**: 2 years

#### Tenant Data
- **Active Tenants**: Retained indefinitely while tenant is active
- **Suspended Tenants**: Retained for 1 year after suspension
- **Deleted Tenants**: Soft delete for 90 days, then permanent deletion

#### Resource Data
- **Active Resources**: Retained indefinitely
- **Deleted Resources**: Retained for 90 days for recovery purposes
- **Resource History**: 1 year

### Audit and Compliance Data

#### Audit Logs
- **Security Events**: 7 years (compliance requirement)
- **Authentication Logs**: 2 years
- **Authorization Logs**: 2 years
- **Data Access Logs**: 2 years
- **Administrative Actions**: 7 years

#### Compliance Data
- **STIG Compliance Reports**: 7 years
- **RMF Documentation**: 7 years
- **Incident Reports**: 7 years
- **Risk Assessments**: 7 years

### Operational Data

#### Application Logs
- **Application Logs (Loki)**: 30 days
- **Access Logs**: 90 days
- **Error Logs**: 90 days
- **Performance Logs**: 30 days

#### Metrics
- **Prometheus Metrics**: 30 days (raw)
- **Aggregated Metrics**: 1 year
- **Custom Metrics**: 90 days

#### Backups
- **Database Backups**: 7 days (daily), 4 weeks (weekly), 12 months (monthly)
- **Configuration Backups**: 90 days
- **Disaster Recovery Backups**: 7 years

### Blockchain Data

#### Transaction History
- **All Transactions**: Retained indefinitely (immutable)
- **Transaction Logs**: 7 years

#### Smart Contract Data
- **Contract State**: Retained indefinitely
- **Contract Events**: 7 years

## Data Deletion Procedures

### Automated Deletion

#### Scheduled Cleanup Jobs
```bash
# Run daily cleanup job
kubectl create cronjob cleanup-old-data \
  --image=postgres:14-alpine \
  --schedule="0 3 * * *" \
  --restart=OnFailure \
  -- /bin/bash -c "psql $DATABASE_URL -f /scripts/cleanup-old-data.sql"
```

#### Cleanup Scripts
- **User Data Cleanup**: Runs monthly, deletes users inactive > 7 years
- **Log Cleanup**: Runs daily, deletes logs older than retention period
- **Backup Cleanup**: Runs daily, deletes backups older than retention period

### Manual Deletion

#### User-Requested Deletion
1. User submits deletion request
2. Account marked for deletion
3. 30-day grace period for account recovery
4. Data anonymized after grace period
5. Permanent deletion after 90 days

#### Administrative Deletion
1. Admin initiates deletion
2. Approval required for sensitive data
3. Data exported for compliance (if required)
4. Data deleted according to retention policy

## Compliance Requirements

### GDPR (General Data Protection Regulation)
- **Right to Erasure**: Users can request data deletion
- **Data Portability**: Users can export their data
- **Retention Limitation**: Data retained only as long as necessary

### SOX (Sarbanes-Oxley Act)
- **Financial Records**: 7 years retention
- **Audit Trails**: 7 years retention

### HIPAA (Health Insurance Portability and Accountability Act)
- **PHI Data**: 6 years minimum retention
- **Access Logs**: 6 years minimum retention

### DoD/MilSpec Compliance
- **Security Logs**: 7 years retention
- **Audit Trails**: 7 years retention
- **Compliance Reports**: 7 years retention

## Implementation

### Database Retention

#### Automated Cleanup Queries
```sql
-- Delete inactive users (7 years)
DELETE FROM users 
WHERE last_login < NOW() - INTERVAL '7 years'
  AND status = 'INACTIVE';

-- Delete old audit logs (after 2 years, archive first)
INSERT INTO audit_logs_archive 
SELECT * FROM audit_logs 
WHERE created_at < NOW() - INTERVAL '2 years';

DELETE FROM audit_logs 
WHERE created_at < NOW() - INTERVAL '2 years';
```

### Log Retention

#### Loki Retention Configuration
```yaml
# gitops/apps/monitoring/loki-config.yaml
retention_period: 30d
retention_stream:
  - selector: '{job="api"}'
    period: 90d
  - selector: '{job="portal"}'
    period: 90d
```

#### Prometheus Retention Configuration
```yaml
# gitops/apps/monitoring/prometheus-config.yaml
retention: 30d
retentionSize: 50GB
```

### Backup Retention

#### Backup Cleanup Script
```bash
# Delete backups older than retention period
find /backups/postgres -name "*.sql.gz" -mtime +7 -delete
find /backups/postgres -name "*.sql.gz" -mtime +30 -delete  # Weekly backups
find /backups/postgres -name "*.sql.gz" -mtime +365 -delete  # Monthly backups
```

## Data Archival

### Long-Term Storage

#### Archived Data Storage
- **Location**: S3 Glacier or equivalent
- **Format**: Compressed, encrypted archives
- **Retention**: Per compliance requirements
- **Access**: On-demand restoration

#### Archive Process
1. Data identified for archival
2. Data compressed and encrypted
3. Data uploaded to archival storage
4. Index updated with archive location
5. Original data deleted after verification

## Monitoring and Compliance

### Retention Policy Compliance

#### Automated Checks
- Daily verification of retention policies
- Alert on data older than retention period
- Report on data deletion activities

#### Compliance Reporting
- Monthly retention compliance report
- Quarterly audit of data retention
- Annual compliance review

## Exceptions and Extensions

### Legal Hold
- Data subject to legal hold cannot be deleted
- Legal hold overrides retention policy
- Legal hold must be documented
- Data released after legal hold lifted

### Business Requirements
- Extended retention for business-critical data
- Approval required for extensions
- Extensions documented and reviewed annually

## Contact

For questions about data retention:
- **Data Protection Officer**: dpo@sankofa.nexus
- **Compliance Team**: compliance@sankofa.nexus
- **Legal Team**: legal@sankofa.nexus