Sankofa/docs/DEPLOYMENT_REQUIREMENTS.md

# Sankofa Phoenix - Deployment Requirements

## Overview

This document outlines all requirements needed to deploy **Sankofa** (the ecosystem) and **Sankofa Phoenix** (the sovereign cloud platform). This includes infrastructure, software, network, security, and operational requirements.

---

## 1. Infrastructure Requirements

### 1.1 Edge Sites (Current Implementation)

**Proxmox VE Infrastructure:**
- ✅ **Proxmox VE 8+** installed on physical hosts
- ✅ **2+ Proxmox nodes** per site (for redundancy)
- ✅ **Network bridge** configured (vmbr0)
- ✅ **Storage pools** configured (local-lvm, ceph-fs, ceph-rbd)
- ✅ **OS Images** available (ubuntu-22.04-cloud.img)

**Current Status:**
- Site 1 (ml110-01): 192.168.11.10 - Operational ✅
- Site 2 (r630-01): 192.168.11.11 - Operational ✅

**Resource Requirements (SMOM-DBIS-138):**
- **Total VMs**: 18 (16 application + 2 infrastructure)
- **Total CPU**: 72 cores
- **Total RAM**: 140 GiB
- **Total Disk**: 278 GiB

### 1.2 Kubernetes Control Plane

**Requirements:**
- **Kubernetes v1.24+** cluster
- **3 master nodes** minimum (for HA)
- **5+ worker nodes** (for production workloads)
- **Container runtime**: containerd or CRI-O
- **CNI plugin**: Calico, Flannel, or Cilium
- **Storage class**: Dynamic provisioning (local-path, NFS, or Ceph)

**Control Plane Components:**
- **Crossplane**: Infrastructure as Code (Proxmox provider)
- **ArgoCD**: GitOps deployment
- **Keycloak**: Identity and access management
- **Prometheus/Grafana**: Monitoring and observability
- **Loki**: Log aggregation
- **Vault**: Secrets management (optional)

### 1.3 Database Infrastructure

**PostgreSQL Requirements:**
- **PostgreSQL 14+** (recommended: 15+)
- **High availability**: Primary + replicas
- **Storage**: NVMe SSD recommended (2TB+ per node)
- **RAM**: 64GB+ per node
- **Backup**: Automated daily backups

**Database Schema:**
- 26 migrations including:
  - Multi-tenancy tables
  - Billing and usage tracking
  - MFA and RBAC
  - Blockchain integration
  - Audit logging

### 1.4 Blockchain Infrastructure (Future)

**Hyperledger Besu Validators:**
- **3-5 validator nodes** per core datacenter
- **CPU**: AMD EPYC 7763 (64 cores) or Intel Xeon Platinum 8380 (40 cores)
- **RAM**: 128GB DDR4 ECC
- **Storage**: 2x 4TB NVMe SSD (RAID 1) for blockchain state
- **Network**: 2x 25GbE network adapters
- **HSM**: Hardware Security Module for key storage

**Read Replica Nodes:**
- **2-3 nodes** per regional datacenter
- **CPU**: AMD EPYC 7543 (32 cores) or Intel Xeon Gold 6338 (32 cores)
- **RAM**: 64GB DDR4 ECC
- **Storage**: 2x 2TB NVMe SSD (RAID 1)

---

## 2. Software Requirements

### 2.1 Development Tools

**Required:**
- **Node.js 18+** (for frontend, API, portal)
- **pnpm** (recommended) or npm/yarn
- **Go 1.21+** (for Crossplane provider)
- **Docker** (for local development and containerization)
- **Git** (version control)

**Optional:**
- **kubectl** (v1.24+) - Kubernetes CLI
- **helm** (v3.0+) - Kubernetes package manager
- **docker-compose** - Local development

### 2.2 Application Components

**Frontend (Next.js):**
- Next.js 14+
- React + TypeScript
- TailwindCSS + shadcn/ui
- TanStack Query

**Backend:**
- GraphQL API (Apollo Server + Fastify)
- PostgreSQL 14+
- WebSocket support
- Real-time subscriptions

**Portal:**
- Next.js portal application
- Keycloak OIDC integration
- Role-based dashboards

**Infrastructure:**
- Crossplane provider for Proxmox
- Kubernetes custom resources (ProxmoxVM)
- GitOps with ArgoCD

### 2.3 Monitoring and Observability

**Required:**
- **Prometheus**: Metrics collection
- **Grafana**: Dashboards and visualization
- **Loki**: Log aggregation
- **Alertmanager**: Alert routing

**Optional:**
- **Jaeger**: Distributed tracing
- **Kiali**: Service mesh visualization

---

## 3. Network Requirements

### 3.1 Edge Sites (Current)

**Network Configuration:**
- **Network bridge**: vmbr0
- **IP range**: 192.168.11.0/24
- **Gateway**: Configured
- **DNS**: Configured

**Connectivity:**
- **Cloudflare Tunnel**: Outbound-only secure connections
- **Nginx Proxy**: SSL/TLS termination and routing
- **Internet**: High-speed with redundancy

### 3.2 Cloudflare Integration

**Required:**
- **Cloudflare account** with Zero Trust
- **Cloudflare Tunnel** configured
- **DNS records** configured
- **Access policies** configured
- **SSL/TLS certificates** (managed by Cloudflare)

**Tunnel Configuration:**
- Tunnel credentials JSON file
- Ingress rules configured
- Health monitoring enabled

### 3.3 Inter-Datacenter Links (Future)

**Core to Core:**
- **Bandwidth**: 100Gbps+ per link
- **Redundancy**: Multiple redundant paths
- **Type**: Dark fiber or high-bandwidth leased lines

**Core to Regional:**
- **Bandwidth**: 10-40Gbps per link
- **Redundancy**: Redundant paths
- **Type**: Leased lines or MPLS

**Regional to Edge:**
- **Bandwidth**: 1-10Gbps per link
- **Redundancy**: Internet with redundancy
- **Type**: Internet connectivity with Cloudflare Tunnels

---

## 4. Security Requirements

### 4.1 Identity and Access Management

**Keycloak:**
- **Keycloak 20+** deployed
- **OIDC clients** configured:
  - `sankofa-api` (backend API)
  - `portal-client` (portal application)
- **Realms** configured (multi-tenant support)
- **MFA** enabled (TOTP, FIDO2, SMS, Email)
- **User federation** configured (optional)

**Access Control:**
- **RBAC**: Role-based access control
- **Tenant isolation**: Multi-tenant data isolation
- **API authentication**: JWT tokens
- **Session management**: Secure session handling

### 4.2 Network Security

**Firewalls:**
- **Next-generation firewalls** (Palo Alto, Fortinet, Check Point)
- **Access policies** configured
- **Intrusion detection/prevention** (IDS/IPS)
- **DDoS protection** (Cloudflare)

**Network Segmentation:**
- **VLANs** for different tiers
- **Network policies** in Kubernetes
- **Service mesh** (optional: Istio, Linkerd)

### 4.3 Application Security

**Security Features:**
- **Rate limiting**: 100 req/min per IP, 1000 req/hour per user
- **Security headers**: CSP, HSTS, X-Frame-Options
- **Input sanitization**: Body sanitization middleware
- **Encryption**: TLS 1.2+ for all connections
- **Secrets management**: Kubernetes secrets or Vault

**Audit Logging:**
- **Comprehensive audit trail** for all operations
- **Log retention** policy configured
- **Compliance** logging (GDPR, SOC 2, ISO 27001)

### 4.4 Blockchain Security

**Key Management:**
- **HSM**: Hardware Security Module for validator keys
- **Key rotation**: Automated key rotation
- **Multi-signature**: Multi-party governance

**Network Security:**
- **Private P2P network**: Encrypted peer-to-peer connections
- **Network overlay**: VPN or dedicated network segment
- **Consensus communication**: Secure channels for validators

---

## 5. Environment Configuration

### 5.1 Environment Variables

**API (.env):**
```env
DB_HOST=postgres
DB_PORT=5432
DB_NAME=sankofa
DB_USER=postgres
DB_PASSWORD=your-password
JWT_SECRET=your-jwt-secret

# Sovereign Identity (Keycloak)
KEYCLOAK_URL=http://keycloak:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=sankofa-api
KEYCLOAK_CLIENT_SECRET=your-keycloak-client-secret
KEYCLOAK_MULTI_REALM=true

# Multi-Tenancy
ENABLE_MULTI_TENANT=true
DEFAULT_TENANT_ID=
BLOCKCHAIN_IDENTITY_ENABLED=true

# Billing
BILLING_GRANULARITY=SECOND
BLOCKCHAIN_BILLING_ENABLED=true

# Blockchain
BLOCKCHAIN_RPC_URL=http://besu:8545
RESOURCE_PROVISIONING_CONTRACT_ADDRESS=0x...
```

**Frontend (.env.local):**
```env
NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://api:4000/graphql
NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://api:4000/graphql-ws
NEXT_PUBLIC_APP_URL=http://localhost:3000
NODE_ENV=development
```

**Portal (.env.local):**
```env
KEYCLOAK_URL=https://keycloak.sankofa.nexus
KEYCLOAK_REALM=sankofa
KEYCLOAK_CLIENT_ID=portal-client
KEYCLOAK_CLIENT_SECRET=your-secret
NEXT_PUBLIC_CROSSPLANE_API=https://crossplane.sankofa.nexus
NEXT_PUBLIC_ARGOCD_URL=https://argocd.sankofa.nexus
NEXT_PUBLIC_GRAFANA_URL=https://grafana.sankofa.nexus
NEXT_PUBLIC_LOKI_URL=https://loki.sankofa.nexus:3100
```

**Proxmox Provider:**
```env
PROXMOX_HOST=192.168.11.10
PROXMOX_USER=root@pam
PROXMOX_PASS=your-password
# OR
PROXMOX_TOKEN=your-api-token
```

### 5.2 Kubernetes Secrets

**Required Secrets:**
- Database credentials
- Keycloak client secrets
- JWT secrets
- Proxmox API credentials
- Cloudflare tunnel credentials
- SSL/TLS certificates (if not using Cloudflare)

---

## 6. Deployment Steps

### 6.1 Prerequisites Checklist

- [ ] Kubernetes cluster deployed and operational
- [ ] PostgreSQL database deployed and accessible
- [ ] Keycloak deployed and configured
- [ ] Proxmox nodes accessible and configured
- [ ] Cloudflare account and tunnel configured
- [ ] Network connectivity verified
- [ ] DNS records configured
- [ ] SSL/TLS certificates configured

### 6.2 Database Setup

```bash
# 1. Create database
createdb sankofa

# 2. Run migrations (26 migrations)
cd api
npm run db:migrate

# 3. Verify migrations
psql -d sankofa -c "\dt"

# 4. Seed initial data (optional)
npm run db:seed
```

### 6.3 Kubernetes Deployment

```bash
# 1. Create namespaces
kubectl create namespace sankofa
kubectl create namespace crossplane-system
kubectl create namespace monitoring

# 2. Deploy Crossplane
kubectl apply -f gitops/apps/crossplane/

# 3. Deploy Proxmox Provider
kubectl apply -f crossplane-provider-proxmox/config/

# 4. Deploy ArgoCD
kubectl apply -f gitops/apps/argocd/

# 5. Deploy Keycloak
kubectl apply -f gitops/apps/keycloak/

# 6. Deploy API
kubectl apply -f gitops/apps/api/

# 7. Deploy Frontend
kubectl apply -f gitops/apps/frontend/

# 8. Deploy Portal
kubectl apply -f gitops/apps/portal/

# 9. Deploy Monitoring
kubectl apply -f gitops/apps/monitoring/
```

### 6.4 Proxmox VM Deployment

```bash
# 1. Deploy infrastructure VMs first
kubectl apply -f examples/production/nginx-proxy-vm.yaml
kubectl apply -f examples/production/cloudflare-tunnel-vm.yaml

# 2. Deploy application VMs
kubectl apply -f examples/production/smom-dbis-138/

# 3. Monitor deployment
kubectl get proxmoxvm -A -w

# 4. Check controller logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f
```

### 6.5 GitOps Setup (ArgoCD)

```bash
# 1. Apply ArgoCD application
kubectl apply -f gitops/apps/argocd/application.yaml

# 2. Sync application
argocd app sync sankofa-phoenix

# 3. Verify sync status
argocd app get sankofa-phoenix
```

### 6.6 Multi-Tenancy Setup

```bash
# 1. Create system tenant (via GraphQL)
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { createTenant(input: { name: \"system\", tier: SOVEREIGN }) { id name billingAccountId } }"
  }'

# 2. Assign admin user to system tenant
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { addUserToTenant(tenantId: \"<system-tenant-id>\", userId: \"<admin-user-id>\", role: TENANT_OWNER) }"
  }'
```

---

## 7. Verification and Testing

### 7.1 Health Checks

```bash
# API health
curl http://api.sankofa.nexus/health

# Frontend
curl http://frontend.sankofa.nexus

# Portal
curl http://portal.sankofa.nexus

# Keycloak health
curl http://keycloak.sankofa.nexus/health

# Proxmox VMs
kubectl get proxmoxvm -A
```

### 7.2 Smoke Tests

```bash
# Run smoke tests
./scripts/smoke-tests.sh
```

### 7.3 Performance Testing

```bash
# Load testing
./scripts/performance-test.sh

# k6 load test
k6 run scripts/k6-load-test.js
```

---

## 8. Operational Requirements

### 8.1 Monitoring and Alerting

**Required:**
- Prometheus metrics collection
- Grafana dashboards
- Alertmanager rules
- Notification channels (email, Slack, PagerDuty)

**Key Metrics:**
- API response times
- Database query performance
- VM resource utilization
- Blockchain network health
- Service availability

### 8.2 Backup and Disaster Recovery

**Database Backups:**
- Daily automated backups
- Retention policy: 30 days minimum
- Off-site backup storage
- Backup verification scripts

**VM Backups:**
- Proxmox backup schedules
- Snapshot management
- Disaster recovery procedures

### 8.3 Support and Operations

**Required:**
- 24/7 on-call rotation
- Incident response procedures
- Runbooks for common issues
- Escalation procedures
- Support team training

---

## 9. Compliance and Governance

### 9.1 Compliance Requirements

**Data Protection:**
- GDPR compliance (EU)
- Data retention policies
- Privacy policy published
- Terms of service published

**Security Standards:**
- SOC 2 Type II (if applicable)
- ISO 27001 (if applicable)
- Security audit procedures
- Penetration testing

### 9.2 Governance

**Multi-Tenancy:**
- Tenant isolation verified
- Resource quotas enforced
- Billing accuracy verified
- Audit logging enabled

**Blockchain Governance:**
- Multi-party governance nodes
- Smart contract upgrade procedures
- Network upgrade procedures

---

## 10. Cost Estimates

### 10.1 Infrastructure Costs

**Edge Sites (Current):**
- Proxmox hardware: $10K-$50K per site
- Network equipment: $5K-$20K per site
- Power and cooling: $1K-$5K per year per site

**Kubernetes Cluster:**
- Control plane: $500-$2K per month
- Worker nodes: $1K-$5K per month
- Storage: $200-$1K per month

**Database:**
- PostgreSQL cluster: $500-$2K per month
- Backup storage: $100-$500 per month

### 10.2 Cloudflare Costs

**Zero Trust:**
- Free tier: Up to 50 users
- Paid tier: $7 per user per month

**Tunnels:**
- Free: Unlimited tunnels
- Paid: Additional features

**Bandwidth:**
- Included in Zero Trust plan
- Additional bandwidth: $0.10-$0.50 per GB

### 10.3 Operational Costs

**Personnel:**
- DevOps engineers: $100K-$200K per year
- SRE engineers: $120K-$250K per year
- Support staff: $50K-$100K per year

**Software Licenses:**
- Most components are open source
- Optional commercial support: $10K-$100K per year

---

## 11. Quick Start Summary

### Minimum Viable Deployment

**For Development/Testing:**
1. Single Kubernetes cluster (3 nodes minimum)
2. PostgreSQL database (single instance)
3. Keycloak (single instance)
4. 2 Proxmox nodes
5. Cloudflare account (free tier)
6. All application components deployed

**For Production:**
1. High-availability Kubernetes cluster (3 masters + 5 workers)
2. PostgreSQL cluster (primary + replicas)
3. Keycloak cluster (HA)
4. Multiple Proxmox sites (2+ sites)
5. Cloudflare Zero Trust (paid tier)
6. Monitoring and alerting configured
7. Backup and disaster recovery configured
8. Security hardening completed

---

## 12. Documentation References

- **[Production Deployment Ready](./PRODUCTION_DEPLOYMENT_READY.md)** - Current deployment status
- **[Launch Checklist](./status/LAUNCH_CHECKLIST.md)** - Pre-launch verification
- **[Deployment Guide](./DEPLOYMENT.md)** - Detailed deployment instructions
- **[Deployment Plan](./deployment_plan.md)** - Phased rollout plan
- **[System Architecture](./system_architecture.md)** - Overall architecture
- **[Hardware BOM](./hardware_bom.md)** - Hardware specifications
- **[VM Deployment Plan](./VM_DEPLOYMENT_PLAN.md)** - VM deployment guide

---

## 13. Next Steps

1. **Review Prerequisites**: Verify all infrastructure and software requirements
2. **Configure Environment**: Set up environment variables and secrets
3. **Deploy Database**: Run migrations and seed data
4. **Deploy Kubernetes**: Deploy control plane components
5. **Deploy Applications**: Deploy API, frontend, and portal
6. **Deploy VMs**: Deploy Proxmox VMs via Crossplane
7. **Configure Monitoring**: Set up Prometheus, Grafana, and Loki
8. **Verify Deployment**: Run health checks and smoke tests
9. **Configure Multi-Tenancy**: Set up initial tenants
10. **Go Live**: Enable production traffic

---

**Last Updated**: 2025-01-XX  
**Status**: Comprehensive deployment requirements documented