Sankofa/docs/DEPLOYMENT.md

# Sankofa Phoenix - Deployment Guide

## Overview

This guide covers the complete deployment process for Sankofa Phoenix, including prerequisites, step-by-step instructions, and post-deployment verification.

## Prerequisites

### Infrastructure Requirements
- Kubernetes cluster (v1.24+)
- PostgreSQL database (v14+)
- Keycloak instance (for portal authentication)
- Prometheus and Grafana (for monitoring)
- Loki (for log aggregation)
- ArgoCD (for GitOps)

### Tools Required
- `kubectl` (v1.24+)
- `helm` (v3.0+)
- `docker` (for local development)
- `pnpm` or `npm`

## Deployment Steps

### 1. Database Setup

```bash
# Run migrations (includes multi-tenancy and billing tables)
cd api
npm run db:migrate

# Verify migration 012_tenants_and_billing ran successfully
psql -d sankofa -c "\dt tenants"
psql -d sankofa -c "\dt billing_accounts"

# Seed initial data (optional)
npm run db:seed
```

### 2. Environment Configuration

Create `.env` files for each component:

**API (.env)**
```env
DB_HOST=postgres
DB_PORT=5432
DB_NAME=sankofa
DB_USER=postgres
DB_PASSWORD=your-password
JWT_SECRET=your-jwt-secret

# Sovereign Identity (Keycloak) - NO Azure dependencies
KEYCLOAK_URL=http://keycloak:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=sankofa-api
KEYCLOAK_CLIENT_SECRET=your-keycloak-client-secret
KEYCLOAK_MULTI_REALM=true

# Multi-Tenancy
ENABLE_MULTI_TENANT=true
DEFAULT_TENANT_ID=
BLOCKCHAIN_IDENTITY_ENABLED=true

# Billing (Superior to Azure Cost Management)
BILLING_GRANULARITY=SECOND
BLOCKCHAIN_BILLING_ENABLED=true

BLOCKCHAIN_RPC_URL=http://besu:8545
RESOURCE_PROVISIONING_CONTRACT_ADDRESS=0x...
```

**Frontend (.env.local)**
```env
NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://api:4000/graphql
NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://api:4000/graphql-ws
```

**Portal (.env.local)**
```env
KEYCLOAK_URL=https://keycloak.sankofa.nexus
KEYCLOAK_REALM=sankofa
KEYCLOAK_CLIENT_ID=portal-client
KEYCLOAK_CLIENT_SECRET=your-secret
NEXT_PUBLIC_CROSSPLANE_API=https://crossplane.sankofa.nexus
NEXT_PUBLIC_ARGOCD_URL=https://argocd.sankofa.nexus
NEXT_PUBLIC_GRAFANA_URL=https://grafana.sankofa.nexus
NEXT_PUBLIC_LOKI_URL=https://loki.sankofa.nexus:3100
```

### 3. Kubernetes Deployment

```bash
# Apply all manifests
kubectl apply -f gitops/apps/api/ -n sankofa
kubectl apply -f gitops/apps/frontend/ -n sankofa
kubectl apply -f gitops/apps/portal/ -n sankofa

# Wait for deployments
kubectl rollout status deployment/api -n sankofa
kubectl rollout status deployment/frontend -n sankofa
kubectl rollout status deployment/portal -n sankofa
```

### 4. GitOps Setup (ArgoCD)

```bash
# Apply ArgoCD application
kubectl apply -f gitops/apps/argocd/application.yaml

# Sync application
argocd app sync sankofa-phoenix
```

### 5. Blockchain Network Setup

```bash
cd blockchain

# Start Besu nodes
docker-compose -f docker-compose.besu.yml up -d

# Deploy smart contracts
pnpm deploy:test
```

### 6. Monitoring Setup

```bash
# Deploy Prometheus
kubectl apply -f gitops/monitoring/prometheus/

# Deploy Grafana
kubectl apply -f gitops/monitoring/grafana/

# Deploy Loki
kubectl apply -f gitops/monitoring/loki/
```

### 7. Multi-Tenancy Setup

After deployment, set up initial tenants:

```bash
# Create system tenant (via GraphQL)
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { createTenant(input: { name: \"system\", tier: SOVEREIGN }) { id name billingAccountId } }"
  }'

# Assign admin user to system tenant
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{
    "query": "mutation { addUserToTenant(tenantId: \"<system-tenant-id>\", userId: \"<admin-user-id>\", role: TENANT_OWNER) }"
  }'
```

See [Tenant Management Guide](./tenants/TENANT_MANAGEMENT.md) for detailed tenant setup.

## Verification

### Health Checks

```bash
# API health
curl http://api.sankofa.nexus/health

# Frontend
curl http://frontend.sankofa.nexus

# Portal
curl http://portal.sankofa.nexus

# Keycloak health
curl http://keycloak.sankofa.nexus/health
```

### Tenant Verification

```bash
# Verify tenant creation works
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "query": "query { myTenant { id name status tier } }"
  }'

# Verify billing tracking works
curl -X POST http://api.sankofa.nexus/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "query": "query { costBreakdown(tenantId: \"<tenant-id>\", groupBy: [\"resource\"]) { total byResource { resourceName cost } } }"
  }'
```

### Smoke Tests

```bash
# Run smoke tests
./scripts/smoke-tests.sh
```

## Rollback Procedure

```bash
# Rollback API
kubectl rollout undo deployment/api -n sankofa

# Rollback Frontend
kubectl rollout undo deployment/frontend -n sankofa

# Rollback Portal
kubectl rollout undo deployment/portal -n sankofa
```

## Troubleshooting

### Common Issues

1. **Database Connection Errors**
   - Verify database credentials
   - Check network connectivity
   - Verify database is running

2. **Kubernetes Pod Issues**
   - Check pod logs: `kubectl logs <pod-name> -n sankofa`
   - Check pod status: `kubectl describe pod <pod-name> -n sankofa`

3. **Blockchain Connection Issues**
   - Verify Besu nodes are running
   - Check RPC endpoint accessibility
   - Verify contract addresses

## Production Checklist

- [ ] All environment variables configured
- [ ] Database migrations completed (including migration 012_tenants_and_billing)
- [ ] Keycloak deployed and configured
- [ ] Keycloak clients created (sankofa-api, portal-client)
- [ ] Secrets created in Kubernetes
- [ ] All services deployed and healthy
- [ ] System tenant created
- [ ] Admin user assigned to system tenant
- [ ] Multi-tenancy verified
- [ ] Billing tracking verified
- [ ] Monitoring and alerting configured
- [ ] Tenant-aware dashboards created
- [ ] Backup procedures in place
- [ ] Security hardening completed
- [ ] Documentation updated

See [Remaining Tasks](./REMAINING_TASKS.md) for complete task list.