proxmox/docs/02-architecture/SANKOFA_PHOENIX_PHASE4_MIGRATION_RUNBOOK.md

Sankofa / Phoenix Phase 4 Migration Runbook

Status: Draft executable runbook for the additive Client / Subscription / Entitlement migration
Last Updated: 2026-03-30
Related: SANKOFA_PHOENIX_COMPLETE_PHASED_EXECUTION_PLAN.md, SANKOFA_PHOENIX_REMAINING_TASKS.md, SANKOFA_PHOENIX_CANONICAL_BOUNDARIES_AND_TAXONOMY.md

Purpose

This runbook describes how to apply, verify, and if needed roll back the additive Phoenix backend migration that introduces:

• clients
• client_users
• service_subscriptions
• entitlements
• tenant.client_id
• billing-table client_id and subscription_id foreign keys

This migration is intentionally additive-first. It does not remove the existing tenant-based reporting shape. It introduces the new commercial boundary while preserving tenant-scoped operations.

Scope of the current tranche

Code already landed for this migration slice in the Sankofa API repo:

• migration 027_client_subscription_entitlements
• operating-model types and service
• GraphQL queries for clients, client, myClient, serviceSubscriptions, mySubscriptions, entitlements, and myEntitlements
• tenant bootstrap hook that creates a default client/subscription/entitlement for new tenants
• identity propagation for clientId and subscriptionId

Preconditions

• Confirm the target database already includes migrations through 026_api_keys.
• Confirm a current backup or snapshot exists for the target Postgres instance.
• Confirm application deploy artifacts are available for the Sankofa API and portal.
• Confirm Keycloak tokens or local JWTs can carry tenant_id, and optionally client_id / subscription_id.
• Confirm the environment has a rollback window and operator coverage.

Deployment order

1. Take a database backup or snapshot.
2. Deploy the API code that understands the new fields before applying the migration.
3. Apply migration 027_client_subscription_entitlements.
4. Restart or reload the Sankofa API.
5. Deploy the updated portal build so the workspace can display clientId and subscriptionId context.
6. Run the verification checks below.

Verification checklist

Database verification

Run checks equivalent to the following:

SELECT to_regclass('public.clients');
SELECT to_regclass('public.client_users');
SELECT to_regclass('public.service_subscriptions');
SELECT to_regclass('public.entitlements');

SELECT COUNT(*) AS tenants_without_client
FROM tenants
WHERE client_id IS NULL;

SELECT COUNT(*) AS subscriptions_without_client
FROM service_subscriptions
WHERE client_id IS NULL;

SELECT COUNT(*) AS entitlements_without_subscription
FROM entitlements
WHERE subscription_id IS NULL;

Expected result:

• all four new tables exist
• tenants_without_client = 0 after backfill completes
• subscriptions_without_client = 0
• entitlements_without_subscription = 0

Data-shape verification

Spot-check a migrated tenant:

SELECT
  t.id AS tenant_id,
  t.name AS tenant_name,
  t.client_id,
  c.name AS client_name,
  s.id AS subscription_id,
  s.offer_code,
  s.commercial_model,
  e.id AS entitlement_id,
  e.entitlement_key
FROM tenants t
LEFT JOIN clients c ON c.id = t.client_id
LEFT JOIN service_subscriptions s ON s.tenant_id = t.id
LEFT JOIN entitlements e ON e.subscription_id = s.id
WHERE t.id = '<tenant-id>';

Expected result:

• one linked client
• at least one tenant-workspace subscription
• at least one tenant.workspace entitlement

API verification

Verify GraphQL with an authenticated token that has tenant context:

query VerifyOperatingModel {
  myTenant { id clientId name }
  myClient { id name status primaryDomain }
  mySubscriptions { id offerName commercialModel status fulfillmentMode }
  myEntitlements { id entitlementKey status }
}

Expected result:

• myTenant.clientId is populated
• myClient resolves for tenant-scoped users
• mySubscriptions and myEntitlements return workspace records

Portal verification

• Sign into portal.sankofa.nexus.
• Confirm the dashboard renders:
  • client boundary card
  • active subscription card
  • entitlement summary card
• Confirm the session still works for users with only tenant-scoped claims.

Post-deploy monitoring

Watch for:

• API errors referencing client_id or subscription_id
• onboarding failures during tenant creation
• billing queries returning null where tenant-linked records should have been backfilled
• portal sessions that include tenant context but fail to resolve myClient

Rollback strategy

Use rollback only if the migration causes runtime or data-integrity issues that cannot be contained quickly.

Application rollback

1. Roll back the API deployment to the previous artifact.
2. Roll back the portal deployment if the new workspace cards cause issues.

Database rollback

If the additive schema itself must be removed, run the down path for migration 027_client_subscription_entitlements only after the application is off the new model.

Rollback effects:

• removes clients, client_users, service_subscriptions, and entitlements
• drops client_id from tenants
• drops client_id / subscription_id additions from billing tables

Safer partial rollback option

Prefer this if the issue is application logic rather than schema:

1. Keep the new tables in place.
2. Disable reads from the new GraphQL queries at the application layer.
3. Disable bootstrap of the operating-model service during tenant creation.
4. Leave the additive columns in place until a corrected deployment is ready.

This avoids destructive churn on newly backfilled data.

Open follow-up work after this migration

• move billing and invoice views to client/subscription ownership
• add onboarding persistence and workflow states
• add subscription management UI and actions
• add entitlement-aware fulfillment and deployment mapping
• add production smoke tests for hostname plus operating-model GraphQL queries