Phoenix Deploy API: alert webhook, .env.example, Site24x7 and E2E docs

- PHOENIX_ALERT_WEBHOOK_URL: POST on alerts when firing
- .env.example: PROXMOX_*, PROMETHEUS_*, webhooks, partner keys
- PHOENIX_SITE24X7_API_KEYS.md: how to issue API keys for Site24x7
- PHOENIX_E2E_PORTAL_API_RAILING.md: E2E test steps and references

Made-with: Cursor

docs/02-architecture/PHOENIX_E2E_PORTAL_API_RAILING.md

@@ -0,0 +1,27 @@
+# E2E: Portal → Phoenix API → Railing
+
+**Purpose:** How to run and extend E2E tests for the flow: Portal (browser) → Sankofa Phoenix API → Phoenix Deploy API (railing).
+
+## Flow
+
+1. User signs in to Portal (NextAuth / Keycloak).
+2. Portal calls Sankofa API with `Authorization: Bearer <accessToken>` (or session cookie).
+3. Sankofa API proxies `/api/v1/infra/*`, `/api/v1/ve/*`, `/api/v1/health/*` to `PHOENIX_RAILING_URL` when set.
+4. Tenant-scoped routes (`/api/v1/tenants/me/resources`, `/api/v1/tenants/me/health`) use JWT or X-API-Key tenant context.
+
+## Manual E2E
+
+1. Start Phoenix Deploy API: `cd phoenix-deploy-api && npm start` (optional: set PROXMOX_* for live data).
+2. Start Sankofa API: set `PHOENIX_RAILING_URL=http://localhost:4001`, then `npm run dev`.
+3. Start Portal: set `NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://localhost:4000/graphql` (or API URL), then `npm run dev`.
+4. Sign in, open Infrastructure page (`/infrastructure`), verify nodes/storage load; open Dashboard and verify Phoenix Health tile; open VMs and verify list (if Crossplane or railing VMs are used).
+
+## Automated E2E (optional)
+
+- **Playwright/Cypress:** Add tests that sign in, navigate to `/infrastructure`, `/vms`, dashboard, and assert presence of data or “Loading”/error states.
+- **API-only:** Use Sankofa API integration tests: start server with test config, `fetch('/api/v1/health/summary', { headers: { Authorization: 'Bearer <test-jwt>' } })` and assert 200 and body shape. See `api/src/__tests__/integration/phoenix-railing.test.ts` for tenant-me routes.
+
+## References
+
+- [PORTAL_RAILING_WIRING.md](../../Sankofa/docs/phoenix/PORTAL_RAILING_WIRING.md) (in Sankofa repo)
+- [PHOENIX_API_RAILING_SPEC.md](PHOENIX_API_RAILING_SPEC.md)

docs/04-configuration/PHOENIX_SITE24X7_API_KEYS.md

@@ -0,0 +1,40 @@
+# Site24x7 API Keys for Phoenix API
+
+**Purpose:** Issue Phoenix API keys for Site24x7 (and similar) users so they can call `/api/v1/*` (Infra, VE, Health) with `X-API-Key` or `Authorization: Bearer <key>`.
+
+## Prerequisites
+
+- API key store is in place (Sankofa API: `api_keys` table + migration 026).
+- Users or service accounts exist in Gitea (e.g. Site24x7 team in Sankofa_Phoenix org).
+
+## Steps
+
+1. **Create API key in Sankofa API**
+   - Use GraphQL mutation `createApiKey` (or future admin REST endpoint) with:
+     - `name`: e.g. `Site24x7-monitor`
+     - `permissions`: `["read"]` for read-only (Infra, VE, Health); `["read","write"]` if VM lifecycle is allowed.
+   - Or insert into `api_keys` (key_hash = SHA256 of the secret key, key_prefix = first 12 chars).
+   - Return the **plain key** once to the operator; store only the hash.
+
+2. **Assign tenant (optional)**
+   - If the key should be tenant-scoped, set `tenant_id` on the `api_keys` row so `/api/v1/tenants/me/*` returns that tenant’s data.
+
+3. **Configure Site24x7**
+   - In Site24x7 monitor config, set:
+     - **URL:** `https://api.sankofa.nexus/api/v1/health/summary` (or your Phoenix API base).
+     - **Header:** `X-API-Key: <key>` or `Authorization: Bearer <key>`.
+
+4. **Phoenix Deploy API (direct)**  
+   If Site24x7 hits phoenix-deploy-api directly (not via Sankofa API), add the same key to `PHOENIX_PARTNER_KEYS` in phoenix-deploy-api `.env`.
+
+## Gitea team → API keys
+
+For each Gitea user in the Site24x7 team who needs API access:
+
+- Ensure the user exists in Sankofa (Keycloak/users table) if using JWT; or create an API key bound to a service user and share the key securely.
+- Prefer one key per service (e.g. one key for Site24x7) rather than per user, unless each user has a distinct integration.
+
+## References
+
+- [PHOENIX_PARTNER_INTEGRATION_SITE24X7_MANAGEENGINE.md](PHOENIX_PARTNER_INTEGRATION_SITE24X7_MANAGEENGINE.md)
+- Sankofa API: `api_keys` table, `verifyApiKey()`, X-API-Key in tenant-auth for `/api/v1/*`.

phoenix-deploy-api/.env.example

@@ -1,17 +1,30 @@
-# =============================================================================
+# Phoenix Deploy API — copy to .env and set values
-# Phoenix Deploy API — Environment (example)
-# =============================================================================
-# Copy to .env and fill in. Do not commit .env.
-# See README.md for Gitea webhook and deploy endpoint usage.
-# =============================================================================
 
-# Listen port
 PORT=4001
-
-# Gitea instance (for commit status API)
 GITEA_URL=https://gitea.d-bis.org
-# Token with repo (or repo:status) scope — create at Gitea → Settings → Applications
+GITEA_TOKEN=
-GITEA_TOKEN=your-gitea-token
+PHOENIX_DEPLOY_SECRET=
 
-# Optional: shared secret for webhook signature and /api/deploy Bearer auth
+# Proxmox (for Infra/VE API; omit for stub responses)
-# PHOENIX_DEPLOY_SECRET=your-webhook-and-deploy-secret
+PROXMOX_HOST=
+PROXMOX_PORT=8006
+PROXMOX_USER=root@pam
+PROXMOX_TOKEN_NAME=
+PROXMOX_TOKEN_VALUE=
+PROXMOX_TLS_VERIFY=1
+
+# VM lifecycle (set to 1 to enable start/stop/reboot)
+PHOENIX_VE_LIFECYCLE_ENABLED=0
+
+# Prometheus (for Health API)
+PROMETHEUS_URL=http://localhost:9090
+PROMETHEUS_ALERTS_URL=
+
+# Outbound webhooks
+PHOENIX_WEBHOOK_URL=
+PHOENIX_WEBHOOK_SECRET=
+PHOENIX_ALERT_WEBHOOK_URL=
+PHOENIX_ALERT_WEBHOOK_SECRET=
+
+# Optional: comma-separated API keys for /api/v1/* (X-API-Key or Bearer)
+PHOENIX_PARTNER_KEYS=

phoenix-deploy-api/server.js

@@ -319,13 +319,27 @@ app.get('/api/v1/health/metrics', async (req, res) => {
 
 /**
  * GET /api/v1/health/alerts — Active alerts (stub or Alertmanager; optional PROMETHEUS_ALERTS_URL)
+ * Optional: POST to PHOENIX_ALERT_WEBHOOK_URL when alerts exist (partner notification).
  */
+const PHOENIX_ALERT_WEBHOOK_URL = process.env.PHOENIX_ALERT_WEBHOOK_URL || '';
+const PHOENIX_ALERT_WEBHOOK_SECRET = process.env.PHOENIX_ALERT_WEBHOOK_SECRET || '';
+
 app.get('/api/v1/health/alerts', async (req, res) => {
   const alertsUrl = process.env.PROMETHEUS_ALERTS_URL || `${PROMETHEUS_URL}/api/v1/alerts`;
   try {
     const data = await fetch(alertsUrl).then((r) => r.json()).catch(() => ({ data: { alerts: [] } }));
     const alerts = data.data?.alerts ?? data.alerts ?? [];
-    res.json({ alerts: Array.isArray(alerts) ? alerts : [], stub: !process.env.PROMETHEUS_URL });
+    const payload = { alerts: Array.isArray(alerts) ? alerts : [], stub: !process.env.PROMETHEUS_URL };
+    res.json(payload);
+    if (PHOENIX_ALERT_WEBHOOK_URL && payload.alerts.length > 0) {
+      const body = JSON.stringify({ event: 'alerts.fired', alerts: payload.alerts, at: new Date().toISOString() });
+      const sig = PHOENIX_ALERT_WEBHOOK_SECRET ? crypto.createHmac('sha256', PHOENIX_ALERT_WEBHOOK_SECRET).update(body).digest('hex') : '';
+      fetch(PHOENIX_ALERT_WEBHOOK_URL, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json', ...(sig && { 'X-Phoenix-Signature': `sha256=${sig}` }) },
+        body,
+      }).catch((e) => console.error('[alert-webhook]', e.message));
+    }
   } catch (err) {
     res.json({ alerts: [], stub: true, message: err.message });
   }