Docs: Phoenix TTS contract, recommendations (TTS/Gitea/Phoenix), push-all note, Gitea labels for virtual-banker

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 16:54:22 -08:00
parent a33fd9e3fe
commit 4945abcf7c
5 changed files with 419 additions and 0 deletions
--- a/docs/04-configuration/EXPLORER_MONOREPO_GITEA_PUSH.md
+++ b/docs/04-configuration/EXPLORER_MONOREPO_GITEA_PUSH.md
@@ -0,0 +1,75 @@
+# Explorer Monorepo — Gitea Push
+
+**Last Updated:** 2026-02-10  
+**Status:** Explorer monorepo is pushed to Gitea. **virtual-banker** is not part of and not referenced by explorer-monorepo; it lives as a separate repo in the projects directory (`/home/intlc/projects/virtual-banker`) and on Gitea as [d-bis/virtual-banker](https://gitea.d-bis.org/d-bis/virtual-banker).
+
+---
+
+## Push all projects (including virtual-banker)
+
+To push **every** repo under the projects directory (including **virtual-banker**, **proxmox**, **explorer-monorepo** only via proxmox, etc.) to Gitea in one go, from the **proxmox** repo root:
+
+```bash
+cd /path/to/proxmox
+# GITEA_TOKEN from .env or export
+bash scripts/dev-vm/push-all-projects-to-gitea.sh
+```
+
+- **Discovered repos:** Direct children of `PROJECTS_DIR` (default `/home/intlc/projects`) that have a `.git` directory.
+- **virtual-banker** is included because it lives at `/home/intlc/projects/virtual-banker`.
+- **explorer-monorepo** is **not** a direct child; push it separately (see “How to push explorer-monorepo” below) if needed.
+
+---
+
+## Why explorer-monorepo isn’t pushed by the usual script
+
+- **explorer-monorepo** is a **submodule** of the proxmox repo (`explorer-monorepo` → `https://github.com/Order-of-Hospitallers/chain-138-explorer.git`).
+- **push-all-projects-to-gitea.sh** only considers **direct children** of `PROJECTS_DIR` (default `/home/intlc/projects`), i.e. sibling repos like `proxmox`, `dbis_core`, not subdirectories/submodules inside proxmox.
+- So explorer-monorepo is never discovered or pushed by that script.
+
+---
+
+## How to push explorer-monorepo to Gitea
+
+### 1. Create the repo on Gitea (if it doesn’t exist)
+
+From the proxmox repo root, using the same token and org as elsewhere:
+
+```bash
+GITEA_TOKEN=xxx bash scripts/dev-vm/gitea-create-orgs-and-repos.sh --dry-run   # optional: check list
+GITEA_TOKEN=xxx bash scripts/dev-vm/gitea-create-orgs-and-repos.sh              # creates d-bis/explorer-monorepo if missing
+```
+
+(`explorer-monorepo` is already in the default `REPO_NAMES` in that script.)
+
+### 2. Add Gitea remote and push from the submodule
+
+From the **explorer-monorepo** directory (submodule has its own `.git`):
+
+```bash
+cd /home/intlc/projects/proxmox/explorer-monorepo
+
+# Optional: load Gitea env from proxmox root
+[ -f ../.env ] && set -a && source ../.env && set +a
+
+# Add Gitea remote if not already present
+git remote get-url gitea 2>/dev/null || \
+  git remote add gitea "https://gitea.d-bis.org/d-bis/explorer-monorepo.git"
+
+# Push current branch (e.g. main)
+BRANCH=$(git branch --show-current)
+git push "https://${GITEA_TOKEN}@gitea.d-bis.org/d-bis/explorer-monorepo.git" "$BRANCH" --set-upstream
+```
+
+Use the same **GITEA_TOKEN** as for other Gitea operations (e.g. from proxmox `.env` or Gitea → Settings → Applications).
+
+### 3. If you get HTTP 413 on push
+
+If the repo is large and the push fails with **HTTP 413**, follow [GITEA_LARGE_PUSH_HTTP_413.md](GITEA_LARGE_PUSH_HTTP_413.md) to raise `HTTP_MAX_REQUEST_BODY` in Gitea and `client_max_body_size` in NPMplus for `gitea.d-bis.org`.
+
+---
+
+## After pushing
+
+- In Gitea, add labels per [GITEA_ORG_STRUCTURE.md](GITEA_ORG_STRUCTURE.md): e.g. `project/explorer`, `domain/blockchain` / `domain/web` as appropriate.
+- To make the proxmox repo point at Gitea for this submodule (optional): update `.gitmodules` and submodule remote to `https://gitea.d-bis.org/d-bis/explorer-monorepo.git` and run `git submodule sync`.
--- a/docs/04-configuration/GITEA_ORG_STRUCTURE.md
+++ b/docs/04-configuration/GITEA_ORG_STRUCTURE.md
@@ -0,0 +1,139 @@
+# Gitea Organizational Structure and Conventions
+
+**Last Updated:** 2026-02-10  
+**Status:** Active Documentation  
+**Gitea Instance:** https://gitea.d-bis.org  
+**Primary Org:** d-bis
+
+---
+
+## 1. Overview
+
+Gitea uses a flat model: **Organization → Teams → Users**. This document defines conventions for virtual hierarchy via labels, team naming, and repository organization to support flexible structuring beyond the built-in flat model.
+
+---
+
+## 2. Label-Based Virtual Hierarchy (Option A)
+
+Use repository labels to create virtual groups without code changes.
+
+### 2.1 Project Labels
+
+| Label | Purpose | Example Repos |
+|-------|---------|---------------|
+| `project/sankofa` | Sankofa Phoenix platform and marketplace | sankofa, as4-411 |
+| `project/omnis` | OMNIS application | omnis |
+| `project/proxmox` | Proxmox infrastructure and docs | proxmox |
+| `project/dbis_core` | DBIS core libraries and services | dbis_core |
+| `project/chain138` | Chain 138 blockchain (Besu, RPC, etc.) | smom-dbis-138, rpc-translator-138 |
+| `project/explorer` | Blockscout, explorers | explorer-monorepo |
+| `project/miracles_in_motion` | MIM web and API | miracles_in_motion |
+| `project/the_order` | Order service | the_order |
+| `project/virtual-banker` | Virtual banker (voice/TTS, widget) | virtual-banker |
+
+### 2.2 Domain Labels
+
+| Label | Purpose | Example Repos |
+|-------|---------|---------------|
+| `domain/blockchain` | Blockchain, smart contracts, RPC | smom-dbis-138, alltra-lifi-settlement |
+| `domain/web` | Web apps, portals, frontends | miracles_in_motion, omnis |
+| `domain/infrastructure` | Proxmox, scripts, configs | proxmox, smom-dbis-138-proxmox |
+| `domain/api` | Backend APIs | dbis_core, the_order |
+
+### 2.3 Label Format Convention
+
+- **Project labels:** `project/<name>` — groups repos by product/project
+- **Domain labels:** `domain/<name>` — groups repos by technical domain
+- Use lowercase, hyphens for multi-word: `project/sankofa-phoenix`
+
+### 2.4 Applying Labels
+
+1. **Create org-level labels:** Run `GITEA_TOKEN=xxx bash scripts/dev-vm/apply-gitea-labels.sh` (includes `project/virtual-banker`).
+2. Repo Settings → Labels — add relevant `project/*` and `domain/*` labels to each repo (e.g. **virtual-banker**: `project/virtual-banker`, `domain/api`).
+3. Filter repos by label in Gitea UI or API
+
+---
+
+## 3. Team Naming Conventions
+
+### 3.1 Format
+
+`org-d-bis/team-<role>` or `org-d-bis/team-<project>-<role>`
+
+### 3.2 Example Teams
+
+| Team Name | Purpose |
+|-----------|---------|
+| `team-frontend` | Frontend developers |
+| `team-backend` | Backend developers |
+| `team-infra` | Infrastructure, DevOps |
+| `team-sankofa` | Sankofa/Phoenix maintainers |
+| `team-blockchain` | Chain 138, Besu, RPC maintainers |
+
+### 3.3 Permissions
+
+- **Owners:** Full org control
+- **Admins:** Repo admin, branch protection
+- **Writers:** Push, PR merge
+- **Readers:** Pull, read-only
+
+---
+
+## 4. Target Hierarchy (Design)
+
+Planned virtual structure for filtering and reporting:
+
+```
+d-bis (org)
+├── project/sankofa
+│   ├── sankofa (main repo)
+│   └── as4-411 (marketplace submodule)
+├── project/omnis
+│   └── omnis
+├── project/proxmox
+│   └── proxmox
+├── project/chain138
+│   ├── smom-dbis-138
+│   ├── rpc-translator-138
+│   └── smom-dbis-138-proxmox
+├── project/explorer
+│   └── explorer-monorepo
+├── project/virtual-banker
+│   └── virtual-banker
+└── domain/*
+    └── (repos tagged by domain)
+```
+
+---
+
+## 5. Option B: Multiple Orgs (Medium Term)
+
+If separation is needed:
+
+| Org | Purpose | Repos |
+|-----|---------|-------|
+| `d-bis` | Main, shared | proxmox, dbis_core |
+| `d-bis-sankofa` | Sankofa Phoenix | sankofa, as4-411 |
+| `d-bis-infra` | Infrastructure | smom-dbis-138-proxmox, configs |
+| `d-bis-apps` | Applications | omnis, miracles_in_motion, the_order |
+
+Requires Keycloak/SAML/LDAP for cross-org identity.
+
+---
+
+## 6. Option C: Subgroups (Long Term)
+
+Track Forgejo/Gitea PR [#35295](https://github.com/go-gitea/gitea/pull/35295) for native subgroups. Target hierarchy:
+
+- `d-bis / sankofa / marketplace`
+- `d-bis / blockchain / chain138`
+- `d-bis / apps / omnis`
+
+---
+
+## 7. References
+
+- [DEV_VM_GITOPS_PLAN.md](DEV_VM_GITOPS_PLAN.md) — Dev VM and Gitea setup
+- [AS4_411_PHOENIX_SUBMODULE_AND_PUSH_ALL.md](AS4_411_PHOENIX_SUBMODULE_AND_PUSH_ALL.md) — Sankofa marketplace
+- [Gitea Organizations](https://docs.gitea.com/usage/organizations)
+- [Gitea Labels](https://docs.gitea.com/usage/labels)
--- a/docs/04-configuration/PHOENIX_TTS_API_CONTRACT.md
+++ b/docs/04-configuration/PHOENIX_TTS_API_CONTRACT.md
@@ -0,0 +1,61 @@
+# Phoenix TTS API contract (ElevenLabs-compatible)
+
+**Last Updated:** 2026-02-10  
+**Purpose:** So virtual-banker (and other apps) can “just change endpoint” from ElevenLabs to a Phoenix-hosted TTS service.
+
+---
+
+## Required endpoints
+
+The Phoenix TTS service **must** implement the same HTTP contract as ElevenLabs for these paths (base path is the app’s `/tts` or similar; below uses prefix `/v1`).
+
+### 1. Sync text-to-speech
+
+- **Method:** `POST`
+- **Path:** `/v1/text-to-speech/:voice_id`
+- **Headers:**
+  - `Content-Type: application/json`
+  - `Accept: audio/mpeg`
+  - Auth: either `xi-api-key: <key>` or `Authorization: Bearer <token>` (configurable in client)
+- **Body (JSON):**
+  ```json
+  {
+    "text": "Hello world",
+    "model_id": "eleven_multilingual_v2",
+    "voice_settings": {
+      "stability": 0.5,
+      "similarity_boost": 0.75,
+      "style": 0,
+      "use_speaker_boost": true
+    }
+  }
+  ```
+- **Response:** `200 OK`, body = raw **mp3** bytes (`audio/mpeg`).
+
+### 2. Streaming text-to-speech
+
+- **Method:** `POST`
+- **Path:** `/v1/text-to-speech/:voice_id/stream`
+- **Headers:** Same as sync.
+- **Body:** Same JSON as sync.
+- **Response:** `200 OK`, body = **streaming** mp3 (same format).
+
+### 3. Health (recommended)
+
+- **Method:** `GET`
+- **Path:** `/health` (at same origin as the TTS base URL, e.g. `https://phoenix.example.com/tts/health` if base is `.../tts/v1`)
+- **Response:** `200 OK` (body optional; used for readiness).
+
+---
+
+## Optional
+
+- **Auth:** If Phoenix uses a different scheme (e.g. Bearer only), clients set `TTS_AUTH_HEADER_NAME` / `TTS_AUTH_HEADER_VALUE`; no API change.
+- **Visemes:** For better lip-sync, a future endpoint could return phoneme/viseme timings; client would call it when available.
+
+---
+
+## Reference
+
+- Virtual-banker TTS client: `virtual-banker/backend/tts` (see `backend/tts/README.md`).
+- ElevenLabs TTS API: [Text-to-speech](https://elevenlabs.io/docs/api-reference/text-to-speech), [Stream](https://elevenlabs.io/docs/api-reference/text-to-speech/stream).
--- a/docs/04-configuration/RECOMMENDATIONS_TTS_GITEA_PHOENIX.md
+++ b/docs/04-configuration/RECOMMENDATIONS_TTS_GITEA_PHOENIX.md
@@ -0,0 +1,91 @@
+# Recommendations: TTS, Gitea, Phoenix, and Operations
+
+**Last Updated:** 2026-02-10  
+**Scope:** virtual-banker TTS, Phoenix deployment, Gitea, and related operations.
+
+---
+
+## 1. TTS and Phoenix
+
+### Implemented
+
+- **Configurable base URL** — Use `TTS_BASE_URL` (or `PHOENIX_TTS_BASE_URL` with `USE_PHOENIX_TTS=true`) to point at Phoenix; no code change.
+- **Optional auth header** — `TTS_AUTH_HEADER_NAME` / `TTS_AUTH_HEADER_VALUE` for Phoenix Bearer (or other) auth.
+- **Health check** — `tts.Service.Health(ctx)` calls `GET {baseURL}/../health` when not using ElevenLabs; wire into readiness.
+- **Decision table** — In the virtual-banker repo see `backend/tts/README.md` for which backend is used given env.
+- **Phoenix API contract** — [PHOENIX_TTS_API_CONTRACT.md](PHOENIX_TTS_API_CONTRACT.md) defines required endpoints (sync, stream, health).
+
+### Additional recommendations
+
+- **Rate limiting / cost** — When using ElevenLabs, track usage (chars or requests) and consider rate limiting in-app or in Phoenix to avoid cost overruns.
+- **Timeouts and resilience** — Client already uses 30s timeout and retries for sync. Consider a **circuit breaker** (e.g. after N failures, stop calling TTS for a cooldown) to avoid cascading failures.
+- **Structured logging** — Log TTS provider (ElevenLabs vs Phoenix), latency, and errors (without logging `TTS_API_KEY` or auth values) for debugging and monitoring.
+- **API versioning** — Phoenix TTS should version the path (e.g. `/v1/...`) so you can add `/v2` later without breaking existing clients.
+- **Fallback** — Optional: if Phoenix is down, fall back to ElevenLabs when both are configured (e.g. try Phoenix first, on failure or health error use ElevenLabs). Requires explicit design (env flags, health checks).
+- **Secrets** — Never log or expose `TTS_API_KEY` / `ELEVENLABS_API_KEY` / `TTS_AUTH_HEADER_VALUE`. In production use a secret manager (e.g. Phoenix env, Vault) and inject into env.
+
+---
+
+## 2. Gitea and repos
+
+### Implemented
+
+- **Push all projects** — From proxmox root: `bash scripts/dev-vm/push-all-projects-to-gitea.sh`; includes virtual-banker. See [EXPLORER_MONOREPO_GITEA_PUSH.md](EXPLORER_MONOREPO_GITEA_PUSH.md).
+- **Labels** — `apply-gitea-labels.sh` creates `project/virtual-banker` (and others). Apply `project/virtual-banker` and `domain/api` to the virtual-banker repo in Gitea (Settings → Labels).
+- **virtual-banker CI** — `.gitea/workflows/ci.yml` in virtual-banker runs `go build` and `go test` on push/PR.
+
+### Additional recommendations
+
+- **Branch protection** — Enable branch protection for `master`/`main` on virtual-banker (and other key repos); require PR and passing CI. See [GITEA_BRANCH_PROTECTION.md](GITEA_BRANCH_PROTECTION.md) if present.
+- **Renovate / Dependabot** — Consider Renovate or similar for virtual-banker (Go deps) so security and version updates are automated. See [RENOVATE_GITEA_SETUP.md](RENOVATE_GITEA_SETUP.md) if present.
+
+---
+
+## 3. virtual-banker deployment and config
+
+### Implemented
+
+- **.env.example** — virtual-banker root has `.env.example` with `TTS_*`, `ELEVENLABS_*`, `USE_PHOENIX_TTS`, `DATABASE_URL`, `REDIS_URL`, `PORT`. Copy to `.env` and set values; do not commit `.env`.
+- **Feature flag** — `USE_PHOENIX_TTS=true` (or `1`) forces Phoenix TTS; use `TTS_BASE_URL` or `PHOENIX_TTS_BASE_URL`.
+
+### Additional recommendations
+
+- **Readiness endpoint** — Expose a `/ready` (or extend `/health`) that calls `tts.Service.Health(ctx)` when TTS is configured, so Kubernetes or load balancers only send traffic when TTS backend is up.
+- **Monitoring** — Track TTS latency (e.g. existing `RecordTTSLatency`), error rate, and provider (ElevenLabs vs Phoenix) in metrics/dashboards so you can detect regressions and compare backends.
+
+---
+
+## 4. Reuse: TTS as a standalone module
+
+### Implemented
+
+- **Same interface, swap endpoint** — Other projects can depend on virtual-banker’s `backend/tts` package and use `NewElevenLabsTTSServiceWithOptions` / `NewElevenLabsTTSServiceWithOptionsFull` with any base URL and auth.
+
+### Additional recommendations
+
+- **Standalone tts-client repo** — If multiple repos (outside virtual-banker) will use this client, consider extracting the `tts` package into a small Go module (e.g. `github.com/d-bis/tts-client`). virtual-banker would then depend on that module, and others can use it without pulling in virtual-banker. Reduces coupling and clarifies ownership.
+- **Visemes** — Current visemes are client-side approximation. For better lip-sync, add a Phoenix endpoint (or separate service) that returns phoneme/viseme timings and extend the client to call it when `TTS_BASE_URL` is set.
+
+---
+
+## 5. Quick reference
+
+| Topic | Doc / location |
+|-------|-----------------|
+| Which TTS backend? | virtual-banker `backend/tts/README.md` (decision table) |
+| Phoenix TTS API contract | [PHOENIX_TTS_API_CONTRACT.md](PHOENIX_TTS_API_CONTRACT.md) |
+| Push all projects to Gitea | [EXPLORER_MONOREPO_GITEA_PUSH.md](EXPLORER_MONOREPO_GITEA_PUSH.md) |
+| Gitea labels and structure | [GITEA_ORG_STRUCTURE.md](GITEA_ORG_STRUCTURE.md) |
+| virtual-banker env vars | virtual-banker repo: `.env.example` |
+| virtual-banker CI | virtual-banker repo: `.gitea/workflows/ci.yml` |
+
+---
+
+## 6. Checklist (optional)
+
+- [ ] Apply Gitea labels to virtual-banker (`project/virtual-banker`, `domain/api`) after running `apply-gitea-labels.sh`
+- [ ] Configure Phoenix TTS to implement the contract in PHOENIX_TTS_API_CONTRACT.md and expose `GET /health`
+- [ ] Set TTS env (or secrets) in Phoenix/deployment so virtual-banker uses Phoenix when desired
+- [ ] Wire `tts.Health()` into readiness probe or `/ready` if you need TTS-up before accepting traffic
+- [ ] Add monitoring for TTS latency and errors
+- [ ] Consider extracting `tts` into a standalone Go module if multiple services will depend on it
--- a/scripts/dev-vm/apply-gitea-labels.sh
+++ b/scripts/dev-vm/apply-gitea-labels.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Create org-level labels and optionally apply to repos per GITEA_ORG_STRUCTURE.md
+# Usage: GITEA_TOKEN=xxx bash apply-gitea-labels.sh [--apply-to-repos]
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+[ -f "$PROJECT_ROOT/.env" ] && set +u && source "$PROJECT_ROOT/.env" 2>/dev/null || true && set -u
+
+GITEA_URL="${GITEA_URL:-https://gitea.d-bis.org}"
+GITEA_TOKEN="${GITEA_TOKEN:-}"
+GITEA_ORG="${GITEA_ORG:-d-bis}"
+APPLY_TO_REPOS=false
+[[ "${1:-}" == "--apply-to-repos" ]] && APPLY_TO_REPOS=true
+
+# Labels from GITEA_ORG_STRUCTURE.md (project/domain)
+LABELS=(
+  "project/sankofa:#7057ff"
+  "project/omnis:#cfd3d7"
+  "project/proxmox:#bfd4f2"
+  "project/dbis_core:#fef2c0"
+  "project/chain138:#0e8a16"
+  "project/explorer:#fbca04"
+  "project/miracles_in_motion:#d93f0b"
+  "project/the_order:#c2e0c6"
+  "project/virtual-banker:#c2e0c6"
+  "domain/blockchain:#1d76db"
+  "domain/web:#5319e7"
+  "domain/infrastructure:#0e8a16"
+  "domain/api:#c5def5"
+)
+
+if [ -z "$GITEA_TOKEN" ]; then
+  echo "Set GITEA_TOKEN"
+  exit 1
+fi
+
+API="${GITEA_URL%/}/api/v1"
+AUTH="Authorization: token $GITEA_TOKEN"
+
+# Create org-level labels (Gitea API: POST /orgs/{org}/labels)
+for entry in "${LABELS[@]}"; do
+  name="${entry%%:*}"
+  color="${entry##*:}"
+  color="${color#\#}"
+  echo "Creating label: $name"
+  curl -sSf -X POST "$API/orgs/$GITEA_ORG/labels" \
+    -H "$AUTH" -H "Content-Type: application/json" \
+    -d "{\"name\":\"$name\",\"color\":\"$color\"}" 2>/dev/null || echo "  (may already exist)"
+done
+
+echo "Labels created. Add them to repos via Gitea UI: Repo → Settings → Labels"