diff --git a/docs/04-configuration/EXPLORER_MONOREPO_GITEA_PUSH.md b/docs/04-configuration/EXPLORER_MONOREPO_GITEA_PUSH.md new file mode 100644 index 0000000..31c50ad --- /dev/null +++ b/docs/04-configuration/EXPLORER_MONOREPO_GITEA_PUSH.md @@ -0,0 +1,75 @@ +# Explorer Monorepo — Gitea Push + +**Last Updated:** 2026-02-10 +**Status:** Explorer monorepo is pushed to Gitea. **virtual-banker** is not part of and not referenced by explorer-monorepo; it lives as a separate repo in the projects directory (`/home/intlc/projects/virtual-banker`) and on Gitea as [d-bis/virtual-banker](https://gitea.d-bis.org/d-bis/virtual-banker). + +--- + +## Push all projects (including virtual-banker) + +To push **every** repo under the projects directory (including **virtual-banker**, **proxmox**, **explorer-monorepo** only via proxmox, etc.) to Gitea in one go, from the **proxmox** repo root: + +```bash +cd /path/to/proxmox +# GITEA_TOKEN from .env or export +bash scripts/dev-vm/push-all-projects-to-gitea.sh +``` + +- **Discovered repos:** Direct children of `PROJECTS_DIR` (default `/home/intlc/projects`) that have a `.git` directory. +- **virtual-banker** is included because it lives at `/home/intlc/projects/virtual-banker`. +- **explorer-monorepo** is **not** a direct child; push it separately (see “How to push explorer-monorepo” below) if needed. + +--- + +## Why explorer-monorepo isn’t pushed by the usual script + +- **explorer-monorepo** is a **submodule** of the proxmox repo (`explorer-monorepo` → `https://github.com/Order-of-Hospitallers/chain-138-explorer.git`). +- **push-all-projects-to-gitea.sh** only considers **direct children** of `PROJECTS_DIR` (default `/home/intlc/projects`), i.e. sibling repos like `proxmox`, `dbis_core`, not subdirectories/submodules inside proxmox. +- So explorer-monorepo is never discovered or pushed by that script. + +--- + +## How to push explorer-monorepo to Gitea + +### 1. Create the repo on Gitea (if it doesn’t exist) + +From the proxmox repo root, using the same token and org as elsewhere: + +```bash +GITEA_TOKEN=xxx bash scripts/dev-vm/gitea-create-orgs-and-repos.sh --dry-run # optional: check list +GITEA_TOKEN=xxx bash scripts/dev-vm/gitea-create-orgs-and-repos.sh # creates d-bis/explorer-monorepo if missing +``` + +(`explorer-monorepo` is already in the default `REPO_NAMES` in that script.) + +### 2. Add Gitea remote and push from the submodule + +From the **explorer-monorepo** directory (submodule has its own `.git`): + +```bash +cd /home/intlc/projects/proxmox/explorer-monorepo + +# Optional: load Gitea env from proxmox root +[ -f ../.env ] && set -a && source ../.env && set +a + +# Add Gitea remote if not already present +git remote get-url gitea 2>/dev/null || \ + git remote add gitea "https://gitea.d-bis.org/d-bis/explorer-monorepo.git" + +# Push current branch (e.g. main) +BRANCH=$(git branch --show-current) +git push "https://${GITEA_TOKEN}@gitea.d-bis.org/d-bis/explorer-monorepo.git" "$BRANCH" --set-upstream +``` + +Use the same **GITEA_TOKEN** as for other Gitea operations (e.g. from proxmox `.env` or Gitea → Settings → Applications). + +### 3. If you get HTTP 413 on push + +If the repo is large and the push fails with **HTTP 413**, follow [GITEA_LARGE_PUSH_HTTP_413.md](GITEA_LARGE_PUSH_HTTP_413.md) to raise `HTTP_MAX_REQUEST_BODY` in Gitea and `client_max_body_size` in NPMplus for `gitea.d-bis.org`. + +--- + +## After pushing + +- In Gitea, add labels per [GITEA_ORG_STRUCTURE.md](GITEA_ORG_STRUCTURE.md): e.g. `project/explorer`, `domain/blockchain` / `domain/web` as appropriate. +- To make the proxmox repo point at Gitea for this submodule (optional): update `.gitmodules` and submodule remote to `https://gitea.d-bis.org/d-bis/explorer-monorepo.git` and run `git submodule sync`. diff --git a/docs/04-configuration/GITEA_ORG_STRUCTURE.md b/docs/04-configuration/GITEA_ORG_STRUCTURE.md new file mode 100644 index 0000000..8107f7d --- /dev/null +++ b/docs/04-configuration/GITEA_ORG_STRUCTURE.md @@ -0,0 +1,139 @@ +# Gitea Organizational Structure and Conventions + +**Last Updated:** 2026-02-10 +**Status:** Active Documentation +**Gitea Instance:** https://gitea.d-bis.org +**Primary Org:** d-bis + +--- + +## 1. Overview + +Gitea uses a flat model: **Organization → Teams → Users**. This document defines conventions for virtual hierarchy via labels, team naming, and repository organization to support flexible structuring beyond the built-in flat model. + +--- + +## 2. Label-Based Virtual Hierarchy (Option A) + +Use repository labels to create virtual groups without code changes. + +### 2.1 Project Labels + +| Label | Purpose | Example Repos | +|-------|---------|---------------| +| `project/sankofa` | Sankofa Phoenix platform and marketplace | sankofa, as4-411 | +| `project/omnis` | OMNIS application | omnis | +| `project/proxmox` | Proxmox infrastructure and docs | proxmox | +| `project/dbis_core` | DBIS core libraries and services | dbis_core | +| `project/chain138` | Chain 138 blockchain (Besu, RPC, etc.) | smom-dbis-138, rpc-translator-138 | +| `project/explorer` | Blockscout, explorers | explorer-monorepo | +| `project/miracles_in_motion` | MIM web and API | miracles_in_motion | +| `project/the_order` | Order service | the_order | +| `project/virtual-banker` | Virtual banker (voice/TTS, widget) | virtual-banker | + +### 2.2 Domain Labels + +| Label | Purpose | Example Repos | +|-------|---------|---------------| +| `domain/blockchain` | Blockchain, smart contracts, RPC | smom-dbis-138, alltra-lifi-settlement | +| `domain/web` | Web apps, portals, frontends | miracles_in_motion, omnis | +| `domain/infrastructure` | Proxmox, scripts, configs | proxmox, smom-dbis-138-proxmox | +| `domain/api` | Backend APIs | dbis_core, the_order | + +### 2.3 Label Format Convention + +- **Project labels:** `project/` — groups repos by product/project +- **Domain labels:** `domain/` — groups repos by technical domain +- Use lowercase, hyphens for multi-word: `project/sankofa-phoenix` + +### 2.4 Applying Labels + +1. **Create org-level labels:** Run `GITEA_TOKEN=xxx bash scripts/dev-vm/apply-gitea-labels.sh` (includes `project/virtual-banker`). +2. Repo Settings → Labels — add relevant `project/*` and `domain/*` labels to each repo (e.g. **virtual-banker**: `project/virtual-banker`, `domain/api`). +3. Filter repos by label in Gitea UI or API + +--- + +## 3. Team Naming Conventions + +### 3.1 Format + +`org-d-bis/team-` or `org-d-bis/team--` + +### 3.2 Example Teams + +| Team Name | Purpose | +|-----------|---------| +| `team-frontend` | Frontend developers | +| `team-backend` | Backend developers | +| `team-infra` | Infrastructure, DevOps | +| `team-sankofa` | Sankofa/Phoenix maintainers | +| `team-blockchain` | Chain 138, Besu, RPC maintainers | + +### 3.3 Permissions + +- **Owners:** Full org control +- **Admins:** Repo admin, branch protection +- **Writers:** Push, PR merge +- **Readers:** Pull, read-only + +--- + +## 4. Target Hierarchy (Design) + +Planned virtual structure for filtering and reporting: + +``` +d-bis (org) +├── project/sankofa +│ ├── sankofa (main repo) +│ └── as4-411 (marketplace submodule) +├── project/omnis +│ └── omnis +├── project/proxmox +│ └── proxmox +├── project/chain138 +│ ├── smom-dbis-138 +│ ├── rpc-translator-138 +│ └── smom-dbis-138-proxmox +├── project/explorer +│ └── explorer-monorepo +├── project/virtual-banker +│ └── virtual-banker +└── domain/* + └── (repos tagged by domain) +``` + +--- + +## 5. Option B: Multiple Orgs (Medium Term) + +If separation is needed: + +| Org | Purpose | Repos | +|-----|---------|-------| +| `d-bis` | Main, shared | proxmox, dbis_core | +| `d-bis-sankofa` | Sankofa Phoenix | sankofa, as4-411 | +| `d-bis-infra` | Infrastructure | smom-dbis-138-proxmox, configs | +| `d-bis-apps` | Applications | omnis, miracles_in_motion, the_order | + +Requires Keycloak/SAML/LDAP for cross-org identity. + +--- + +## 6. Option C: Subgroups (Long Term) + +Track Forgejo/Gitea PR [#35295](https://github.com/go-gitea/gitea/pull/35295) for native subgroups. Target hierarchy: + +- `d-bis / sankofa / marketplace` +- `d-bis / blockchain / chain138` +- `d-bis / apps / omnis` + +--- + +## 7. References + +- [DEV_VM_GITOPS_PLAN.md](DEV_VM_GITOPS_PLAN.md) — Dev VM and Gitea setup +- [AS4_411_PHOENIX_SUBMODULE_AND_PUSH_ALL.md](AS4_411_PHOENIX_SUBMODULE_AND_PUSH_ALL.md) — Sankofa marketplace +- [Gitea Organizations](https://docs.gitea.com/usage/organizations) +- [Gitea Labels](https://docs.gitea.com/usage/labels) diff --git a/docs/04-configuration/PHOENIX_TTS_API_CONTRACT.md b/docs/04-configuration/PHOENIX_TTS_API_CONTRACT.md new file mode 100644 index 0000000..6c67834 --- /dev/null +++ b/docs/04-configuration/PHOENIX_TTS_API_CONTRACT.md @@ -0,0 +1,61 @@ +# Phoenix TTS API contract (ElevenLabs-compatible) + +**Last Updated:** 2026-02-10 +**Purpose:** So virtual-banker (and other apps) can “just change endpoint” from ElevenLabs to a Phoenix-hosted TTS service. + +--- + +## Required endpoints + +The Phoenix TTS service **must** implement the same HTTP contract as ElevenLabs for these paths (base path is the app’s `/tts` or similar; below uses prefix `/v1`). + +### 1. Sync text-to-speech + +- **Method:** `POST` +- **Path:** `/v1/text-to-speech/:voice_id` +- **Headers:** + - `Content-Type: application/json` + - `Accept: audio/mpeg` + - Auth: either `xi-api-key: ` or `Authorization: Bearer ` (configurable in client) +- **Body (JSON):** + ```json + { + "text": "Hello world", + "model_id": "eleven_multilingual_v2", + "voice_settings": { + "stability": 0.5, + "similarity_boost": 0.75, + "style": 0, + "use_speaker_boost": true + } + } + ``` +- **Response:** `200 OK`, body = raw **mp3** bytes (`audio/mpeg`). + +### 2. Streaming text-to-speech + +- **Method:** `POST` +- **Path:** `/v1/text-to-speech/:voice_id/stream` +- **Headers:** Same as sync. +- **Body:** Same JSON as sync. +- **Response:** `200 OK`, body = **streaming** mp3 (same format). + +### 3. Health (recommended) + +- **Method:** `GET` +- **Path:** `/health` (at same origin as the TTS base URL, e.g. `https://phoenix.example.com/tts/health` if base is `.../tts/v1`) +- **Response:** `200 OK` (body optional; used for readiness). + +--- + +## Optional + +- **Auth:** If Phoenix uses a different scheme (e.g. Bearer only), clients set `TTS_AUTH_HEADER_NAME` / `TTS_AUTH_HEADER_VALUE`; no API change. +- **Visemes:** For better lip-sync, a future endpoint could return phoneme/viseme timings; client would call it when available. + +--- + +## Reference + +- Virtual-banker TTS client: `virtual-banker/backend/tts` (see `backend/tts/README.md`). +- ElevenLabs TTS API: [Text-to-speech](https://elevenlabs.io/docs/api-reference/text-to-speech), [Stream](https://elevenlabs.io/docs/api-reference/text-to-speech/stream). diff --git a/docs/04-configuration/RECOMMENDATIONS_TTS_GITEA_PHOENIX.md b/docs/04-configuration/RECOMMENDATIONS_TTS_GITEA_PHOENIX.md new file mode 100644 index 0000000..c9736d9 --- /dev/null +++ b/docs/04-configuration/RECOMMENDATIONS_TTS_GITEA_PHOENIX.md @@ -0,0 +1,91 @@ +# Recommendations: TTS, Gitea, Phoenix, and Operations + +**Last Updated:** 2026-02-10 +**Scope:** virtual-banker TTS, Phoenix deployment, Gitea, and related operations. + +--- + +## 1. TTS and Phoenix + +### Implemented + +- **Configurable base URL** — Use `TTS_BASE_URL` (or `PHOENIX_TTS_BASE_URL` with `USE_PHOENIX_TTS=true`) to point at Phoenix; no code change. +- **Optional auth header** — `TTS_AUTH_HEADER_NAME` / `TTS_AUTH_HEADER_VALUE` for Phoenix Bearer (or other) auth. +- **Health check** — `tts.Service.Health(ctx)` calls `GET {baseURL}/../health` when not using ElevenLabs; wire into readiness. +- **Decision table** — In the virtual-banker repo see `backend/tts/README.md` for which backend is used given env. +- **Phoenix API contract** — [PHOENIX_TTS_API_CONTRACT.md](PHOENIX_TTS_API_CONTRACT.md) defines required endpoints (sync, stream, health). + +### Additional recommendations + +- **Rate limiting / cost** — When using ElevenLabs, track usage (chars or requests) and consider rate limiting in-app or in Phoenix to avoid cost overruns. +- **Timeouts and resilience** — Client already uses 30s timeout and retries for sync. Consider a **circuit breaker** (e.g. after N failures, stop calling TTS for a cooldown) to avoid cascading failures. +- **Structured logging** — Log TTS provider (ElevenLabs vs Phoenix), latency, and errors (without logging `TTS_API_KEY` or auth values) for debugging and monitoring. +- **API versioning** — Phoenix TTS should version the path (e.g. `/v1/...`) so you can add `/v2` later without breaking existing clients. +- **Fallback** — Optional: if Phoenix is down, fall back to ElevenLabs when both are configured (e.g. try Phoenix first, on failure or health error use ElevenLabs). Requires explicit design (env flags, health checks). +- **Secrets** — Never log or expose `TTS_API_KEY` / `ELEVENLABS_API_KEY` / `TTS_AUTH_HEADER_VALUE`. In production use a secret manager (e.g. Phoenix env, Vault) and inject into env. + +--- + +## 2. Gitea and repos + +### Implemented + +- **Push all projects** — From proxmox root: `bash scripts/dev-vm/push-all-projects-to-gitea.sh`; includes virtual-banker. See [EXPLORER_MONOREPO_GITEA_PUSH.md](EXPLORER_MONOREPO_GITEA_PUSH.md). +- **Labels** — `apply-gitea-labels.sh` creates `project/virtual-banker` (and others). Apply `project/virtual-banker` and `domain/api` to the virtual-banker repo in Gitea (Settings → Labels). +- **virtual-banker CI** — `.gitea/workflows/ci.yml` in virtual-banker runs `go build` and `go test` on push/PR. + +### Additional recommendations + +- **Branch protection** — Enable branch protection for `master`/`main` on virtual-banker (and other key repos); require PR and passing CI. See [GITEA_BRANCH_PROTECTION.md](GITEA_BRANCH_PROTECTION.md) if present. +- **Renovate / Dependabot** — Consider Renovate or similar for virtual-banker (Go deps) so security and version updates are automated. See [RENOVATE_GITEA_SETUP.md](RENOVATE_GITEA_SETUP.md) if present. + +--- + +## 3. virtual-banker deployment and config + +### Implemented + +- **.env.example** — virtual-banker root has `.env.example` with `TTS_*`, `ELEVENLABS_*`, `USE_PHOENIX_TTS`, `DATABASE_URL`, `REDIS_URL`, `PORT`. Copy to `.env` and set values; do not commit `.env`. +- **Feature flag** — `USE_PHOENIX_TTS=true` (or `1`) forces Phoenix TTS; use `TTS_BASE_URL` or `PHOENIX_TTS_BASE_URL`. + +### Additional recommendations + +- **Readiness endpoint** — Expose a `/ready` (or extend `/health`) that calls `tts.Service.Health(ctx)` when TTS is configured, so Kubernetes or load balancers only send traffic when TTS backend is up. +- **Monitoring** — Track TTS latency (e.g. existing `RecordTTSLatency`), error rate, and provider (ElevenLabs vs Phoenix) in metrics/dashboards so you can detect regressions and compare backends. + +--- + +## 4. Reuse: TTS as a standalone module + +### Implemented + +- **Same interface, swap endpoint** — Other projects can depend on virtual-banker’s `backend/tts` package and use `NewElevenLabsTTSServiceWithOptions` / `NewElevenLabsTTSServiceWithOptionsFull` with any base URL and auth. + +### Additional recommendations + +- **Standalone tts-client repo** — If multiple repos (outside virtual-banker) will use this client, consider extracting the `tts` package into a small Go module (e.g. `github.com/d-bis/tts-client`). virtual-banker would then depend on that module, and others can use it without pulling in virtual-banker. Reduces coupling and clarifies ownership. +- **Visemes** — Current visemes are client-side approximation. For better lip-sync, add a Phoenix endpoint (or separate service) that returns phoneme/viseme timings and extend the client to call it when `TTS_BASE_URL` is set. + +--- + +## 5. Quick reference + +| Topic | Doc / location | +|-------|-----------------| +| Which TTS backend? | virtual-banker `backend/tts/README.md` (decision table) | +| Phoenix TTS API contract | [PHOENIX_TTS_API_CONTRACT.md](PHOENIX_TTS_API_CONTRACT.md) | +| Push all projects to Gitea | [EXPLORER_MONOREPO_GITEA_PUSH.md](EXPLORER_MONOREPO_GITEA_PUSH.md) | +| Gitea labels and structure | [GITEA_ORG_STRUCTURE.md](GITEA_ORG_STRUCTURE.md) | +| virtual-banker env vars | virtual-banker repo: `.env.example` | +| virtual-banker CI | virtual-banker repo: `.gitea/workflows/ci.yml` | + +--- + +## 6. Checklist (optional) + +- [ ] Apply Gitea labels to virtual-banker (`project/virtual-banker`, `domain/api`) after running `apply-gitea-labels.sh` +- [ ] Configure Phoenix TTS to implement the contract in PHOENIX_TTS_API_CONTRACT.md and expose `GET /health` +- [ ] Set TTS env (or secrets) in Phoenix/deployment so virtual-banker uses Phoenix when desired +- [ ] Wire `tts.Health()` into readiness probe or `/ready` if you need TTS-up before accepting traffic +- [ ] Add monitoring for TTS latency and errors +- [ ] Consider extracting `tts` into a standalone Go module if multiple services will depend on it diff --git a/scripts/dev-vm/apply-gitea-labels.sh b/scripts/dev-vm/apply-gitea-labels.sh new file mode 100755 index 0000000..258513d --- /dev/null +++ b/scripts/dev-vm/apply-gitea-labels.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Create org-level labels and optionally apply to repos per GITEA_ORG_STRUCTURE.md +# Usage: GITEA_TOKEN=xxx bash apply-gitea-labels.sh [--apply-to-repos] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +[ -f "$PROJECT_ROOT/.env" ] && set +u && source "$PROJECT_ROOT/.env" 2>/dev/null || true && set -u + +GITEA_URL="${GITEA_URL:-https://gitea.d-bis.org}" +GITEA_TOKEN="${GITEA_TOKEN:-}" +GITEA_ORG="${GITEA_ORG:-d-bis}" +APPLY_TO_REPOS=false +[[ "${1:-}" == "--apply-to-repos" ]] && APPLY_TO_REPOS=true + +# Labels from GITEA_ORG_STRUCTURE.md (project/domain) +LABELS=( + "project/sankofa:#7057ff" + "project/omnis:#cfd3d7" + "project/proxmox:#bfd4f2" + "project/dbis_core:#fef2c0" + "project/chain138:#0e8a16" + "project/explorer:#fbca04" + "project/miracles_in_motion:#d93f0b" + "project/the_order:#c2e0c6" + "project/virtual-banker:#c2e0c6" + "domain/blockchain:#1d76db" + "domain/web:#5319e7" + "domain/infrastructure:#0e8a16" + "domain/api:#c5def5" +) + +if [ -z "$GITEA_TOKEN" ]; then + echo "Set GITEA_TOKEN" + exit 1 +fi + +API="${GITEA_URL%/}/api/v1" +AUTH="Authorization: token $GITEA_TOKEN" + +# Create org-level labels (Gitea API: POST /orgs/{org}/labels) +for entry in "${LABELS[@]}"; do + name="${entry%%:*}" + color="${entry##*:}" + color="${color#\#}" + echo "Creating label: $name" + curl -sSf -X POST "$API/orgs/$GITEA_ORG/labels" \ + -H "$AUTH" -H "Content-Type: application/json" \ + -d "{\"name\":\"$name\",\"color\":\"$color\"}" 2>/dev/null || echo " (may already exist)" +done + +echo "Labels created. Add them to repos via Gitea UI: Repo → Settings → Labels"