proxmox/AGENTS.md

# Proxmox workspace — agent instructions

Single canonical copy for Cursor/Codex. (If your editor also loads `.cursor/rules`, treat those as overlays.)

## Scope

Orchestration for Proxmox VE, Chain 138 (`smom-dbis-138/`), explorers, NPMplus, and deployment runbooks.

## Quick pointers

| Need | Location |
|------|-----------|
| Doc index | `docs/MASTER_INDEX.md` |
| Chain 138 PMM swap quote (CLI) | `bash scripts/verify/pmm-swap-quote-chain138.sh --token-in … --amount-in …` — on-chain `querySellBase`/`querySellQuote` + suggested `minOut` for `DODOPMMIntegration.swapExactIn` (REST `/quote` is xy=k only). |
| Chain 138 info site (`info.defi-oracle.io`) | Dedicated nginx LXC (default VMID **2410** / `IP_INFO_DEFI_ORACLE_WEB`): `provision-info-defi-oracle-web-lxc.sh` then `sync-info-defi-oracle-to-vmid2400.sh` (sync asserts `/token-aggregation` proxy); NPM fleet `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`; Cloudflare DNS `scripts/cloudflare/set-info-defi-oracle-dns-to-vmid2400-tunnel.sh`; cache `pnpm run cloudflare:purge-info-defi-oracle-cache`; runbook `docs/04-configuration/INFO_DEFI_ORACLE_IO_DEPLOYMENT.md`; `pnpm run verify:info-defi-oracle-public` (SPA routes including `/governance`, `/ecosystem`, `/documentation`, `/solacenet`, `llms.txt`, `agent-hints.json`, **same-origin** token-aggregation JSON; `INFO_SITE_BASE=…` optional); CI `info-defi-oracle-138.yml` (build) and `verify-info-defi-oracle-public.yml` (weekly + manual smoke); optional `pnpm run audit:info-defi-oracle-site` (`pnpm exec playwright install chromium`) |
| **SolaceNet + gateway rails** (dbis_core) | Hub map: `docs/04-configuration/SOLACENET_PUBLIC_HUB.md`. Backlog: `dbis_core/docs/solacenet/REMAINING_TASKS_FULL_LIST.md`. Gap IDs: `dbis_core/docs/solacenet/PROTOCOL_GAPS_CHECKLIST.md`. **Delta audit** (missing wiring, naming drift, CI): `dbis_core/docs/solacenet/AUDIT_GAPS_INCONSISTENCIES_MISSING.md`. Enforce rails runbook: `dbis_core/docs/solacenet/SOLACENET_GATEWAY_RAILS_ENFORCE_RUNBOOK.md`. Tests: `cd dbis_core && npm run test:gateway` (unit + HTTP integration). **Provider seed:** `cd dbis_core && npm run seed:gateway-provider` (needs `DATABASE_URL`). **Smoke (auth):** `bash scripts/verify/check-dbis-core-gateway-rails.sh`. **Outbox worker:** `cd dbis_core && npm run worker:gateway-outbox` (`DATABASE_URL`). CI: `.github/workflows/dbis-core-gateway-ci.yml`. API: `GET/POST /api/v1/gateway/rails*` (optional `SOLACENET_GATEWAY_RAILS_ENFORCE`) — `dbis_core/src/core/gateway/routes/gateway.routes.ts`. |
| cXAUC/cXAUT unit | 1 full token = 1 troy oz Au — `docs/11-references/EXPLORER_TOKEN_LIST_CROSSCHECK.md` (section 5.1) |
| GRU / UTRNF token naming (`c*` vs collateral prefix) | `docs/04-configuration/naming-conventions/README.md`, `docs/04-configuration/naming-conventions/02_DBIS_NAMESPACE_AND_UTRNF_MAPPING.md` |
| PMM mesh 6s tick | `smom-dbis-138/scripts/reserve/pmm-mesh-6s-automation.sh` — `docs/integration/ORACLE_AND_KEEPER_CHAIN138.md` (PMM mesh automation) |
| Mainnet cWUSD\* peg, TRUU PMM, bot readiness | `docs/03-deployment/MAINNET_PMM_TRUU_CWUSD_PEG_AND_BOT_RUNBOOK.md` (§11 live inventory) — `scripts/verify/check-mainnet-pmm-peg-bot-readiness.sh`, `scripts/deployment/deploy-mainnet-pmm-cw-truu-pool.sh`, `scripts/deployment/add-mainnet-truu-pmm-topup.sh`, `scripts/deployment/compute-mainnet-truu-liquidity-amounts.sh`, `scripts/deployment/compute-mainnet-truu-pmm-seed-amounts.sh`; `cross-chain-pmm-lps/config/deployment-status.json` `pmmPoolsVolatile`; `docs/11-references/CONTRACT_ADDRESSES_REFERENCE.md` (Mainnet TRUU PMM); `check-full-deployment-status.sh` when `ETHEREUM_MAINNET_RPC` + `DODO_PMM_INTEGRATION_MAINNET` are set |
| VMID / IP / FQDN | `docs/04-configuration/ALL_VMIDS_ENDPOINTS.md` |
| Proxmox Mail Proxy (LAN SMTP) | VMID **100** `192.168.11.32` (`proxmox-mail-gateway`) — submission **587** / **465**; see Mail Proxy note in `ALL_VMIDS_ENDPOINTS.md` |
| Spare R630 storage + optional tune-up | `scripts/proxmox/ensure-r630-spare-node-storage.sh`, `scripts/proxmox/provision-r630-03-six-ssd-thinpools.sh`, `scripts/proxmox/pve-spare-host-optional-tuneup.sh` · load balance / migrate: `docs/04-configuration/PROXMOX_LOAD_BALANCING_RUNBOOK.md` |
| Ops template + JSON | `docs/03-deployment/PROXMOX_VE_OPERATIONAL_DEPLOYMENT_TEMPLATE.md`, `config/proxmox-operational-template.json` (`proxmox_nodes[].mgmt_fqdn` = `*.sankofa.nexus`; `config/ip-addresses.conf` `PROXMOX_FQDN_*`) |
| Live vs template (read-only SSH) | `bash scripts/verify/audit-proxmox-operational-template.sh` — defaults to ML110 + **r630-01..04** (`PROXMOX_HOSTS` overrides) |
| Proxmox mgmt FQDN DNS + `/etc/hosts` snippet | `bash scripts/verify/check-proxmox-mgmt-fqdn.sh` (`--print-hosts`, optional `--ssh`) |
| Proxmox SSH check (all 5 nodes) | `bash scripts/security/ensure-proxmox-ssh-access.sh` (`--fqdn`, optional `--copy` for `ssh-copy-id`) |
| Proxmox cluster hardware poll (LAN, key SSH) | `bash scripts/verify/poll-proxmox-cluster-hardware.sh` — writes `reports/status/hardware_poll_*.txt`; companion narrative + ARP/edge: `reports/status/hardware_and_connected_inventory_*.md` |
| IT live inventory + IPAM drift (LAN, Phase 0) | `bash scripts/it-ops/export-live-inventory-and-drift.sh` → `reports/status/live_inventory.json`, `drift.json` (exit **2** only if duplicate guest IPs; merges `ip-addresses.conf` + `ALL_VMIDS_ENDPOINTS.md`). [SANKOFA_IT_OPS_LIVE_INVENTORY_SCRIPTS.md](docs/03-deployment/SANKOFA_IT_OPS_LIVE_INVENTORY_SCRIPTS.md). Spec: [SANKOFA_IT_OPERATIONS_CONTROLLER_SPEC.md](docs/02-architecture/SANKOFA_IT_OPERATIONS_CONTROLLER_SPEC.md) |
| IT inventory read API (Phase 0 stub) | `python3 services/sankofa-it-read-api/server.py` — GET `/health`, `/v1/inventory/live`, `/v1/inventory/drift`; optional `IT_READ_API_KEY` + `X-API-Key`; optional `IT_READ_API_CORS_ORIGINS` (comma-separated). [services/sankofa-it-read-api/README.md](services/sankofa-it-read-api/README.md), systemd [config/systemd/sankofa-it-read-api.service.example](config/systemd/sankofa-it-read-api.service.example) |
| **IT read API LAN bootstrap** | `bash scripts/deployment/bootstrap-sankofa-it-read-api-lan.sh` — rsync → `/opt/proxmox` on seed PVE, systemd + `/etc/sankofa-it-read-api.env`, repo `.env` + portal CT 7801 merge, weekly export timer on PVE. NPM: [upsert-it-read-api-proxy-host.sh](scripts/nginx-proxy-manager/upsert-it-read-api-proxy-host.sh); DNS: [add-it-api-sankofa-dns.sh](scripts/cloudflare/add-it-api-sankofa-dns.sh). [SANKOFA_IT_OPS_KEYCLOAK_PORTAL_NEXT_STEPS.md](docs/03-deployment/SANKOFA_IT_OPS_KEYCLOAK_PORTAL_NEXT_STEPS.md) |
| Keycloak realm role for portal `/it` | `bash scripts/deployment/keycloak-sankofa-ensure-it-admin-role.sh` (CT 7802 via SSH); assign `sankofa-it-admin` to IT users. Portal: `IT_READ_API_URL` + optional `IT_READ_API_KEY` on CT 7801. Weekly export timer: [config/systemd/sankofa-it-inventory-export.timer.example](config/systemd/sankofa-it-inventory-export.timer.example) |
| IT admin UI next steps (Keycloak + portal `/it`) | [docs/03-deployment/SANKOFA_IT_OPS_KEYCLOAK_PORTAL_NEXT_STEPS.md](docs/03-deployment/SANKOFA_IT_OPS_KEYCLOAK_PORTAL_NEXT_STEPS.md) |
| Config validation | `bash scripts/validation/validate-config-files.sh` (optional: `python3 -m pip install check-jsonschema` for `validate-dbis-institutional-schemas.sh`, `validate-naming-convention-registry-examples.sh`, `validate-jvmtm-regulatory-closure-schemas.sh`, `validate-reserve-provenance-package.sh`; includes explorer Chain 138 inventory vs `config/smart-contracts-master.json`) |
| Chain 138 contract addresses (JSON + bytecode) | `config/smart-contracts-master.json` — `bash scripts/verify/check-contracts-on-chain-138.sh` (expect **75/75** when Core RPC reachable; jq uses JSON when file present) |
| OMNL + Core + Chain 138 + RTGS + Smart Vaults | `docs/03-deployment/OMNL_DBIS_CORE_CHAIN138_SMART_VAULT_RTGS_RUNBOOK.md`; identifiers (UETR vs DLT-primary): `docs/03-deployment/OJK_BI_AUDIT_JVMTM_REMEDIATION_AND_UETR_POLICY.md`; JVMTM Tables B/C/D closure matrix: `config/jvmtm-regulatory-closure/INAAUDJVMTM_2025_AUDIT_CLOSURE_MATRIX.md`; **dual-anchor attestation:** `scripts/omnl/omnl-chain138-attestation-tx.sh` (138 + optional mainnet via `ETHEREUM_MAINNET_RPC`); E2E zip: `AUDIT_PROOF.json` `chainAttestationMainnet`; machine-readable: `config/dbis-institutional/` |
| Blockscout address labels from registry | `bash scripts/verify/sync-blockscout-address-labels-from-registry.sh` (plan); `--apply` with `BLOCKSCOUT_*` env when explorer API confirmed |
| ISO-20022 on-chain methodology + intake gateway | `docs/04-configuration/SMART_CONTRACTS_ISO20022_FIN_METHODOLOGY.md`, `ISO20022_INTAKE_GATEWAY_CONTRACT_MULTI_NETWORK.md`; Rail: `docs/dbis-rail/ISO_GATEWAY_AND_RELAYER_SPEC.md` |
| FQDN / NPM E2E verifier | `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` — inventory: `docs/04-configuration/E2E_ENDPOINTS_LIST.md`. Gitea Actions URLs (no API): `bash scripts/verify/print-gitea-actions-urls.sh` |
| **Gitea** (org forge **VMID 104**, upgrades, NPM) | `docs/04-configuration/GITEA_PLATFORM_AND_UPGRADE_RUNBOOK.md` — `scripts/operator/upgrade-gitea-lxc.sh` (`--dry-run`, `GITEA_VERSION=`); `config/ip-addresses.conf` **`IP_GITEA_INFRA`**, **`GITEA_PUBLIC_UPSTREAM_*`**; `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`, `update-npmplus-fourth-proxy-hosts.sh` |
| Chain 138 LAN RPC health + nonce/gas parity | `bash scripts/verify/check-chain138-rpc-health.sh` (fleet + public capability); `bash scripts/verify/check-chain138-rpc-nonce-gas-parity.sh` (LAN: aligned chainId / deployer nonces / gasPrice); offline/CI: `bash scripts/verify/self-test-chain138-rpc-verify.sh`; shared VMID list: `scripts/lib/chain138-lan-rpc-inventory.sh` |
| RPC FQDN batch (`eth_chainId` + WSS) | `bash scripts/verify/check-rpc-fqdns-e2e.sh` — after DNS + `update-npmplus-proxy-hosts-api.sh`; includes `rpc-core.d-bis.org` |
| Submodule trees clean (CI / post-merge) | `bash scripts/verify/submodules-clean.sh` |
| Submodule + explorer remotes | `docs/00-meta/SUBMODULE_HYGIENE.md` |
| smom-dbis-138 `.env` in bash scripts | Prefer `source smom-dbis-138/scripts/lib/deployment/dotenv.sh` + `load_deployment_env --repo-root "$PROJECT_ROOT"` (trims RPC URL line endings). From an interactive shell: `source smom-dbis-138/scripts/load-env.sh`. Proxmox root scripts: `source scripts/lib/load-project-env.sh` (also trims common RPC vars). |
| Sankofa portal → CT 7801 (build + restart) | `./scripts/deployment/sync-sankofa-portal-7801.sh` (`--dry-run` first); default `NEXTAUTH_URL=https://portal.sankofa.nexus` via `sankofa-portal-ensure-nextauth-on-ct.sh`; IT `/it` env: `sankofa-portal-merge-it-read-api-env-from-repo.sh` (`IT_READ_API_URL` in repo `.env`) |
| Portal Keycloak OIDC secret on CT 7801 | After client exists: `./scripts/deployment/sankofa-portal-merge-keycloak-env-from-repo.sh` (needs `KEYCLOAK_CLIENT_SECRET` in repo `.env`; base64-safe over SSH) |
| Sankofa corporate web → CT 7806 | Provision: `./scripts/deployment/provision-sankofa-public-web-lxc-7806.sh`. Sync: `./scripts/deployment/sync-sankofa-public-web-to-ct.sh`. systemd: `config/systemd/sankofa-public-web.service`. Set `IP_SANKOFA_PUBLIC_WEB` in `.env`, then `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` |
| CCIP relay (r630-01 host) | WETH lane: `config/systemd/ccip-relay.service`. Mainnet cW lane: `config/systemd/ccip-relay-mainnet-cw.service` (health `http://192.168.11.11:9863/healthz`). Public edge: set `CCIP_RELAY_MAINNET_CW_PUBLIC_HOST`, run `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`, relay-only `scripts/nginx-proxy-manager/upsert-ccip-relay-mainnet-cw-proxy-host.sh`, or SSH hop `scripts/nginx-proxy-manager/upsert-ccip-relay-mainnet-cw-via-ssh.sh`; DNS `scripts/cloudflare/configure-relay-mainnet-cw-dns.sh`. Use `NPM_URL=https://…:81` for API scripts (HTTP on :81 301s to HTTPS). |
| XDC Zero + Chain 138 (parallel to CCIP) | `bash scripts/xdc-zero/run-xdc-zero-138-operator-sequence.sh` · `docs/03-deployment/CHAIN138_XDC_ZERO_BRIDGE_RUNBOOK.md` · `CHAIN138_XDC_ZERO_DEPLOYMENT_TROUBLESHOOTING.md` · `config/xdc-zero/` · `scripts/xdc-zero/` · systemd `node dist/server.js` template — **XDC mainnet RPC:** `https://rpc.xinfin.network` (chain id 50; more endpoints: [chainid.network/chain/50](https://chainid.network/chain/50/)); **Chain 138 side:** Core `http://192.168.11.211:8545` is operator-only, relayer/services use `https://rpc-http-pub.d-bis.org` |
| OP Stack Standard Rollup (Ethereum mainnet, Superchain) | `docs/03-deployment/OP_STACK_STANDARD_ROLLUP_SUPERCHAIN_RUNBOOK.md` · optional L2↔Besu notes `docs/03-deployment/OP_STACK_L2_AND_BESU138_BRIDGE_NOTES.md` · `config/op-stack-superchain/` · `scripts/op-stack/` (e.g. `fetch-standard-mainnet-toml.sh`, checklist scripts) · `config/systemd/op-stack-*.example.service` — **distinct L2 chain ID from Besu 138**; follow [Optimism superchain-registry](https://github.com/ethereum-optimism/superchain-registry) for listing |
| Wormhole protocol (LLM / MCP) vs Chain 138 facts | Wormhole NTT/Connect/VAAs/etc.: `docs/04-configuration/WORMHOLE_AI_RESOURCES_LLM_PLAYBOOK.md`, mirror `scripts/doc/sync-wormhole-ai-resources.sh`, MCP `mcp-wormhole-docs/` + `docs/04-configuration/MCP_SETUP.md`. **Chain 138 addresses, PMM, CCIP:** repo `docs/11-references/` + `docs/07-ccip/` — not Wormhole bundles. Cursor overlay: `.cursor/rules/wormhole-ai-resources.mdc`. |
| TsunamiSwap VM 5010 check | `./scripts/deployment/tsunamiswap-vm-5010-provision.sh` (inventory only until VM exists) |
| The Order portal (`https://the-order.sankofa.nexus`) | OSJ management UI (secure auth); source repo **the_order** at `~/projects/the_order`. NPM upstream defaults to **order-haproxy** CT **10210** (`IP_ORDER_HAPROXY:80`); use `THE_ORDER_UPSTREAM_*` to point at the Sankofa portal if 10210 is down. Provision HAProxy: `scripts/deployment/provision-order-haproxy-10210.sh`. **`www.the-order.sankofa.nexus`** → **301** apex (same as www.sankofa / www.phoenix). |
| Portal login + Keycloak systemd + `.env` (prints password once) | `./scripts/deployment/enable-sankofa-portal-login-7801.sh` (`--dry-run` first); preserves `KEYCLOAK_*` from repo `.env` and runs merge script when `KEYCLOAK_CLIENT_SECRET` is set |
| Keycloak redirect URIs (portal + admin) | `./scripts/deployment/keycloak-sankofa-ensure-client-redirects-via-proxmox-pct.sh` (or `keycloak-sankofa-ensure-client-redirects.sh` for LAN URL) — needs `KEYCLOAK_ADMIN_PASSWORD` in `.env` |
| NPM TLS for hosts missing certs | `./scripts/request-npmplus-certificates.sh` — optional `CERT_DOMAINS_FILTER='portal\\.sankofa|admin\\.sankofa'` |
| Token-aggregation API (Chain 138) | `pnpm run verify:token-aggregation-api` — tokens, pools, quote (prints `quoteEngine` when `jq` installed), `bridge/routes`, networks. Build + env: `scripts/deploy-token-aggregation-for-publication.sh` (sets `RPC_URL_138`, `TOKEN_AGGREGATION_CHAIN138_RPC_URL`, optional `TOKEN_AGGREGATION_PMM_*`). LAN push + restart: `scripts/deployment/push-token-aggregation-bundle-to-explorer.sh`. Nginx gaps: `scripts/fix-explorer-http-api-v1-proxy.sh` (apex `/api/v1/`), `scripts/fix-explorer-token-aggregation-api-v2-proxy.sh` (planner POST). Runbook: `docs/04-configuration/TOKEN_AGGREGATION_REPORT_API_RUNBOOK.md`. |
| **Chain 138 Open Snap** (MetaMask, open Snap permissions only; stable MetaMask requires MetaMask install allowlist for npm Snaps) | Source repo: [Defi-Oracle-Tooling/chain138-snap-minimal](https://github.com/Defi-Oracle-Tooling/chain138-snap-minimal). Vendored in this workspace: `metamask-integration/chain138-snap-minimal/`. Snap ID `npm:chain138-open-snap`; **`npm run verify`** = `npm audit --omit=dev` + build. **Publish:** token in `chain138-snap/.env` or `npm login`, then `./scripts/deployment/publish-chain138-open-snap.sh`. **Full-feature Snap** (API quotes, allowlist): `metamask-integration/chain138-snap/`. Explorer `/wallet` install works on stable MetaMask only after allowlisting; use Flask or local serve for dev. |
| Completable (no LAN) | `./scripts/run-completable-tasks-from-anywhere.sh` |
| Operator (LAN + secrets) | `./scripts/run-all-operator-tasks-from-lan.sh` (use `--skip-backup` if `NPM_PASSWORD` unset) |
| Cloudflare bulk DNS → `PUBLIC_IP` | `./scripts/update-all-dns-to-public-ip.sh` — use **`--dry-run`** and **`--zone-only=sankofa.nexus`** (or `d-bis.org` / `mim4u.org` / `defi-oracle.io`) to limit scope; see script header. Prefer scoped **`CLOUDFLARE_API_TOKEN`** (see `.env.master.example`). |
| IRU marketplace surfaces + Turnstile (Captcha) | [docs/03-deployment/SANKOFA_MARKETPLACE_SURFACES.md](docs/03-deployment/SANKOFA_MARKETPLACE_SURFACES.md) — **native** (VMs, IPs, app hosting, etc.) vs **partner** (e.g. SolaceNet IRU) methodology; Turnstile **secret** on API (`CLOUDFLARE_TURNSTILE_SECRET_KEY` or aliases), **site key** on frontend build (`VITE_*`); not the same as Cloudflare DNS keys. [docs/04-configuration/MASTER_SECRETS.md](docs/04-configuration/MASTER_SECRETS.md) (Cloudflare table). |

## Git submodules

Most submodules are **pinned commits**; `git submodule update --init --recursive` often leaves **detached HEAD** — that is normal. To **change** a submodule: check out a branch inside it, commit, **push the submodule first**, then commit and push the **parent** submodule pointer. Do not embed credentials in `git remote` URLs; use SSH or a credential helper. Explorer Gitea vs GitHub and token cleanup: `docs/00-meta/SUBMODULE_HYGIENE.md`.

## Production safety (Proxmox / shared config)

- **Scoped LXC starts:** use `scripts/operator/start-stopped-lxc-scoped.sh --host <PVE> --vmid <N> [--vmid …]`; default is **dry-run**; add **`--apply`** or **`PROXMOX_OPS_APPLY=1`** to mutate. Optional **`PROXMOX_OPS_ALLOWED_VMIDS`** enforces an allowlist. Do **not** use cluster-wide “start every stopped CT” patterns for production.
- **Maintenance scripts (SSH + pct):** set **`PROXMOX_SAFE_DEFAULTS=1`** so `fix-core-rpc-2101.sh`, `make-rpc-vmids-writable-via-ssh.sh`, and `ensure-legacy-monitor-networkd-via-ssh.sh` default to **plan-only** unless **`--apply`** or **`PROXMOX_OPS_APPLY=1`**. Without that env, behavior stays **legacy** (mutate unless `--dry-run`) so existing docs/commands keep working.
- **Guard helpers** for new SSH+pct scripts: `scripts/lib/proxmox-production-guard.sh`.
- **VMID → host** for automation: `get_host_for_vmid` in `scripts/lib/load-project-env.sh` must match live placement (`docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`).
- **Shared config:** avoid drive-by edits to `config/ip-addresses.conf` or root `.env` when the task only affects one workload; prefer flags, workload-specific env files, or small dedicated scripts.
- Cursor overlay: `.cursor/rules/proxmox-production-safety.mdc`.

## Rules of engagement

- Review scripts before running; prefer `--dry-run` where supported.
- Do not run the full operator flow when everything is healthy unless the user explicitly wants broad fixes (NPM/nginx/RPC churn).
- Chain 138 deploy RPC: `http://192.168.11.211:8545` (Core). Read-only / non-deploy checks may use public RPC per project rules.

Full detail: see embedded workspace rules and `docs/00-meta/OPERATOR_READY_CHECKLIST.md`.