diff --git a/AGENTS.md b/AGENTS.md index ccc2d87..3fb7f13 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,7 @@ Orchestration for Proxmox VE, Chain 138 (`smom-dbis-138/`), explorers, NPMplus, | Sankofa portal → CT 7801 (build + restart) | `./scripts/deployment/sync-sankofa-portal-7801.sh` (`--dry-run` first); sets `NEXTAUTH_URL` on CT via `sankofa-portal-ensure-nextauth-on-ct.sh` | | CCIP relay (r630-01 host) | Unit: `config/systemd/ccip-relay.service` → `/etc/systemd/system/ccip-relay.service`; `systemctl enable --now ccip-relay` | | TsunamiSwap VM 5010 check | `./scripts/deployment/tsunamiswap-vm-5010-provision.sh` (inventory only until VM exists) | -| The Order portal (`https://the-order.sankofa.nexus`) | OSJ management UI (secure auth); source repo **the_order** at `~/projects/the_order`. NPM upstream defaults to portal 7801 until order-haproxy (10210) is set via `THE_ORDER_UPSTREAM_*` in `update-npmplus-proxy-hosts-api.sh`. **`www.the-order.sankofa.nexus`** is updated to **301** to the apex hostname (same as www.sankofa / www.phoenix). | +| The Order portal (`https://the-order.sankofa.nexus`) | OSJ management UI (secure auth); source repo **the_order** at `~/projects/the_order`. NPM upstream defaults to **order-haproxy** CT **10210** (`IP_ORDER_HAPROXY:80`); use `THE_ORDER_UPSTREAM_*` to point at the Sankofa portal if 10210 is down. Provision HAProxy: `scripts/deployment/provision-order-haproxy-10210.sh`. **`www.the-order.sankofa.nexus`** → **301** apex (same as www.sankofa / www.phoenix). | | Portal login + Keycloak systemd + `.env` (prints password once) | `./scripts/deployment/enable-sankofa-portal-login-7801.sh` (`--dry-run` first) | | Completable (no LAN) | `./scripts/run-completable-tasks-from-anywhere.sh` | | Operator (LAN + secrets) | `./scripts/run-all-operator-tasks-from-lan.sh` (use `--skip-backup` if `NPM_PASSWORD` unset) | diff --git a/config/haproxy/order-haproxy-10210.cfg.template b/config/haproxy/order-haproxy-10210.cfg.template new file mode 100644 index 0000000..4492944 --- /dev/null +++ b/config/haproxy/order-haproxy-10210.cfg.template @@ -0,0 +1,27 @@ +# HAProxy on VMID 10210 (order-haproxy @ 192.168.11.39). +# NPMplus terminates TLS and forwards HTTP to :80 here; we proxy to the Sankofa/Order Next.js portal. +# Deploy: scripts/deployment/provision-order-haproxy-10210.sh (substitutes __BACKEND_HOST__ / __BACKEND_PORT__). + +global + log stdout format raw local0 + maxconn 4096 + +defaults + log global + mode http + option httplog + option dontlognull + option forwardfor + timeout connect 10s + timeout client 300s + timeout server 300s + timeout tunnel 3600s + +frontend fe_http + bind *:80 + # Client used HTTPS at NPM; help Next.js / auth callbacks + http-request set-header X-Forwarded-Proto https if !{ hdr(X-Forwarded-Proto) -m found } + default_backend be_portal + +backend be_portal + server portal __BACKEND_HOST__:__BACKEND_PORT__ check inter 10s fall 3 rise 2 maxconn 1000 diff --git a/docs/04-configuration/ALL_VMIDS_ENDPOINTS.md b/docs/04-configuration/ALL_VMIDS_ENDPOINTS.md index 73dcc15..971278c 100644 --- a/docs/04-configuration/ALL_VMIDS_ENDPOINTS.md +++ b/docs/04-configuration/ALL_VMIDS_ENDPOINTS.md @@ -283,7 +283,7 @@ The following VMIDs have been permanently removed: | 10200 | 192.168.11.46 | order-prometheus | ✅ Running | 9090 | Metrics (`IP_ORDER_PROMETHEUS`; not Order Redis) | | 10201 | 192.168.11.47 | order-grafana | ✅ Running | 3000 | Dashboards | | 10202 | 192.168.11.48 | order-opensearch | ✅ Running | 9200 | Search | -| 10210 | 192.168.11.39 | order-haproxy | ✅ Running | 80, 443 | Edge for **the-order.sankofa.nexus** (NPMplus upstream HTTP :80) | +| 10210 | 192.168.11.39 | order-haproxy | ✅ Running | 80 (HAProxy → portal :3000) | Edge for **the-order.sankofa.nexus**; HAProxy config via `config/haproxy/order-haproxy-10210.cfg.template` + `scripts/deployment/provision-order-haproxy-10210.sh` | **Gov portals vs Order:** VMID **7804** alone uses **192.168.11.54** (`IP_GOV_PORTALS_DEV`). Order-legal must not use .54. diff --git a/docs/04-configuration/E2E_ENDPOINTS_LIST.md b/docs/04-configuration/E2E_ENDPOINTS_LIST.md index 4fc076a..934a2a5 100644 --- a/docs/04-configuration/E2E_ENDPOINTS_LIST.md +++ b/docs/04-configuration/E2E_ENDPOINTS_LIST.md @@ -164,7 +164,7 @@ When running from outside LAN or when backends are down, the following endpoints | mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | 502 — MIM4U web backends (192.168.11.37:80); non-blocking for contract/pool | | studio.sankofa.nexus | Historically 404 when the proxy misses `/studio/` or backend `192.168.11.72:8000`; verifier checks `/studio/`. Passed on 2026-03-26 after the NPMplus host update | | phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; `verify-end-to-end-routing.sh` checks `https://…/health` (200), not `/`. A separate **marketing** site on the apex hostname (if desired) needs another upstream or app routes—NPM still points `phoenix.sankofa.nexus` at the Fastify API today. | -| the-order.sankofa.nexus | 502 when NPM still points at empty **10210** / **10090**. `update-npmplus-proxy-hosts-api.sh` defaults **THE_ORDER_UPSTREAM_IP/PORT** to the Sankofa portal (7801) until you set `THE_ORDER_UPSTREAM_IP=192.168.11.39` and `THE_ORDER_UPSTREAM_PORT=80` once order-haproxy serves. Passed on 2026-03-26 with the interim portal target | +| the-order.sankofa.nexus | 502 if **10210** HAProxy or backend portal is down. NPM defaults upstream to **192.168.11.39:80** (order-haproxy). Fallback: `THE_ORDER_UPSTREAM_IP` / `THE_ORDER_UPSTREAM_PORT` = portal **192.168.11.51:3000** | **Verifier behavior (2026-03):** `openssl s_client` is wrapped with `timeout` (`E2E_OPENSSL_TIMEOUT` default 15s, `E2E_OPENSSL_X509_TIMEOUT` default 5s) so `--profile=private` / `--profile=all` cannot hang. **`--profile=all`** merges private and public `E2E_OPTIONAL_WHEN_FAIL` lists for temporary regressions. Install **`wscat`** (`npm install -g wscat`) for full WSS JSON-RPC checks; the script uses `wscat -n` to match `curl -k`, and now treats a clean `wscat` exit as a successful full WebSocket check even when the tool prints no JSON output. @@ -180,4 +180,4 @@ When running from outside LAN or when backends are down, the following endpoints |------|--------| | **502s (dbis-admin, dbis-api, secure, mifos)** | From LAN: `./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e]` or `./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e`. If NPMplus API is unreachable: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh`. Runbook: [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](../00-meta/502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md). | | **404 studio.sankofa.nexus** | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for `studio.sankofa.nexus` points to it. See [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md), [SANKOFA_STUDIO_E2E_FLOW.md](../03-deployment/SANKOFA_STUDIO_E2E_FLOW.md), [SANKOFA_STUDIO_DEPLOYMENT.md](../03-deployment/SANKOFA_STUDIO_DEPLOYMENT.md). | -| **the-order 502** | From LAN with `.env`: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` (interim upstream = portal). When **order-haproxy** (VMID 10210, `192.168.11.39:80`) answers HTTP, switch: `THE_ORDER_UPSTREAM_IP=192.168.11.39 THE_ORDER_UPSTREAM_PORT=80` for that run. If `:80` does not connect from LAN, keep portal upstream until HAProxy is serving. | +| **the-order 502** | Check **10210** HAProxy (`curl http://192.168.11.39:80/` with `Host: the-order.sankofa.nexus`) and portal **192.168.11.51:3000**. Re-provision: `bash scripts/deployment/provision-order-haproxy-10210.sh`. NPM refresh: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Direct portal bypass: `THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000` for that run. | diff --git a/scripts/deployment/provision-order-haproxy-10210.sh b/scripts/deployment/provision-order-haproxy-10210.sh new file mode 100755 index 0000000..ed51144 --- /dev/null +++ b/scripts/deployment/provision-order-haproxy-10210.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Install HAProxy in LXC 10210 (order-haproxy) and proxy :80 → Sankofa/Order portal (Next.js). +# Requires SSH to Proxmox host that runs CT 10210 (default: r630-01). See config/ip-addresses.conf. +# Usage: ./scripts/deployment/provision-order-haproxy-10210.sh [--dry-run] +# +# One-time repair (unprivileged CT with host uid 0 on disk → "nobody" inside, apt broken): on Proxmox host, +# pct stop 10210 && pct mount 10210 && chown -R 100000:100000 /var/lib/lxc/10210/rootfs && pct unmount 10210 && pct start 10210 +# (Default Proxmox idmap: container root = 100000 on host.) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +# shellcheck source=/dev/null +source "$PROJECT_ROOT/config/ip-addresses.conf" + +DRY_RUN=false +for a in "$@"; do [[ "$a" == "--dry-run" ]] && DRY_RUN=true; done + +PROXMOX="${PROXMOX_ORDER_HAPROXY_NODE:-${PROXMOX_HOST_R630_01:-192.168.11.11}}" +VMID="${ORDER_HAPROXY_VMID:-10210}" +BACKEND_HOST="${ORDER_HAPROXY_BACKEND_HOST:-${IP_SANKOFA_PORTAL:-192.168.11.51}}" +BACKEND_PORT="${ORDER_HAPROXY_BACKEND_PORT:-${SANKOFA_PORTAL_PORT:-3000}}" +TEMPLATE="$PROJECT_ROOT/config/haproxy/order-haproxy-10210.cfg.template" + +if [[ ! -r "$TEMPLATE" ]]; then + echo "❌ Missing template: $TEMPLATE" + exit 1 +fi + +CFG=$(sed -e "s/__BACKEND_HOST__/${BACKEND_HOST}/g" -e "s/__BACKEND_PORT__/${BACKEND_PORT}/g" "$TEMPLATE") + +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "Provision order-haproxy (CT $VMID on $PROXMOX)" +echo " Backend: http://${BACKEND_HOST}:${BACKEND_PORT}" +echo " Dry-run: $DRY_RUN" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + +if [[ "$DRY_RUN" == true ]]; then + echo "$CFG" + exit 0 +fi + +remote_run() { + ssh -o BatchMode=yes -o ConnectTimeout=15 -o StrictHostKeyChecking=accept-new \ + "${PROXMOX_SSH_USER:-root}@$PROXMOX" "$@" +} + +if ! remote_run "pct status $VMID" 2>/dev/null | grep -q running; then + echo "❌ CT $VMID is not running on $PROXMOX" + exit 1 +fi + +remote_run "pct exec $VMID -- bash -c ' +set -e +export DEBIAN_FRONTEND=noninteractive +if ! dpkg -s haproxy >/dev/null 2>&1; then + apt-get update -qq + apt-get install -y -qq haproxy +fi +'" + +echo "$CFG" | remote_run "pct exec $VMID -- bash -c 'cat > /etc/haproxy/haproxy.cfg'" + +remote_run "pct exec $VMID -- bash -c ' +set -e +haproxy -c -f /etc/haproxy/haproxy.cfg +systemctl enable haproxy +systemctl restart haproxy +sleep 1 +systemctl is-active --quiet haproxy +echo OK: haproxy active +command -v ss >/dev/null && ss -lntp | grep -E \":80|:443\" || true +'" + +IP_ORDER="${IP_ORDER_HAPROXY:-192.168.11.39}" +echo "" +echo "✅ Done. From LAN: curl -sS -o /dev/null -w '%{http_code}\\n' http://${IP_ORDER}:80/" +echo " Then NPM: THE_ORDER_UPSTREAM_IP=${IP_ORDER} THE_ORDER_UPSTREAM_PORT=80 bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh" diff --git a/scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh b/scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh index 3b6173d..44a8e76 100755 --- a/scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh +++ b/scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh @@ -389,12 +389,12 @@ IP_KEYCLOAK="${IP_KEYCLOAK:-192.168.11.52}" update_proxy_host "keycloak.sankofa.nexus" "http://${IP_KEYCLOAK}:8080" false false && updated_count=$((updated_count + 1)) || { add_proxy_host "keycloak.sankofa.nexus" "${IP_KEYCLOAK}" 8080 false false && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1)) # the-order.sankofa.nexus — public hostname for the Sovereign Military Order of Malta (OSJ) management portal (secure auth). # Application source (operator workstation): repo the_order at ~/projects/the_order (e.g. /home/intlc/projects/the_order). -# Ideal upstream: VMID 10210 order-haproxy @ IP_ORDER_HAPROXY:80. When HAProxy/order edge is not serving, NPM may 502. -# Interim default: same Next.js upstream as sankofa.nexus (7801). Switch: THE_ORDER_UPSTREAM_IP=192.168.11.39 THE_ORDER_UPSTREAM_PORT=80. +# Default upstream: VMID 10210 order-haproxy @ IP_ORDER_HAPROXY:80 (provision: scripts/deployment/provision-order-haproxy-10210.sh). +# If 10210 is down: THE_ORDER_UPSTREAM_IP=${IP_SANKOFA_PORTAL} THE_ORDER_UPSTREAM_PORT=${SANKOFA_PORTAL_PORT} (direct portal 7801). # www.the-order.sankofa.nexus → 301 https://the-order.sankofa.nexus$request_uri (same pattern as www.sankofa / www.phoenix). IP_ORDER_HAPROXY="${IP_ORDER_HAPROXY:-192.168.11.39}" -THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_SANKOFA_PORTAL}}" -THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-${SANKOFA_PORTAL_PORT}}" +THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_ORDER_HAPROXY}}" +THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-80}" # block_exploits false — same policy as sankofa.nexus portal (Next/API-friendly; avoid 405 on some POST paths) update_proxy_host "the-order.sankofa.nexus" "http://${THE_ORDER_UPSTREAM_IP}:${THE_ORDER_UPSTREAM_PORT}" false false && updated_count=$((updated_count + 1)) || { add_proxy_host "the-order.sankofa.nexus" "${THE_ORDER_UPSTREAM_IP}" "${THE_ORDER_UPSTREAM_PORT}" false false && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1)) update_proxy_host "www.the-order.sankofa.nexus" "http://${THE_ORDER_UPSTREAM_IP}:${THE_ORDER_UPSTREAM_PORT}" false false "https://the-order.sankofa.nexus" && updated_count=$((updated_count + 1)) || { add_proxy_host "www.the-order.sankofa.nexus" "${THE_ORDER_UPSTREAM_IP}" "${THE_ORDER_UPSTREAM_PORT}" false false "https://the-order.sankofa.nexus" && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1)) diff --git a/scripts/update-sankofa-npmplus-proxy-hosts.sh b/scripts/update-sankofa-npmplus-proxy-hosts.sh index 8c159d6..ff035cb 100755 --- a/scripts/update-sankofa-npmplus-proxy-hosts.sh +++ b/scripts/update-sankofa-npmplus-proxy-hosts.sh @@ -27,8 +27,9 @@ IP_SANKOFA_PORTAL="${IP_SANKOFA_PORTAL:-${IP_SERVICE_51:-192.168.11.51}}" IP_SANKOFA_PHOENIX_API="${IP_SANKOFA_PHOENIX_API:-${IP_SERVICE_50:-192.168.11.50}}" SANKOFA_PORTAL_PORT="${SANKOFA_PORTAL_PORT:-3000}" SANKOFA_PHOENIX_API_PORT="${SANKOFA_PHOENIX_API_PORT:-4000}" -THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_SANKOFA_PORTAL}}" -THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-${SANKOFA_PORTAL_PORT}}" +IP_ORDER_HAPROXY="${IP_ORDER_HAPROXY:-192.168.11.39}" +THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_ORDER_HAPROXY}}" +THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-80}" # NPM proxy host IDs: sankofa=3, www.sankofa=4, phoenix=5, www.phoenix=6; the-order=7, www.the-order=59 (typical; verify in NPM UI) SANKOFA_NPM_ID_ROOT="${SANKOFA_NPM_ID_ROOT:-3}"