feat(order): HAProxy on 10210, NPM → 192.168.11.39:80
- Add order-haproxy config template and provision-order-haproxy-10210.sh (SSH to r630-01) - Document one-time unprivileged CT idmap chown repair when apt fails - Default THE_ORDER_UPSTREAM_* to IP_ORDER_HAPROXY:80; portal bypass via env - Align update-sankofa-npmplus-proxy-hosts.sh, AGENTS, ALL_VMIDS, E2E notes Made-with: Cursor
This commit is contained in:
defiQUG
@@ -21,7 +21,7 @@ Orchestration for Proxmox VE, Chain 138 (`smom-dbis-138/`), explorers, NPMplus,
|
||||
| Sankofa portal → CT 7801 (build + restart) | `./scripts/deployment/sync-sankofa-portal-7801.sh` (`--dry-run` first); sets `NEXTAUTH_URL` on CT via `sankofa-portal-ensure-nextauth-on-ct.sh` |
|
||||
| CCIP relay (r630-01 host) | Unit: `config/systemd/ccip-relay.service` → `/etc/systemd/system/ccip-relay.service`; `systemctl enable --now ccip-relay` |
|
||||
| TsunamiSwap VM 5010 check | `./scripts/deployment/tsunamiswap-vm-5010-provision.sh` (inventory only until VM exists) |
|
||||
| The Order portal (`https://the-order.sankofa.nexus`) | OSJ management UI (secure auth); source repo **the_order** at `~/projects/the_order`. NPM upstream defaults to portal 7801 until order-haproxy (10210) is set via `THE_ORDER_UPSTREAM_*` in `update-npmplus-proxy-hosts-api.sh`. **`www.the-order.sankofa.nexus`** is updated to **301** to the apex hostname (same as www.sankofa / www.phoenix). |
|
||||
| The Order portal (`https://the-order.sankofa.nexus`) | OSJ management UI (secure auth); source repo **the_order** at `~/projects/the_order`. NPM upstream defaults to **order-haproxy** CT **10210** (`IP_ORDER_HAPROXY:80`); use `THE_ORDER_UPSTREAM_*` to point at the Sankofa portal if 10210 is down. Provision HAProxy: `scripts/deployment/provision-order-haproxy-10210.sh`. **`www.the-order.sankofa.nexus`** → **301** apex (same as www.sankofa / www.phoenix). |
|
||||
| Portal login + Keycloak systemd + `.env` (prints password once) | `./scripts/deployment/enable-sankofa-portal-login-7801.sh` (`--dry-run` first) |
|
||||
| Completable (no LAN) | `./scripts/run-completable-tasks-from-anywhere.sh` |
|
||||
| Operator (LAN + secrets) | `./scripts/run-all-operator-tasks-from-lan.sh` (use `--skip-backup` if `NPM_PASSWORD` unset) |
|
||||
|
||||
27
config/haproxy/order-haproxy-10210.cfg.template
Normal file
27
config/haproxy/order-haproxy-10210.cfg.template
Normal file
@@ -0,0 +1,27 @@
|
||||
# HAProxy on VMID 10210 (order-haproxy @ 192.168.11.39).
|
||||
# NPMplus terminates TLS and forwards HTTP to :80 here; we proxy to the Sankofa/Order Next.js portal.
|
||||
# Deploy: scripts/deployment/provision-order-haproxy-10210.sh (substitutes __BACKEND_HOST__ / __BACKEND_PORT__).
|
||||
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 4096
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
option httplog
|
||||
option dontlognull
|
||||
option forwardfor
|
||||
timeout connect 10s
|
||||
timeout client 300s
|
||||
timeout server 300s
|
||||
timeout tunnel 3600s
|
||||
|
||||
frontend fe_http
|
||||
bind *:80
|
||||
# Client used HTTPS at NPM; help Next.js / auth callbacks
|
||||
http-request set-header X-Forwarded-Proto https if !{ hdr(X-Forwarded-Proto) -m found }
|
||||
default_backend be_portal
|
||||
|
||||
backend be_portal
|
||||
server portal __BACKEND_HOST__:__BACKEND_PORT__ check inter 10s fall 3 rise 2 maxconn 1000
|
||||
@@ -283,7 +283,7 @@ The following VMIDs have been permanently removed:
|
||||
| 10200 | 192.168.11.46 | order-prometheus | ✅ Running | 9090 | Metrics (`IP_ORDER_PROMETHEUS`; not Order Redis) |
|
||||
| 10201 | 192.168.11.47 | order-grafana | ✅ Running | 3000 | Dashboards |
|
||||
| 10202 | 192.168.11.48 | order-opensearch | ✅ Running | 9200 | Search |
|
||||
| 10210 | 192.168.11.39 | order-haproxy | ✅ Running | 80, 443 | Edge for **the-order.sankofa.nexus** (NPMplus upstream HTTP :80) |
|
||||
| 10210 | 192.168.11.39 | order-haproxy | ✅ Running | 80 (HAProxy → portal :3000) | Edge for **the-order.sankofa.nexus**; HAProxy config via `config/haproxy/order-haproxy-10210.cfg.template` + `scripts/deployment/provision-order-haproxy-10210.sh` |
|
||||
|
||||
**Gov portals vs Order:** VMID **7804** alone uses **192.168.11.54** (`IP_GOV_PORTALS_DEV`). Order-legal must not use .54.
|
||||
|
||||
|
||||
@@ -164,7 +164,7 @@ When running from outside LAN or when backends are down, the following endpoints
|
||||
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | 502 — MIM4U web backends (192.168.11.37:80); non-blocking for contract/pool |
|
||||
| studio.sankofa.nexus | Historically 404 when the proxy misses `/studio/` or backend `192.168.11.72:8000`; verifier checks `/studio/`. Passed on 2026-03-26 after the NPMplus host update |
|
||||
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; `verify-end-to-end-routing.sh` checks `https://…/health` (200), not `/`. A separate **marketing** site on the apex hostname (if desired) needs another upstream or app routes—NPM still points `phoenix.sankofa.nexus` at the Fastify API today. |
|
||||
| the-order.sankofa.nexus | 502 when NPM still points at empty **10210** / **10090**. `update-npmplus-proxy-hosts-api.sh` defaults **THE_ORDER_UPSTREAM_IP/PORT** to the Sankofa portal (7801) until you set `THE_ORDER_UPSTREAM_IP=192.168.11.39` and `THE_ORDER_UPSTREAM_PORT=80` once order-haproxy serves. Passed on 2026-03-26 with the interim portal target |
|
||||
| the-order.sankofa.nexus | 502 if **10210** HAProxy or backend portal is down. NPM defaults upstream to **192.168.11.39:80** (order-haproxy). Fallback: `THE_ORDER_UPSTREAM_IP` / `THE_ORDER_UPSTREAM_PORT` = portal **192.168.11.51:3000** |
|
||||
|
||||
**Verifier behavior (2026-03):** `openssl s_client` is wrapped with `timeout` (`E2E_OPENSSL_TIMEOUT` default 15s, `E2E_OPENSSL_X509_TIMEOUT` default 5s) so `--profile=private` / `--profile=all` cannot hang. **`--profile=all`** merges private and public `E2E_OPTIONAL_WHEN_FAIL` lists for temporary regressions. Install **`wscat`** (`npm install -g wscat`) for full WSS JSON-RPC checks; the script uses `wscat -n` to match `curl -k`, and now treats a clean `wscat` exit as a successful full WebSocket check even when the tool prints no JSON output.
|
||||
|
||||
@@ -180,4 +180,4 @@ When running from outside LAN or when backends are down, the following endpoints
|
||||
|------|--------|
|
||||
| **502s (dbis-admin, dbis-api, secure, mifos)** | From LAN: `./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e]` or `./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e`. If NPMplus API is unreachable: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh`. Runbook: [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](../00-meta/502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md). |
|
||||
| **404 studio.sankofa.nexus** | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for `studio.sankofa.nexus` points to it. See [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md), [SANKOFA_STUDIO_E2E_FLOW.md](../03-deployment/SANKOFA_STUDIO_E2E_FLOW.md), [SANKOFA_STUDIO_DEPLOYMENT.md](../03-deployment/SANKOFA_STUDIO_DEPLOYMENT.md). |
|
||||
| **the-order 502** | From LAN with `.env`: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` (interim upstream = portal). When **order-haproxy** (VMID 10210, `192.168.11.39:80`) answers HTTP, switch: `THE_ORDER_UPSTREAM_IP=192.168.11.39 THE_ORDER_UPSTREAM_PORT=80` for that run. If `:80` does not connect from LAN, keep portal upstream until HAProxy is serving. |
|
||||
| **the-order 502** | Check **10210** HAProxy (`curl http://192.168.11.39:80/` with `Host: the-order.sankofa.nexus`) and portal **192.168.11.51:3000**. Re-provision: `bash scripts/deployment/provision-order-haproxy-10210.sh`. NPM refresh: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Direct portal bypass: `THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000` for that run. |
|
||||
|
||||
78
scripts/deployment/provision-order-haproxy-10210.sh
Executable file
78
scripts/deployment/provision-order-haproxy-10210.sh
Executable file
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install HAProxy in LXC 10210 (order-haproxy) and proxy :80 → Sankofa/Order portal (Next.js).
|
||||
# Requires SSH to Proxmox host that runs CT 10210 (default: r630-01). See config/ip-addresses.conf.
|
||||
# Usage: ./scripts/deployment/provision-order-haproxy-10210.sh [--dry-run]
|
||||
#
|
||||
# One-time repair (unprivileged CT with host uid 0 on disk → "nobody" inside, apt broken): on Proxmox host,
|
||||
# pct stop 10210 && pct mount 10210 && chown -R 100000:100000 /var/lib/lxc/10210/rootfs && pct unmount 10210 && pct start 10210
|
||||
# (Default Proxmox idmap: container root = 100000 on host.)
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "$PROJECT_ROOT/config/ip-addresses.conf"
|
||||
|
||||
DRY_RUN=false
|
||||
for a in "$@"; do [[ "$a" == "--dry-run" ]] && DRY_RUN=true; done
|
||||
|
||||
PROXMOX="${PROXMOX_ORDER_HAPROXY_NODE:-${PROXMOX_HOST_R630_01:-192.168.11.11}}"
|
||||
VMID="${ORDER_HAPROXY_VMID:-10210}"
|
||||
BACKEND_HOST="${ORDER_HAPROXY_BACKEND_HOST:-${IP_SANKOFA_PORTAL:-192.168.11.51}}"
|
||||
BACKEND_PORT="${ORDER_HAPROXY_BACKEND_PORT:-${SANKOFA_PORTAL_PORT:-3000}}"
|
||||
TEMPLATE="$PROJECT_ROOT/config/haproxy/order-haproxy-10210.cfg.template"
|
||||
|
||||
if [[ ! -r "$TEMPLATE" ]]; then
|
||||
echo "❌ Missing template: $TEMPLATE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CFG=$(sed -e "s/__BACKEND_HOST__/${BACKEND_HOST}/g" -e "s/__BACKEND_PORT__/${BACKEND_PORT}/g" "$TEMPLATE")
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "Provision order-haproxy (CT $VMID on $PROXMOX)"
|
||||
echo " Backend: http://${BACKEND_HOST}:${BACKEND_PORT}"
|
||||
echo " Dry-run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$CFG"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
remote_run() {
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=15 -o StrictHostKeyChecking=accept-new \
|
||||
"${PROXMOX_SSH_USER:-root}@$PROXMOX" "$@"
|
||||
}
|
||||
|
||||
if ! remote_run "pct status $VMID" 2>/dev/null | grep -q running; then
|
||||
echo "❌ CT $VMID is not running on $PROXMOX"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
remote_run "pct exec $VMID -- bash -c '
|
||||
set -e
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
if ! dpkg -s haproxy >/dev/null 2>&1; then
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq haproxy
|
||||
fi
|
||||
'"
|
||||
|
||||
echo "$CFG" | remote_run "pct exec $VMID -- bash -c 'cat > /etc/haproxy/haproxy.cfg'"
|
||||
|
||||
remote_run "pct exec $VMID -- bash -c '
|
||||
set -e
|
||||
haproxy -c -f /etc/haproxy/haproxy.cfg
|
||||
systemctl enable haproxy
|
||||
systemctl restart haproxy
|
||||
sleep 1
|
||||
systemctl is-active --quiet haproxy
|
||||
echo OK: haproxy active
|
||||
command -v ss >/dev/null && ss -lntp | grep -E \":80|:443\" || true
|
||||
'"
|
||||
|
||||
IP_ORDER="${IP_ORDER_HAPROXY:-192.168.11.39}"
|
||||
echo ""
|
||||
echo "✅ Done. From LAN: curl -sS -o /dev/null -w '%{http_code}\\n' http://${IP_ORDER}:80/"
|
||||
echo " Then NPM: THE_ORDER_UPSTREAM_IP=${IP_ORDER} THE_ORDER_UPSTREAM_PORT=80 bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh"
|
||||
@@ -389,12 +389,12 @@ IP_KEYCLOAK="${IP_KEYCLOAK:-192.168.11.52}"
|
||||
update_proxy_host "keycloak.sankofa.nexus" "http://${IP_KEYCLOAK}:8080" false false && updated_count=$((updated_count + 1)) || { add_proxy_host "keycloak.sankofa.nexus" "${IP_KEYCLOAK}" 8080 false false && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1))
|
||||
# the-order.sankofa.nexus — public hostname for the Sovereign Military Order of Malta (OSJ) management portal (secure auth).
|
||||
# Application source (operator workstation): repo the_order at ~/projects/the_order (e.g. /home/intlc/projects/the_order).
|
||||
# Ideal upstream: VMID 10210 order-haproxy @ IP_ORDER_HAPROXY:80. When HAProxy/order edge is not serving, NPM may 502.
|
||||
# Interim default: same Next.js upstream as sankofa.nexus (7801). Switch: THE_ORDER_UPSTREAM_IP=192.168.11.39 THE_ORDER_UPSTREAM_PORT=80.
|
||||
# Default upstream: VMID 10210 order-haproxy @ IP_ORDER_HAPROXY:80 (provision: scripts/deployment/provision-order-haproxy-10210.sh).
|
||||
# If 10210 is down: THE_ORDER_UPSTREAM_IP=${IP_SANKOFA_PORTAL} THE_ORDER_UPSTREAM_PORT=${SANKOFA_PORTAL_PORT} (direct portal 7801).
|
||||
# www.the-order.sankofa.nexus → 301 https://the-order.sankofa.nexus$request_uri (same pattern as www.sankofa / www.phoenix).
|
||||
IP_ORDER_HAPROXY="${IP_ORDER_HAPROXY:-192.168.11.39}"
|
||||
THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_SANKOFA_PORTAL}}"
|
||||
THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-${SANKOFA_PORTAL_PORT}}"
|
||||
THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_ORDER_HAPROXY}}"
|
||||
THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-80}"
|
||||
# block_exploits false — same policy as sankofa.nexus portal (Next/API-friendly; avoid 405 on some POST paths)
|
||||
update_proxy_host "the-order.sankofa.nexus" "http://${THE_ORDER_UPSTREAM_IP}:${THE_ORDER_UPSTREAM_PORT}" false false && updated_count=$((updated_count + 1)) || { add_proxy_host "the-order.sankofa.nexus" "${THE_ORDER_UPSTREAM_IP}" "${THE_ORDER_UPSTREAM_PORT}" false false && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1))
|
||||
update_proxy_host "www.the-order.sankofa.nexus" "http://${THE_ORDER_UPSTREAM_IP}:${THE_ORDER_UPSTREAM_PORT}" false false "https://the-order.sankofa.nexus" && updated_count=$((updated_count + 1)) || { add_proxy_host "www.the-order.sankofa.nexus" "${THE_ORDER_UPSTREAM_IP}" "${THE_ORDER_UPSTREAM_PORT}" false false "https://the-order.sankofa.nexus" && updated_count=$((updated_count + 1)); } || failed_count=$((failed_count + 1))
|
||||
|
||||
@@ -27,8 +27,9 @@ IP_SANKOFA_PORTAL="${IP_SANKOFA_PORTAL:-${IP_SERVICE_51:-192.168.11.51}}"
|
||||
IP_SANKOFA_PHOENIX_API="${IP_SANKOFA_PHOENIX_API:-${IP_SERVICE_50:-192.168.11.50}}"
|
||||
SANKOFA_PORTAL_PORT="${SANKOFA_PORTAL_PORT:-3000}"
|
||||
SANKOFA_PHOENIX_API_PORT="${SANKOFA_PHOENIX_API_PORT:-4000}"
|
||||
THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_SANKOFA_PORTAL}}"
|
||||
THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-${SANKOFA_PORTAL_PORT}}"
|
||||
IP_ORDER_HAPROXY="${IP_ORDER_HAPROXY:-192.168.11.39}"
|
||||
THE_ORDER_UPSTREAM_IP="${THE_ORDER_UPSTREAM_IP:-${IP_ORDER_HAPROXY}}"
|
||||
THE_ORDER_UPSTREAM_PORT="${THE_ORDER_UPSTREAM_PORT:-80}"
|
||||
|
||||
# NPM proxy host IDs: sankofa=3, www.sankofa=4, phoenix=5, www.phoenix=6; the-order=7, www.the-order=59 (typical; verify in NPM UI)
|
||||
SANKOFA_NPM_ID_ROOT="${SANKOFA_NPM_ID_ROOT:-3}"
|
||||
|
||||
Reference in New Issue
Block a user