7ac74f432b
- Institutional / JVMTM / reserve-provenance / GRU transport + standards JSON - Validation and verify scripts (Blockscout labels, x402, GRU preflight, P1 local path) - Wormhole wiring in AGENTS, MCP_SETUP, MASTER_INDEX, 04-configuration README - Meta docs, integration gaps, live verification log, architecture updates - CI validate-config workflow updates Operator/LAN items, submodule working trees, and public token-aggregation edge routes remain follow-up (see TODOS_CONSOLIDATED P1). Made-with: Cursor
20 KiB
E2E verification — endpoint inventory and profiles
Source: scripts/verify/verify-end-to-end-routing.sh (DOMAIN_TYPES).
List from CLI (public): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=public
List from CLI (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=private
Run E2E (public profile recommended): ./scripts/verify/verify-end-to-end-routing.sh --profile=public (from LAN with DNS or use E2E_USE_SYSTEM_RESOLVER=1 and /etc/hosts per E2E_DNS_FROM_LAN_RUNBOOK.md).
Run E2E (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --profile=private.
Gitea Actions (umbrella / cc-*): no stable unauthenticated REST for all Gitea versions — print UI URLs with ./scripts/verify/print-gitea-actions-urls.sh and confirm jobs in the browser after push.
What each hostname should present (operator narrative): FQDN_EXPECTED_CONTENT.md.
Latest verified public pass: 2026-03-30 via bash scripts/verify/verify-end-to-end-routing.sh --profile=public with report at verification_report.md. Result: exit 0, DNS passed: 45, Failed: 0, HTTPS passed: 32, Skipped / optional: 13 — includes d-bis.org, www.d-bis.org, admin.d-bis.org, core.d-bis.org (NPM + Cloudflare + Let’s Encrypt after fleet script).
Previous: 2026-03-29 — verification_report.md; older: 20260329_045210, 20260327.
Latest verified private/admin pass: 2026-03-27 via bash scripts/verify/verify-end-to-end-routing.sh --profile=private with report at verification_report.md. Result: exit 0, DNS passed: 4, Failed: 0.
Evidence folders: Each run creates verification-evidence/e2e-verification-YYYYMMDD_HHMMSS/. Commit the runs you want on record; older dirs can be removed locally to reduce noise (scripts/maintenance/prune-e2e-verification-evidence.sh --dry-run lists candidates). Routing truth is not inferred from old reports—use ALL_VMIDS_ENDPOINTS.md.
Verification profiles
- Public profile (default for routine E2E): web, api, public RPC endpoints.
- Private/admin profile: private RPC and Fireblocks RPC endpoints. Run separately for internal operations.
Full endpoint inventory (combined)
| Endpoint | Type | URL | Description (content provided) |
|---|---|---|---|
| explorer.d-bis.org | web | https://explorer.d-bis.org | Blockscout-style blockchain explorer for Chain 138: blocks, transactions, addresses, contracts, tokens, verification. |
| d-bis.org | web | https://d-bis.org | Public DBIS web presence — institutional portal (Gov Portals Next app when deployed behind NPM). |
| admin.d-bis.org | web | https://admin.d-bis.org | Admin console for DBIS operations staff; typical upstream VMID 10130. |
| dbis-admin.d-bis.org | web | https://dbis-admin.d-bis.org | Legacy admin hostname; same upstream intent as admin.d-bis.org if still in DNS. |
| secure.d-bis.org | web | https://secure.d-bis.org | Member secure portal (authenticated institutions); path-based routing on 10130 per ALL_VMIDS_ENDPOINTS.md. |
| core.d-bis.org | web | https://core.d-bis.org | DBIS Core banking application — client portal (dbis_core); NPM upstream TBD (often co-located with API 10150/10151 when live). |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org | DBIS core API: token aggregation, Crypto.com OTC, exchange endpoints (VMID 10150). |
| dbis-api-2.d-bis.org | api | https://dbis-api-2.d-bis.org | DBIS API secondary instance (VMID 10151). |
| mim4u.org | web | https://mim4u.org | MIM4U main site. |
| www.mim4u.org | web | https://www.mim4u.org | MIM4U www. |
| secure.mim4u.org | web | https://secure.mim4u.org | MIM4U secure portal. |
| training.mim4u.org | web | https://training.mim4u.org | MIM4U training site. |
| sankofa.nexus | web | https://sankofa.nexus | Sankofa Nexus root / web. |
| www.sankofa.nexus | web | https://www.sankofa.nexus | 301 to https://sankofa.nexus (canonical apex; NPM advanced_config). |
| phoenix.sankofa.nexus | web | https://phoenix.sankofa.nexus | Phoenix API (7800); E2E uses /health for HTTPS check. |
| www.phoenix.sankofa.nexus | web | https://www.phoenix.sankofa.nexus | 301 to https://phoenix.sankofa.nexus (canonical apex; NPM advanced_config). |
| the-order.sankofa.nexus | web | https://the-order.sankofa.nexus | OSJ management portal (secure auth); app the_order at ~/projects/the_order. NPM upstream default: order-haproxy VMID 10210 http://192.168.11.39:80 → portal 192.168.11.51:3000 (provision-order-haproxy-10210.sh). Override with THE_ORDER_UPSTREAM_* for direct portal if 10210 is down. |
| www.the-order.sankofa.nexus | web | https://www.the-order.sankofa.nexus | 301 to https://the-order.sankofa.nexus (canonical apex; NPM advanced_config). |
| studio.sankofa.nexus | web | https://studio.sankofa.nexus | Sankofa Studio (FusionAI Creator) at VMID 7805. |
| keycloak.sankofa.nexus | web | https://keycloak.sankofa.nexus | Keycloak IdP (VMID 7802); client SSO for admin/portal. |
| admin.sankofa.nexus | web | https://admin.sankofa.nexus | Client SSO: access administration (hostname intent; NPM upstream TBD). |
| portal.sankofa.nexus | web | https://portal.sankofa.nexus | Client SSO: portal / marketplace (typical upstream VMID 7801). Add DNS + NPM row via update-npmplus-proxy-hosts-api.sh; NextAuth public URL https://portal.sankofa.nexus. |
| dash.sankofa.nexus | web | https://dash.sankofa.nexus | Operator systems dashboard (IP allowlist + MFA intent; upstream TBD). |
| docs.d-bis.org | web | https://docs.d-bis.org | Docs on explorer nginx where configured. |
| blockscout.defi-oracle.io | web | https://blockscout.defi-oracle.io | Generic Blockscout hostname (often VMID 5000); not canonical Chain 138 explorer.d-bis.org. |
| cacti-alltra.d-bis.org | web | https://cacti-alltra.d-bis.org | Cacti monitoring UI for Alltra. |
| cacti-hybx.d-bis.org | web | https://cacti-hybx.d-bis.org | Cacti monitoring UI for HYBX. |
| mifos.d-bis.org | web | https://mifos.d-bis.org | Mifos X / Fineract banking and microfinance platform (VMID 5800). |
| dapp.d-bis.org | web | https://dapp.d-bis.org | DApp frontend for Chain 138 bridge (VMID 5801). |
| gitea.d-bis.org | web | https://gitea.d-bis.org | Gitea git repository and CI (Dev VM 5700). |
| dev.d-bis.org | web | https://dev.d-bis.org | Dev VM web / Codespaces entry. |
| codespaces.d-bis.org | web | https://codespaces.d-bis.org | Codespaces / dev environment entry. |
| rpc-http-pub.d-bis.org | rpc-http | https://rpc-http-pub.d-bis.org | Chain 138 public JSON-RPC HTTP (VMID 2201). |
| rpc-ws-pub.d-bis.org | rpc-ws | wss://rpc-ws-pub.d-bis.org | Chain 138 public JSON-RPC WebSocket. |
| rpc.d-bis.org | rpc-http | https://rpc.d-bis.org | Chain 138 RPC HTTP (alias). |
| rpc2.d-bis.org | rpc-http | https://rpc2.d-bis.org | Chain 138 RPC HTTP (second). |
| ws.rpc.d-bis.org | rpc-ws | wss://ws.rpc.d-bis.org | Chain 138 RPC WebSocket. |
| ws.rpc2.d-bis.org | rpc-ws | wss://ws.rpc2.d-bis.org | Chain 138 RPC WebSocket (second). |
| rpc-http-prv.d-bis.org | rpc-http | https://rpc-http-prv.d-bis.org | Chain 138 private/admin RPC HTTP (VMID 2101). |
| rpc-ws-prv.d-bis.org | rpc-ws | wss://rpc-ws-prv.d-bis.org | Chain 138 private RPC WebSocket. |
| rpc-fireblocks.d-bis.org | rpc-http | https://rpc-fireblocks.d-bis.org | Chain 138 RPC for Fireblocks Web3 (VMID 2301). |
| ws.rpc-fireblocks.d-bis.org | rpc-ws | wss://ws.rpc-fireblocks.d-bis.org | Chain 138 RPC WebSocket for Fireblocks. |
| rpc.public-0138.defi-oracle.io | rpc-http | https://rpc.public-0138.defi-oracle.io | Defi Oracle Chain 138 public RPC. |
| rpc.defi-oracle.io | rpc-http | https://rpc.defi-oracle.io | Defi Oracle RPC. |
| wss.defi-oracle.io | rpc-ws | wss://wss.defi-oracle.io | Defi Oracle RPC WebSocket. |
| rpc-alltra.d-bis.org | rpc-http | https://rpc-alltra.d-bis.org | Alltra chain RPC HTTP. |
| rpc-alltra-2.d-bis.org | rpc-http | https://rpc-alltra-2.d-bis.org | Alltra chain RPC HTTP (2). |
| rpc-alltra-3.d-bis.org | rpc-http | https://rpc-alltra-3.d-bis.org | Alltra chain RPC HTTP (3). |
| rpc-hybx.d-bis.org | rpc-http | https://rpc-hybx.d-bis.org | HYBX chain RPC HTTP. |
| rpc-hybx-2.d-bis.org | rpc-http | https://rpc-hybx-2.d-bis.org | HYBX chain RPC HTTP (2). |
| rpc-hybx-3.d-bis.org | rpc-http | https://rpc-hybx-3.d-bis.org | HYBX chain RPC HTTP (3). |
Planned DBIS institutional subdomains (multi-portal program)
Registered in verify-end-to-end-routing.sh as optional-when-fail until DNS and upstreams are live. Detail: DBIS_INSTITUTIONAL_SUBDOMAINS.md, blueprint: DBIS_WEB_AND_INSTITUTION_MASTER_BLUEPRINT.md.
| Endpoint | Type | URL | Description |
|---|---|---|---|
| www.d-bis.org | web | https://www.d-bis.org | Optional www → apex d-bis.org redirect. |
| members.d-bis.org | web | https://members.d-bis.org | Member institution portal (OIDC BFF). |
| developers.d-bis.org | web | https://developers.d-bis.org | Developer hub; links to Gitea + OpenAPI. |
| data.d-bis.org | api | https://data.d-bis.org | Public data API (openapi.yaml). |
| research.d-bis.org | web | https://research.d-bis.org | Research and working papers. |
| policy.d-bis.org | web | https://policy.d-bis.org | Policy publications + manifests. |
| ops.d-bis.org | web | https://ops.d-bis.org | Staff operations (SSO). |
| identity.d-bis.org | web | https://identity.d-bis.org | Trust anchors + DID registry documentation/API. |
| status.d-bis.org | web | https://status.d-bis.org | Public status / SLOs. |
| sandbox.d-bis.org | web | https://sandbox.d-bis.org | Sandbox console (isolated test). |
| interop.d-bis.org | web | https://interop.d-bis.org | Interoperability lab (CBDC / cross-chain). |
Endpoints by type
Web
API
| Domain | URL |
|---|---|
| dbis-api.d-bis.org | https://dbis-api.d-bis.org |
| dbis-api-2.d-bis.org | https://dbis-api-2.d-bis.org |
RPC HTTP (public)
| Domain | URL |
|---|---|
| rpc-http-pub.d-bis.org | https://rpc-http-pub.d-bis.org |
| rpc.d-bis.org | https://rpc.d-bis.org |
| rpc2.d-bis.org | https://rpc2.d-bis.org |
| rpc.public-0138.defi-oracle.io | https://rpc.public-0138.defi-oracle.io |
| rpc.defi-oracle.io | https://rpc.defi-oracle.io |
| rpc-alltra.d-bis.org | https://rpc-alltra.d-bis.org |
| rpc-alltra-2.d-bis.org | https://rpc-alltra-2.d-bis.org |
| rpc-alltra-3.d-bis.org | https://rpc-alltra-3.d-bis.org |
| rpc-hybx.d-bis.org | https://rpc-hybx.d-bis.org |
| rpc-hybx-2.d-bis.org | https://rpc-hybx-2.d-bis.org |
| rpc-hybx-3.d-bis.org | https://rpc-hybx-3.d-bis.org |
RPC WebSocket (public)
| Domain | URL |
|---|---|
| rpc-ws-pub.d-bis.org | wss://rpc-ws-pub.d-bis.org |
| ws.rpc.d-bis.org | wss://ws.rpc.d-bis.org |
| ws.rpc2.d-bis.org | wss://ws.rpc2.d-bis.org |
| wss.defi-oracle.io | wss://wss.defi-oracle.io |
RPC HTTP (private/admin profile)
| Domain | URL |
|---|---|
| rpc-http-prv.d-bis.org | https://rpc-http-prv.d-bis.org |
| rpc-fireblocks.d-bis.org | https://rpc-fireblocks.d-bis.org |
RPC WebSocket (private/admin profile)
| Domain | URL |
|---|---|
| rpc-ws-prv.d-bis.org | wss://rpc-ws-prv.d-bis.org |
| ws.rpc-fireblocks.d-bis.org | wss://ws.rpc-fireblocks.d-bis.org |
Report content
After each run, the verification report includes:
- All endpoints — table of every domain, type, and URL.
- Summary — counts (DNS pass, HTTPS pass, failed, skipped) and average response time.
- Results overview — table of each domain with DNS | SSL | HTTPS | RPC status.
- Test Results by Domain — per-domain detail (DNS, SSL, HTTPS, Blockscout API, RPC).
Output directory: docs/04-configuration/verification-evidence/e2e-verification-<timestamp>/
Files: verification_report.md, all_e2e_results.json, *_https_headers.txt, *_rpc_response.txt.
Known E2E warnings (public profile)
When running from outside LAN or when backends are down, the following endpoints commonly show HTTPS warn (not fail, due to E2E_OPTIONAL_WHEN_FAIL).
These known items do not block contract or pool completion. Fix when convenient; E2E still passes when they are in E2E_OPTIONAL_WHEN_FAIL.
2026-03-26 note: after recovering NPMplus CT 10233 and re-running update-npmplus-proxy-hosts-api.sh, the latest public profile passed for all currently tested public domains, including Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U.
2026-03-29 update: public profile passed again with Failed: 0 after fixing the explorer /api/v1 proxy, removing the stale 192.168.11.52 address from CT 10232, and moving VMID 10092 off 192.168.11.37 so MIM4U owns that IP exclusively. Current evidence: docs/04-configuration/verification-evidence/e2e-verification-20260329_170619/.
| Endpoint | Typical cause |
|---|---|
| admin.d-bis.org, dbis-admin.d-bis.org | 502 — admin frontend (VMID 10130) unreachable from public |
| core.d-bis.org | DNS/502 until NPM row and dbis_core client upstream are provisioned |
| dbis-api.d-bis.org, dbis-api-2.d-bis.org | 502 — API backends (10150/10151) unreachable |
| secure.d-bis.org | 502 — secure portal backend unreachable |
| mifos.d-bis.org | 502 — Mifos (VMID 5800) unreachable from public |
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | Resolved on 2026-03-29. If these regress to 502, first check for IP ownership conflicts on 192.168.11.37 before debugging nginx. |
| studio.sankofa.nexus | Historically 404 when the proxy misses /studio/ or backend 192.168.11.72:8000; verifier checks /studio/. Passed on 2026-03-26 after the NPMplus host update |
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; verify-end-to-end-routing.sh checks https://…/health (200), not /. A separate marketing site on the apex hostname (if desired) needs another upstream or app routes—NPM still points phoenix.sankofa.nexus at the Fastify API today. |
| the-order.sankofa.nexus | 502 if 10210 HAProxy or backend portal is down. NPM defaults upstream to 192.168.11.39:80 (order-haproxy). Fallback: THE_ORDER_UPSTREAM_IP / THE_ORDER_UPSTREAM_PORT = portal 192.168.11.51:3000 |
| keycloak.sankofa.nexus, admin.sankofa.nexus, portal.sankofa.nexus | Resolved on 2026-03-29 after removing the duplicate 192.168.11.52 address from CT 10232. If these regress, verify ARP ownership of 192.168.11.52 before restarting Keycloak or NPMplus. |
| dash.sankofa.nexus | Still optional / unprovisioned. DNS/SSL/HTTPS may warn or skip until IP_SANKOFA_DASH and its app upstream are intentionally wired. |
| docs.d-bis.org, blockscout.defi-oracle.io | Same optional-when-fail behavior; blockscout.defi-oracle.io also runs optional /api/v2/stats like explorer.d-bis.org. |
Verifier behavior (2026-03): openssl s_client is wrapped with timeout (E2E_OPENSSL_TIMEOUT default 15s, E2E_OPENSSL_X509_TIMEOUT default 5s) so --profile=private / --profile=all cannot hang. --profile=all merges private and public E2E_OPTIONAL_WHEN_FAIL lists for temporary regressions. Install wscat (npm install -g wscat) for full WSS JSON-RPC checks; the script uses wscat -n to match curl -k, and now treats a clean wscat exit as a successful full WebSocket check even when the tool prints no JSON output.
Canonical www redirects (2026-03): For www.sankofa.nexus, www.phoenix.sankofa.nexus, and www.the-order.sankofa.nexus, HTTP 301/308 must include a Location whose host matches the expected apex (E2E_WWW_CANONICAL_BASE in verify-end-to-end-routing.sh). Wrong apex → HTTPS fail. Missing Location → warn.
Cloudflare bulk DNS: scripts/update-all-dns-to-public-ip.sh supports --dry-run (no API calls) and --zone-only=sankofa.nexus (or d-bis.org | mim4u.org | defi-oracle.io) to limit blast radius. Env: CLOUDFLARE_DNS_DRY_RUN=1, DNS_ZONE_ONLY=….
WebSocket test-format warnings: Older runs may show "connection established but RPC test failed" when wscat is used: the upgrade succeeded but the verifier expected printable "result" output. The script now accepts either explicit JSON output or a clean wscat exit, so current runs treat those WS checks as pass when the connection completes successfully. The script also accepts Chain 138 chainId 0x8a in output.
Remediation (when you want these to pass from public)
| Goal | Action |
|---|---|
| 502s (dbis-admin, dbis-api, secure, mifos) | From LAN: ./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e] or ./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e. If NPMplus API is unreachable: ./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh. Runbook: 502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md. |
| 404 studio.sankofa.nexus | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for studio.sankofa.nexus points to it. See ALL_VMIDS_ENDPOINTS.md, SANKOFA_STUDIO_E2E_FLOW.md, SANKOFA_STUDIO_DEPLOYMENT.md. |
| the-order 502 | Check 10210 HAProxy (curl http://192.168.11.39:80/ with Host: the-order.sankofa.nexus) and portal 192.168.11.51:3000. Re-provision: bash scripts/deployment/provision-order-haproxy-10210.sh. NPM refresh: bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh. Direct portal bypass: THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000 for that run. |