dedb55e05c
Made-with: Cursor
15 KiB
E2E verification — endpoint inventory and profiles
Source: scripts/verify/verify-end-to-end-routing.sh (DOMAIN_TYPES).
List from CLI (public): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=public
List from CLI (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=private
Run E2E (public profile recommended): ./scripts/verify/verify-end-to-end-routing.sh --profile=public (from LAN with DNS or use E2E_USE_SYSTEM_RESOLVER=1 and /etc/hosts per E2E_DNS_FROM_LAN_RUNBOOK.md).
Run E2E (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --profile=private.
What each hostname should present (operator narrative): FQDN_EXPECTED_CONTENT.md.
Latest verified public pass: 2026-03-27 via bash scripts/verify/verify-end-to-end-routing.sh --profile=public with report at verification_report.md. Result: exit 0, DNS passed: 38, Failed: 0, HTTPS passed: 19, Skipped / optional: 1 (after run-all-operator-tasks-from-lan.sh NPM sync; rpc.defi-oracle.io may log HTTP 405 on the verifier probe but stays non-failing for the profile).
Latest verified private/admin pass: 2026-03-27 via bash scripts/verify/verify-end-to-end-routing.sh --profile=private with report at verification_report.md. Result: exit 0, DNS passed: 4, Failed: 0.
Evidence folders: Each run creates verification-evidence/e2e-verification-YYYYMMDD_HHMMSS/. Commit the runs you want on record; older dirs can be removed locally to reduce noise (scripts/maintenance/prune-e2e-verification-evidence.sh --dry-run lists candidates). Routing truth is not inferred from old reports—use ALL_VMIDS_ENDPOINTS.md.
Verification profiles
- Public profile (default for routine E2E): web, api, public RPC endpoints.
- Private/admin profile: private RPC and Fireblocks RPC endpoints. Run separately for internal operations.
Full endpoint inventory (combined)
| Endpoint | Type | URL | Description (content provided) |
|---|---|---|---|
| explorer.d-bis.org | web | https://explorer.d-bis.org | Blockscout-style blockchain explorer for Chain 138: blocks, transactions, addresses, contracts, tokens, verification. |
| dbis-admin.d-bis.org | web | https://dbis-admin.d-bis.org | DBIS admin dashboard and frontend (VMID 10130). |
| secure.d-bis.org | web | https://secure.d-bis.org | Secure DBIS frontend / authenticated portal. |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org | DBIS core API: token aggregation, Crypto.com OTC, exchange endpoints (VMID 10150). |
| dbis-api-2.d-bis.org | api | https://dbis-api-2.d-bis.org | DBIS API secondary instance (VMID 10151). |
| mim4u.org | web | https://mim4u.org | MIM4U main site. |
| www.mim4u.org | web | https://www.mim4u.org | MIM4U www. |
| secure.mim4u.org | web | https://secure.mim4u.org | MIM4U secure portal. |
| training.mim4u.org | web | https://training.mim4u.org | MIM4U training site. |
| sankofa.nexus | web | https://sankofa.nexus | Sankofa Nexus root / web. |
| www.sankofa.nexus | web | https://www.sankofa.nexus | 301 to https://sankofa.nexus (canonical apex; NPM advanced_config). |
| phoenix.sankofa.nexus | web | https://phoenix.sankofa.nexus | Phoenix API (7800); E2E uses /health for HTTPS check. |
| www.phoenix.sankofa.nexus | web | https://www.phoenix.sankofa.nexus | 301 to https://phoenix.sankofa.nexus (canonical apex; NPM advanced_config). |
| the-order.sankofa.nexus | web | https://the-order.sankofa.nexus | OSJ management portal (secure auth); app the_order at ~/projects/the_order. NPM upstream default: order-haproxy VMID 10210 http://192.168.11.39:80 → portal 192.168.11.51:3000 (provision-order-haproxy-10210.sh). Override with THE_ORDER_UPSTREAM_* for direct portal if 10210 is down. |
| www.the-order.sankofa.nexus | web | https://www.the-order.sankofa.nexus | 301 to https://the-order.sankofa.nexus (canonical apex; NPM advanced_config). |
| studio.sankofa.nexus | web | https://studio.sankofa.nexus | Sankofa Studio (FusionAI Creator) at VMID 7805. |
| cacti-alltra.d-bis.org | web | https://cacti-alltra.d-bis.org | Cacti monitoring UI for Alltra. |
| cacti-hybx.d-bis.org | web | https://cacti-hybx.d-bis.org | Cacti monitoring UI for HYBX. |
| mifos.d-bis.org | web | https://mifos.d-bis.org | Mifos X / Fineract banking and microfinance platform (VMID 5800). |
| dapp.d-bis.org | web | https://dapp.d-bis.org | DApp frontend for Chain 138 bridge (VMID 5801). |
| gitea.d-bis.org | web | https://gitea.d-bis.org | Gitea git repository and CI (Dev VM 5700). |
| dev.d-bis.org | web | https://dev.d-bis.org | Dev VM web / Codespaces entry. |
| codespaces.d-bis.org | web | https://codespaces.d-bis.org | Codespaces / dev environment entry. |
| rpc-http-pub.d-bis.org | rpc-http | https://rpc-http-pub.d-bis.org | Chain 138 public JSON-RPC HTTP (VMID 2201). |
| rpc-ws-pub.d-bis.org | rpc-ws | wss://rpc-ws-pub.d-bis.org | Chain 138 public JSON-RPC WebSocket. |
| rpc.d-bis.org | rpc-http | https://rpc.d-bis.org | Chain 138 RPC HTTP (alias). |
| rpc2.d-bis.org | rpc-http | https://rpc2.d-bis.org | Chain 138 RPC HTTP (second). |
| ws.rpc.d-bis.org | rpc-ws | wss://ws.rpc.d-bis.org | Chain 138 RPC WebSocket. |
| ws.rpc2.d-bis.org | rpc-ws | wss://ws.rpc2.d-bis.org | Chain 138 RPC WebSocket (second). |
| rpc-http-prv.d-bis.org | rpc-http | https://rpc-http-prv.d-bis.org | Chain 138 private/admin RPC HTTP (VMID 2101). |
| rpc-ws-prv.d-bis.org | rpc-ws | wss://rpc-ws-prv.d-bis.org | Chain 138 private RPC WebSocket. |
| rpc-fireblocks.d-bis.org | rpc-http | https://rpc-fireblocks.d-bis.org | Chain 138 RPC for Fireblocks Web3 (VMID 2301). |
| ws.rpc-fireblocks.d-bis.org | rpc-ws | wss://ws.rpc-fireblocks.d-bis.org | Chain 138 RPC WebSocket for Fireblocks. |
| rpc.public-0138.defi-oracle.io | rpc-http | https://rpc.public-0138.defi-oracle.io | Defi Oracle Chain 138 public RPC. |
| rpc.defi-oracle.io | rpc-http | https://rpc.defi-oracle.io | Defi Oracle RPC. |
| wss.defi-oracle.io | rpc-ws | wss://wss.defi-oracle.io | Defi Oracle RPC WebSocket. |
| rpc-alltra.d-bis.org | rpc-http | https://rpc-alltra.d-bis.org | Alltra chain RPC HTTP. |
| rpc-alltra-2.d-bis.org | rpc-http | https://rpc-alltra-2.d-bis.org | Alltra chain RPC HTTP (2). |
| rpc-alltra-3.d-bis.org | rpc-http | https://rpc-alltra-3.d-bis.org | Alltra chain RPC HTTP (3). |
| rpc-hybx.d-bis.org | rpc-http | https://rpc-hybx.d-bis.org | HYBX chain RPC HTTP. |
| rpc-hybx-2.d-bis.org | rpc-http | https://rpc-hybx-2.d-bis.org | HYBX chain RPC HTTP (2). |
| rpc-hybx-3.d-bis.org | rpc-http | https://rpc-hybx-3.d-bis.org | HYBX chain RPC HTTP (3). |
Endpoints by type
Web
| Domain | URL |
|---|---|
| explorer.d-bis.org | https://explorer.d-bis.org |
| dbis-admin.d-bis.org | https://dbis-admin.d-bis.org |
| secure.d-bis.org | https://secure.d-bis.org |
| mim4u.org | https://mim4u.org |
| www.mim4u.org | https://www.mim4u.org |
| secure.mim4u.org | https://secure.mim4u.org |
| training.mim4u.org | https://training.mim4u.org |
| sankofa.nexus | https://sankofa.nexus |
| www.sankofa.nexus | https://www.sankofa.nexus |
| phoenix.sankofa.nexus | https://phoenix.sankofa.nexus |
| www.phoenix.sankofa.nexus | https://www.phoenix.sankofa.nexus |
| the-order.sankofa.nexus | https://the-order.sankofa.nexus |
| www.the-order.sankofa.nexus | https://www.the-order.sankofa.nexus |
| studio.sankofa.nexus | https://studio.sankofa.nexus |
| cacti-alltra.d-bis.org | https://cacti-alltra.d-bis.org |
| cacti-hybx.d-bis.org | https://cacti-hybx.d-bis.org |
| mifos.d-bis.org | https://mifos.d-bis.org |
| dapp.d-bis.org | https://dapp.d-bis.org |
| gitea.d-bis.org | https://gitea.d-bis.org |
| dev.d-bis.org | https://dev.d-bis.org |
| codespaces.d-bis.org | https://codespaces.d-bis.org |
API
| Domain | URL |
|---|---|
| dbis-api.d-bis.org | https://dbis-api.d-bis.org |
| dbis-api-2.d-bis.org | https://dbis-api-2.d-bis.org |
RPC HTTP (public)
| Domain | URL |
|---|---|
| rpc-http-pub.d-bis.org | https://rpc-http-pub.d-bis.org |
| rpc.d-bis.org | https://rpc.d-bis.org |
| rpc2.d-bis.org | https://rpc2.d-bis.org |
| rpc.public-0138.defi-oracle.io | https://rpc.public-0138.defi-oracle.io |
| rpc.defi-oracle.io | https://rpc.defi-oracle.io |
| rpc-alltra.d-bis.org | https://rpc-alltra.d-bis.org |
| rpc-alltra-2.d-bis.org | https://rpc-alltra-2.d-bis.org |
| rpc-alltra-3.d-bis.org | https://rpc-alltra-3.d-bis.org |
| rpc-hybx.d-bis.org | https://rpc-hybx.d-bis.org |
| rpc-hybx-2.d-bis.org | https://rpc-hybx-2.d-bis.org |
| rpc-hybx-3.d-bis.org | https://rpc-hybx-3.d-bis.org |
RPC WebSocket (public)
| Domain | URL |
|---|---|
| rpc-ws-pub.d-bis.org | wss://rpc-ws-pub.d-bis.org |
| ws.rpc.d-bis.org | wss://ws.rpc.d-bis.org |
| ws.rpc2.d-bis.org | wss://ws.rpc2.d-bis.org |
| wss.defi-oracle.io | wss://wss.defi-oracle.io |
RPC HTTP (private/admin profile)
| Domain | URL |
|---|---|
| rpc-http-prv.d-bis.org | https://rpc-http-prv.d-bis.org |
| rpc-fireblocks.d-bis.org | https://rpc-fireblocks.d-bis.org |
RPC WebSocket (private/admin profile)
| Domain | URL |
|---|---|
| rpc-ws-prv.d-bis.org | wss://rpc-ws-prv.d-bis.org |
| ws.rpc-fireblocks.d-bis.org | wss://ws.rpc-fireblocks.d-bis.org |
Report content
After each run, the verification report includes:
- All endpoints — table of every domain, type, and URL.
- Summary — counts (DNS pass, HTTPS pass, failed, skipped) and average response time.
- Results overview — table of each domain with DNS | SSL | HTTPS | RPC status.
- Test Results by Domain — per-domain detail (DNS, SSL, HTTPS, Blockscout API, RPC).
Output directory: docs/04-configuration/verification-evidence/e2e-verification-<timestamp>/
Files: verification_report.md, all_e2e_results.json, *_https_headers.txt, *_rpc_response.txt.
Known E2E warnings (public profile)
When running from outside LAN or when backends are down, the following endpoints commonly show HTTPS warn (not fail, due to E2E_OPTIONAL_WHEN_FAIL).
These known items do not block contract or pool completion. Fix when convenient; E2E still passes when they are in E2E_OPTIONAL_WHEN_FAIL.
2026-03-26 note: after recovering NPMplus CT 10233 and re-running update-npmplus-proxy-hosts-api.sh, the latest public profile passed for all currently tested public domains, including Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U.
| Endpoint | Typical cause |
|---|---|
| dbis-admin.d-bis.org | 502 — backend (VMID 10130) unreachable from public |
| dbis-api.d-bis.org, dbis-api-2.d-bis.org | 502 — API backends (10150/10151) unreachable |
| secure.d-bis.org | 502 — secure portal backend unreachable |
| mifos.d-bis.org | 502 — Mifos (VMID 5800) unreachable from public |
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | 502 — MIM4U web backends (192.168.11.37:80); non-blocking for contract/pool |
| studio.sankofa.nexus | Historically 404 when the proxy misses /studio/ or backend 192.168.11.72:8000; verifier checks /studio/. Passed on 2026-03-26 after the NPMplus host update |
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; verify-end-to-end-routing.sh checks https://…/health (200), not /. A separate marketing site on the apex hostname (if desired) needs another upstream or app routes—NPM still points phoenix.sankofa.nexus at the Fastify API today. |
| the-order.sankofa.nexus | 502 if 10210 HAProxy or backend portal is down. NPM defaults upstream to 192.168.11.39:80 (order-haproxy). Fallback: THE_ORDER_UPSTREAM_IP / THE_ORDER_UPSTREAM_PORT = portal 192.168.11.51:3000 |
Verifier behavior (2026-03): openssl s_client is wrapped with timeout (E2E_OPENSSL_TIMEOUT default 15s, E2E_OPENSSL_X509_TIMEOUT default 5s) so --profile=private / --profile=all cannot hang. --profile=all merges private and public E2E_OPTIONAL_WHEN_FAIL lists for temporary regressions. Install wscat (npm install -g wscat) for full WSS JSON-RPC checks; the script uses wscat -n to match curl -k, and now treats a clean wscat exit as a successful full WebSocket check even when the tool prints no JSON output.
Canonical www redirects (2026-03): For www.sankofa.nexus, www.phoenix.sankofa.nexus, and www.the-order.sankofa.nexus, HTTP 301/308 must include a Location whose host matches the expected apex (E2E_WWW_CANONICAL_BASE in verify-end-to-end-routing.sh). Wrong apex → HTTPS fail. Missing Location → warn.
Cloudflare bulk DNS: scripts/update-all-dns-to-public-ip.sh supports --dry-run (no API calls) and --zone-only=sankofa.nexus (or d-bis.org | mim4u.org | defi-oracle.io) to limit blast radius. Env: CLOUDFLARE_DNS_DRY_RUN=1, DNS_ZONE_ONLY=….
WebSocket test-format warnings: Older runs may show "connection established but RPC test failed" when wscat is used: the upgrade succeeded but the verifier expected printable "result" output. The script now accepts either explicit JSON output or a clean wscat exit, so current runs treat those WS checks as pass when the connection completes successfully. The script also accepts Chain 138 chainId 0x8a in output.
Remediation (when you want these to pass from public)
| Goal | Action |
|---|---|
| 502s (dbis-admin, dbis-api, secure, mifos) | From LAN: ./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e] or ./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e. If NPMplus API is unreachable: ./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh. Runbook: 502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md. |
| 404 studio.sankofa.nexus | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for studio.sankofa.nexus points to it. See ALL_VMIDS_ENDPOINTS.md, SANKOFA_STUDIO_E2E_FLOW.md, SANKOFA_STUDIO_DEPLOYMENT.md. |
| the-order 502 | Check 10210 HAProxy (curl http://192.168.11.39:80/ with Host: the-order.sankofa.nexus) and portal 192.168.11.51:3000. Re-provision: bash scripts/deployment/provision-order-haproxy-10210.sh. NPM refresh: bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh. Direct portal bypass: THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000 for that run. |