proxmox/docs/03-deployment/PUBLIC_SECTOR_LIVE_DEPLOYMENT_CHECKLIST.md

Public sector live deployment checklist (Complete Credential, SMOA, Phoenix)

Last updated: 2026-03-26
Related: PUBLIC_SECTOR_TENANCY_MARKETPLACE_AND_DEPLOYMENT_BASELINE.md, COMPLETE_CREDENTIAL_EIDAS_PROGRAM_REPOS.md, DEPLOY_CONFIRM_AND_FULL_E2E_RUNBOOK.md, config/public-sector-program-manifest.json

This checklist tracks proxmox-repo automation and sibling repos (../complete-credential, ../smoa). Rows marked Done (session) were executed from an operator host with LAN access unless noted.

---

Execution log (2026-03-23)

Action

Result

Sankofa api + portal (workstation)

API: websocket.ts imports logger; GraphQL/schema fixes under api/src. Portal: Apollo + dashboard GraphQL, UI primitives, root: true .eslintrc.json with @typescript-eslint + strict no-explicit-any / no-console / a11y / import/order (optional hardening: lib clients typed with unknown, form htmlFor/id, escaped entities). pnpm exec tsc --noEmit + pnpm build clean. Deploy: sync portal/ (+ lockfile) to CT 7801, pnpm install && pnpm build, restart sankofa-portal; sync 7800 API if needed

Execution log (2026-03-26)

Action	Result
./scripts/run-all-operator-tasks-from-lan.sh (live, no --dry-run)	Exit 0 (~36 min); W0-1 NPMplus RPC/proxy host updates; W0-3 live NPMplus backup; Blockscout verification step ran
NPMplus update script	Some hosts logged duplicate-create then PUT recovery; rpc.tw-core.d-bis.org and *.tw-core.d-bis.org showed repeated failures — review those rows in NPM UI if traffic depends on them
scripts/maintenance/diagnose-vm-health-via-proxmox-ssh.sh	Completed: Phoenix CTs 7800–7803 running on r630-01; NPMplus 10233 up; port 81 check OK
scripts/maintenance/npmplus-verify-port81.sh	Restored in repo; loopback :81 returns HTTP 301 (redirect) — treated as reachable
pct exec 7800 / 7801: ss -tlnp	As of 2026-03-26 session: no listeners. As of 2026-03-23 follow-up: 7800 API can reach active + /health on :4000 when sankofa-api is deployed; 7801 portal needs current portal tree + successful pnpm build on the CT (see 2026-03-23 log row above)
pct exec 7802 Keycloak	http://127.0.0.1:8080/ → 200; /health/ready → 404 (version may use different health path)
./scripts/run-completable-tasks-from-anywhere.sh	Exit 0
E2E_ACCEPT_502_INTERNAL=1 ./scripts/verify/verify-end-to-end-routing.sh	0 failed; report docs/04-configuration/verification-evidence/e2e-verification-20260325_182512/
./scripts/verify/run-contract-verification-with-proxy.sh	Exit 0
complete-credential Phase 1 compose + run-phase1-synthetic.sh	OK (operator console 8087 = 200)
../smoa: ./gradlew :app:assembleDebug	BUILD SUCCESSFUL; APK: smoa/app/build/outputs/apk/debug/app-debug.apk
scripts/deployment/sync-sankofa-portal-7801.sh + NPM alignment	Portal tree synced to CT 7801, pnpm install + pnpm build, sankofa-portal active (*:3000). NPM proxy IDs 3–6: sankofa.nexus / www → 192.168.11.51:3000; phoenix.sankofa.nexus / www → 192.168.11.50:4000. Repeatable deploy: ./scripts/deployment/sync-sankofa-portal-7801.sh (--dry-run first).
validate-config-files.sh / run-completable-tasks-from-anywhere.sh	Exit 0

---

Execution log (2026-03-25)

Action	Result
RPC 192.168.11.221:8545 / 192.168.11.211:8545	HTTP 201
SSH root@192.168.11.10 / .11	OK (BatchMode)
./scripts/run-completable-tasks-from-anywhere.sh	Exit 0
./scripts/verify/check-contracts-on-chain-138.sh	59/59 present
E2E_ACCEPT_502_INTERNAL=1 ./scripts/verify/verify-end-to-end-routing.sh	37 domains, 0 failed; report under docs/04-configuration/verification-evidence/e2e-verification-20260325_165153/
https://phoenix.sankofa.nexus/, https://sankofa.nexus/	HTTP 200
http://192.168.11.50:4000/health, :51:3000, :52:8080/health/ready	No HTTP response from operator host (hosts ping; services may be down, firewalled, or not bound) — re-check on Proxmox / in-container
./scripts/verify/backup-npmplus.sh --dry-run	OK
./scripts/verify/run-contract-verification-with-proxy.sh	Exit 0
./scripts/run-all-operator-tasks-from-lan.sh --dry-run	Printed wave0 + verify sequence
cd smom-dbis-138 && forge test --match-path 'test/e2e/*.sol'	Exit 0
cd ../smoa && ./gradlew smoaVerify --no-daemon	Exit 0
complete-credential: git submodule status	Submodules present on commits
docker compose -f integration/docker-compose.phase1.yml config	Valid
docker compose -f integration/docker-compose.phase1.yml up -d	All Phase 1 containers up
Rebuild + recreate cc-operator-console; ./integration/run-phase1-synthetic.sh	OK

---

Checklist

ID	Task	Status
A1	LAN / VPN; Proxmox SSH	Done (session)
A2	Root .env + smom-dbis-138/.env for operator	Operator to confirm secrets present
A3	config/public-sector-program-manifest.json valid	Done (completable)
B1	NPMplus proxy + TLS for public FQDNs	Done (2026-03-26) — run-wave0-from-lan.sh / update script applied; spot-check rpc.tw-core / *.tw-core in NPM if needed
B2	scripts/verify/backup-npmplus.sh (live)	Done (2026-03-26) — W0-3 as part of run-all-operator-tasks-from-lan.sh
B3	scripts/maintenance/npmplus-verify-port81.sh	Done — script restored; SSH pct exec 10233 loopback :81
C1	Phoenix stack VMIDs 7800–7803 per SERVICE_DESCRIPTIONS.md	7802 Keycloak: HTTP 200 on / inside CT. 7800 API: listener :4000 (/health OK). 7801 portal: sankofa-portal active, Next on :3000 (sync via scripts/deployment/sync-sankofa-portal-7801.sh)
C2	Keycloak realms: admin / tenant / org-unit RBAC	Product + IdP work — not automated here
C3	Phoenix API + portal wired; GraphQL /graphql, /health	API: curl -sS http://192.168.11.50:4000/health. Portal: curl -sS http://192.168.11.51:3000/ (Next HTML). NPM: apex sankofa / phoenix hosts → .51:3000 / .50:4000 (not Blockscout)
C4	Service catalog SKUs + entitlements (billing optional)	Product — see tenancy baseline G2
D1	SMOA LXC per smoa/backend/docs/LXC-PROXMOX-CONTAINERS.md	Deploy on Proxmox
D2	SMOA API behind NPM	After D1
D3	Release APK + download URL or MDM	Debug APK built (2026-03-26): ../smoa/app/build/outputs/apk/debug/app-debug.apk — publish via CI signed release + NPM/static URL or MDM
D4	Device E2E against prod API	After D2–D3
E1	complete-credential submodules initialized	Done (session)
E2	Phase 1 Docker stack local/CI	Done (session) — not yet Proxmox production
E3	./integration/run-phase1-synthetic.sh after console rebuild	Done (session)
E4	Production slice / dedicated LXC for cc-*	Architecture choice (profile A/B/C)
F1	Chain 138 on-chain contract check	Done (session)
F2	Blockscout verification	Done (session)
F3	Public E2E routing	Done (session, 502-tolerant flag)
G1	Logs, metrics, DB backups for Phoenix + SMOA + CC DBs	Operational runbooks
G2	Incident ownership per stack	Process

---

Quick commands (repo root unless noted)

./scripts/run-completable-tasks-from-anywhere.sh
source scripts/lib/load-project-env.sh && ./scripts/verify/check-contracts-on-chain-138.sh
E2E_ACCEPT_502_INTERNAL=1 ./scripts/verify/verify-end-to-end-routing.sh
./scripts/verify/run-contract-verification-with-proxy.sh
./scripts/deployment/sync-sankofa-portal-7801.sh --dry-run   # then run without --dry-run (portal → CT 7801)
./scripts/verify/backup-npmplus.sh --dry-run   # then run without --dry-run

Complete Credential (sibling clone):

cd ../complete-credential
docker compose -f integration/docker-compose.phase1.yml up -d --build
./integration/run-phase1-synthetic.sh

SMOA:

cd ../smoa && ./gradlew smoaVerify --no-daemon

---

Follow-ups

1. Phoenix LAN services: Curl 192.168.11.50:4000/health and 192.168.11.51:3000/; if portal is down, run sync-sankofa-portal-7801.sh or systemctl status sankofa-portal on CT 7801.
2. Operator full wave: ./scripts/run-all-operator-tasks-from-lan.sh only when NPM RPC fix + backup + verify are intentionally desired (mutates NPM).
3. Production Complete Credential: Move from laptop Docker to dedicated LXC and NPM routes per deployment profile.