d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
7df27bde328c3751e20493eb79289bb2a35df084
proxmox/docs/00-meta/REMAINING_WORK_DETAILED_STEPS.md
defiQUG b3a8fe4496
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
Details
chore: sync all changes to Gitea 
- Config, docs, scripts, and backup manifests
- Submodule refs unchanged (m = modified content in submodules)

Made-with: Cursor
2026-03-02 11:37:34 -08:00

21 KiB
Raw Blame History

Remaining Work — Detailed Steps for Each Task

Last Updated: 2026-02-28
Purpose: Single list of all remaining work with step-by-step instructions.
Sources: E2E_COMPLETION_TASKS_DETAILED_LIST.md, WAVE2_WAVE3_OPERATOR_CHECKLIST.md, TODO_TASK_LIST_MASTER.md.

Copy-paste runbook: For a single page of ready-to-run commands, see NEXT_STEPS_OPERATOR.md.

Full plan (required / optional / recommended): COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md — Wave 0 gates, required phases/codebase/security, optional, recommended (139+ items).

Execution order: Wave 0 → Wave 1 → Wave 2 → Wave 3 → Ongoing. Within each wave, run tasks in parallel where possible.

Infra deployment readiness: For a single checklist of what is already in place (templates on all hosts, deps, scripts) vs what unblocks completion (LAN, SSH, creds), see 03-deployment/INFRA_DEPLOYMENT_LOCKED_AND_LOADED.md.

✅ Can Be Accomplished Now (No LAN / Proxmox / Creds Required)

These can be done from your current environment (e.g. dev machine, WSL, CI) without being on LAN, SSH to Proxmox, or setting NPM_PASSWORD/PRIVATE_KEY.

Item What to do
W1-11 Doc consolidation; archive — move/refactor per ARCHIVE_CANDIDATES.md; consolidate by folder (01-, 02-, …).
W1-12 Quick reference cards; decision trees — edit QUICK_REFERENCE_CARDS.md, CONFIGURATION_DECISION_TREE, 04-configuration README.
W1-9, W1-10, W1-13 Docs/design — review or refine NETWORK_ARCHITECTURE §3–7, VLAN migration plan, UDM_PRO_VLAN_* docs, IP assignments, connectivity matrix, runbook cross-links.
W1-20 Shellcheck — run bash scripts/verify/run-shellcheck.sh --optional; or install shellcheck (apt install shellcheck / brew install shellcheck) and run without --optional to fix reported issues.
W1-21 Config validation / env standardization — extend validate-config-files.sh or ENV_STANDARDIZATION docs if needed.
W1-22 Token-aggregation; CoinGecko — follow COINGECKO_SUBMISSION.md; code/docs in repo.
W1-23 Chain 138 Snap — market data UI, swap quotes, bridge routes in metamask-integration.
W1-24 Explorer — dark mode, network selector, sync indicator in explorer-monorepo.
W1-26 API keys — obtain keys (sign up at URLs in reports/API_KEYS_REQUIRED.md); set in root and subproject .env for any keys you have or can get.
API Keys & Secrets Same: open report, sign up where needed, add values to .env; restart services only after you have access to run them.
W1-14 dbis_core TypeScript — fix ~1186 TS errors by module: run npx prisma generate in dbis_core (fixes @prisma/client); then add explicit types for implicit any (e.g. callback params). Sample fix applied in cbdc-fx.service.ts.
W1-15 – W1-17 Placeholders / code — smom canonical addresses env-only, AlltraAdapter fee, smart accounts kit, quote service Fabric 999, .bak deprecation; see PLACEHOLDERS_AND_* and E2E Part 6.
Placeholders & Code (E2E) Code/docs in smom-dbis-138, dbis_core, the-order (e-signature docs, document security design), OMNIS, Tezos relay — any work that doesn’t require running infra.
CCIP checklist (dry) Run bash scripts/ccip/ccip-deploy-checklist.sh to validate env and print deployment order (no deploy).
Validation commands Re-run anytime: run-all-validation, validate-config-files, validate-genesis, verify-end-to-end-routing, run-wave0-from-lan.sh --dry-run, phase4 --show-steps/--dry-run, schedule-*-cron.sh --show.

Not doable now (need LAN, Proxmox, or creds): W0-1, W0-2, W0-3, crontab --install, W1-1, W1-2, W1-8 (backup run), W1-19, W2-* (all deploy), W3-* (all), CT-1a, O-4 (explorer logs via SSH). Deferred/backlog (W1-3, W1-4) are “assign to backlog,” not execute now.

Completed (2026-02-05): W1-11 (32 files archived to docs/archive/00-meta-status/), W1-12 (decision tree links, 04-config README, QUICK_REFERENCE_CARDS), W1-9/10/13 (NETWORK_ARCHITECTURE runbook cross-links), W1-20 (shellcheck --optional run), W1-21 (ENV_STANDARDIZATION + validate-config-files ref), W1-22–W1-24 (CoinGecko/Snap/Explorer refs in QUICK_REFERENCE_CARDS), W1-26/API keys (report + .env.example pointer), W1-14 (dbis_core: sample TS fix in cbdc-fx.service.ts; doc for prisma generate + implicit any), W1-15–W1-17 (PLACEHOLDERS canonical env note), CCIP checklist + all validation commands run.

Completed (2026-02-20): Doc consolidation continued — NEXT_STEPS_INDEX, DOCUMENTATION_CONSOLIDATION_PLAN; Batch 4+5 → 00-meta-pruned; ALL_TASKS_COMPLETE → root-status-reports; project root cleanup → archive/root-cleanup-20260220; fix-wsl-ip.sh → scripts/. Completable-from-anywhere run: config validation OK, on-chain check 45/45, run-all-validation --skip-genesis OK, reconcile-env --print. ARCHIVE_CANDIDATES "Last reviewed" set.

Completed (plan implementation): COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md added; cross-links from PHASES_AND_TASKS_MASTER, TODO_TASK_LIST_MASTER, RECOMMENDATIONS_OPERATOR_CHECKLIST, REMAINING_WORK_DETAILED_STEPS, OPTIONAL_RECOMMENDATIONS_INDEX, RUNBOOKS_MASTER_INDEX, ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST, OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST, FULL_PARALLEL_EXECUTION_ORDER, NEXT_STEPS_INDEX, MASTER_INDEX. Validation: run-all-validation --skip-genesis OK; run-completable-tasks-from-anywhere.sh OK (config, on-chain 36/36, reconcile-env); phase4-sovereign-tenants.sh --show-steps and schedule-daily-weekly-cron.sh --show run.

Wave 0 — Gates (Do First When Credentials Allow)

W0-1: NPMplus RPC fix (405)

Blocker: Must run from a host on the same LAN as NPMplus (192.168.11.x).

Detailed steps:

  1. From a machine on LAN (e.g. 192.168.11.x), open a terminal in the project root.
  2. Option A — Run the combined Wave 0 script (RPC fix + backup): 
    cd /path/to/proxmox
bash scripts/run-wave0-from-lan.sh
    (Use --skip-backup if you only want the RPC fix.)
  3. Option B — Run only the RPC fix script: 
    bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh
  4. Verify: run bash scripts/verify/verify-end-to-end-routing.sh — RPC domains should pass (no longer 405).

W0-2: sendCrossChain (real)

Blocker: PRIVATE_KEY and LINK approved for fee in .env; bridge contract: 0x971cD9D156f193df8051E48043C476e53ECd4693.

Detailed steps:

  1. In project root, ensure .env has:
    • PRIVATE_KEY — wallet that will send and pay gas/fees.
    • LINK or equivalent approved for the bridge fee token if required.
  2. Run the bridge script without --dry-run: 
    bash scripts/bridge/run-send-cross-chain.sh <amount> [recipient]
    Example: bash scripts/bridge/run-send-cross-chain.sh 0.01 0x...
  3. Confirm transaction on chain; check bridge contract and destination chain as needed.

W0-3: NPMplus backup

Blocker: NPM_PASSWORD in .env; NPMplus container reachable (run from LAN or where NPMplus API is reachable).

Detailed steps:

  1. Set NPM_PASSWORD in .env (and optionally NPM_HOST if not default).
  2. From a host that can reach NPMplus (e.g. on LAN): 
    bash scripts/verify/backup-npmplus.sh
    Or run the combined script: bash scripts/run-wave0-from-lan.sh (omit --skip-backup).
  3. Backup artifacts are written to the path reported by the script (e.g. under logs/ or verification evidence).

Crontab installs (operator host)

Blocker: Run on the host where the crontab should be installed (e.g. jump host or Proxmox node).

NPMplus backup cron (W1-8 part)

Detailed steps:

  1. On the target host: cd /path/to/proxmox.
  2. Show the line: bash scripts/maintenance/schedule-npmplus-backup-cron.sh --show.
  3. Install: bash scripts/maintenance/schedule-npmplus-backup-cron.sh --install.
  4. Default: daily at 03:00; log: logs/npmplus-backup.log.

Daily/weekly checks cron (O-1, O-2, O-3)

Detailed steps:

  1. On the target host: cd /path/to/proxmox.
  2. Show lines: bash scripts/maintenance/schedule-daily-weekly-cron.sh --show.
  3. Install: bash scripts/maintenance/schedule-daily-weekly-cron.sh --install.
  4. Defaults: daily 08:00 (explorer sync, RPC 2201); weekly Sunday 09:00 (Config API); log: logs/daily-weekly-checks.log.

Wave 1 — Operator / Code / Doc (Parallel Where Possible)

W1-1: SSH key-based auth; disable password

Blocker: Proxmox/SSH access; coordinate to avoid lockout.

Detailed steps:

  1. Deploy your SSH public key(s) to all Proxmox hosts (e.g. ssh-copy-id root@<host>).
  2. Test key-based login: ssh root@<host> (no password).
  3. Dry-run: bash scripts/security/setup-ssh-key-auth.sh --dry-run.
  4. Apply: bash scripts/security/setup-ssh-key-auth.sh --apply (disables password auth).
  5. Keep a break-glass method (console/out-of-band) in case of lockout.
    Runbook: OPERATIONAL_RUNBOOKS.md § Access Control.

W1-2: Firewall — restrict Proxmox API 8006

Blocker: Proxmox host or SSH from admin network.

Detailed steps:

  1. Decide allowed CIDR(s) for Proxmox API (e.g. admin VPN or office IP).
  2. Dry-run: bash scripts/security/firewall-proxmox-8006.sh --dry-run [CIDR].
  3. Apply: bash scripts/security/firewall-proxmox-8006.sh --apply [CIDR].
  4. Verify: access https://:8006 from an allowed IP only.

W1-8: Automated backup; NPMplus backup run; cron (see above)

Detailed steps (one-time backup run):

  1. When NPMplus is up and NPM_PASSWORD is set: bash scripts/verify/backup-npmplus.sh.
  2. For full automated backup (validators, configs): bash scripts/backup/automated-backup.sh [--with-npmplus].
  3. Cron: see Crontab installs above for NPMplus backup and daily/weekly.

W1-19: Secure validator key permissions

Blocker: Run on Proxmox host as root (or via SSH from LAN).

Detailed steps:

  1. SSH to each Proxmox host that runs validators (VMIDs 1000–1004 or per your layout).
  2. From project on that host (or copy script and run): 
    bash scripts/secure-validator-keys.sh --dry-run   # review
bash scripts/secure-validator-keys.sh             # apply chmod 600, chown besu
  3. Confirm Besu still starts and can read keys (e.g. pct exec <vmid> -- systemctl status besu).

W1-3, W1-4: smom security audits; bridge integrations (Deferred)

  • W1-3: smom Security audits VLT-024, ISO-024 — assign to smom backlog.
  • W1-4: smom Bridge integrations BRG-VLT, BRG-ISO — assign to smom backlog.
    No detailed steps here; track in smom/backlog.

W1-5 – W1-7: Monitoring config (no deploy)

  • W1-5: Prometheus scrape (Besu 9545), alert rules — configs: scripts/monitoring/prometheus-besu-config.yml, smom-dbis-138/monitoring/prometheus/; export-prometheus-targets.sh.
  • W1-6: Grafana dashboards; Alertmanager config — smom-dbis-138/monitoring/grafana/, alertmanager/alertmanager.yml.
  • W1-7: Loki/Alertmanager config — smom-dbis-138/monitoring/loki/, alertmanager/.
    Steps: Copy or merge configs into the monitoring stack when you deploy (Wave 2).

W1-9 – W1-13: Docs / design (mostly done)

  • W1-9: VLAN enablement design — NETWORK_ARCHITECTURE.md §3–5.
  • W1-10: VLAN migration plan — UDM_PRO_VLAN_MIGRATION_PLAN.md, MISSING_CONTAINERS_LIST.md.
  • W1-11: Doc consolidation; archive — ARCHIVE_CANDIDATES.md; move agreed items.
  • W1-12: Quick reference cards — QUICK_REFERENCE_CARDS.md, CONFIGURATION_DECISION_TREE.
  • W1-13: IP assignments; connectivity matrix; runbooks — NETWORK_ARCHITECTURE §7, OPERATIONAL_RUNBOOKS, MISSING_CONTAINERS_LIST.

W1-14 – W1-17: Codebase (deferred / backlog)

W1-20 – W1-21: Shellcheck; config validation

  • W1-20: bash scripts/verify/run-shellcheck.sh [--optional] or run-shellcheck-docker.sh; install shellcheck if desired.
  • W1-21: Config validation and env standardization — already in place: validate-config-files.sh, ENV_STANDARDIZATION docs.

W1-22 – W1-26: MetaMask / explorer / API keys (optional)

  • W1-22: Token-aggregation hardening; CoinGecko — COINGECKO_SUBMISSION.md.
  • W1-23: Chain 138 Snap — market data UI, swap quotes, bridge routes; metamask-integration.
  • W1-24: Explorer — dark mode, network selector, sync indicator; explorer-monorepo.
  • W1-25: Paymaster (optional): forge script script/smart-accounts/DeployPaymaster.s.sol --rpc-url $RPC_URL_138 --broadcast from smom-dbis-138; see SMART_ACCOUNTS_DEPLOYMENT_NOTE.
  • W1-26: API keys — obtain Li.Fi, Jumper, 1inch (and others in reports/API_KEYS_REQUIRED.md); set in .env.

Wave 2 — Infra / Deploy (Parallel by Host or Component)

W2-1: Deploy monitoring stack (Prometheus, Grafana, Loki, Alertmanager)

Detailed steps:

  1. Use configs: smom-dbis-138/monitoring/, scripts/monitoring/.
  2. Run or adapt: scripts/deployment/phase2-observability.sh (or deploy manually per runbook).
  3. Ensure Prometheus scrapes Besu 9545; add targets from export-prometheus-targets.sh if used.
  4. Runbook: OPERATIONAL_RUNBOOKS.md § Phase 2.

W2-2: Grafana via Cloudflare Access; alerts

Detailed steps:

  1. After W2-1 is up, publish Grafana via Cloudflare Access (or your chosen ingress).
  2. Configure Alertmanager routes (email/Slack/PagerDuty) in alertmanager/alertmanager.yml.
  3. Test alert routing (e.g. test alert or drill).

W2-3: VLAN enablement (UDM Pro + Proxmox; migrate services)

Detailed steps:

  1. Configure sovereign VLANs on UDM Pro (e.g. 200–203 per design).
  2. Enable VLAN-aware bridge on Proxmox; attach VMs/containers to VLANs.
  3. Migrate services to VLANs per NETWORK_ARCHITECTURE.md §3–5 and UDM_PRO_VLAN_* docs.
  4. Verify connectivity and firewall between VLANs.

W2-4: Phase 3 CCIP — Ops/Admin (5400-5401); NAT pools; scripts

Detailed steps:

  1. Run checklist: bash scripts/ccip/ccip-deploy-checklist.sh (validates env, prints order).
  2. Deploy CCIP Ops/Admin nodes (VMIDs 5400, 5401) per CCIP_DEPLOYMENT_SPEC.md.
  3. Configure NAT pools on ER605 (Blocks #2–4 for commit/execute/RMN).
  4. Expand/create commit/execute/RMN scripts for the full fleet (used in Wave 3).

W2-5: Phase 4 — Sovereign tenant VLANs; isolation

Detailed steps:

  1. Show steps: bash scripts/deployment/phase4-sovereign-tenants.sh --show-steps.
  2. Dry-run: bash scripts/deployment/phase4-sovereign-tenants.sh --dry-run.
  3. Execute manual steps per runbook: OPERATIONAL_RUNBOOKS.md § Phase 4; UDM_PRO_FIREWALL_MANUAL_CONFIGURATION.md.
  4. Steps: (1) UDM Pro VLANs 200–203, (2) Proxmox VLAN-aware bridge, (3) migrate tenant containers, (4) access control / firewall, (5) Block #6 egress NAT and verify isolation.

W2-6: Missing containers (2506, 2507, 2508) — Destroyed 2026-02-08

Detailed steps:

  1. Canonical list: MISSING_CONTAINERS_LIST.md.
  2. Create three LXC containers:
    • 2506, 2507, 2508 — Destroyed 2026-02-08 on all hosts. RPC range: 2500–2505 only.
  3. Specs: 16GB RAM, 4 CPU, 200GB disk; discovery disabled; JWT auth via nginx.
  4. Use existing RPC container templates/scripts where available; configure permissioning and nginx per docs.

W2-7: DBIS services (10100–10151); Hyperledger

Detailed steps:

  1. Follow deployment runbooks for DBIS service VMIDs (10100–10151).
  2. Start/configure Hyperledger services per runbook and MISSING_CONTAINERS_LIST.md (Firefly etc.).
  3. Parallelize by host where multiple hosts are used.

W2-8: NPMplus HA (Keepalived, 10234) — Optional

Detailed steps:

  1. Follow NPMPLUS_HA_SETUP_GUIDE.md.
  2. Deploy secondary NPMplus (e.g. VMID 10234); configure Keepalived/HAProxy for failover.
  3. Test failover and revert.

Wave 3 — After Wave 2

W3-1: CCIP Fleet (16 commit, 16 execute, 7 RMN)

Depends on: W2-4 (Ops/Admin, NAT pools).

Detailed steps:

  1. Deploy 16 commit nodes: VMIDs 5410–5425 (CCIP-COMMIT-01 … CCIP-COMMIT-16).
  2. Deploy 16 execute nodes: VMIDs 5440–5455 (CCIP-EXEC-01 … CCIP-EXEC-16).
  3. Deploy 7 RMN nodes: VMIDs 5470–5476 (CCIP-RMN-01 … CCIP-RMN-07).
  4. Use scripts/runbooks from W2-4; full spec: CCIP_DEPLOYMENT_SPEC.md.

W3-2: Phase 4 tenant isolation enforcement

Depends on: W2-3 / W2-5 (VLANs and sovereign tenant setup).

Detailed steps:

  1. Apply firewall rules and ACLs to enforce east-west denial between tenants.
  2. Verify tenant isolation (no cross-tenant access); verify egress NAT (Block #6) per design.
  3. Document any exceptions and review periodically.

Ongoing (No Wave)

ID Task Frequency Detailed steps
O-1 Monitor explorer sync Daily Cron runs daily-weekly-checks.sh daily (or run manually).
O-2 Monitor RPC 2201 Daily Same script.
O-3 Config API uptime Weekly Cron runs daily-weekly-checks.sh weekly.
O-4 Review explorer logs Weekly Runbook: OPERATIONAL_RUNBOOKS § Maintenance [138]; e.g. ssh root@<host> journalctl -u blockscout -n 200.
O-5 Update token list As needed Runbook [139]; update token-list.json / explorer config.

One-off: CT-1a Restore (if backup exists)

Task: Restore container 2301 (besu-rpc-private-1) from backup instead of recreating.

Detailed steps:

  1. Locate backup file (e.g. backup.tar.zst for CT 2301).
  2. On Proxmox host (e.g. ml110): pct restore 2301 /path/to/backup.tar.zst --storage local-lvm.
  3. Adjust network/storage if needed; start container and verify service.

Deferred / Backlog (No Steps Here)

  • W1-3, W1-4: smom security audits; bridge integrations — smom backlog.
  • W1-14: dbis_core TypeScript fixes — backlog; parallelize by module.
  • W1-15 – W1-17: smom placeholders; IRU; Fabric 999; .bak deprecation — see PLACEHOLDERS_AND_* docs.
  • Improvements index 1–139: Work through ALL_IMPROVEMENTS_AND_GAPS_INDEX.md by cohort; many overlap with W1/W2/W3 above.

API Keys & Secrets (Obtain and Set)

Full list: reports/API_KEYS_REQUIRED.md. Variable names are in .env.example.

Detailed steps:

  1. Open reports/API_KEYS_REQUIRED.md and note required keys per category (DeFi, fiat ramp, e-signature, alerts, explorers, OTC, etc.).
  2. Obtain each key (sign-up URLs in report); set in root .env and in subproject .env where used (e.g. dbis_core, the-order, metamask-integration).
  3. Restart or redeploy services that depend on those env vars.

Placeholders & Code Completions (E2E)

See E2E_COMPLETION_TASKS_DETAILED_LIST.md Part 6 for:

  • smom-dbis-138: canonical addresses env-only, AlltraAdapter fee, smart accounts kit, quote service Fabric 999, EnhancedSwapRouter/DODOPMMProvider, WETH bridges, .bak deprecation.
  • dbis_core: Prometheus/Redis/PagerDuty/AS4; TypeScript errors.
  • the-order: E-signature, court e-filing, document security/export.
  • OMNIS: Sankofa Phoenix SDK when available.
  • multi-chain-execution / Tezos: TezosRelayService when implemented.

Validation commands (re-run anytime)

Check Command
All validation bash scripts/verify/run-all-validation.sh [--skip-genesis]
Full verification bash scripts/verify/run-full-verification.sh
E2E routing bash scripts/verify/verify-end-to-end-routing.sh
Config files bash scripts/validation/validate-config-files.sh
Genesis bash smom-dbis-138/scripts/validation/validate-genesis.sh
Wave 0 (dry-run) bash scripts/run-wave0-from-lan.sh --dry-run

Related: E2E_COMPLETION_TASKS_DETAILED_LIST.md, WAVE2_WAVE3_OPERATOR_CHECKLIST.md, FULL_PARALLEL_EXECUTION_ORDER.md.

Reference in New Issue View Git Blame Copy Permalink