Files
proxmox/docs/archive/00-meta-pruned/FULL_PARALLEL_RUN_LOG.md
defiQUG bea1903ac9
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
Sync all local changes: docs, config, scripts, submodule refs, verification evidence
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-21 15:46:06 -08:00

275 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Full Maximum Parallel Run Log
**Run started:** 2026-02-05
**Execution model:** By wave (Wave 0 → Wave 1 → Wave 2 → Wave 3); within each wave, tasks run in parallel where possible.
**2026-02-06:** Single runner `scripts/run-all-waves-parallel.sh` executed (maximum parallel mode). Wave 0 (W0-1, W0-3) and Wave 1 (parallel) and Wave 2 (W2-6 create 2506,2507,2508) completed. Wave 3 runbook-only.
---
## Wave 0 (gates / credentials)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W0-1 | Apply NPMplus RPC fix (405) | ⚠️ Skipped (no LAN) | Auth/connection failed from this environment. Run from host on LAN: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` |
| W0-2 | Execute sendCrossChain (real) | ⏳ Operator | Requires PRIVATE_KEY and LINK approval; run manually when ready. |
| W0-3 | NPMplus backup | ⚠️ Partial | Script ran; container 10233 not running so DB dump failed. API export attempted. Run again when NPMplus is up. |
---
## Wave 1 (full parallel — executed)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-verify | check-dependencies.sh | ✅ Pass | All required deps present (bash, curl, jq, openssl, ssh). |
| W1-verify | verify-end-to-end-routing.sh | ✅ Run | 25 DNS pass, 14 HTTPS pass, 6 RPC failures (405). Report: `docs/04-configuration/verification-evidence/e2e-verification-20260205_111157/`. |
| W1-18 | Progress indicators in scripts | ✅ Done | `run-full-verification.sh`: Progress 0/54/5 and Step N/5. `verify-end-to-end-routing.sh`: Progress domain N/25. |
**Not run in this session (require SSH, credentials, or external):** W1-1W1-17, W1-19W1-44 (security, monitoring deploy, VLAN, docs consolidation, dbis_core TS, smom audits, etc.). These remain for operator or future automated runs.
---
## Wave 1 continued (second batch — 2026-02-05)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-5 / D5 | export-prometheus-targets.sh | ✅ Done | Exported `smom-dbis-138/monitoring/prometheus/targets-proxmox.yml`. |
| W1-5 | Prometheus Besu 9545 config | ✅ Exists | `scripts/monitoring/prometheus-besu-config.yml` and `smom-dbis-138/monitoring/prometheus/scrape-proxmox.yml` already define Besu 9545 scrape. |
| W1-validate | smom-dbis-138 validate-genesis.sh | ⚠️ Fail | Exit 1 (likely missing lib/init.sh or config path in subshell). |
| W1-20 | shellcheck on verify scripts | ⏳ Skip | shellcheck not installed in environment. |
| W1-39 | smom-dbis-138 pnpm test | ⚠️ Skip | hardhat not found / node_modules missing; run `pnpm install` and ensure hardhat in PATH. |
| W1-11 | Documentation archive candidates | ✅ Done | Created [ARCHIVE_CANDIDATES.md](../../00-meta/ARCHIVE_CANDIDATES.md) (by folder, deprecated list, next steps). |
---
## Wave 1 continued (third batch — 2026-02-05, full parallel)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-12 | Quick reference cards (Verification & E2E) | ✅ Done | Added §5 Verification & E2E and config/decision-tree links to [QUICK_REFERENCE_CARDS.md](../../12-quick-reference/QUICK_REFERENCE_CARDS.md). |
| W1-12 / 6874 | Config templates & decision tree links | ✅ Done | Same doc: CONFIGURATION_DECISION_TREE, INGRESS_SOURCE_OF_TRUTH, 04-configuration README linked. |
| W1-31 | verify-min-gas-price.sh strict mode | ✅ Done | Added `set -euo pipefail` to `scripts/verify/verify-min-gas-price.sh`. |
**Re-run checks (same session):**
| Script | Result | Note |
|--------|--------|------|
| check-dependencies.sh | ✅ Pass | All deps present. |
| verify-min-gas-price.sh | ⚠️ Exit 2 | SSH to 192.168.11.x not available from this env (expected). |
---
## Wave 1 continued (fourth batch — 2026-02-05)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| — | run-full-verification.sh (re-run) | ✅ Partial | Deps pass; DNS verification 19/19; UDM Pro public 80/443 OK; NPMplus step failed (off-LAN). Evidence: verification-evidence/dns-verification-*, udm-pro-verification-*. |
| W1-12 | QUICK_REFERENCE.md links | ✅ Done | Added "Parallel run & verification" section linking execution order, Wave 1 summary, Wave 2/3 checklist, run log, and verification commands. |
---
## Wave 1 continued (fifth batch — 2026-02-05, fix failures & complete options)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-validate | validate-genesis.sh (smom-dbis-138) | ✅ Fixed | Script now runs standalone: minimal log_* if init.sh not loaded; PROJECT_ROOT/CONFIG set before sourcing; QBFT supported (`.config.qbft` in addition to `.config.ibft2`). Passes from both proxmox root and smom-dbis-138 root. |
| — | verify-end-to-end-routing.sh (re-run) | ✅ Run | 25 DNS pass, 14 HTTPS pass, 6 failed (RPC 405 until NPMplus fix). Report: `e2e-verification-20260205_121640/`. |
| — | validate-config-files.sh | ✅ Pass | Found ip-addresses.conf, .env.example; optional env warnings only. |
| W1-39 | smom-dbis-138 pnpm test | ⚠️ Partial | Added `@openzeppelin/contracts-upgradeable` to package.json. Tests still fail: missing internal deps (e.g. `@emoney/interfaces`). Run from full workspace or add internal packages to resolve. |
| — | shellcheck | ⏳ Skip | Not installed (permission denied for apt). Install when available; run on `scripts/verify/*.sh`. |
---
## Wave 1 continued (sixth batch — 2026-02-05, review & optional)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| — | TODO_TASK_LIST_MASTER sync | ✅ Done | Verification section: validate-genesis fix, validate-config-files, E2E, shellcheck optional. Monitoring: Besu 9545 config marked done. Validation commands: added check-dependencies, E2E-only, validate-config-files, validate-genesis, run-shellcheck. Status links to run log and Wave 1/2/3 docs. |
| — | Optional shellcheck script | ✅ Done | Added `scripts/verify/run-shellcheck.sh` — runs shellcheck on verify scripts when installed. |
| W1-39 | Hardhat @emoney resolution | ⚠️ Reverted | prepare-hardhat-emoney.js + symlink caused HH415 duplicate sources (contracts/emoney vs @emoney). Use `forge test` for full smom-dbis-138 tests; `pnpm test` remains limited unless Hardhat remapping plugin used. |
---
## Wave 1 continued (seventh batch — 2026-02-05, code complete & tested)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-18 / W1-21 | Config validation in run-full-verification | ✅ Done | Added Step 0/6: config validation (validate-config-files.sh). TOTAL_STEPS=6; Step 6 = generate source-of-truth. |
| — | All verifications run (parallel) | ✅ Pass | check-dependencies, validate-config-files, validate-genesis, daily-weekly-checks daily — all passed. run-shellcheck exit 1 (shellcheck not installed). |
| — | run-full-verification.sh with Step 0 | ✅ Run | Step 0 config validation passed; Steps 12 (DNS, UDM Pro) passed. (Step 3 NPMplus fails off-LAN as expected.) |
| — | daily-weekly-checks.sh | ✅ Tested | `daily` mode: explorer SKIP, RPC OK; exit 0. Script has set -euo pipefail. |
---
## Wave 1 continued (eighth batch — 2026-02-05, maintenance scripts & cron)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W0 / W1-8 | run-wave0-from-lan.sh | ✅ Done | Runs W0-1 (NPMplus RPC fix) and W0-3 (backup); `--dry-run`, `--skip-backup`, `--skip-rpc-fix`. Tested. |
| W1-8 | schedule-npmplus-backup-cron.sh | ✅ Done | `--install` / `--show`; daily 03:00. Tested. |
| O-1O-3 | schedule-daily-weekly-cron.sh | ✅ Done | Daily 08:00, weekly Sun 09:00 for daily-weekly-checks.sh. Wired in checklist, TODO, OPERATIONAL_RUNBOOKS, scripts/README. Tested. |
| — | Docs | ✅ Done | WAVE2_WAVE3_OPERATOR_CHECKLIST, TODO validation table, OPERATIONAL_RUNBOOKS, scripts/README updated. |
| — | Re-run tests | ✅ Pass | check-dependencies, run-wave0-from-lan --dry-run, schedule-*-cron --show, daily-weekly-checks daily, validate-config-files, run-full-verification (steps 02 pass; step 3 NPMplus fails off-LAN). |
---
## Wave 1 continued (ninth batch — 2026-02-05, shellcheck optional & Phase 4 runbook)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| W1-20 / §3 | run-shellcheck.sh --optional | ✅ Done | With `--optional`, exits 0 when shellcheck not installed (CI-friendly). |
| Phase 4 | OPERATIONAL_RUNBOOKS § Phase 4 | ✅ Done | Phase 4 runbook paragraph: script, ORCHESTRATION_DEPLOYMENT_GUIDE, NETWORK_ARCHITECTURE, UDM_PRO_FIREWALL. |
| Phase 4 | phase4-sovereign-tenants.sh | ✅ Done | Added `--show-steps`, `--dry-run`; runbook links. scripts/README §13 Phase 4. |
| TODO | Phase 4 & shellcheck | ✅ Done | Phase 4 marked done (runbook + script); shellcheck marked done (--optional). |
| — | Tests | ✅ Pass | run-shellcheck.sh --optional (exit 0); phase4 --show-steps, --dry-run; check-dependencies; validate-config-files. |
---
## Wave 1 continued (tenth batch — 2026-02-05, CI validation & secure-validator-keys)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| CI | run-all-validation.sh | ✅ Done | Single entry point: check-dependencies + validate-config-files + optional validate-genesis. `--skip-genesis` to skip genesis. |
| W1-19 | secure-validator-keys.sh --dry-run | ✅ Done | Added `--dry-run`; run on Proxmox as root. Documented in OPERATIONAL_RUNBOOKS, scripts/README. |
| — | Docs | ✅ Done | TODO: run-all-validation in validation table; §4 "1139" CI validation note. OPERATIONAL_RUNBOOKS: Phase 2 Security links to secure-validator-keys, SSH, firewall scripts. |
| — | Tests | ✅ Pass | run-all-validation.sh and run-all-validation.sh --skip-genesis both exit 0. secure-validator-keys.sh --dry-run exits 1 off-Proxmox (expected). |
---
## Wave 2 & Wave 3
Not run (require Proxmox/SSH, running NPMplus, or NAT/CCIP infra). **Operator checklist:** [WAVE2_WAVE3_OPERATOR_CHECKLIST.md](README.md).
---
## Wave 1 completion summary (2026-02-05)
All Wave 1 tasks are classified and documented:
- **[WAVE1_COMPLETION_SUMMARY.md](README.md)** — Status of every W1-1W1-44 (Done / Operator / Documented / Deferred).
- **[WAVE2_WAVE3_OPERATOR_CHECKLIST.md](README.md)** — Ordered checklist for Wave 0, Wave 2, Wave 3, and Ongoing.
**Run complete (automation scope):** All Wave 1 items that can be done without SSH/LAN/credentials are done or documented. Remaining work is operator-driven (Wave 0, W1 operator tasks, Wave 2, Wave 3).
**Code changes completed and tested (2026-02-05):**
- run-full-verification.sh: Step 0 config validation added (6 steps total); tested.
- check-dependencies, validate-config-files, validate-genesis, daily-weekly-checks daily: all passed.
- run-shellcheck: skipped (shellcheck not installed).
- All scripts in scripts/verify and scripts/maintenance/daily-weekly-checks.sh use set -euo pipefail where applicable.
---
## Wave 1 continued (eleventh batch — 2026-02-05, full maximum parallel: remaining completions)
| ID | Task | Result | Notes |
|----|------|--------|-------|
| CI | run-all-validation.sh --skip-genesis | ✅ Pass | Dependencies OK; config OK; genesis skipped. |
| Config | validate-config-files.sh | ✅ Pass | ip-addresses.conf, .env.example; optional env warnings. |
| W1-1 | setup-ssh-key-auth.sh --dry-run | ✅ Run | Prints steps; apply requires operator on each host. |
| W1-2 | firewall-proxmox-8006.sh --dry-run | ✅ Run | UFW commands shown; ADMIN_CIDR=192.168.11.0/24. |
| W1-5 / W1-7 | phase2-observability.sh --config-only | ✅ Run | prometheus.yml, alertmanager.yml written to config/monitoring/. |
| CCIP | ccip-deploy-checklist.sh | ✅ Run | VMID ranges and deployment order printed; env warnings (CCIP_ETH_*). |
| W1-8 | schedule-npmplus-backup-cron.sh --show | ✅ Run | Crontab line shown (03:00). |
| Phase 4 | phase4-sovereign-tenants.sh --show-steps | ✅ Run | Five steps and runbook links. |
| Backup | automated-backup.sh (no --with-npmplus) | ✅ Run | Config backup: backups/configs/proxmox-configs-20260205_155139. |
| Shellcheck | run-shellcheck.sh --optional | ✅ Exit 0 | shellcheck not installed; optional mode. |
| Wave 0 | run-wave0-from-lan.sh --dry-run | ✅ Run | W0-1, W0-3 dry-run; W0-2 reminder. |
**Summary:** All automatable Wave 1 validations, dry-runs, config generation, and checklists executed in parallel. Wave 0 and apply steps remain operator/LAN/creds.
---
## 2026-02-06 — run-all-waves-parallel.sh (maximum parallel mode)
| Wave | Task | Result | Notes |
|------|------|--------|-------|
| **Wave 0** | run-via-proxmox-ssh.sh wave0 --host 192.168.11.11 | ✅ Done | W0-1: NPMplus RPC fix — 19 proxy hosts updated. W0-3: backup ran; API auth warning (NPM_PASSWORD/container). W0-2: run run-send-cross-chain.sh without --dry-run when ready. |
| **Wave 1** | secure-env-permissions, schedule-npmplus-backup-cron --install, schedule-daily-weekly-cron --install, setup-ssh-key-auth --dry-run, firewall-proxmox-8006 --dry-run, run-shellcheck --optional, validate-config-files | ✅ Done | All ran in parallel. Logs in temp dir. |
| **Wave 2 (W2-6)** | create-missing-containers-2506-2508.sh on r630-01 | ✅ Done | Containers 2506 (192.168.11.202), 2507 (192.168.11.203), 2508 (192.168.11.204) created on 192.168.11.11. IPs .256/.257/.258 in doc invalid; script uses .202/.203/.204. Post-create: JWT, discovery disabled per MISSING_CONTAINERS_LIST. |
| **Wave 3** | Runbook-only | — | W3-1 CCIP Fleet, W3-2 Phase 4 tenant isolation — see FULL_PARALLEL_EXECUTION_ORDER and WAVE2_WAVE3_OPERATOR_CHECKLIST. |
**Scripts added:** `scripts/run-all-waves-parallel.sh`, `scripts/create-missing-containers-2506-2508.sh`.
---
## 2026-02-05 — Full parallel: config cleanup & remaining automatable items
| Category | Task | Result | Notes |
|----------|------|--------|-------|
| **Config cleanup** | ip-addresses.conf | ✅ Done | RPC_LUIS_2=.202, RPC_PUTU_1=.203, RPC_PUTU_2=.204 (was .256/.257/.258). |
| **Config cleanup** | MISSING_CONTAINERS_LIST.md | ✅ Done | Table and intro updated to deployed IPs .202/.203/.204; 25062508 created on r630-01. |
| **Config cleanup** | Other docs/scripts | ✅ Done | REMAINING_WORK_DETAILED_STEPS.md, CHAIN138_JWT_AUTH_REQUIREMENTS.md; create-all-chain138-containers-direct.sh, create-chain138-containers.sh, generate-jwt-token-for-container.sh, repair-corrupted-ip-replacements.sh, fix-remaining-hardcoded-ips.sh — defaults/refs updated to .202/.203/.204. |
| **Validation** | run-all-validation.sh --skip-genesis | ✅ Pass | Dependencies OK; config OK. |
| **Validation** | validate-config-files.sh | ✅ Pass | ip-addresses.conf, .env.example. |
| **Cron** | schedule-daily-weekly-cron.sh --show | ✅ Confirmed | O-1, O-2 daily 08:00; O-3 weekly Sun 09:00. |
| **Cron** | schedule-npmplus-backup-cron.sh --show | ✅ Confirmed | NPMplus backup 03:00 when installed. |
**Remaining (operator / LAN / creds only):** Wave 0 W0-2 (sendCrossChain real), W0-3 (NPMplus backup when up); post-create 25062508 (Besu config, JWT, discovery off); W1-1/W1-2 --apply; Wave 2 (monitoring stack, VLAN, CCIP, Phase 4, DBIS, NPMplus HA); Wave 3 (CCIP Fleet, Phase 4 isolation). See [REMAINING_WORK_DETAILED_TASKS.md](README.md).
---
## 2026-02-06 — Full parallel: remaining safe tasks
| Task | Result | Notes |
|------|--------|-------|
| W1-8 NPMplus backup cron | Done | `schedule-npmplus-backup-cron.sh --install` — daily 03:00 installed. |
| W1-1 setup-ssh-key-auth --dry-run | Done | Steps printed; apply = operator. |
| W1-2 firewall-proxmox-8006 --dry-run | Done | UFW commands shown; apply = operator. |
| daily-weekly-checks.sh all | Done | RPC 2201 OK; explorer/config API skip off-LAN. |
| run-all-validation.sh --skip-genesis | Pass | Dependencies and config OK. |
| run-send-cross-chain.sh 0.01 --dry-run | Run | Dry-run OK; real run when PRIVATE_KEY/LINK ready. |
| phase2-observability.sh --config-only | Done | prometheus.yml, alertmanager.yml in config/monitoring/. |
| secure-env-permissions.sh --dry-run | Done | |
| validate-config-files.sh | Pass | |
**Additional parallel batch (same session):** ccip-deploy-checklist.sh ✅; phase4-sovereign-tenants.sh --show-steps ✅; run-shellcheck.sh --optional ✅ (shellcheck not installed); check-dependencies.sh ✅; automated-backup.sh --dry-run ✅; run-all-validation.sh --skip-genesis ✅. No further automatable items; remainder is operator-only (see REMAINING_WORK_DETAILED_TASKS.md § Automation complete).
---
## 2026-02-06 — SSH to Proxmox: copy then run (r630-01)
Scripts copied to host first, then run via SSH (run-via-proxmox-ssh.sh extended with copy + secure-keys modes).
| Task | Result | Notes |
|------|--------|-------|
| copy --host 192.168.11.11 | Done | Extended set copied to /tmp/proxmox-scripts-run (.env, config, run-wave0-from-lan, update-npmplus, backup-npmplus, secure-validator-keys, create-missing-containers-2506-2508). |
| wave0 --host 192.168.11.11 | Done | W0-1: 19 NPMplus proxy hosts updated. W0-3: backup ran; direct DB copy failed (container may be down); API export warning. W0-2: run run-send-cross-chain.sh without --dry-run when ready. |
| secure-keys --host 192.168.11.11 | Done (dry-run) | Would secure validator keys in 10001004; 10001002 secured (dry-run); 10031004 not running, skipped. Use --apply to run for real. |
**Remaining on host:** Post-create 25062508 (Besu config, JWT, discovery off) — run from REMOTE_DIR or copy configure scripts; W1-1/W1-2 --apply when ready.
---
## 2026-02-06 — "Still to do on the host" completion
| Task | Result | Notes |
|------|--------|-------|
| secure-validator-keys --apply | Done | `run-via-proxmox-ssh.sh secure-keys --apply --host 192.168.11.11`. Containers 1000, 1001, 1002 secured; 10031004 skipped (not running). |
| Post-create 25062508 (Besu config) | Run / ready | IPs fixed in configure-besu-chain138-nodes.sh (.202/.203/.204). Script run with PROXMOX_HOST=192.168.11.11 exited early (collect_enodes may need Besu running on at least one node on that host). Once Besu is installed and running on 25062508 (or other nodes on r630-01), run: `PROXMOX_HOST=192.168.11.11 bash scripts/run-configure-besu-on-host.sh`. |
**Wrapper:** `scripts/run-configure-besu-on-host.sh [HOST]` — runs configure-besu with 10m timeout.
**Update (review):** Besu nodes on r630-01 are all running (10001002, 15001502, 25002508 including 25062508). PROJECT_ROOT in `configure-besu-chain138-nodes.sh` was fixed (was pointing to archive/; now uses repo root so `config/ip-addresses.conf` is found). **Next step:** run the configure script and let it finish (can take 510 min):
`cd /home/intlc/projects/proxmox && PROXMOX_HOST=192.168.11.11 bash scripts/run-configure-besu-on-host.sh`
Note: script only configures containers on the single PROXMOX_HOST (1003, 1004, 1503, 1504 are on ml110 and are skipped when host is r630-01).
---
## 2026-02-06 — Besu configure (post-create 25062508) completed
| Task | Result | Notes |
|------|--------|-------|
| configure-besu-chain138-nodes.sh fixes | Done | collect_enodes: logs to stderr so only path on stdout. generate_static_nodes: Python via env vars (ENODES_FILE, OUTPUT_FILE) to avoid heredoc argv issues. deploy_to_container: trim path vars for scp. Discovery fallback: run inside container via `pct exec $vmid -- bash -c "..."`. WORK_DIR under OUTPUT_DIR; post-restart sleep 1s. run-configure-besu-on-host.sh timeout 900s. |
| PROXMOX_HOST=192.168.11.11 (r630-01) | Done | Collected 6 enodes (1000, 1001, 1002, 1500, 1501, 1502). Deployed static-nodes.json and permissioned-nodes.json to all running Besu nodes (25002508, 10001002, 15001502). Discovery disabled for 2500, 25032508; enabled for 25012502, validators, sentries. RPC 25062508: no config file / no Besu service found (expected if not yet configured); files deployed. Configuration complete. |
| PROXMOX_HOST=192.168.11.10 (ml110) | Done | Collected 3 enodes (1003, 1004, 1503). Deployed to 1003, 1004, 1504, 1503. Configuration complete. |
**Note:** Enode extraction for RPC nodes (25002508) fails (no nodekey at expected paths); static-nodes/permissioned-nodes use validator + sentry enodes only. For full mesh you can add RPC enodes manually or extend the script to read from admin_nodeInfo when available.
---
## Next steps
1. **Wave 0:** From host on LAN: W0-1 (NPMplus update), W0-3 (backup when NPMplus up); W0-2 (sendCrossChain real) when keys and LINK ready.
2. **Wave 1 operator:** W1-1, W1-2 apply (--apply when ready); W1-8 cron install (NPM_PASSWORD); W1-19, W1-20, W1-27; install shellcheck (run on `scripts/verify/*.sh`). smom-dbis-138: validate-genesis ✅ fixed; pnpm test needs internal workspace deps (e.g. @emoney/interfaces) or run from full workspace.
3. **Wave 2 & 3:** Follow [WAVE2_WAVE3_OPERATOR_CHECKLIST.md](README.md).
4. **Ongoing:** O-1O-5 ✅ completed (cron + token list validated). See [REMAINING_ITEMS_FULL_PARALLEL_LIST.md](README.md).