Files
proxmox/docs/00-meta/OPERATOR_READY_CHECKLIST.md

198 lines
9.4 KiB
Markdown
Raw Normal View History

# Operator Ready Checklist — Copy-Paste Commands
**Last Updated:** 2026-03-04
**Purpose:** Single page with exact commands to complete every pending todo. Run from **repo root** on a host with **LAN** access (and `smom-dbis-138/.env` with `PRIVATE_KEY`, `NPM_PASSWORD` where noted).
**Do you have all necessary creds?** See [OPERATOR_CREDENTIALS_CHECKLIST.md](OPERATOR_CREDENTIALS_CHECKLIST.md) — per-task list of LAN, PRIVATE_KEY, NPM_PASSWORD, RPC_URL_138, SSH, LINK, gas, token balance.
**From anywhere (no LAN):** `./scripts/run-completable-tasks-from-anywhere.sh`
**If deployer needs gas on public chains or Wemix:** Run `./scripts/deployment/deployer-gas-auto-route.sh` (optional: `--dry-run`, `--chain 138` or `--chain 1111`). See [DEPLOYER_GAS_AUTO_ROUTE_RUNBOOK.md](../03-deployment/DEPLOYER_GAS_AUTO_ROUTE_RUNBOOK.md).
**Remaining for full network coverage (13-chain max execution):** [REMAINING_DEPLOYMENTS_FOR_FULL_NETWORK_COVERAGE.md](../03-deployment/REMAINING_DEPLOYMENTS_FOR_FULL_NETWORK_COVERAGE.md) — Phase A (mint + add liquidity 138) → B (**Cronos**, Celo, **Wemix** CCIP + LINK) → C (cW* + edge pools). **2026-03-04:** Celo + Gnosis CCIP ✅; Cronos and Wemix need deployer gas (CRO ~15, WEMIX ~0.4) then deploy + complete-config. See `./scripts/deployment/acquire-cro-and-wemix-gas.sh`.
---
## 1. High: Gnosis, Cronos, Celo, Wemix CCIP bridges
**Ref:** [CONFIG_READY_CHAINS_COMPLETION_RUNBOOK](../07-ccip/CONFIG_READY_CHAINS_COMPLETION_RUNBOOK.md)
**Prereqs:** Confirm [CCIP supports](https://docs.chain.link/ccip/supported-networks) 100, **25**, 42220, 1111. Per chain: RPC, CCIP Router, LINK, WETH9/WETH10, deployer with native gas (xDAI, **CRO ~15**, CELO, **WEMIX ~0.4**). To acquire CRO and WEMIX: `./scripts/deployment/acquire-cro-and-wemix-gas.sh`.
```bash
cd smom-dbis-138
source .env
# Per chain (set RPC_URL, CCIP_ROUTER_ADDRESS, LINK_TOKEN_ADDRESS, WETH9_ADDRESS, WETH10_ADDRESS, PRIVATE_KEY)
forge script script/deploy/bridge/DeployWETHBridges.s.sol:DeployWETHBridges --rpc-url "$RPC_URL" --broadcast -vvvv
```
Then add destinations (Chain 138 ↔ each chain) and fund with LINK — use:
```bash
DRY_RUN=1 ./scripts/deployment/complete-config-ready-chains.sh # print commands
./scripts/deployment/complete-config-ready-chains.sh # run (requires bridge addresses in .env)
```
**Add Cronos and Wemix:** Copy Cronos/Wemix vars from `smom-dbis-138/docs/deployment/ENV_CONFIG_READY_CHAINS.example` into `smom-dbis-138/.env`; fund deployer with CRO and WEMIX (see `acquire-cro-and-wemix-gas.sh`); then:
```bash
cd smom-dbis-138
./scripts/deployment/deploy-bridges-config-ready-chains.sh cronos
./scripts/deployment/deploy-bridges-config-ready-chains.sh wemix
# Set CCIPWETH9_BRIDGE_CRONOS, CCIPWETH10_BRIDGE_CRONOS, CCIPWETH9_BRIDGE_WEMIX, CCIPWETH10_BRIDGE_WEMIX in .env from output
./scripts/deployment/complete-config-ready-chains.sh
```
**Full steps:** See runbook § Step 14.
---
## 2. Medium: LINK support on Mainnet relay
**Ref:** [RELAY_BRIDGE_ADD_LINK_SUPPORT_RUNBOOK](../07-ccip/RELAY_BRIDGE_ADD_LINK_SUPPORT_RUNBOOK.md)
**Options:** A = extend CCIPRelayBridge to accept LINK; B = deploy separate LINK receiver. After implement + deploy + fund:
```bash
# In config/token-mapping.json set relaySupported: true for LINK
# Update TOKEN_MAPPING_AND_MAINNET_ADDRESSES.md and CCIP_BRIDGE_MAINNET_CONNECTION.md
# Restart relay service on r630-01: /opt/smom-dbis-138/services/relay
```
---
## 3. LAN: Blockscout verification
```bash
source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh
```
Single contract retry: `./scripts/verify/run-contract-verification-with-proxy.sh --only ContractName`
---
## 4. LAN: Fix E2E 502s
```bash
./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e
# Or lighter:
./scripts/maintenance/address-all-remaining-502s.sh --run-besu-fix --e2e
```
**Runbook:** [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md)
---
## 5. LAN: Run all operator tasks (backup + verify ± deploy ± create-vms)
```bash
./scripts/run-all-operator-tasks-from-lan.sh --dry-run # print steps
./scripts/run-all-operator-tasks-from-lan.sh # backup + Blockscout verify
./scripts/run-all-operator-tasks-from-lan.sh --deploy # + contract deploy
./scripts/run-all-operator-tasks-from-lan.sh --create-vms # + create DBIS Core containers
./scripts/run-all-operator-tasks-from-lan.sh --deploy --create-vms
```
---
## 5a. LAN: Token-aggregation DB and migrations (VMID 5000)
If `/health` returns "database token_aggregation does not exist":
```bash
./scripts/apply-token-aggregation-fix.sh # create DB, run migrations, restart (via Proxmox)
./scripts/apply-token-aggregation-fix.sh --dry-run # print steps only
```
If VMID 5000 has no `postgres` user, run `createdb` and migrations on the host where PostgreSQL runs, or set token-aggregation `DATABASE_URL` to `explorer_db` and run `smom-dbis-138/services/token-aggregation/scripts/run-migrations.sh` there.
---
## 5b. LAN: Chain 138 next steps (Phase 2: preflight → mirror+pool → register c* as GRU → verify)
**Ref:** [DEPLOYMENT_ORDER_OF_OPERATIONS](../03-deployment/DEPLOYMENT_ORDER_OF_OPERATIONS.md) Phase 2. Use when mirror/pool/GRU registration or verify are pending.
```bash
./scripts/deployment/run-all-next-steps-chain138.sh --dry-run # print steps only
./scripts/deployment/run-all-next-steps-chain138.sh # run all (preflight, deploy mirror+pool, register c*, verify)
./scripts/deployment/run-all-next-steps-chain138.sh --skip-mirror # pool + register + verify only (set TRANSACTION_MIRROR_ADDRESS in smom-dbis-138/.env first)
```
If TransactionMirror deploy fails with **CreateCollision:** set `TRANSACTION_MIRROR_ADDRESS=0xC7f2Cf4845C6db0e1a1e91ED41Bcd0FcC1b0E141` in `smom-dbis-138/.env` and re-run with `--skip-mirror`. See [TRANSACTION_MIRROR_CHAIN138_COLLISION_FIX](../03-deployment/TRANSACTION_MIRROR_CHAIN138_COLLISION_FIX.md).
---
## 6. Low: DODO PMM on Chain 138
**Ref:** [OPTIONAL_DEPLOYMENTS_START_HERE](../07-ccip/OPTIONAL_DEPLOYMENTS_START_HERE.md) §2B
**Prereqs:** Set in `smom-dbis-138/.env`: `DODO_VENDING_MACHINE_ADDRESS`, `COMPLIANT_USDT_ADDRESS`, `COMPLIANT_USDC_ADDRESS`.
```bash
./scripts/run-optional-deployments.sh --execute --phases 7
# Or from smom-dbis-138: ./scripts/deployment/deploy-optional-future-all.sh (Phase 7 = DODO)
```
---
## 7. Low: Mainnet trustless stack (Lockbox138 + Mainnet)
**Ref:** [OPTIONAL_DEPLOYMENTS_START_HERE](../07-ccip/OPTIONAL_DEPLOYMENTS_START_HERE.md) §2C
**Prereqs:** `ETHEREUM_MAINNET_RPC`, Mainnet ETH for deployer.
```bash
cd smom-dbis-138
source .env
forge script script/bridge/trustless/DeployTrustlessBridge.s.sol:DeployTrustlessBridge \
--rpc-url "$ETHEREUM_MAINNET_RPC" --broadcast --via-ir --verify
# Then: Lockbox138 on 138; configure Lockbox138↔InboxETH; fund liquidity. See runbook §C.
```
---
## 8. Wave 0: sendCrossChain (real) and NPMplus backup
**sendCrossChain (real):** Requires `PRIVATE_KEY` and LINK approved in `.env`. Bridge: `0x971cD9D156f193df8051E48043C476e53ECd4693`.
```bash
bash scripts/bridge/run-send-cross-chain.sh 0.01 [recipient_address]
# Omit --dry-run to execute. Example: bash scripts/bridge/run-send-cross-chain.sh 0.01 0x...
```
**NPMplus backup:** Requires `NPM_PASSWORD` in `.env` and host on LAN.
```bash
bash scripts/verify/backup-npmplus.sh
# Or combined Wave 0: bash scripts/run-wave0-from-lan.sh
```
**NPMplus RPC fix (405):** From LAN: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Verify: `bash scripts/verify/verify-end-to-end-routing.sh`.
**NPMplus API unreachable (167/169):** Restart Docker inside NPMplus LXC: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh` (SSH to r630-01, restarts npmplus in 10233 and 10235).
**E2E from LAN (no public DNS):** If E2E fails at DNS (`Could not resolve host`), use [E2E_DNS_FROM_LAN_RUNBOOK.md](../04-configuration/E2E_DNS_FROM_LAN_RUNBOOK.md): append `config/e2e-hosts-append.txt` to `/etc/hosts`, then run `E2E_USE_SYSTEM_RESOLVER=1 ./scripts/verify/verify-end-to-end-routing.sh --profile=public`. Revert with `sudo ./scripts/verify/remove-e2e-hosts-from-etc-hosts.sh`.
**E2E profiles:** Use `--profile=public` for public endpoints (default) or `--profile=private` for private/admin RPC only. Run sequentially to avoid timestamp collision in evidence dirs. **Known E2E warnings** (502/404 and WS): [E2E_ENDPOINTS_LIST.md](../04-configuration/E2E_ENDPOINTS_LIST.md) § Known E2E warnings and Remediation. MIM4U web 502s and WS test-format warnings are **non-blocking** for contract/pool completion.
**Pre-PR validation:** Before opening PRs (Chainlist, token list, Trust Wallet), run `./scripts/run-before-pr-validations.sh` from repo root.
---
## 9. Wemix token verification (High)
Open [scan.wemix.com/tokens](https://scan.wemix.com/tokens); confirm WETH, USDT, USDC addresses. If different, update `config/token-mapping-multichain.json` and [WEMIX_TOKEN_VERIFICATION.md](../07-ccip/WEMIX_TOKEN_VERIFICATION.md). Then:
```bash
./scripts/validation/validate-config-files.sh
```
---
## References
- [COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md) — full plan (required, optional, recommended)
- [TODOS_CONSOLIDATED.md](TODOS_CONSOLIDATED.md) — full task list
- [NEXT_STEPS_AND_REMAINING_TODOS.md](NEXT_STEPS_AND_REMAINING_TODOS.md) — detail and completed items
- [STEPS_FROM_PROXMOX_OR_LAN_WITH_SECRETS.md](STEPS_FROM_PROXMOX_OR_LAN_WITH_SECRETS.md) — full LAN steps