Files
proxmox/docs/00-meta/ALL_RECOMMENDATIONS_OPERATOR_ONLY.md

141 lines
6.8 KiB
Markdown
Raw Normal View History

# All Recommendations — Operator-Only Checklist
**Purpose:** Single checklist for all recommendations that require **LAN/Proxmox access**, **operator credentials**, **external services**, or **ongoing maintenance**. Use when you have operator or LAN access.
**Sources:** [ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md](ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md) (items 111, 7581, 9899, 106121, 135139, R1R24), [OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST.md](OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST.md), [RECOMMENDATIONS_OPERATOR_CHECKLIST.md](RECOMMENDATIONS_OPERATOR_CHECKLIST.md).
---
## 1. Proxmox / security (LAN or host access)
| # | Action | Command or notes |
|---|--------|------------------|
| 1 | Secure .env permissions | `chmod 600 .env` (and subproject .env files) |
| 2 | Secure validator key permissions | On host: `chmod 600` keys, `chown -R besu:besu` validator dirs |
| 3 | SSH key-only auth | On Proxmox/containers: `PasswordAuthentication no`, `PubkeyAuthentication yes` in sshd_config |
| 4 | Firewall for Proxmox API (8006) | Restrict to specific IPs (iptables or UDM Pro rules) |
| 5 | Network segmentation (VLANs) | Per NETWORK_ARCHITECTURE; switches, Proxmox bridges, ER605 |
| 6 | Metrics (Prometheus, Besu 9545) | Deploy Prometheus scrape; enable Besu metrics port |
| 7 | Health check + alerting | Configure Alertmanager / PagerDuty / Slack per MASTER_SECRETS |
| 8 | Automated backup + encrypted validator keys | Run backup script; store encrypted copies off-host |
| 9 | Backup configs + version control | Commit configs to repo or backup store from LAN |
---
## 2. Deployment and runbooks (operator)
| # | Action | When |
|---|--------|------|
| **Priority** | **Mainnet liquidity + ramps** | [MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md](MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md), [MAINNET_RAMP_USER_FLOWS.md](../04-configuration/MAINNET_RAMP_USER_FLOWS.md) |
| 10 | Integration tests for deployment scripts | Run from LAN against staging/dev |
| 11 | Runbooks: add/remove validator, upgrade Besu, key rotation, recovery, consensus | Document and run from LAN when needed |
| 1220 | Error handling, logging, Loki, resource/network/DB tuning, CI/CD, CLI tool | Implement on infra from operator environment |
---
## 3. Contracts and verification (R1R3, R8R9, R14)
| # | Action | Command |
|---|--------|--------|
| R1 | Verify every deployed contract on Blockscout | After each deploy when Blockscout reachable |
| R2 | Keep CONTRACT_ADDRESSES_REFERENCE and ADDRESS_MATRIX_AND_STATUS updated | When new contracts deployed or deprecated |
| R3 | Run on-chain check | `./scripts/verify/check-contracts-on-chain-138.sh` (set RPC_URL_138 from LAN) |
| R8 | Set RPC_URL_138; run from LAN if 192.168.11.x not reachable | Before any deploy |
| R9 | Use GAS_PRICE=1000000000 (or current min) on Chain 138 | Every forge script on 138 |
| R14 | Run verification after deploy in CI | When Blockscout reachable from runner |
---
## 4. Security and secrets (R4R7, 4852)
| # | Action | Notes |
|---|--------|-------|
| R4 | Do not use deprecated CCIPWETH9Bridge; use `0xcacfd227A040002e49e2e01626363071324f820a` and set env | Always |
| R5 | Never commit .env or private keys; rotate exposed keys | Always |
| R6 | API keys in .env.example placeholders | — |
| R7 | Restrict deployer key and RPC admin access | Access review |
| 4852 | Secret audit, input validation, security scanning, RBAC, config validation | git-secrets, gitleaks, bandit, trivy as applicable |
---
## 5. Documentation and runbooks (R12R13, R15R16)
| # | Action | When |
|---|--------|------|
| R12 | Keep CONTRACT_DEPLOYMENT_RUNBOOK, BLOCKSCOUT_VERIFICATION_GUIDE in sync | After script/URL changes |
| R13 | Document addresses in CONTRACT_ADDRESSES_REFERENCE per chain | Per-chain deploy |
| R15 | Consider single script: check env → deploy → verify → update config | Automation |
| R16 | Use .env.development / .env.staging / .env.production or JSON per chain | Config hygiene |
---
## 6. Monitoring and operations (R17R18, 135139)
| # | Task | Frequency |
|---|------|-----------|
| R17 | Monitor critical bridge/oracle events | Ongoing |
| R18 | Ensure Blockscout (VMID 5000) up and /api reachable | Health checks |
| 135 | Monitor explorer sync status | Daily |
| 136 | Monitor RPC node health (e.g. VMID 2201) | Daily |
| 137 | Check config API uptime | Weekly |
| 138 | Review explorer logs | Weekly |
| 139 | Update token list | As needed |
---
## 7. Testing and quality (R19R20)
| # | Action | When |
|---|--------|------|
| R19 | Run forge test before deploying; integration tests where available | Pre-deploy |
| R20 | NatSpec on public contract functions | Code quality |
---
## 8. Configuration and DNS (R21R22, infrastructure 7581)
| # | Action | When |
|---|--------|------|
| R21 | The Order / Sankofa NPMplus proxy host | When The Order portal deployed: add proxy; document in RPC_ENDPOINTS_MASTER, ALL_VMIDS_ENDPOINTS |
| R22 | Document or configure blocks #2#6 in NETWORK_ARCHITECTURE | When decided |
| Sankofa cutover | Replace <TARGET_IP>, <TARGET_PORT>, TBDs in SANKOFA_CUTOVER_PLAN | When deployed |
| 7581 | VLAN enablement, observability stack, CCIP fleet, sovereign tenants, missing containers | Per NEXT_STEPS_MASTER and deployment phases |
---
## 9. Quick wins and token mapping (R23R24)
| # | Action | When |
|---|--------|------|
| R23 | Scripts: progress indicators; --dry-run; config validation | Script updates |
| R24 | Keep config/token-mapping.json as single source of truth for 138↔Mainnet | Adding tokens |
---
## 10. External services and submissions
| # | Action | Where |
|---|--------|-------|
| 98 | CoinGecko submission (Chain 138) | [CoinGecko](https://www.coingecko.com/) |
| 99 | Consensys outreach (Swaps/Bridge support) | MetaMask/Consensys channels |
| 106108 | Verify Etherlink/Jumper/LiFi for chains 138, 651940, 42793 | External APIs / docs |
| 109121 | Tezos/Etherlink contracts and relay services | Deploy and run from operator env |
---
## 11. Optional tools (install for full automation)
- **shellcheck** — `apt install shellcheck` or `brew install shellcheck`
- **wscat** — `npm i -g wscat` or `npx -y wscat` for WebSocket RPC tests
- **sqlite3, websocat, sshpass, dig, parallel** — per verification evidence NEXT_STEPS_RUN_*.md
---
## Where to read more
- **Full recommendations list:** [ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md](ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md)
- **High-priority only:** [ALL_RECOMMENDATIONS_HIGH_PRIORITY.md](ALL_RECOMMENDATIONS_HIGH_PRIORITY.md)
- **Operator runbook (copy-paste):** [NEXT_STEPS_OPERATOR.md](NEXT_STEPS_OPERATOR.md)
- **Operator ready checklist:** [OPERATOR_READY_CHECKLIST.md](../04-configuration/OPERATOR_READY_CHECKLIST.md)
- **Detailed operator/external:** [OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST.md](OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST.md)