proxmox/scripts/README.md

# Scripts Directory

**Last Updated:** 2026-01-31

---

## Overview

This directory contains automation scripts for Proxmox VE management. Scripts have been consolidated into unified frameworks for better maintainability.

**Current Count:** 381 scripts (down from 759 - 50% reduction)

---

## Unified Frameworks

### 1. verify-all.sh
Verification framework consolidating all check/verify/validate scripts.

**Usage:**
```bash
./scripts/verify-all.sh [component] [type] [host]
```

**Examples:**
```bash
./scripts/verify-all.sh all
./scripts/verify-all.sh service status
./scripts/verify-all.sh network connectivity
```

**See:** `docs/00-meta/FRAMEWORK_USAGE_GUIDE.md` for complete documentation.

---

### 2. list.sh
Listing framework consolidating all list/show/get scripts.

**Usage:**
```bash
./scripts/list.sh [type] [filter] [host]
```

**Examples:**
```bash
./scripts/list.sh containers
./scripts/list.sh containers running
./scripts/list.sh vms r630-01
```

---

### 3. fix-all.sh
Fix framework consolidating all fix-*.sh scripts.

**Usage:**
```bash
./scripts/fix-all.sh [issue-type] [component] [host] [--dry-run]
```

**Examples:**
```bash
./scripts/fix-all.sh all
./scripts/fix-all.sh service postgresql 10100
./scripts/fix-all.sh network all --dry-run
```

---

### 4. configure.sh
Configuration framework consolidating all configure/config scripts.

**Usage:**
```bash
./scripts/configure.sh [component] [action] [host]
```

**Examples:**
```bash
./scripts/configure.sh all setup
./scripts/configure.sh network update
./scripts/configure.sh ssl validate
```

---

### 5. deploy.sh
Deployment framework consolidating all deploy/setup/install scripts.

**Usage:**
```bash
./scripts/deploy.sh [component] [options] [host]
```

**Examples:**
```bash
./scripts/deploy.sh all
./scripts/deploy.sh service postgresql
./scripts/deploy.sh all --phase=1
```

---

### 6. CCIP WETH9 Bridge (Chain 138) – router mismatch fix

Deploy and configure a new WETH9 bridge using the **working** CCIP router (fixes router mismatch where the old bridge pointed to an address with no code).

**Usage:**
```bash
# Dry-run (no PRIVATE_KEY): simulate deploy and config
./scripts/deploy-and-configure-weth9-bridge-chain138.sh --dry-run

# Real run (set in smom-dbis-138/.env or export)
# PRIVATE_KEY=0x...   # required
# RPC_URL_138=http://192.168.11.211:8545   # Chain 138 Core (admin/deploy)
./scripts/deploy-and-configure-weth9-bridge-chain138.sh
# Then: export CCIPWETH9_BRIDGE_CHAIN138=<printed address>
```

All bridge scripts use `CCIPWETH9_BRIDGE_CHAIN138` when set; otherwise they fall back to the previous bridge address. See `COMPREHENSIVE_STATUS_BRIDGE_READY.md` and `.env.example` (CCIP section).

---

### 7. Contract Verification (Blockscout)

Verify deployed contracts on Blockscout (Chain 138) using the **Forge Verification Proxy** (required for Forge/Blockscout API compatibility).

**Preferred: orchestrated script (starts proxy if needed):**
```bash
source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh
```

**Manual (proxy + verify):**
```bash
# 1. Start proxy (separate terminal)
BLOCKSCOUT_URL=http://192.168.11.140:4000 node forge-verification-proxy/server.js

# 2. Run verification
./scripts/verify-contracts-blockscout.sh
```

**Env:** `FORGE_VERIFY_TIMEOUT=600` (default; set to `0` for no limit). Uses `scripts/lib/load-project-env.sh` for config.

**See:** `forge-verification-proxy/README.md`, `docs/03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md`

### 8. CCIP WETH9 Bridge — send ETH (WETH) to mainnet

Send WETH cross-chain via CCIP (Chain 138 → Ethereum mainnet or other destination). Uses `PRIVATE_KEY` and `CCIPWETH9_BRIDGE_CHAIN138` from env (load-project-env).

**Send to mainnet (exact command):**
```bash
cd /home/intlc/projects/proxmox
source smom-dbis-138/.env
export CCIP_DEST_CHAIN_SELECTOR=5009297550715157269   # Ethereum mainnet
./scripts/bridge/run-send-cross-chain.sh <amount_eth> [recipient]
# Example: ./scripts/bridge/run-send-cross-chain.sh 0.005
# With recipient:  ./scripts/bridge/run-send-cross-chain.sh 0.005 0xYourMainnetAddress
```

**Dry-run (simulate only):**
```bash
./scripts/bridge/run-send-cross-chain.sh <amount_eth> [recipient] --dry-run
```

Default bridge in `.env` is the **LINK-fee** bridge (pay fee in Chain 138 LINK). To pay fee in **native ETH**, set `CCIPWETH9_BRIDGE_CHAIN138=0x63cbeE010D64ab7F1760ad84482D6cC380435ab5`.

**Requirements:** Sender must have (1) WETH on Chain 138 (balance ≥ amount), (2) for LINK-fee bridge: LINK on Chain 138 approved for the bridge; for native-ETH bridge: sufficient ETH for fee. When using a **new** bridge address, approve both WETH and LINK to that bridge. Recipient defaults to sender address if omitted.

**If send reverts** (e.g. `0x9996b315` with fee-token address): the CCIP router on Chain 138 may not accept the bridge’s fee token (LINK at `0xb772...`). See [docs/07-ccip/SEND_ETH_TO_MAINNET_REVERT_TRACE.md](../docs/07-ccip/SEND_ETH_TO_MAINNET_REVERT_TRACE.md) for the revert trace and fix options.

**Env:** `CCIP_DEST_CHAIN_SELECTOR` (default: 5009297550715157269 = Ethereum mainnet); `GAS_PRICE` (default: 1000000000); `CONFIRM_ABOVE_ETH` (optional; prompt for confirmation above this amount).

### 9. DBIS Frontend Deploy to Container

Deploy dbis-frontend build to Proxmox container VMID 10130. Builds locally, pushes dist, reloads nginx.

**Usage:**
```bash
./scripts/dbis/deploy-dbis-frontend-to-container.sh
```

**Env:** Uses `load-project-env.sh` and `get_host_for_vmid()`. `DBIS_FRONTEND_DEPLOY_PATH` overrides container deploy path (e.g. `/opt/dbis-core/frontend/dist`).

### 10. CT 2301 Corrupted Rootfs Recovery

CT 2301 (besu-rpc-private-1) may fail to start with `lxc.hook.pre-start` due to corrupted rootfs.

**Scripts:**
- `./scripts/fix-ct-2301-corrupted-rootfs.sh` — documents recovery options
- `./scripts/recreate-ct-2301.sh` — destroys and recreates CT 2301 (data loss; use after corrupted rootfs). Uses `load-project-env.sh` for config.

### 11. Backup and Security

- **Config backup:** `./scripts/backup-proxmox-configs.sh [--dry-run]` — backs up local config and .env
- **NPMplus backup:** `./scripts/verify/backup-npmplus.sh [--dry-run]` — requires NPM_PASSWORD in .env
- **Wave 0 from LAN:** `./scripts/run-wave0-from-lan.sh [--dry-run] [--skip-backup] [--skip-rpc-fix]` — runs NPMplus RPC fix (W0-1) and NPMplus backup (W0-3); W0-2 (sendCrossChain) run separately without `--dry-run`.
- **All waves (max parallel):** `./scripts/run-all-waves-parallel.sh [--dry-run] [--skip-wave0] [--skip-wave2] [--host HOST]` — Wave 0 via SSH, Wave 1 parallel (env, cron, SSH/firewall dry-run, shellcheck, validate), Wave 2 W2-6 (create 2506/2507/2508). See `docs/00-meta/FULL_PARALLEL_EXECUTION_ORDER.md` and `FULL_PARALLEL_RUN_LOG.md`.
- **NPMplus backup cron:** `./scripts/maintenance/schedule-npmplus-backup-cron.sh [--install|--show]` — add or print daily 03:00 cron for backup-npmplus.sh.
- **Security:** `./scripts/security/secure-env-permissions.sh [--dry-run]` or `chmod 600 .env smom-dbis-138/.env dbis_core/.env` — secure env files. **Validator keys (W1-19):** On Proxmox host as root: `./scripts/secure-validator-keys.sh [--dry-run]` (VMIDs 1000–1004).

### 12. Maintenance (135–139)

- **Daily/weekly checks:** `./scripts/maintenance/daily-weekly-checks.sh [daily|weekly|all]` — explorer sync (135), RPC health (136), config API (137). **Cron:** `./scripts/maintenance/schedule-daily-weekly-cron.sh [--install|--show]` (daily 08:00, weekly Sun 09:00). See [OPERATIONAL_RUNBOOKS.md](../docs/03-deployment/OPERATIONAL_RUNBOOKS.md) § Maintenance.
- **Start firefly-ali-1 (6201):** `./scripts/maintenance/start-firefly-6201.sh [--dry-run] [--host HOST]` — start CT 6201 on r630-02 when needed (optional ongoing).
- **Config validation (pre-deploy):** `./scripts/validation/validate-config-files.sh` — set `VALIDATE_REQUIRED_FILES` for required paths. **CI / all validation:** `./scripts/verify/run-all-validation.sh [--skip-genesis]` — dependencies + config + optional genesis (no LAN/SSH).

### 13. Phase 2, 3 & 4 Deployment Scripts

- **Monitoring (Phase 2):** `./scripts/deployment/phase2-observability.sh [--config-only]` — writes `config/monitoring/` (prometheus.yml, alertmanager.yml).
- **Security (Phase 2):** `./scripts/security/setup-ssh-key-auth.sh [--dry-run|--apply]`, `./scripts/security/firewall-proxmox-8006.sh [--dry-run|--apply] [CIDR]`.
- **Backup (Phase 2):** `./scripts/backup/automated-backup.sh [--dry-run] [--with-npmplus]` — config + optional NPMplus; cron in header.
- **CCIP (Phase 3):** `./scripts/ccip/ccip-deploy-checklist.sh` — env check and deployment order from spec.
- **Sovereign tenants (Phase 4):** `./scripts/deployment/phase4-sovereign-tenants.sh [--show-steps|--dry-run]` — checklist; full runbook in OPERATIONAL_RUNBOOKS § Phase 4.
- **Full verification (6 steps):** `./scripts/verify/run-full-verification.sh` — Step 0: config validation; Steps 1–5: DNS, UDM Pro, NPMplus, backend VMs, E2E routing; Step 6: source-of-truth JSON. Run from project root.

---

## Utility Modules

Shared utility functions are available in `scripts/utils/`:

- `container-utils.sh` - Container operations
- `network-utils.sh` - Network operations
- `service-utils.sh` - Service operations
- `config-utils.sh` - Configuration operations
- `proxmox-utils.sh` - Proxmox operations

**Usage:**
```bash
source "$(dirname "${BASH_SOURCE[0]}")/../utils/container-utils.sh"
container_status 5000
container_restart 5000
```

---

## Shared Libraries

Core shared modules in `scripts/lib/`:

- **`load-project-env.sh`** — Load project environment (.env, config/ip-addresses.conf, smom-dbis-138/.env). **Use this** instead of hardcoding IPs or sourcing multiple files. Scripts that need config should `source "${SCRIPT_DIR}/lib/load-project-env.sh"`.
- `ip-config.sh` - Centralized IP address configuration
- `logging.sh` - Consistent logging functions
- `proxmox-api.sh` - Proxmox API helpers
- `ssh-helpers.sh` - SSH utility functions

---

## Migration

Old scripts have been archived to `scripts/archive/consolidated/`. Use the frameworks instead.

**Migration Guide:** `docs/00-meta/FRAMEWORK_MIGRATION_GUIDES.md`  
**Migration Examples:** `docs/00-meta/MIGRATION_EXAMPLES.md`  
**Migration Checklist:** `docs/00-meta/MIGRATION_CHECKLIST.md`

---

## Directory Structure

```
scripts/
├── lib/              # Shared libraries (load-project-env.sh, etc.)
├── bridge/           # CCIP bridge scripts
│   └── run-send-cross-chain.sh
├── dbis/             # DBIS Core deployment scripts
│   └── deploy-dbis-frontend-to-container.sh
├── verify/           # Verification scripts
│   ├── check-contracts-on-chain-138.sh   # On-chain bytecode check (Chain 138)
│   ├── run-contract-verification-with-proxy.sh
│   └── ...           # Other verify scripts
├── utils/            # Utility modules
├── archive/          # Archived scripts
│   ├── consolidated/ # Migrated scripts
│   ├── small-scripts/# Merged small scripts
│   ├── test/        # Test scripts
│   └── backups/     # Backup scripts
├── verify-all.sh     # Verification framework
├── list.sh           # Listing framework
├── fix-all.sh        # Fix framework
├── configure.sh      # Configuration framework
└── deploy.sh         # Deployment framework
```

---

## Documentation

- **Framework Usage:** `docs/00-meta/FRAMEWORK_USAGE_GUIDE.md`
- **Migration Guides:** `docs/00-meta/FRAMEWORK_MIGRATION_GUIDES.md`
- **Final Report:** `docs/00-meta/FINAL_REDUCTION_REPORT.md`
- **Script Inventory:** `docs/00-meta/SCRIPT_INVENTORY.md`

---

**Status:** ✅ Scripts consolidated and documented