# Next Steps Completion — RPC Stability Hardening

**Date**: 2026-01-05

## What we found

### 1) Storage node restriction mismatch (startup blocker)

- VMIDs **2400–2402** and **2500–2508** (RPC nodes) use **`local-lvm:*`** as `rootfs`.
- The Proxmox node **`ml110`** is hosting these VMIDs, but **`local-lvm`** in `/etc/pve/storage.cfg` was restricted to **`r630-01`** only.
- Result: containers could fail to start on `ml110` with:
  - `storage 'local-lvm' is not available on node 'ml110'`

### 2) Besu heap oversizing (runtime instability)

- VMIDs **2506–2508** had **4GB memory** but `BESU_OPTS=-Xmx8g -Xms8g` → high risk of swap/IO thrash.
- VMID **2505** had the same symptom earlier and already caused a failure.

## Actions taken

### Storage fix (cluster config)

- Updated `/etc/pve/storage.cfg` on `ml110` to allow `local-lvm` on **`ml110`**.
- `pvesm status` now shows `local-lvm` **active** on `ml110`.

### RPC stability fix (node configs)

- VMID **2505**:
  - Container resources: memory **6144MB**, swap **1024MB**
  - Besu heap: `BESU_OPTS=-Xms2g -Xmx4g`
- VMIDs **2506–2508**:
  - Besu heap right-sized to: `BESU_OPTS=-Xms1g -Xmx2g`
  - Restarted `besu-rpc` and confirmed listeners `:8545/:8546/:9545`

## Verification

- Full RPC fleet retest: **12/12 reachable + authorized**, **block spread Δ0**
  - Report: `reports/rpc_nodes_test_20260105_064904.md`

## New reusable scripts added

- `scripts/audit-proxmox-rpc-storage.sh`
- `scripts/audit-proxmox-rpc-besu-heap.sh`

