proxmox/docs/04-configuration/STORAGE_RECOMMENDATIONS_BY_FILL_RATE.md

Storage Recommendations by Fill Rate and Growth

Last updated: 2026-02-28

Based on current usage, history in logs/storage-growth/history.csv, and physical drive layout across ml110, r630-01, and r630-02.

Completed (2026-02-28): Storage growth cron verified; prune (VMID 5000 + r630-01 CTs) run; ml110 sdb added to VG pve and data thin pool extended to ~1.7 TB (ml110 data now ~11% used). Phase 1 migration (r630-01 data → thin1): 8 CTs migrated (10233, 10120, 10100, 10101, 10235, 10236, 7804, 8640); r630-01 data 65.8% (was 72%), thin1 50.6%.

---

1. Thresholds and monitoring

Level	Use %	Action
Healthy	< 75%	Continue normal collection; review quarterly.
Watch	75–84%	Weekly review; plan prune or migration.
WARN	85–94%	Prune and/or migrate within 1–2 weeks; do not add new large CTs.
CRIT	≥ 95%	Immediate action; LVM thin pools can fail or go read-only.

Current scripts: check-disk-all-vmids.sh uses WARN 85%, CRIT 95% for container root usage. These recommendations apply to host storage (pvesm / LVM) as well.

---

2. Observed fill behavior (from history)

Host	Storage	Trend (recent)	Implied rate / note
ml110	data	~28.7% → ~25% (Feb 15 → 27)	Slight decrease (prune/dedup). Plenty of free space.
r630-01	data	88% → 100% → 72% → 65.8% (Phase 1 migration)	After Phase 1 (8 CTs data→thin1). Main growth host (validators, RPCs, many CTs).
r630-02	thin1-r630-02	~26.5% stable	Low growth.
r630-02	thin2	~4.8% → ~9% after 5000 migration	Now holds Blockscout (5000); monitor.
r630-02	thin5	Was 84.6% → 0% after migration	Empty; available for future moves.

Conclusion: The pool that fills fastest and needs the most attention is r630-01 data (72% now; many CTs, Besu/DB growth). ml110 data is stable and has headroom. r630-02 is manageable if you avoid concentrating more large CTs on a single thin pool.

---

3. Recommendations by host and pool

ml110

• data / local-lvm (~25%)

  • Rate: Low/slow.
  • Recommendations:
    • Keep running collect-storage-growth-data.sh --append (e.g. cron every 6h).
    • Prune logs in CTs periodically (e.g. with fix-storage-r630-01-and-thin5.sh-style logic for ml110 or a dedicated prune script).
    • No urgency; review again when approaching 70%.

• Unused sdb (931G)

  • Recommendation: Use it before adding new disks elsewhere.
    • Option A: Add sdb to VG pve and extend the data thin pool (or create a second thin pool). Frees pressure on sda and doubles effective data capacity.
    • Option B: Create a separate VG + thin pool on sdb for new or migrated CTs.
  • Document the chosen layout and any new Proxmox storage names in storage.cfg and in PHYSICAL_DRIVES_AND_CONFIG.md.

r630-01

• data / local-lvm (~72%)

  • Rate: Highest risk; this pool has the most CTs and Besu/DB growth.
  • Recommendations:
    1. Short term:
       • Run log/journal prune on all r630-01 CTs regularly (e.g. fix-storage-r630-01-and-thin5.sh Phase 2, or a cron job).
       • Keep storage growth collection (e.g. every 6h) and review weekly when > 70%.
    2. Before 85%:
       • Move one or more large CTs to thin1 on r630-01 (thin1 ~43% used, has space) if VMIDs allow, or plan migration to r630-02 thin pools.
       • Identify biggest CTs: check-disk-all-vmids.sh and lvs on r630-01 (data pool).
    3. Before 90%:
       • Decide on expansion (e.g. add disks to RAID10 and extend md0/LVM) or permanent migration of several CTs to r630-02.
  • Do not let this pool sit above 85% for long; it has already hit 100% once.

• thin1 (~43%)

  • Rate: Moderate.
  • Recommendations: Use as spillover for data pool migrations when possible. Monitor monthly; act if > 75%.

r630-02

• thin1-r630-02 (~26%)

  • Rate: Low.
  • Recommendation: Monitor; no change needed unless you add many CTs here.

• thin2 (~9% after 5000 migration)

  • Rate: May grow with Blockscout (5000) and other CTs.
  • Recommendations:
    • Run VMID 5000 prune periodically: vmid5000-free-disk-and-logs.sh.
    • If thin2 approaches 75%, consider moving one CT to thin5 (now empty) or thin6.

• thin3, thin4, thin6 (roughly 11–22%)

  • Rate: Low to moderate.
  • Recommendation: Include in weekly pvesm/lvs review; no special action unless one pool trends > 75%.

• thin5 (0% after migration)

  • Recommendation: Keep as reserve for migrations from thin2 or other pools when they approach WARN.

---

4. Operational schedule (by fill rate)

When	Action
Always	Cron: collect-storage-growth-data.sh --append every 6h; weekly: prune-storage-snapshots.sh (e.g. Sun 08:00).
Weekly	Review pvesm status and lvs (or run audit-proxmox-rpc-storage.sh); check any pool > 70%.
75% ≤ use < 85%	Plan and run prune; plan migration for largest CTs on that pool; consider using ml110 sdb (if not yet in use).
85% ≤ use < 95%	Execute prune and migration within 1–2 weeks; do not add new large VMs/CTs to that pool.
≥ 95%	Immediate prune + migration; consider emergency migration to ml110 (after adding sdb) or r630-02.

---

5. Scripts to support these recommendations

Script	Purpose
scripts/monitoring/collect-storage-growth-data.sh --append	Record fill over time (for rate).
scripts/maintenance/schedule-storage-growth-cron.sh --install	Install 6h collect + weekly prune.
scripts/audit-proxmox-rpc-storage.sh	Current pvesm + RPC rootfs mapping.
scripts/maintenance/check-disk-all-vmids.sh	Per-CT disk usage (find big consumers).
scripts/maintenance/fix-storage-r630-01-and-thin5.sh	Prune 5000 + r630-01 CT logs; optional migrate 5000.
scripts/maintenance/migrate-ct-r630-01-data-to-thin1.sh <VMID>	Migrate one CT from r630-01 data → thin1 (same host).
scripts/maintenance/vmid5000-free-disk-and-logs.sh	Prune Blockscout (5000) only.

---

6. Adding ml110 sdb to increase capacity (suggested steps)

1. On ml110: vgextend pve /dev/sdb (if sdb is already a PV) or pvcreate /dev/sdb && vgextend pve /dev/sdb.
2. Extend the data thin pool: lvextend -L +900G /dev/pve/data (or use lvextend -l +100%FREE and adjust as needed).
3. Re-run pvesm status and update documentation.
4. No CT migration required; existing LVs on data can use the new space.

(If sdb is a raw disk with no PV, partition or use full disk as PV per your policy; then add to pve and extend the data LV as above.)

---

7. Summary table by risk

Host	Pool	Current (approx)	Risk	Priority recommendation
ml110	data	~11% (post-extension)	Low	Done: sdb added; pool ~1.7 TB. Monitor as before.
ml110	sdb	In use (extended data)	—	Done: sdb added to pve, data thin pool extended (~1.7 TB total).
r630-01	data	~72%	High	Prune weekly; plan migrations before 85%; consider thin1 spillover.
r630-01	thin1	~43%	Medium	Use for migrations from data; monitor monthly.
r630-02	thin1-r630-02	~26%	Low	Monitor.
r630-02	thin2	~9%	Low	Prune 5000 periodically; watch growth.
r630-02	thin5	0%	Low	Keep as reserve for migrations.
r630-02	thin3, thin4, thin6	~11–22%	Low	Include in weekly review.

These recommendations are based on the rate of filling observed in history and current configurations; adjust thresholds or schedule if your growth pattern changes.