proxmox/docs/04-configuration/VMID5000_STORAGE_AND_LOGS_RUNBOOK.md

VMID 5000 (Explorer) — Storage and Logs Runbook

Last updated: 2026-02-21
Host: 192.168.11.12 (r630-02)
Purpose: Free disk space, maintain ≥70% free where possible, and keep logs on a separate volume.

---

1. Current layout

Mount	Size	Purpose	Notes
/ (rootfs)	196G	thin5 (LVM thin)	Root + Docker (Blockscout/Postgres). Docker container logs are capped (see §2.3).
/var/log-remote	220G (rpool)	Host bind mount /var/lib/vz/logs-vmid5000	Log volume — nginx and future logs; 193G free

• Docker container logs were the main cause of root filling (Blockscout *-json.log grew to ~178G). This is now limited by daemon.json (max-size 100m, max-file 3). The maintenance script also truncates any container log over 500M.
• Log volume ensures nginx and other service logs grow on rpool, not on root.

---

2. Free space on root (quick wins)

Run from Proxmox host (or via SSH to 192.168.11.12) so pct exec 5000 is available.

2.1 One-shot cleanup (script)

From proxmox repo (with SSH to 192.168.11.12):

# Option A: from repo
PROXMOX_HOST_R630_02=192.168.11.12 ./scripts/maintenance/vmid5000-free-disk-and-logs.sh

# Option B: from explorer-monorepo
EXPLORER_VM_HOST=root@192.168.11.12 bash scripts/free-disk-vmid5000.sh

2.2 Docker container log limit (prevents root filling)

Configured on VM 5000: /etc/docker/daemon.json:

{"log-driver": "json-file", "log-opts": {"max-size": "100m", "max-file": "3"}}

• Restart Docker after changing: pct exec 5000 -- systemctl restart docker
• Existing containers keep the limit after restart. New containers use it by default.

2.3 Emergency: root full (e.g. one container log grew before limit)

If root is 100% and nginx/config updates fail, truncate the large container log from the host:

ssh root@192.168.11.12 "pct exec 5000 -- bash -c '
  for f in /var/lib/docker/containers/*/*-json.log; do
    [ -f \"\$f\" ] && [ \$(stat -c%s \"\$f\" 2>/dev/null) -gt 536870912 ] && truncate -s 0 \"\$f\"
  done
'"

(536870912 = 512M; adjust if needed.) Then run §2.1 and ensure §2.2 is applied.

2.4 Manual steps (if you need to free more)

# On Proxmox host
HOST=192.168.11.12
VMID=5000
ssh root@$HOST "pct exec $VMID -- bash -c '
  # Journal: keep 1 day or 100M
  journalctl --vacuum-time=1d
  journalctl --vacuum-size=100M

  # Old backups (keep last 2)
  ls -t /var/www/html/index.html.backup.* 2>/dev/null | tail -n +3 | xargs -r rm -f
  ls -t /etc/nginx/sites-available/blockscout.backup.* 2>/dev/null | tail -n +3 | xargs -r rm -f

  # Truncate syslog and force logrotate
  : > /var/log/syslog
  logrotate -f /etc/logrotate.conf

  # Docker (safe: no container/volume prune)
  docker image prune -af
  docker builder prune -af
'"

---

3. Log volume (/var/log-remote)

3.1 How it was set up

• Host path: /var/lib/vz/logs-vmid5000 (on rpool; 193G free).
• In CT 5000: Bind mount as /var/log-remote.
• CT config (on host): mp1: /var/lib/vz/logs-vmid5000,mp=/var/log-remote

Nginx is configured to log to /var/log-remote/nginx/. Logrotate for that directory: /etc/logrotate.d/nginx-remote inside the container (daily, rotate 7, compress).

3.2 Journal size limit

To prevent journal from refilling root:

• Config: /etc/systemd/journald.conf.d/size-limit.conf in the container:
  • SystemMaxUse=100M
  • SystemMaxFileSize=20M
• Restart journald to apply: systemctl restart systemd-journald (inside CT).

3.3 Add more services to log volume

Point other services to /var/log-remote/<service>/ and add logrotate entries under /etc/logrotate.d/ so logs stay on the log volume.

---

4. Reaching ~70% free on root (optional, future)

With Docker log limits (§2.2), root should stay well below 100%. If you need more headroom:

1. Expand rootfs: On the Proxmox host, grow the LV and then inside the CT: resize2fs /dev/mapper/... (or the appropriate command for the root device). Requires free space in the thin pool.
2. Second disk for Docker: Add a bind mount (e.g. new LV) to the container and move /var/lib/docker there (stop Docker, move, adjust fstab or mount, start Docker).
3. Migrate Blockscout/Postgres to a dedicated data volume.

This runbook focuses on freeing space and capping Docker logs; expansion is optional.

---

5. Checks

Daily/weekly: scripts/maintenance/daily-weekly-checks.sh daily runs check-disk-all-vmids.sh, which reports VMID 5000 root usage (WARN ≥85%, CRIT ≥95%).

# Disk usage
ssh root@192.168.11.12 "pct exec 5000 -- df -h / /var/log-remote"

# Docker container log sizes (should be small with daemon.json limit)
ssh root@192.168.11.12 "pct exec 5000 -- du -sh /var/lib/docker/containers/*"

# Log volume usage
ssh root@192.168.11.12 "pct exec 5000 -- du -sh /var/log /var/log-remote"

# Journal size
ssh root@192.168.11.12 "pct exec 5000 -- journalctl --disk-usage"

---

6. Related

• LOGROTATE_AUDIT_RUNBOOK.md — logrotate for other VMIDs
• STORAGE_GROWTH_AND_HEALTH.md — growth factors and thresholds
• explorer-monorepo/scripts/free-disk-vmid5000.sh — lightweight Docker prune only
• scripts/maintenance/vmid5000-free-disk-and-logs.sh — full cleanup (journal, backups, logs, Docker container log truncation >500M, image/builder prune)
• Docker daemon.json on VM 5000: log-opts max-size 100m, max-file 3 (§2.2)