proxmox/docs/04-configuration/FULL_FIXES_PREPARED.md

Full Fixes Prepared — Consolidated Checklist

Last Updated: 2026-02-07
Purpose: Single master list of all fixes (infra, network, optional) with copy-paste commands.
Sources: FIXES_PREPARED.md, STUCK_TX_AND_BLOCK_STATUS_20260207.md, SENTRY_RPC_REVIEW_20260207.md, NEXT_STEPS_OPERATOR.md.
Applied run (2026-02-07): verification-evidence/APPLIED_FIXES_20260207.md.

---

Master checklist

#	Priority	Area	Issue	Fix	Section
1	Required	Validators	Only 1/5 active; block production stalled	Ensure validators 1000–1004 are up; re-run validator fix if needed	§ 1
2	Required	Stuck tx	Deployer nonce 13178 stuck (not in chain/txpool)	Use nonce 13178 for next send, or clear RPC DB	§ 2
3	Required	Sentry 1504	besu-sentry inactive; restart failed	Start service (try besu-sentry-5 or correct unit name)	§ 3
4	Required	RPC 2301	besu-rpc inactive; no RPC response	Start RPC service on ml110; check unit name and logs	§ 4
5	Required	Network	UDM Pro port forward Alltra/HYBX	Add 76.53.10.38 → 192.168.11.169 (80, 81, 443) in UniFi	§ 5
6	Required	Network	Alltra/HYBX 502 (rpc-alltra*, rpc-hybx*, cacti-*)	Verify backends (2500–2502, 2503–2505, 5201, 5202); fix NPMplus or deploy	§ 6
7	Optional	Sentry 1503	Container unknown	Confirm if 1503 exists on r630-01 or elsewhere	§ 3
8	Optional	RPC 2402, 2503–2508	Containers unknown	Create or map to correct hosts if needed	§ 4
9	Optional	NPMplus certs	Remaining Alltra/HYBX hosts	Request Let's Encrypt via script or UI	§ 7
10	Optional	Explorer SSL	explorer.d-bis.org cert	NPMplus UI: SSL for explorer.d-bis.org	§ 7
11	Optional	NPMplus cert 134	cross-all.defi-oracle.io files missing	NPMplus UI: re-save or re-request cert	§ 7
12	Optional	Shellcheck / env / verification	Code quality, permissions, E2E	Run scripts per table below	§ 7

---

1. Validators & block production (required)

Goal: Get 4/5 validators active so QBFT quorum is met and blocks are produced.

Hosts: 1000, 1001, 1002 → r630-01 (192.168.11.11); 1003, 1004 → ml110 (192.168.11.10).

Commands (from project root, SSH to hosts required):

cd /path/to/proxmox

# 1a. Re-apply validator config + tx-pool eviction and restart (if not already done)
bash scripts/fix-all-validators-and-txpool.sh

# 1b. Check validator status (after a few minutes)
source config/ip-addresses.conf 2>/dev/null
bash scripts/monitoring/monitor-blockchain-health.sh

# 1c. Per-host: check and start validators if needed
ssh root@192.168.11.11 "for v in 1000 1001 1002; do echo \"VMID \$v:\"; pct exec \$v -- systemctl is-active besu-validator 2>/dev/null || echo 'not found'; done"
ssh root@192.168.11.10 "for v in 1003 1004; do echo \"VMID \$v:\"; pct exec \$v -- systemctl is-active besu-validator 2>/dev/null || echo 'not found'; done"

# 1d. If any are inactive, start and check logs
ssh root@<host> "pct exec <vmid> -- systemctl start besu-validator"
ssh root@<host> "pct exec <vmid> -- journalctl -u besu-validator.service -n 50 --no-pager"

Docs: BLOCK_PRODUCTION_MONITORING.md, QBFT_TROUBLESHOOTING.md, TXPOOL_EVICTION_PREVENT_STUCK.md.

---

2. Stuck transaction (required)

Issue: Deployer 0x4A666F96fC8764181194447A7dFdb7d471b301C8 has pending nonce 13178 not in chain or txpool.

Fix A — Use next nonce (recommended): Use nonce 13178 for the next transaction from this account.

cd /path/to/proxmox
source config/ip-addresses.conf 2>/dev/null
bash scripts/skip-stuck-transactions.sh
# Then in cast/forge: --nonce 13178 --gas-price 10000000000 --rpc-url http://192.168.11.211:8545

Fix B — Clear RPC tx state (nuclear): Only if you need to reset RPC’s internal state.

PROXMOX_USER=root RPC_HOST=192.168.11.11 bash scripts/clear-rpc-database-complete.sh

Docs: STUCK_TRANSACTIONS_SOLUTION.md.

---

3. Sentry nodes (required)

1504 (ml110): Config already updated; service restart failed. Start with correct unit name:

ssh root@192.168.11.10 "pct exec 1504 -- systemctl list-units --type=service | grep -i besu"
# Then start the correct unit, e.g.:
ssh root@192.168.11.10 "pct exec 1504 -- systemctl start besu-sentry"   # or besu-sentry-5

1503: Container status unknown on r630-01. Confirm if VMID 1503 exists:

ssh root@192.168.11.11 "pct list | grep 1503"

Re-review all sentries:

bash scripts/review-sentry-and-rpc-nodes.sh

---

4. RPC nodes (required)

2301 (ml110, 192.168.11.232): RPC inactive. Start service and check config:

ssh root@192.168.11.10 "pct exec 2301 -- systemctl list-units --type=service | grep -i besu"
ssh root@192.168.11.10 "pct exec 2301 -- systemctl start besu-rpc"   # or besu-rpc-private
ssh root@192.168.11.10 "pct exec 2301 -- journalctl -u besu-rpc -n 30 --no-pager"

2402, 2503–2508: Containers unknown. If these nodes are required, create them per MISSING_CONTAINERS_LIST.md or confirm correct Proxmox hosts and VMIDs.

1504 / 2301 (Besu not installed): See INSTALL_BESU_1504_2301_RUNBOOK.md to install Besu and configs, then start services (or run scripts/fix-besu-services-on-host.sh on ml110 after install).

Re-review all RPC nodes:

bash scripts/review-sentry-and-rpc-nodes.sh --apply-txpool

---

5. UDM Pro & Alltra/HYBX network (required)

UDM Pro port forward: Add rules in UniFi Network → Firewall & Security (or Port Forwarding):

Rule Name	Dest IP	Dest Port	Forward to IP	Forward to Port
NPMplus Alltra/HYBX HTTP	76.53.10.38	80	192.168.11.169	80
NPMplus Alltra/HYBX HTTPS	76.53.10.38	443	192.168.11.169	443
NPMplus Alltra/HYBX Admin	76.53.10.38	81	192.168.11.169	81

Verify from LAN:

curl -s -o /dev/null -w "%{http_code}" http://192.168.11.169:80/
curl -s -o /dev/null -w "%{http_code}" -k https://192.168.11.169:81/

Doc: UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md.

---

6. Alltra/HYBX 502 (required)

Domains returning 502: rpc-alltra*.d-bis.org, rpc-hybx*.d-bis.org, cacti-alltra.d-bis.org, cacti-hybx.d-bis.org.

Step 1 — Verify backends from LAN:

# Alltra RPC
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.172:8545
# HYBX RPC
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.246:8545
# Cacti
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.177:80/
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.251:80/

Step 2: If backends respond → in NPMplus (https://192.168.11.167:81) fix Proxy Hosts (forward to correct IP:port). If backends do not respond → deploy/start VMIDs 2500–2502, 2503–2505, 5201, 5202 per NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md.

---

7. Optional fixes

Goal	Command
NPMplus certs (first only)	FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
NPMplus certs (all remaining)	NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
Explorer SSL	NPMplus UI → SSL Certificates → explorer.d-bis.org; Proxy Hosts → SSL tab
Cert 134 (cross-all.defi-oracle.io)	NPMplus UI → SSL Certificates → re-save or re-request
Shellcheck	bash scripts/verify/run-shellcheck.sh --optional
Env permissions	bash scripts/security/secure-env-permissions.sh
Full verification	bash scripts/verify/run-full-verification.sh
NPMplus backup	bash scripts/verify/backup-npmplus.sh

---

Execution order (recommended)

1. Validators & block production — fix validators, confirm 4/5 active, block production resuming.
2. Stuck transaction — use nonce 13178 for next deployer tx (or clear RPC if required).
3. Sentry 1504 — start besu-sentry (or correct unit) on ml110.
4. RPC 2301 — start besu-rpc on ml110; fix config if needed.
5. UDM Pro — add port forward 76.53.10.38 → 192.168.11.169 if direct Alltra/HYBX access is needed.
6. Alltra/HYBX 502 — verify backends, then fix NPMplus or deploy containers.
7. Optional — certs, Explorer SSL, shellcheck, env permissions, full verification, backup.

---

Quick script index

Purpose	Script
Validators + tx-pool eviction	scripts/fix-all-validators-and-txpool.sh
Sentry & RPC review (+ tx-pool)	scripts/review-sentry-and-rpc-nodes.sh [--apply-txpool]
Blockchain health	scripts/monitoring/monitor-blockchain-health.sh
Stuck tx / next nonce	scripts/skip-stuck-transactions.sh
Stuck tx investigation	scripts/investigate-transaction-persistence.sh
Full verification (6 steps)	scripts/verify/run-full-verification.sh

All scripts run from project root; source config/ip-addresses.conf (and optionally .env) where the script does not do so.