proxmox/docs/06-besu/NEXT_ACTIONS_EXECUTION_COMPLETE.md

Next Actions Execution Complete

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-21
Status: ✅ Executed

---

Summary

All requested next actions were performed:

1. Validator tx-pool configuration – Attempted update, then reverted due to Besu layered pool incompatibility.
2. Validator restarts – All 5 validators restarted; block production resumed.
3. Verification – Configs checked, RPC/chain state queried, scripts updated for root@ SSH.

---

1. Validator Tx-Pool Update (Attempted → Reverted)

What Was Done

• SSH: Automated script used intlc@ and failed (config “not found”); root cause was SSH auth. Switched to root@ for Proxmox hosts (192.168.11.10 ml110, 192.168.11.11 r630-01).
• Config path: Validator config confirmed at /etc/besu/config-validator.toml inside each container.
• Tx-pool block added on all 5 validators (1000–1004):

  # Transaction Pool Configuration
  tx-pool-max-size=8192
  tx-pool-limit-by-account-percentage=0.5
  tx-pool-price-bump=10
  

• Restart: All validators restarted.

Result: Validators Crashed

Logs showed:

Could not use legacy transaction pool options with layered implementation

• Cause: Besu 23.10+ uses the layered transaction pool by default. The options above are legacy and are not supported with the layered implementation.
• Effect: Validators failed on startup, block production stopped at block 1203105.

Revert

• Legacy tx-pool lines were removed from /etc/besu/config-validator.toml on all 5 validators.
• Validators were restarted again.
• All came back active; block production resumed (e.g. 1203143 → 1203166+).

---

2. Validator Status After Revert

Validator	Host	Config Reverted	Service
1000	r630-01	✅	active
1001	r630-01	✅	active
1002	r630-01	✅	active
1003	ml110	✅	active
1004	ml110	✅	active

---

3. Chain State (Post–Revert)

• RPC: http://192.168.11.211:8545 – up, chain ID 138.
• Block production: Resuming (e.g. 1203143 → 1203166 in ~25 s).
• Latest block tx count: Still 0 in sampled blocks (1203166, 1203165, …).
• Deployer: 0x4A666F96fC8764181194447A7dFdb7d471b301C8
  • Latest nonce: 13104 (0x3330)
  • Pending nonce: 13113 (0x3339)
  • 9 pending transactions (nonces 13105–13113).

---

4. RPC Peers

• admin_peers on RPC returns 12 peers, including validator IPs (e.g. 192.168.11.100–104, 221, 233–238).
• P2P connectivity RPC ↔ validators appears OK.

---

5. Script Updates

• scripts/update-all-validators-txpool.sh:
  • Uses root@ for SSH (PROXMOX_USER=root).
  • Warning in header: do not add legacy tx-pool options; they crash validators with the layered pool.
  • SSH targets use PROXMOX_USER and PROXMOX_ML110 / PROXMOX_R630.
• scripts/verify-validator-configs.sh:
  • Uses root@ for SSH.
  • Treats absence of legacy tx-pool-max-size as correct; reports legacy options if present.

---

6. Layered vs Legacy Tx-Pool (Besu 23.10+)

Aspect	Legacy	Layered (default)
Options	tx-pool-max-size, tx-pool-limit-by-account-percentage, etc.	tx-pool-max-future-by-sender, tx-pool-layer-max-capacity, tx-pool-max-prioritized
Usage	Deprecated	Default
Adding legacy options with layered	Crashes	N/A

• Do not add tx-pool-max-size, tx-pool-limit-by-account-percentage, or tx-pool-retention-hours when using the default layered pool.
• If tuning is needed, use layered options only (see Besu docs / script header).

---

7. Empty Blocks and Pending Transactions

• Observation: Blocks remain empty (0 txs) while 9 txs are pending at RPC.
• Possible causes (not fully confirmed):
  1. RPC tx-pool: RPC still has legacy tx-pool options in config-rpc-core.toml. If RPC uses layered, similar conflicts might apply (RPC is running, so behavior may differ).
  2. Propagation: Validators might not be receiving or retaining those txs in their layered pool (eviction, limits, or gas rules).
  3. Gas / min fee: Validators might reject txs (e.g. below min gas price).

Next steps to consider:

• Confirm whether RPC uses legacy or layered pool and align config.
• Check validator logs for tx propagation / inclusion / drops.
• Optionally add layered tx-pool options on validators (e.g. tx-pool-max-future-by-sender, tx-pool-max-prioritized) and test.
• Verify min gas price / fee settings on validators vs. pending txs.

---

8. Files Touched

• scripts/update-all-validators-txpool.sh – root@, layered warning, SSH targets.
• scripts/verify-validator-configs.sh – root@, legacy vs layered checks.
• docs/06-besu/NEXT_ACTIONS_EXECUTION_COMPLETE.md – this report.

---

9. Commands Reference

# Verify validators (uses root@)
PROXMOX_USER=root bash scripts/verify-validator-configs.sh

# Check RPC and chain
cast chain-id --rpc-url http://192.168.11.211:8545
cast block-number --rpc-url http://192.168.11.211:8545
cast rpc admin_peers --rpc-url http://192.168.11.211:8545

# Deployer nonces
cast rpc eth_getTransactionCount 0x4A666F96fC8764181194447A7dFdb7d471b301C8 latest --rpc-url http://192.168.11.211:8545
cast rpc eth_getTransactionCount 0x4A666F96fC8764181194447A7dFdb7d471b301C8 pending --rpc-url http://192.168.11.211:8545

---

Conclusion: Validator tx-pool update was attempted, then reverted because legacy options crash Besu’s layered pool. All validators are active, block production has resumed, and scripts now use root@ and document the layered vs legacy behavior. Empty blocks and 9 pending txs remain to be addressed via RPC/validator tx-pool alignment and propagation analysis.