proxmox/docs/06-besu/TRANSACTION_PERSISTENCE_FINDINGS.md

Transaction Persistence Investigation - Findings

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Status: ✅ ROOT CAUSE IDENTIFIED

---

🔍 Investigation Results

Key Finding: Pending Transaction in Mempool

Latest Nonce (Confirmed): 0x3330 (13104 decimal)
Pending Nonce (Includes Pending): 0x3331 (13105 decimal)

Difference: 1 pending transaction

---

📊 Root Cause Analysis

The Problem

1. Transaction with nonce 13104 is stuck in mempool

   • NOT in blockchain (searched 1000 blocks, not found)
   • IS in mempool (pending nonce is 13105, indicating nonce 13104 is pending)
   • Prevents new transactions from being sent

2. Why it persists after mempool flush

   • Transaction was re-added to mempool after restart
   • May be in validator mempools, not just RPC node
   • Besu may have transaction retention/replay mechanisms

3. Why "Known transaction" error occurs

   • Besu recognizes the transaction hash
   • Transaction exists in mempool or transaction pool database
   • Cannot replace without higher gas price

---

🔍 Investigation Details

Check 1: Pending vs Latest Nonce ✅

• Result: Pending nonce (13105) > Latest nonce (13104)
• Conclusion: 1 pending transaction exists

Check 2: Transaction Pool Status ⚠️

• Result: eth_pendingTransactions method not available or returned error
• Conclusion: Cannot query transaction pool directly via RPC

Check 3: Last Transaction from Deployer ⚠️

• Result: No transactions found in last 500 blocks
• Conclusion: Deployer account has been inactive (or transactions are very old)

Check 4: Nonce 13104 in Blockchain ✅

• Result: Transaction with nonce 13104 NOT found in blockchain
• Conclusion: Transaction is in mempool, not blockchain

Check 5: Besu Configuration

• Result: Configuration files not found in standard locations
• Conclusion: Configuration may be in different location or embedded in service file

---

💡 Why Transactions Persist

Possible Reasons

1. Transaction Replay After Restart

   • Besu may replay transactions from peers after restart
   • Network sync may re-add transactions to mempool
   • Validators may propagate transactions back to RPC nodes

2. Transaction Pool Database Persistence

   • Besu may store transactions in database
   • Database not cleared by service restart
   • Transactions reloaded from database on startup

3. Network Propagation

   • Transaction may be in validator mempools
   • Validators propagate to RPC nodes
   • Transaction pool sync across network

4. Transaction Retention Settings

   • Besu may have transaction retention period
   • Transactions kept in pool for extended time
   • Not cleared until expiry or explicit clear

---

✅ Solutions

Solution 1: Wait for Transaction Expiry (Recommended)

Approach: Wait for Besu's transaction retention period to expire

Pros:

• No downtime
• Automatic cleanup
• No risk of nonce gaps

Cons:

• Takes time (1-6 hours typically)
• May not work if retention is very long

Implementation:

# Monitor pending nonce
while true; do
  PENDING=$(cast rpc eth_getTransactionCount $DEPLOYER pending --rpc-url $RPC)
  LATEST=$(cast rpc eth_getTransactionCount $DEPLOYER latest --rpc-url $RPC)
  if [ "$PENDING" = "$LATEST" ]; then
    echo "✅ Pending transaction cleared!"
    break
  fi
  echo "⏳ Still pending... ($PENDING vs $LATEST)"
  sleep 60
done

Solution 2: Clear Transaction Pool Database (Aggressive)

Approach: Stop Besu, clear transaction pool database files, restart

Pros:

• Immediate cleanup
• Most effective method
• Clears all pending transactions

Cons:

• Requires downtime
• May need to clear on all nodes
• Risk of losing legitimate pending transactions

Implementation:

# Stop Besu
systemctl stop besu-rpc.service

# Clear transaction pool database
find /data/besu -type d -name "*pool*" -exec rm -rf {} \; 2>/dev/null
find /data/besu -type f -name "*transaction*" -delete 2>/dev/null
find /data/besu -type f -name "*pool*" -delete 2>/dev/null

# Restart Besu
systemctl start besu-rpc.service

Solution 3: Use Next Nonce (Skip Pending)

Approach: Use nonce 13105 to skip the pending transaction

Pros:

• Immediate deployment
• Bypasses stuck transaction
• No downtime

Cons:

• Risk of nonce gap if pending transaction eventually succeeds
• May cause issues if transaction is eventually mined
• Not recommended for production

Implementation:

# Use explicit nonce
NEXT_NONCE=13105
forge script ... --nonce $NEXT_NONCE

Solution 4: Replace with Higher Gas Price

Approach: Send replacement transaction with much higher gas price

Pros:

• May replace pending transaction
• Uses correct nonce
• No nonce gaps

Cons:

• Higher cost
• May not work if gas price is already very high
• Requires knowing current gas price of pending transaction

Implementation:

# Use very high gas price (10x normal)
MAX_FEE="20000000000"  # 20 gwei
PRIORITY="19000000000"  # 19 gwei

forge script ... --with-gas-price $MAX_FEE --priority-gas-price $PRIORITY

---

🎯 Recommended Approach

Primary: Wait for Transaction Expiry (Solution 1)

1. Monitor pending nonce every 5-10 minutes
2. Wait for pending nonce to match latest nonce
3. Proceed with deployment once cleared

Fallback: Clear Transaction Pool Database (Solution 2)

If waiting doesn't work:

1. Stop Besu service
2. Clear transaction pool database files
3. Restart Besu
4. Verify pending transactions cleared
5. Proceed with deployment

---

📋 Next Steps

1. Monitor pending nonce to see if transaction expires
2. If transaction persists, clear transaction pool database
3. Once cleared, proceed with deployment using nonce 13105
4. Verify deployment before proceeding to next contract

---

🔍 Diagnostic Commands

Check Pending Transactions

# Compare pending vs latest nonce
LATEST=$(cast rpc eth_getTransactionCount $DEPLOYER latest --rpc-url $RPC)
PENDING=$(cast rpc eth_getTransactionCount $DEPLOYER pending --rpc-url $RPC)
echo "Latest: $LATEST, Pending: $PENDING"

Monitor Transaction Expiry

# Watch for pending transaction to clear
watch -n 10 "cast rpc eth_getTransactionCount $DEPLOYER pending --rpc-url $RPC"

Clear Transaction Pool

# Stop, clear, restart
systemctl stop besu-rpc.service
find /data/besu -name "*pool*" -delete
systemctl start besu-rpc.service

---

Status: Root cause identified - Pending transaction in mempool
Next: Wait for expiry or clear transaction pool database