# Flush All Stuck Transactions - Complete Guide

**Date**: $(date)  
**Status**: ✅ **SCRIPT READY**

---

## 🎯 Objective

Flush all stuck transactions from all Besu nodes using multiple methods.

---

## 📋 Available Methods

The script `flush-all-stuck-transactions.sh` tries three methods in order:

### Method 1: TXPOOL API Clear (Non-Invasive)
- Uses `txpool_besuClear` or `txpool_clear` RPC methods
- **Requires**: TXPOOL API enabled on RPC node (VMID 2500 has this enabled)
- **Status**: ⚠️ Methods not available in this Besu version
- **Impact**: None - just clears in-memory transaction pool

### Method 2: Restart All Services (Recommended)
- Restarts all Besu services (validators, sentries, RPC nodes)
- **Requires**: Proxmox host access (`pct` command)
- **Impact**: Services restart, clears in-memory transaction pools
- **Duration**: ~30 seconds

### Method 3: Clear Database Files (Most Aggressive)
- Stops services, clears transaction pool database files, restarts
- **Requires**: Proxmox host access and confirmation
- **Impact**: All pending transactions lost, requires re-sync
- **Duration**: ~2-3 minutes

---

## 🚀 Quick Start

### Step 1: Copy Scripts to Proxmox Host

**From your local machine**, copy the scripts to the Proxmox host:

```bash
cd /home/intlc/projects/proxmox
./scripts/copy-flush-scripts-to-proxmox.sh
```

This will copy:
- `flush-all-mempools-proxmox.sh`
- `flush-all-stuck-transactions.sh`
- `clear-transaction-pool-database.sh`

### Step 2: Run on Proxmox Host

**SSH to the Proxmox host** and run:

```bash
ssh root@192.168.11.10  # or your Proxmox host IP
cd /home/intlc/projects/proxmox
./scripts/flush-all-stuck-transactions.sh
```

Or run directly via SSH:

```bash
ssh root@192.168.11.10 "cd /home/intlc/projects/proxmox && ./scripts/flush-all-stuck-transactions.sh"
```

The script will:
1. Try Method 1 (TXPOOL API) - usually fails as methods aren't available
2. Automatically try Method 2 (restart services)
3. Offer Method 3 (database clear) if needed

---

## 📋 Nodes Affected

### Validators (5 nodes)
- VMID 1000-1004
- Service: `besu-validator.service`

### Sentries (4 nodes)
- VMID 1500-1503
- Service: `besu-sentry.service`

### RPC Nodes (3 nodes)
- VMID 2500-2502
- Service: `besu-rpc.service`

**Total**: 12 nodes

---

## 🔧 Manual Methods

### Option 1: Simple Restart (Quick)

```bash
cd /home/intlc/projects/proxmox
./scripts/flush-all-mempools-proxmox.sh
```

This just restarts all services - simplest and fastest method.

### Option 2: Database Clear (Aggressive)

```bash
cd /home/intlc/projects/proxmox
./scripts/clear-transaction-pool-database.sh
```

This stops services, clears transaction pool database files, then restarts.

---

## ✅ Verification

After flushing, verify transactions are cleared:

```bash
# Check transaction pool via RPC (if TXPOOL enabled)
curl -X POST http://192.168.11.250:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' | jq '.result | length'

# Should return 0 or empty result if pool is clear
```

---

## ⚠️ Important Notes

1. **Transaction Retention**: If transactions persist after restart, they may be stored in blockchain state (not just mempool)
2. **Network Sync**: After restart, nodes will re-sync with peers - transactions may reappear
3. **Nonce State**: If nonce is stuck, it's part of blockchain state, not just transaction pool
4. **Best Practice**: Use Method 2 (restart) first - it's the safest and most effective

---

## 🔍 Troubleshooting

### Transactions Persist After Restart

If transactions persist after restarting services:

1. **Check if transaction was already mined**
   ```bash
   cast tx <tx_hash> --rpc-url http://192.168.11.250:8545
   ```

2. **Check blockchain state (nonce)**
   ```bash
   cast nonce <address> --rpc-url http://192.168.11.250:8545
   ```

3. **Transaction may be in blockchain state, not mempool**
   - Nonce 23 means 23 transactions have been sent
   - This is blockchain state, not just transaction pool
   - Cannot be cleared by flushing mempool

### Solution for Blockchain State Issues

If nonce is stuck due to blockchain state:
- Use a different account
- Wait for transaction to be processed or expire
- Accept the nonce and continue with next nonce

---

## 📚 Related Documentation

- [Flush Mempools Instructions](./FLUSH_MEMPOOLS_INSTRUCTIONS.md)
- [Transaction Pool Clear Results](./TRANSACTION_POOL_CLEAR_RESULTS.md)
- [Blockchain Database Clear Results](./BLOCKCHAIN_DATABASE_CLEAR_RESULTS.md)
- [Ethereum Mainnet Blocking Issue](./ETHEREUM_MAINNET_BLOCKING_ISSUE.md)

---

**Last Updated**: $(date)

