# Restart Besu Node (Clears Mempool)

**Purpose**: Restart a Besu node to clear the mempool and resolve stuck transactions  
**Script**: `scripts/restart-besu-node.sh`

---

## Overview

Restarting a Besu node clears the mempool, which removes all pending transactions. This is useful for resolving:

- Stuck transactions in mempool
- Mempool-related errors
- Transaction validation issues
- Network synchronization problems

**⚠️ Important**: Restarting clears the mempool, so all pending transactions will be lost and need to be resubmitted.

---

## Usage

### Basic Usage

```bash
# Restart by VMID (auto-detect service type)
./scripts/restart-besu-node.sh 1000

# Restart by hostname
./scripts/restart-besu-node.sh validator-1

# Specify service type explicitly
./scripts/restart-besu-node.sh 2500 rpc
```

### Service Types

- `validator` - Besu validator node
- `sentry` - Besu sentry node
- `rpc` - Besu RPC node
- `auto` - Auto-detect (default)

---

## What Happens When You Restart

1. **Service Stops**
   - Besu process terminates
   - Mempool is cleared (all pending transactions removed)
   - Network connections close

2. **Service Starts**
   - Besu process restarts
   - Node resyncs with network
   - Peer connections rebuild
   - Mempool is empty (ready for new transactions)

3. **Recovery Time**
   - Service typically starts within 10-30 seconds
   - Network resync may take 1-5 minutes depending on network state
   - Node will catch up to latest block height

---

## When to Use

### ✅ Use Restart When:

- Transactions are stuck in mempool
- Getting "replacement transaction underpriced" errors
- Mempool appears blocked
- Node shows mempool-related errors in logs
- Need to clear transaction state

### ❌ Don't Use Restart For:

- Normal operation (only when troubleshooting)
- Frequent issues (investigate root cause instead)
- Production during high-traffic periods (if avoidable)
- Multiple nodes simultaneously (restart one at a time)

---

## Examples

### Example 1: Restart Validator Node

```bash
# Restart validator node (VMID 1000)
./scripts/restart-besu-node.sh 1000

# Output:
# [INFO] VMID: 1000
# [INFO] Hostname: validator-1
# [INFO] Detected service type: validator
# [INFO] Service: besu-validator
# [WARN] ⚠️  Restarting Besu node will clear the mempool
# Continue with restart? (y/N): y
# [✓] Service restart command executed
# [✓] Service is active
# [✓] Besu node restart completed!
```

### Example 2: Restart RPC Node

```bash
# Restart RPC node (VMID 2500)
./scripts/restart-besu-node.sh 2500 rpc

# The script will restart besu-rpc.service
```

### Example 3: Restart by Hostname

```bash
# Restart using hostname
./scripts/restart-besu-node.sh sentry-2

# Script finds VMID automatically
```

---

## Manual Restart (Alternative)

If you prefer to restart manually:

```bash
# For validator node
pct exec 1000 -- systemctl restart besu-validator.service

# For sentry node
pct exec 1500 -- systemctl restart besu-sentry.service

# For RPC node
pct exec 2500 -- systemctl restart besu-rpc.service
```

---

## Monitoring After Restart

### Check Service Status

```bash
# Check if service is active
pct exec <VMID> -- systemctl status besu-validator.service

# Follow logs
pct exec <VMID> -- journalctl -u besu-validator.service -f
```

### Verify Node Recovery

1. **Check service is running**
   ```bash
   pct exec <VMID> -- systemctl is-active besu-validator.service
   # Should return: active
   ```

2. **Check node is syncing**
   ```bash
   # Via RPC
   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
   
   # Should return: {"jsonrpc":"2.0","id":1,"result":false}
   # (false means fully synced)
   ```

3. **Check block height**
   ```bash
   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   ```

---

## Troubleshooting

### Service Won't Start

If the service fails to start after restart:

1. **Check logs**
   ```bash
   pct exec <VMID> -- journalctl -u besu-validator.service -n 50
   ```

2. **Check configuration**
   ```bash
   # Verify config file exists
   pct exec <VMID> -- ls -la /etc/besu/config-*.toml
   ```

3. **Check permissions**
   ```bash
   pct exec <VMID> -- ls -la /opt/besu
   pct exec <VMID> -- ls -la /var/lib/besu
   ```

### Node Not Syncing

If the node doesn't sync after restart:

1. **Check peer connections**
   ```bash
   # Via admin RPC (if enabled)
   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}'
   ```

2. **Check static nodes**
   ```bash
   pct exec <VMID> -- cat /etc/besu/static-nodes.json
   ```

3. **Wait longer**
   - Network resync can take several minutes
   - Validator nodes may need to catch up to latest block

---

## Impact on Network

### Single Node Restart

- **Minimal impact**: Other nodes continue operating
- **Validation**: If restarting a validator, other validators continue block production
- **RPC**: If restarting RPC node, use other RPC nodes during downtime

### Multiple Node Restart

- **⚠️ Warning**: Avoid restarting multiple nodes simultaneously
- **Best practice**: Restart one node at a time, wait for recovery
- **Validator quorum**: Ensure enough validators remain running for consensus

---

## Related Scripts

- `scripts/check-besu-transaction-pool.sh` - Check mempool status
- `scripts/check-mempool-status.sh` - Monitor mempool and block production
- `scripts/network/bootstrap-network.sh` - Restart all nodes in order

---

## References

- [Besu Documentation](https://besu.hyperledger.org/)
- [Transaction Pool Documentation](https://besu.hyperledger.org/en/stable/public-networks/concepts/transactions-pool/)
- [Mempool Issue Resolution](../docs/MEMPOOL_ISSUE_RESOLUTION.md)

---

**Last Updated**: 2025-12-22

