proxmox/docs/06-besu/DEPLOYMENT_STRATEGY_EVALUATION.md

# Deployment Strategy Evaluation

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date**: 2025-01-20  
**Status**: Comprehensive Analysis Complete  
**Purpose**: Evaluate best methods to accomplish all deployment tasks successfully

---

## Current Situation

### ✅ Resolved Issues

1. **Network Configuration**
   - Problem: VMID 2101 had no IP address
   - Solution: Configured IP 192.168.11.211/24 with gateway
   - Status: ✅ FIXED

2. **Internet Access**
   - Problem: VM had no internet connectivity
   - Solution: Gateway configured, internet restored
   - Status: ✅ FIXED

3. **Dependencies**
   - Problem: Missing lib/ directory (forge-std, openzeppelin)
   - Solution: Copied lib/ directory to VM
   - Status: ✅ FIXED

4. **Compilation Strategy**
   - Problem: 266 files too large for single compilation
   - Solution: Incremental deployment script (compile per script)
   - Status: ✅ SOLVED

### ❌ Current Blocking Issues

1. **"Replacement transaction underpriced" Errors**
   - Multiple deployment attempts creating pending transactions
   - Same nonce being reused
   - New transactions can't replace pending ones

2. **Transaction Pool Issues**
   - Transactions stuck in mempool
   - Unable to verify transaction status reliably
   - Besu network may have transaction pool limitations

3. **Deployment Verification**
   - Contracts not appearing on-chain despite transaction submission
   - Transaction hashes exist but contracts not deployed

---

## Root Cause Analysis

### Issue 1: Nonce Management

**Problem**: 
- Each deployment attempt uses the same nonce
- Previous transactions with that nonce are still pending
- New transactions can't replace them without higher gas price

**Evidence**:
- Transaction hash: `0x421f8e0a8265fd46ccb0f18502a355aaf7c375216a200281444d14bd7e6cea08`
- Current nonce: 13104
- Error: "Replacement transaction underpriced"

### Issue 2: Gas Price Strategy

**Problem**:
- Gas prices may not be high enough to replace pending transactions
- EIP-1559 parameters may not be optimal
- Besu network may have specific gas price requirements

**Current Settings**:
- Max fee: 1.5 gwei (1500000000 wei)
- Priority fee: 1.4 gwei (1400000000 wei)
- Base fee: 7 wei

### Issue 3: Transaction Pool Management

**Problem**:
- Besu transaction pool may not be clearing properly
- Pending transactions blocking new deployments
- No reliable way to check/clear mempool

---

## Solution Strategies

### Strategy 1: Wait and Retry (RECOMMENDED)

**Approach**:
1. Wait for pending transactions to be mined
2. Check transaction status periodically
3. Once mined or failed, proceed with next deployment

**Pros**:
- No risk of transaction conflicts
- Respects network state
- Most reliable approach

**Cons**:
- Requires waiting (may take time)
- Need to monitor transaction status

**Implementation**:
```bash
# Wait for transaction to be mined
while true; do
  STATUS=$(cast receipt $TXHASH --rpc-url $RPC 2>&1)
  if echo "$STATUS" | grep -q "status.*0x1"; then
    echo "Transaction successful!"
    break
  elif echo "$STATUS" | grep -q "status.*0x0"; then
    echo "Transaction failed, retrying..."
    break
  fi
  sleep 10
done
```

### Strategy 2: Clear Broadcast Cache and Use Higher Nonce

**Approach**:
1. Clear all broadcast cache files
2. Manually increment nonce
3. Deploy with explicit nonce

**Pros**:
- Bypasses pending transaction issues
- Full control over nonce

**Cons**:
- Requires manual nonce management
- Risk of nonce gaps if transaction eventually succeeds

**Implementation**:
```bash
# Clear cache
rm -rf broadcast/*/138/run-*.json

# Get current nonce
NONCE=$(cast nonce $DEPLOYER --rpc-url $RPC)

# Deploy with explicit nonce (if forge supports it)
forge script ... --nonce $NONCE
```

### Strategy 3: Significantly Increase Gas Price

**Approach**:
1. Use much higher gas prices (10x or more)
2. Force replacement of pending transactions
3. Accept higher deployment costs

**Pros**:
- May replace pending transactions
- Faster deployment

**Cons**:
- Higher costs
- May not work if transaction already mined
- Wastes gas if pending tx eventually succeeds

**Implementation**:
```bash
# Use very high gas price
MAX_FEE="15000000000"  # 15 gwei
PRIORITY="14000000000"  # 14 gwei

forge script ... --with-gas-price $MAX_FEE --priority-gas-price $PRIORITY
```

### Strategy 4: Use Besu Admin API to Clear Mempool

**Approach**:
1. Use Besu admin API to inspect/clear mempool
2. Remove pending transactions
3. Deploy fresh

**Pros**:
- Direct control over mempool
- Can clear specific transactions

**Cons**:
- Requires admin access
- May not be available on all Besu nodes
- Could affect other pending transactions

**Implementation**:
```bash
# Check if admin API available
cast rpc txpool_besuPendingTransactions --rpc-url $RPC

# Clear mempool (if supported)
cast rpc txpool_clearPendingTransactions --rpc-url $RPC
```

### Strategy 5: Deploy Using Direct cast send

**Approach**:
1. Compile contracts locally
2. Extract bytecode
3. Use `cast send --create` directly
4. Full control over transaction parameters

**Pros**:
- Complete control over transaction
- Can set explicit nonce
- Can use very high gas prices
- Bypasses forge script limitations

**Cons**:
- More complex setup
- Need to extract bytecode manually
- More error-prone

**Implementation**:
```bash
# Extract bytecode
BYTECODE=$(jq -r '.bytecode.object' out/CCIPWETH9Bridge.sol/CCIPWETH9Bridge.json)

# Deploy with cast send
cast send --create $BYTECODE \
  --rpc-url $RPC \
  --private-key $PRIVATE_KEY \
  --nonce $NONCE \
  --max-fee-per-gas $MAX_FEE \
  --priority-fee-per-gas $PRIORITY
```

### Strategy 6: Deploy One at a Time with Verification

**Approach**:
1. Deploy WETH9 Bridge
2. Wait and verify deployment
3. Only proceed to WETH10 after WETH9 confirmed
4. Repeat for each contract

**Pros**:
- Clear progress tracking
- Can verify each step
- Easier to debug issues

**Cons**:
- Slower overall process
- Requires manual intervention

---

## Recommended Approach

### Primary Strategy: Wait and Verify (Strategy 1 + Strategy 6)

**Step-by-Step**:

1. **Check Current Transaction Status**
   ```bash
   # Check if WETH9 transaction is mined
   TXHASH="0x421f8e0a8265fd46ccb0f18502a355aaf7c375216a200281444d14bd7e6cea08"
   cast receipt $TXHASH --rpc-url http://localhost:8545
   ```

2. **If Transaction Pending**
   - Wait for it to be mined (check every 30 seconds)
   - If successful, verify contract deployment
   - If failed, proceed to retry

3. **If Transaction Failed or Not Found**
   - Clear broadcast cache
   - Retry deployment with higher gas price
   - Use explicit nonce management

4. **Deploy Each Contract Sequentially**
   - WETH9 Bridge → Verify → WETH10 Bridge → Verify → LINK Token → Verify

### Fallback Strategy: Direct cast send (Strategy 5)

If forge script continues to fail:
1. Compile contracts
2. Extract bytecode
3. Use `cast send --create` with explicit nonce and high gas prices

---

## Implementation Plan

### Phase 1: Transaction Status Verification

1. Check all pending transaction hashes
2. Wait for them to be mined (with timeout)
3. Verify contract deployments
4. Document results

### Phase 2: Clean Deployment

1. Clear all broadcast caches
2. Get current nonce
3. Deploy with explicit nonce tracking
4. Verify each deployment before proceeding

### Phase 3: Alternative Methods

If Phase 2 fails:
1. Use `cast send --create` method
2. Extract bytecode from compiled contracts
3. Deploy directly with full control

---

## Best Practices

1. **Always Verify Before Proceeding**
   - Check transaction receipt
   - Verify contract code on-chain
   - Confirm deployment address

2. **Manage Nonces Explicitly**
   - Track nonce manually if needed
   - Don't rely on automatic nonce management
   - Clear cache between attempts

3. **Use Appropriate Gas Prices**
   - Check network minimum gas price
   - Use 2-3x minimum for safety
   - Monitor base fee for EIP-1559

4. **Monitor Transaction Pool**
   - Check for pending transactions
   - Wait for clearance before retrying
   - Use admin APIs if available

5. **Incremental Deployment**
   - Deploy one contract at a time
   - Verify each before proceeding
   - Keep detailed logs

---

## Tools and Scripts Needed

1. **Transaction Status Checker**
   - Monitor transaction status
   - Wait for confirmation
   - Report results

2. **Nonce Manager**
   - Track current nonce
   - Increment safely
   - Handle gaps

3. **Gas Price Calculator**
   - Dynamic gas price calculation
   - Safety multipliers
   - EIP-1559 support

4. **Deployment Verifier**
   - Check contract code on-chain
   - Verify deployment addresses
   - Confirm functionality

---

## Success Criteria

✅ All contracts deployed:
- WETH9 Bridge at expected address
- WETH10 Bridge at generated address
- LINK Token at canonical address

✅ All transactions confirmed:
- Transaction receipts show success
- Contracts have code on-chain
- Addresses match expected values

✅ Configuration complete:
- Bridge destinations configured
- Mainnet chain selector set
- All settings verified

---

**Status**: Ready for implementation  
**Next**: Execute Phase 1 - Transaction Status Verification