proxmox/docs/06-besu/DEPLOYMENT_TROUBLESHOOTING.md

Deployment Troubleshooting

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Status: 🔍 INVESTIGATING

---

Issues Identified

1. Nonce Conversion Failure

Problem: Hex to decimal conversion showing "0" instead of actual values

• Latest nonce hex: "0x3330" (should be 13104)
• After conversion: Shows as 0

Root Cause:

• Quotes in hex values ("0x3330" instead of 0x3330)
• Python conversion failing due to string formatting
• cast --to-dec may also be affected

Impact:

• Cannot properly track nonce status
• Cannot determine if transactions are confirmed
• Deployment verification failing

2. Transactions Not Confirming

Problem: Transactions sent but not being mined

• WETH9 Bridge (nonce 13105): Sent, but not confirmed
• WETH10 Bridge (nonce 13106): Sent, but not confirmed
• Pending nonce: 0x3333 (13107) indicates transactions in mempool

Possible Causes:

1. Block production stalled: Validators not producing blocks
2. Nonce gap blocking: Stuck transaction at nonce 13104 preventing later transactions
3. Gas price too low: Transactions not being prioritized
4. Network sync issues: Transactions not reaching validators

3. Block Production Status

Check Needed: Verify blocks are being produced

• If blocks not being produced → Network stalled
• If blocks are being produced → Transactions may be queued

---

Troubleshooting Steps

Step 1: Verify Nonce Conversion

# Test hex conversion
HEX="0x3330"
python3 -c "print(int('$HEX', 16))"  # Should be 13104

# Check if quotes are the issue
HEX_QUOTED='"0x3330"'
HEX_CLEAN=$(echo "$HEX_QUOTED" | tr -d '"')
python3 -c "print(int('$HEX_CLEAN', 16))"  # Should work

Step 2: Check Block Production

# Monitor block production
BLOCK1=$(cast block-number --rpc-url $RPC)
sleep 10
BLOCK2=$(cast block-number --rpc-url $RPC)
# Should increase

Step 3: Check Transaction Mempool

# Check pending transactions
cast rpc eth_pendingTransactions --rpc-url $RPC

# Check if our transactions are there
# Look for nonces 13105 and 13106

Step 4: Check Validator Status

# Check if validators are running
# Check validator logs for errors
# Verify network consensus

Step 5: Check Gas Price

# Verify gas price is sufficient
# Check minimum gas price on network
# Compare with transaction gas prices

---

Diagnostic Commands

Check Nonce Status (Fixed)

# Get hex value
LATEST_HEX=$(cast rpc eth_getTransactionCount $DEPLOYER latest --rpc-url $RPC)

# Remove quotes if present
LATEST_CLEAN=$(echo "$LATEST_HEX" | tr -d '"')

# Convert to decimal
LATEST_DEC=$(python3 -c "print(int('$LATEST_CLEAN', 16))")

Check Block Production

# Monitor blocks
watch -n 5 "cast block-number --rpc-url $RPC"

Check Pending Transactions

# Get pending transactions
cast rpc eth_pendingTransactions --rpc-url $RPC | jq '.result[] | select(.from == "$DEPLOYER")'

Check Transaction Receipts

# Search for transactions in recent blocks
LATEST=$(cast block-number --rpc-url $RPC)
for i in {0..50}; do
  BLOCK=$((LATEST - i))
  cast block $BLOCK --rpc-url $RPC | grep $DEPLOYER
done

---

Potential Solutions

Solution 1: Fix Nonce Conversion

• Remove quotes from hex values before conversion
• Use proper hex conversion method
• Update scripts to handle quoted hex values

Solution 2: Check Block Production

• Verify validators are producing blocks
• Check validator logs for errors
• Restart validators if needed

Solution 3: Wait for Transaction Processing

• Transactions may be queued
• Wait for network to process
• Check if stuck transaction at nonce 13104 is blocking

Solution 4: Increase Gas Price

• Transactions may be too low priority
• Increase gas price to prioritize
• Check network minimum gas price

Solution 5: Clear and Retry

• Clear transaction pool again
• Wait for stuck transaction to expire
• Retry with higher gas price

---

Next Steps

1. ✅ Fix nonce conversion issue (remove quotes)
2. ✅ Verify block production is working
3. ✅ Check if transactions are in mempool
4. ⏳ Determine why transactions aren't confirming
5. ⏳ Take corrective action based on findings

---

Status: Investigation in progress
Priority: Fix nonce conversion first, then investigate block production