Files
proxmox/docs/12-quick-reference/TROUBLESHOOTING_QUICK_REFERENCE.md
defiQUG e4c9dda0fd
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
chore: update submodule references and documentation
- Marked submodules ai-mcp-pmm-controller, explorer-monorepo, and smom-dbis-138 as dirty to reflect recent changes.
- Updated documentation to clarify operator script usage, including dotenv loading and task execution instructions.
- Enhanced the README and various index files to provide clearer navigation and task completion guidance.

Made-with: Cursor
2026-03-04 02:03:08 -08:00

6.2 KiB

Troubleshooting Quick Reference

Last Updated: 2026-03-04
Document Version: 1.0
Status: Active Documentation


Quick Reference: Common Issues and Solutions

Container Issues

Issue Quick Check Quick Fix
Container won't start pct status <vmid> pct start <vmid>
Container out of disk pct exec <vmid> df -h pct set <vmid> -rootfs <size>
Container out of memory pct list --full pct set <vmid> -memory <mb>
Container network issue pct exec <vmid> ping 8.8.8.8 Check network config
Container service down pct exec <vmid> systemctl status <service> pct exec <vmid> systemctl restart <service>

Network Issues

Issue Quick Check Quick Fix
Can't reach container ping <container-ip> Check firewall rules
DNS not working pct exec <vmid> nslookup google.com Check DNS config
Port not accessible nc -zv <ip> <port> Check port forwarding
VLAN not working ip addr show Check VLAN config
Cloudflare tunnel down pct status 102 Restart cloudflared container

RPC / Chain 138 transaction errors

Issue Quick Check Quick Fix
RPC -32001 (Nonce too low) cast nonce $DEPLOYER --rpc-url $RPC --block pending Use NEXT_NONCE=$(cast nonce ...) before forge script; or run preflight and clear tx pool.
RPC -32602 (Invalid params) Check eth_chainId and RPC URL Use correct RPC_URL_138; use explicit --gas-limit; see RPC_ERRORS_32001_32602.md.
RPC -32xxx gas (deploy failed) cast balance $DEPLOYER --rpc-url $RPC Use --gas-estimate-multiplier 150 (or 200) with forge script ... --broadcast; fund deployer; see RPC_ERRORS_32001_32602.md.
Wrong token address (Explorer / wallet) Blockscout shows multiple contracts per symbol Use canonical addresses only: EXPLORER_TOKEN_LIST_CROSSCHECK.md §5. Do not use non-canonical LINK/cUSDT/cUSDC. PMM token alignment verified on-chain 2026-03-04 — see §8.

Service Issues

Issue Quick Check Quick Fix
Besu node not syncing pct exec <vmid> curl http://localhost:8545 Check logs, restart
RPC endpoint down curl https://rpc-http-pub.d-bis.org Check Nginx, restart RPC
Blockscout not loading curl http://192.168.11.140:80 Check Blockscout status
Cloudflare tunnel error pct logs 102 Check tunnel config

Performance Issues

Issue Quick Check Quick Fix
High CPU usage pct list --full Check processes, limit CPU
High memory usage pct list --full Increase memory or optimize
Disk I/O high iostat -x 1 Check disk usage, optimize
Network latency ping <target> Check network, optimize routing

Quick Diagnostic Commands

Container Health Check

# Check container status
pct status <vmid>

# Check container resources
pct list --full | grep <vmid>

# Check container logs
pct logs <vmid> --tail 50

# Check container network
pct exec <vmid> ip addr show

Network Health Check

# Test connectivity
ping -c 4 192.168.11.1

# Check DNS
nslookup google.com

# Check routes
ip route show

# Test port
nc -zv <ip> <port>

Service Health Check

# Check Besu RPC
curl -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Check Blockscout
curl http://192.168.11.140:80

# Check Cloudflare tunnel
pct status 102

Common Error Messages

Error Message Likely Cause Solution
"No space left on device" Disk full Free up space or expand disk
"Connection refused" Service not running Start service or check firewall
"Network unreachable" Network config issue Check network configuration
"Permission denied" Permission issue Check file permissions
"Container not found" Wrong VMID Verify VMID with pct list

Emergency Procedures

Container Won't Start

  1. Check status: pct status <vmid>
  2. Check logs: pct logs <vmid>
  3. Check config: pct config <vmid>
  4. Try manual start: pct start <vmid>
  5. If still failing, check resources: pct list --full

Network Completely Down

  1. Check router: ping 192.168.11.1
  2. Check switch: Physical connection
  3. Check Proxmox network: ip addr show
  4. Check firewall: iptables -L
  5. Restart network: systemctl restart networking

Service Critical Failure

  1. Check service status: systemctl status <service>
  2. Check logs: journalctl -u <service> -n 50
  3. Restart service: systemctl restart <service>
  4. If persistent, check configuration
  5. Escalate to detailed troubleshooting guide

Quick Fixes by Service

Besu Node

# Restart Besu container
pct restart <vmid>

# Check Besu logs
pct logs <vmid> | grep -i error

# Check Besu RPC
curl -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

RPC Endpoint

# Check RPC container
pct status 2500

# Check Nginx
pct exec 105 nginx -t

# Restart Nginx
pct exec 105 systemctl restart nginx

# Test RPC endpoint
curl https://rpc-http-pub.d-bis.org

Cloudflare Tunnel

# Check tunnel status
pct status 102

# Check tunnel logs
pct logs 102 | tail -50

# Restart tunnel
pct restart 102

# Verify tunnel health
# Check Cloudflare dashboard


Last Updated: 2025-01-20
Review Cycle: Quarterly