e4c9dda0fd
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- Marked submodules ai-mcp-pmm-controller, explorer-monorepo, and smom-dbis-138 as dirty to reflect recent changes. - Updated documentation to clarify operator script usage, including dotenv loading and task execution instructions. - Enhanced the README and various index files to provide clearer navigation and task completion guidance. Made-with: Cursor
207 lines
6.2 KiB
Markdown
207 lines
6.2 KiB
Markdown
# Troubleshooting Quick Reference
|
|
|
|
**Last Updated:** 2026-03-04
|
|
**Document Version:** 1.0
|
|
**Status:** Active Documentation
|
|
|
|
---
|
|
|
|
## Quick Reference: Common Issues and Solutions
|
|
|
|
### Container Issues
|
|
|
|
| Issue | Quick Check | Quick Fix |
|
|
|-------|-------------|-----------|
|
|
| Container won't start | `pct status <vmid>` | `pct start <vmid>` |
|
|
| Container out of disk | `pct exec <vmid> df -h` | `pct set <vmid> -rootfs <size>` |
|
|
| Container out of memory | `pct list --full` | `pct set <vmid> -memory <mb>` |
|
|
| Container network issue | `pct exec <vmid> ping 8.8.8.8` | Check network config |
|
|
| Container service down | `pct exec <vmid> systemctl status <service>` | `pct exec <vmid> systemctl restart <service>` |
|
|
|
|
---
|
|
|
|
### Network Issues
|
|
|
|
| Issue | Quick Check | Quick Fix |
|
|
|-------|-------------|-----------|
|
|
| Can't reach container | `ping <container-ip>` | Check firewall rules |
|
|
| DNS not working | `pct exec <vmid> nslookup google.com` | Check DNS config |
|
|
| Port not accessible | `nc -zv <ip> <port>` | Check port forwarding |
|
|
| VLAN not working | `ip addr show` | Check VLAN config |
|
|
| Cloudflare tunnel down | `pct status 102` | Restart cloudflared container |
|
|
|
|
---
|
|
|
|
### RPC / Chain 138 transaction errors
|
|
|
|
| Issue | Quick Check | Quick Fix |
|
|
|-------|-------------|-----------|
|
|
| RPC -32001 (Nonce too low) | `cast nonce $DEPLOYER --rpc-url $RPC --block pending` | Use `NEXT_NONCE=$(cast nonce ...)` before forge script; or run [preflight](09-troubleshooting/RPC_ERRORS_32001_32602.md) and clear tx pool. |
|
|
| RPC -32602 (Invalid params) | Check `eth_chainId` and RPC URL | Use correct `RPC_URL_138`; use explicit `--gas-limit`; see [RPC_ERRORS_32001_32602.md](09-troubleshooting/RPC_ERRORS_32001_32602.md). |
|
|
| RPC -32xxx gas (deploy failed) | `cast balance $DEPLOYER --rpc-url $RPC` | Use `--gas-estimate-multiplier 150` (or 200) with `forge script ... --broadcast`; fund deployer; see [RPC_ERRORS_32001_32602.md](09-troubleshooting/RPC_ERRORS_32001_32602.md). |
|
|
| Wrong token address (Explorer / wallet) | Blockscout shows multiple contracts per symbol | Use **canonical** addresses only: [EXPLORER_TOKEN_LIST_CROSSCHECK.md](11-references/EXPLORER_TOKEN_LIST_CROSSCHECK.md) §5. Do not use non-canonical LINK/cUSDT/cUSDC. PMM token alignment verified on-chain 2026-03-04 — see §8. |
|
|
|
|
### Service Issues
|
|
|
|
| Issue | Quick Check | Quick Fix |
|
|
|-------|-------------|-----------|
|
|
| Besu node not syncing | `pct exec <vmid> curl http://localhost:8545` | Check logs, restart |
|
|
| RPC endpoint down | `curl https://rpc-http-pub.d-bis.org` | Check Nginx, restart RPC |
|
|
| Blockscout not loading | `curl http://192.168.11.140:80` | Check Blockscout status |
|
|
| Cloudflare tunnel error | `pct logs 102` | Check tunnel config |
|
|
|
|
---
|
|
|
|
### Performance Issues
|
|
|
|
| Issue | Quick Check | Quick Fix |
|
|
|-------|-------------|-----------|
|
|
| High CPU usage | `pct list --full` | Check processes, limit CPU |
|
|
| High memory usage | `pct list --full` | Increase memory or optimize |
|
|
| Disk I/O high | `iostat -x 1` | Check disk usage, optimize |
|
|
| Network latency | `ping <target>` | Check network, optimize routing |
|
|
|
|
---
|
|
|
|
### Quick Diagnostic Commands
|
|
|
|
#### Container Health Check
|
|
```bash
|
|
# Check container status
|
|
pct status <vmid>
|
|
|
|
# Check container resources
|
|
pct list --full | grep <vmid>
|
|
|
|
# Check container logs
|
|
pct logs <vmid> --tail 50
|
|
|
|
# Check container network
|
|
pct exec <vmid> ip addr show
|
|
```
|
|
|
|
#### Network Health Check
|
|
```bash
|
|
# Test connectivity
|
|
ping -c 4 192.168.11.1
|
|
|
|
# Check DNS
|
|
nslookup google.com
|
|
|
|
# Check routes
|
|
ip route show
|
|
|
|
# Test port
|
|
nc -zv <ip> <port>
|
|
```
|
|
|
|
#### Service Health Check
|
|
```bash
|
|
# Check Besu RPC
|
|
curl -X POST http://localhost:8545 -H "Content-Type: application/json" \
|
|
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
|
|
|
|
# Check Blockscout
|
|
curl http://192.168.11.140:80
|
|
|
|
# Check Cloudflare tunnel
|
|
pct status 102
|
|
```
|
|
|
|
---
|
|
|
|
### Common Error Messages
|
|
|
|
| Error Message | Likely Cause | Solution |
|
|
|---------------|--------------|----------|
|
|
| "No space left on device" | Disk full | Free up space or expand disk |
|
|
| "Connection refused" | Service not running | Start service or check firewall |
|
|
| "Network unreachable" | Network config issue | Check network configuration |
|
|
| "Permission denied" | Permission issue | Check file permissions |
|
|
| "Container not found" | Wrong VMID | Verify VMID with `pct list` |
|
|
|
|
---
|
|
|
|
### Emergency Procedures
|
|
|
|
#### Container Won't Start
|
|
1. Check status: `pct status <vmid>`
|
|
2. Check logs: `pct logs <vmid>`
|
|
3. Check config: `pct config <vmid>`
|
|
4. Try manual start: `pct start <vmid>`
|
|
5. If still failing, check resources: `pct list --full`
|
|
|
|
#### Network Completely Down
|
|
1. Check router: `ping 192.168.11.1`
|
|
2. Check switch: Physical connection
|
|
3. Check Proxmox network: `ip addr show`
|
|
4. Check firewall: `iptables -L`
|
|
5. Restart network: `systemctl restart networking`
|
|
|
|
#### Service Critical Failure
|
|
1. Check service status: `systemctl status <service>`
|
|
2. Check logs: `journalctl -u <service> -n 50`
|
|
3. Restart service: `systemctl restart <service>`
|
|
4. If persistent, check configuration
|
|
5. Escalate to detailed troubleshooting guide
|
|
|
|
---
|
|
|
|
### Quick Fixes by Service
|
|
|
|
#### Besu Node
|
|
```bash
|
|
# Restart Besu container
|
|
pct restart <vmid>
|
|
|
|
# Check Besu logs
|
|
pct logs <vmid> | grep -i error
|
|
|
|
# Check Besu RPC
|
|
curl -X POST http://localhost:8545 -H "Content-Type: application/json" \
|
|
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
|
|
```
|
|
|
|
#### RPC Endpoint
|
|
```bash
|
|
# Check RPC container
|
|
pct status 2500
|
|
|
|
# Check Nginx
|
|
pct exec 105 nginx -t
|
|
|
|
# Restart Nginx
|
|
pct exec 105 systemctl restart nginx
|
|
|
|
# Test RPC endpoint
|
|
curl https://rpc-http-pub.d-bis.org
|
|
```
|
|
|
|
#### Cloudflare Tunnel
|
|
```bash
|
|
# Check tunnel status
|
|
pct status 102
|
|
|
|
# Check tunnel logs
|
|
pct logs 102 | tail -50
|
|
|
|
# Restart tunnel
|
|
pct restart 102
|
|
|
|
# Verify tunnel health
|
|
# Check Cloudflare dashboard
|
|
```
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- **[../09-troubleshooting/TROUBLESHOOTING_FAQ.md](/docs/09-troubleshooting/TROUBLESHOOTING_FAQ.md)** ⭐⭐⭐ - Complete troubleshooting FAQ
|
|
- **[../09-troubleshooting/TROUBLESHOOTING_FAQ.md](../09-troubleshooting/TROUBLESHOOTING_FAQ.md)** ⭐⭐ - Troubleshooting FAQ and flow
|
|
- **[../03-deployment/OPERATIONAL_RUNBOOKS.md](../03-deployment/OPERATIONAL_RUNBOOKS.md)** ⭐⭐ - Operational runbooks
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-01-20
|
|
**Review Cycle:** Quarterly
|