proxmox/scripts/cloudflare-tunnels/docs/TROUBLESHOOTING.md

Troubleshooting Guide

Common issues and solutions for Cloudflare Tunnel setup.

Quick Diagnostics

Run the health check script first:

./scripts/check-tunnel-health.sh

Common Issues

1. Tunnel Service Not Running

Symptoms:

• systemctl status cloudflared-ml110 shows inactive
• DNS resolves but connection times out

Solutions:

# Check service status
systemctl status cloudflared-ml110

# Check logs for errors
journalctl -u cloudflared-ml110 -n 50

# Restart service
systemctl restart cloudflared-ml110

# Check if credentials file exists
ls -la /etc/cloudflared/tunnel-ml110.json

# Verify configuration
cat /etc/cloudflared/tunnel-ml110.yml

Common Causes:

• Missing credentials file
• Invalid tunnel ID in config
• Network connectivity issues
• Incorrect configuration syntax

2. DNS Not Resolving

Symptoms:

• dig ml110-01.d-bis.org returns no results
• Browser shows "DNS_PROBE_FINISHED_NXDOMAIN"

Solutions:

1. Verify DNS record exists:

   • Go to Cloudflare Dashboard → DNS → Records
   • Check CNAME record exists for ml110-01
   • Verify proxy is enabled (orange cloud)

2. Verify DNS record target:

   • Target should be: <tunnel-id>.cfargotunnel.com
   • Not an IP address

3. Wait for DNS propagation:

   • DNS changes can take up to 5 minutes
   • Clear DNS cache: sudo systemd-resolve --flush-caches

4. Test DNS resolution:

   dig ml110-01.d-bis.org
   nslookup ml110-01.d-bis.org
   

3. Connection Timeout

Symptoms:

• DNS resolves correctly
• Connection times out when accessing URL
• Browser shows "ERR_CONNECTION_TIMED_OUT"

Solutions:

1. Check tunnel is running:

   systemctl status cloudflared-ml110
   

2. Check tunnel logs:

   journalctl -u cloudflared-ml110 -f
   

3. Verify internal connectivity:

   # From VMID 102, test connection to Proxmox host
   curl -k https://192.168.11.10:8006
   

4. Check tunnel configuration:

   • Verify service URL is correct: https://192.168.11.10:8006
   • Check tunnel ID matches Cloudflare dashboard

5. Verify Cloudflare tunnel status:

   • Go to Cloudflare Zero Trust → Networks → Tunnels
   • Check tunnel shows "Healthy" status

4. SSL Certificate Errors

Symptoms:

• Browser shows SSL certificate warnings
• "NET::ERR_CERT_AUTHORITY_INVALID"

Solutions:

1. This is expected - Proxmox uses self-signed certificates

2. Cloudflare handles SSL termination at the edge

3. Verify tunnel config has:

   originRequest:
     noTLSVerify: true
   

4. If issues persist, try HTTP instead:

   service: http://192.168.11.10:8006
   

   (Note: Less secure, but may work if HTTPS issues persist)

5. Cloudflare Access Not Showing

Symptoms:

• Direct access to Proxmox UI
• No Cloudflare Access login page

Solutions:

1. Verify DNS proxy is enabled:

   • Cloudflare Dashboard → DNS → Records
   • Ensure orange cloud is ON (proxied)

2. Verify Access application exists:

   • Zero Trust → Access → Applications
   • Check application for ml110-01.d-bis.org exists

3. Verify Access policy:

   • Check policy is configured correctly
   • Ensure policy is active

4. Check SSL/TLS settings:

   • Cloudflare Dashboard → SSL/TLS
   • Set to "Full" or "Full (strict)"

5. Clear browser cache:

   • Hard refresh: Ctrl+Shift+R (or Cmd+Shift+R on Mac)

6. MFA Not Working

Symptoms:

• Login works but MFA prompt doesn't appear
• Can't complete authentication

Solutions:

1. Verify MFA is enabled in policy:

   • Zero Trust → Access → Applications
   • Edit application → Edit policy
   • Check "Require" includes MFA

2. Check identity provider settings:

   • Verify MFA is configured in your identity provider
   • Test MFA with another application

3. Check user MFA status:

   • Zero Trust → My Team → Users
   • Verify user has MFA enabled

7. Service Fails to Start

Symptoms:

• systemctl start cloudflared-ml110 fails
• Service immediately stops after starting

Solutions:

1. Check service logs:

   journalctl -u cloudflared-ml110 -n 100
   

2. Verify credentials file:

   ls -la /etc/cloudflared/tunnel-ml110.json
   cat /etc/cloudflared/tunnel-ml110.json
   

3. Verify configuration syntax:

   cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml validate
   

4. Check file permissions:

   chmod 600 /etc/cloudflared/tunnel-ml110.json
   chmod 644 /etc/cloudflared/tunnel-ml110.yml
   

5. Test tunnel manually:

   cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml run
   

8. Multiple Tunnels Conflict

Symptoms:

• Only one tunnel works
• Other tunnels fail to start

Solutions:

1. Check for port conflicts:

   • Each tunnel uses different metrics port
   • Verify ports 9091, 9092, 9093 are not in use

2. Verify separate credentials:

   • Each tunnel needs its own credentials file
   • Verify files are separate: tunnel-ml110.json, tunnel-r630-01.json, etc.

3. Check systemd services:

   systemctl status cloudflared-*
   

4. Verify separate config files:

   • Each tunnel has its own config file
   • Tunnel IDs are different in each config

Advanced Troubleshooting

Check Tunnel Connectivity

# From VMID 102, test each Proxmox host
curl -k -I https://192.168.11.10:8006
curl -k -I https://192.168.11.11:8006
curl -k -I https://192.168.11.12:8006

Verify Tunnel Configuration

# Validate config file
cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml validate

# List tunnels
cloudflared tunnel list

# Get tunnel info
cloudflared tunnel info <tunnel-id>

Check Network Connectivity

# Test DNS resolution
dig ml110-01.d-bis.org

# Test HTTPS connectivity
curl -v https://ml110-01.d-bis.org

# Check Cloudflare IPs
nslookup ml110-01.d-bis.org

View Real-time Logs

# Follow logs for all tunnels
journalctl -u cloudflared-ml110 -f
journalctl -u cloudflared-r630-01 -f
journalctl -u cloudflared-r630-02 -f

Diagnostic Commands

Complete Health Check

# Run comprehensive health check
./scripts/check-tunnel-health.sh

# Check all services
systemctl status cloudflared-ml110 cloudflared-r630-01 cloudflared-r630-02

# Check tunnel connectivity
for tunnel in ml110 r630-01 r630-02; do
    echo "Checking $tunnel..."
    systemctl is-active cloudflared-$tunnel && echo "✓ Running" || echo "✗ Not running"
done

Reset Tunnel

If a tunnel is completely broken:

# Stop service
systemctl stop cloudflared-ml110

# Remove old credentials (backup first!)
cp /etc/cloudflared/tunnel-ml110.json /etc/cloudflared/tunnel-ml110.json.backup

# Re-authenticate (if needed)
cloudflared tunnel login

# Re-create tunnel (if needed)
cloudflared tunnel create tunnel-ml110

# Start service
systemctl start cloudflared-ml110

Getting Help

If issues persist:

1. Collect diagnostic information:

   ./scripts/check-tunnel-health.sh > /tmp/tunnel-health.txt
   journalctl -u cloudflared-* > /tmp/tunnel-logs.txt
   

2. Check Cloudflare dashboard:

   • Zero Trust → Networks → Tunnels → Status
   • Access → Logs → Recent authentication attempts

3. Review documentation:

   • Cloudflare Tunnel Docs
   • Cloudflare Access Docs

4. Common issues:

   • Invalid tunnel token
   • Network connectivity issues
   • Incorrect configuration
   • DNS propagation delays

Prevention

To avoid issues:

1. ✅ Regular health checks - Run check-tunnel-health.sh daily
2. ✅ Monitor logs - Set up log monitoring
3. ✅ Backup configs - Keep backups of config files
4. ✅ Test after changes - Test after any configuration changes
5. ✅ Document changes - Keep track of what was changed