# Proxmox VE Issues on pve and pve2

**Date:** 2025-01-20  
**Hosts:** 
- pve (192.168.11.11) - r630-01
- pve2 (192.168.11.12) - r630-02

---

## Issues Identified

### Critical Issues

1. **SSL Certificate/Key Loading Failure**
   - **Error:** `/etc/pve/local/pve-ssl.key: failed to load local private key`
   - **Impact:** pveproxy workers are crashing/exiting
   - **Affects:** Both pve and pve2
   - **Symptom:** Web interface not accessible (HTTP Status: 000)

2. **pve-cluster Service Failed (pve only)**
   - **Status:** `failed (Result: exit-code)`
   - **Error:** `pmxcfs` exited with status 255/EXCEPTION
   - **Impact:** Cluster filesystem not accessible, `/etc/pve` may not be mounted
   - **Last Failure:** 8 hours ago (restart counter at 5)

3. **pvestatd Service Failed (pve only)**
   - **Status:** `failed (Result: exit-code)`
   - **Error:** `ipcc_send_rec[2] failed: Connection refused`
   - **Impact:** Status daemon cannot communicate with cluster

### Service Status Summary

#### pve (192.168.11.11)
- ✅ **pveproxy:** Active but workers crashing
- ✅ **pvedaemon:** Active
- ❌ **pvestatd:** Failed
- ❌ **pve-cluster:** Failed

#### pve2 (192.168.11.12)
- ✅ **pveproxy:** Active but workers crashing
- ✅ **pvedaemon:** Active
- ⚠️ **pvestatd:** Status unknown (not shown in output)
- ⚠️ **pve-cluster:** Status unknown (not shown in output)

---

## Root Cause Analysis

### SSL Certificate Issue

The primary issue is that pveproxy workers cannot load the SSL private key from `/etc/pve/local/pve-ssl.key`. This could be caused by:

1. **Cluster Filesystem Not Mounted**
   - `/etc/pve` is typically a FUSE filesystem (pmxcfs)
   - If pve-cluster service is down, `/etc/pve` may not be accessible
   - SSL certificates are stored in `/etc/pve/local/`

2. **Corrupted SSL Certificates**
   - SSL key file may be corrupted or have wrong permissions
   - Certificate regeneration may be needed

3. **Cluster Configuration Issues**
   - If nodes are in a cluster, cluster quorum may be lost
   - Cluster filesystem may need to be reset

### Cluster Filesystem Issue (pve)

The pve-cluster service is failing, which means:
- The cluster filesystem (pmxcfs) is not running
- `/etc/pve` directory may not be properly mounted
- Cluster configuration may be corrupted

---

## Diagnostic Commands

### Check SSL Certificate Status
```bash
ssh root@192.168.11.11
ls -la /etc/pve/local/
cat /etc/pve/local/pve-ssl.key | head -5
```

### Check Cluster Status
```bash
ssh root@192.168.11.11
systemctl status pve-cluster
journalctl -u pve-cluster -n 50
```

### Check /etc/pve Mount
```bash
ssh root@192.168.11.11
mount | grep /etc/pve
df -h /etc/pve
```

### Check pveproxy Logs
```bash
ssh root@192.168.11.11
journalctl -u pveproxy -n 50 | grep -E "worker exit|failed to load"
```

---

## Fix Procedures

### Option 1: Regenerate SSL Certificates (Recommended First Step)

```bash
# On each host
ssh root@192.168.11.11
systemctl stop pveproxy pvedaemon
pvecm updatecerts --force
systemctl start pvedaemon pveproxy
```

### Option 2: Fix Cluster Filesystem (pve only)

If pve-cluster service is failing:

```bash
ssh root@192.168.11.11

# Stop cluster service
systemctl stop pve-cluster

# Kill any remaining pmxcfs processes
killall -9 pmxcfs

# Try to start cluster service
systemctl start pve-cluster

# Check status
systemctl status pve-cluster
```

### Option 3: Reset Cluster (If Not in Active Cluster)

If these nodes are standalone (not in a cluster), you may need to:

```bash
# Remove cluster configuration
systemctl stop pve-cluster
rm -rf /etc/pve/.members
rm -rf /var/lib/pve-cluster

# Restart services
systemctl start pve-cluster
systemctl start pvestatd pvedaemon pveproxy
```

**⚠️ WARNING:** Only do this if nodes are NOT in an active cluster!

### Option 4: Manual Certificate Regeneration

If automatic regeneration fails:

```bash
ssh root@192.168.11.11

# Stop services
systemctl stop pveproxy pvedaemon

# Backup existing certificates
cp -r /etc/pve/local /root/pve-local-backup

# Regenerate certificates
pvecm updatecerts --force

# Or manually regenerate
pvecm updatecerts --silent

# Start services
systemctl start pvedaemon pveproxy
```

---

## Automated Fix Script

A fix script has been created: `scripts/fix-proxmox-ssl-cluster.sh`

**Usage:**
```bash
# Fix both hosts
./scripts/fix-proxmox-ssl-cluster.sh both

# Fix individual host
./scripts/fix-proxmox-ssl-cluster.sh pve
./scripts/fix-proxmox-ssl-cluster.sh pve2
```

**What it does:**
1. Stops Proxmox services
2. Attempts to regenerate SSL certificates
3. Fixes cluster filesystem (if needed)
4. Restarts services in correct order
5. Verifies service status

---

## Verification Steps

After applying fixes, verify:

1. **Services are running:**
   ```bash
   ssh root@192.168.11.11
   systemctl status pve-cluster pvestatd pvedaemon pveproxy
   ```

2. **Web interface is accessible:**
   ```bash
   curl -k https://192.168.11.11:8006/
   # Should return HTML or redirect
   ```

3. **No worker exits:**
   ```bash
   ssh root@192.168.11.11
   journalctl -u pveproxy -f
   # Watch for "worker exit" messages
   ```

4. **SSL certificate is valid:**
   ```bash
   ssh root@192.168.11.11
   ls -la /etc/pve/local/pve-ssl.key
   # Should exist and be readable
   ```

---

## Next Steps

1. **Run diagnostic script:**
   ```bash
   ./scripts/diagnose-proxmox-hosts.sh both
   ```

2. **Run fix script:**
   ```bash
   ./scripts/fix-proxmox-ssl-cluster.sh both
   ```

3. **Verify fixes:**
   - Check service status
   - Test web interface
   - Monitor logs for errors

4. **If issues persist:**
   - Check if nodes are in a cluster
   - Verify network connectivity between nodes
   - Check for disk space issues
   - Review Proxmox logs in detail

---

## Related Documentation

- [R630-04 Proxmox Troubleshooting](/docs/09-troubleshooting/R630-04-PROXMOX-TROUBLESHOOTING.md) - Similar issues on r630-04
- [Proxmox Cluster Configuration](./docs/02-architecture/CLUSTER_MIGRATION_PLAN.md) - Cluster setup information

---

**Last Updated:** 2025-01-20  
**Status:** Issues Identified - Fixes Pending
