proxmox/reports/r630-02-startup-failures-final-analysis.md

R630-02 Container Startup Failures - Final Analysis

Date: January 19, 2026
Status: ⚠️ STORAGE VOLUMES MISSING ON BOTH NODES

---

Complete Picture

Initial Problem

• 33 containers failed to start on r630-02
• Error messages indicated missing logical volumes and startup failures

Root Cause Discovery

1. Containers don't exist on r630-02

   • All 33 containers have missing configuration files
   • No logical volumes exist for these containers
   • Containers were migrated or never created on r630-02

2. Containers exist on pve2 but can't start

   • All 33 containers are present on pve2 (192.168.11.11)
   • Containers are in "stopped" state
   • Storage volumes are missing on pve2 as well
   • Error: no such logical volume pve/vm-XXXX-disk-X

Conclusion

The containers exist on pve2, but their storage volumes are missing. This means:

• Container configurations exist
• Logical volumes (storage) were deleted or never created
• Containers cannot start without their storage volumes

---

Storage Status

On r630-02:

• Storage pools: thin1-r630-02 (active), thin1 (inactive), data (inactive)
• No logical volumes for failed containers
• Containers don't exist

On pve2:

• Containers exist but storage volumes missing
• Need to check available storage pools
• Need to recreate or migrate storage volumes

---

Resolution Options

Option 1: Recreate Storage Volumes on pve2 (If Data Not Critical)

If the data in these containers is not critical or can be restored:

1. Delete and recreate containers:

   # For each container
   ssh root@192.168.11.11 "pct destroy <VMID>"
   # Then recreate with proper storage
   

2. Or recreate just the storage volumes:

   # Check container storage config
   ssh root@192.168.11.11 "pct config <VMID> | grep rootfs"
   
   # Recreate volume on correct storage pool
   # (requires knowing correct storage pool and size)
   

Option 2: Restore from Backup

If backups exist:

# Restore containers from backup
vzdump --restore <backup_file> <VMID>

Option 3: Migrate to Different Storage

If storage pools are misconfigured:

# Migrate to working storage pool
pct migrate <VMID> <target_node> --storage <working_storage> --restart

Option 4: Check for Volumes on Other Storage Pools

Volumes might exist but on different storage pools:

# Check all storage pools
ssh root@192.168.11.11 "lvs | grep vm-3000"
ssh root@192.168.11.11 "pvesm status"

---

Next Steps

1. Check storage configuration on pve2:

   • Identify available storage pools
   • Check if volumes exist on different pools
   • Verify container storage requirements

2. Determine data criticality:

   • Are these containers critical?
   • Can data be restored from backup?
   • Can containers be recreated?

3. Execute resolution:

   • Recreate volumes if data not critical
   • Restore from backup if available
   • Migrate to working storage if misconfigured

---

Container List (All on pve2, Storage Missing)

VMID	Hostname	Status	Storage Issue
3000-3003	ml110 (x4)	stopped	Missing LV
3500-3501	oracle/ccip	stopped	Missing LV
5200	cacti-1	stopped	Missing LV
6000	fabric-1	stopped	Missing LV
6400	indy-1	stopped	Missing LV
10000-10092	order-* (12)	stopped	Missing LV
10100-10151	dbis-* (6)	stopped	Missing LV
10200-10230	order-* (5)	stopped	Missing LV
10232	CT10232	stopped	Missing LV + Lock

Total: 33 containers with missing storage volumes

---

Recommendation

Immediate Action: Investigate storage configuration on pve2 to determine:

1. Which storage pools are available
2. Whether volumes exist on different pools
3. Whether containers can be recreated or need restoration

Long-term: Implement storage monitoring and backup procedures to prevent this issue.