proxmox/reports/r630-02-startup-failures-review-summary.md

R630-02 Container Startup Failures - Review Summary

Date: January 19, 2026
Reviewer: AI Assistant
Status: ✅ ANALYSIS COMPLETE - TOOLS CREATED

---

Review Summary

I've completed a comprehensive review of the container startup failures on r630-02. The analysis identified 33 failed containers across three distinct failure categories.

---

Failure Categories

1. Logical Volume Errors (8 containers)

Error: no such logical volume pve/vm-XXXX-disk-X

Affected Containers:

• CT 3000, 3001, 3002, 3003
• CT 3500, 3501
• CT 6000, 6400

Root Cause: Storage volumes are missing or containers reference incorrect storage pools.

Likely Causes:

• Volumes deleted during storage migration
• Containers migrated but configs not updated
• Storage pool recreated/reset
• Wrong storage pool reference (e.g., thin1 vs thin1-r630-02)

2. Startup Failures (24 containers)

Error: startup for container 'XXXX' failed

Affected Containers:

• CT 5200
• CT 10000-10092 (multiple)
• CT 10100-10151 (multiple)
• CT 10200-10230 (multiple)

Root Cause: Multiple potential causes requiring individual diagnosis.

Possible Causes:

• Missing configuration files
• Storage corruption or misconfiguration
• Network configuration issues
• Resource constraints (memory/CPU)
• Container filesystem corruption
• Missing dependencies

3. Lock Error (1 container)

Error: CT is locked (create)

Affected Container:

• CT 10232

Root Cause: Container stuck in creation state, likely from interrupted operation.

---

Created Tools

1. Analysis Document

File: reports/r630-02-container-startup-failures-analysis.md

Contents:

• Detailed breakdown of all failures
• Root cause analysis for each category
• Diagnostic steps and commands
• Resolution options
• Recommended actions

2. Diagnostic Script

File: scripts/diagnose-r630-02-startup-failures.sh

Features:

• Checks container status and configuration
• Verifies logical volume existence
• Identifies storage configuration issues
• Captures detailed startup errors
• Checks for lock files
• Provides system resource information
• Generates comprehensive diagnostic report

Usage:

./scripts/diagnose-r630-02-startup-failures.sh

3. Fix Script

File: scripts/fix-r630-02-startup-failures.sh

Features:

• Automatically fixes logical volume issues where possible
• Updates storage pool references
• Clears lock files
• Attempts container starts after fixes
• Supports dry-run mode
• Provides detailed fix summary

Usage:

# Dry run (no changes)
./scripts/fix-r630-02-startup-failures.sh --dry-run

# Apply fixes
./scripts/fix-r630-02-startup-failures.sh

---

Recommended Next Steps

Step 1: Run Diagnostic Script

cd /home/intlc/projects/proxmox
./scripts/diagnose-r630-02-startup-failures.sh

This will:

• Identify root causes for each failure
• Check storage status and configuration
• Verify logical volume existence
• Capture detailed error messages
• Provide system resource information

Step 2: Review Diagnostic Output

Review the diagnostic output to understand:

• Which containers have missing logical volumes
• Which containers have configuration issues
• Which containers have other startup problems
• System resource availability

Step 3: Run Fix Script (Dry Run First)

# First, run in dry-run mode to see what would be fixed
./scripts/fix-r630-02-startup-failures.sh --dry-run

# Review the dry-run output, then apply fixes
./scripts/fix-r630-02-startup-failures.sh

Step 4: Manual Resolution

For containers that the fix script cannot automatically resolve:

• Review diagnostic output for specific error messages
• Check if volumes need to be recreated
• Verify container configurations
• Recreate containers if configs are missing
• Check for resource constraints

Step 5: Verification

After fixes are applied:

# Check container status
ssh root@192.168.11.12 "pct list | grep -E '3000|3001|3002|3003|3500|3501|5200|6000|6400|10000|10001|10020|10030|10040|10050|10060|10070|10080|10090|10091|10092|10100|10101|10120|10130|10150|10151|10200|10201|10202|10210|10230|10232'"

---

Key Findings

1. Storage Issues: 8 containers have missing logical volumes, likely due to storage migration or pool recreation.

2. Configuration Issues: 24 containers fail to start, many likely due to missing or corrupted configuration files.

3. Lock Issues: 1 container is stuck in creation state and needs lock clearing.

4. Pattern Recognition: Many failures appear to be from containers that were migrated or had storage reorganized, but configurations weren't properly updated.

---

Related Files

• Analysis Document: reports/r630-02-container-startup-failures-analysis.md
• Diagnostic Script: scripts/diagnose-r630-02-startup-failures.sh
• Fix Script: scripts/fix-r630-02-startup-failures.sh
• Previous Logs Review: reports/r630-02-logs-review.txt

---

Notes

• The diagnostic script provides detailed information but may take a few minutes to run for all containers.
• The fix script attempts automated resolution but some issues may require manual intervention.
• Always run the fix script in dry-run mode first to review proposed changes.
• Some containers may need to be recreated if their configurations are missing or corrupted.
• Storage volumes may need to be recreated if they were lost during migration.

---

Conclusion

The review is complete with comprehensive analysis and automated tools created. The next step is to run the diagnostic script to gather detailed information about each failure, then use the fix script to resolve issues where possible.