proxmox/docs/OUTSTANDING_ISSUES_RESOLUTION_GUIDE.md

Outstanding Issues Resolution Guide

Date: 2026-01-03
Status: Requires Physical/Console Access

---

Summary

Two servers require physical or console access to complete setup:

1. r630-03 (192.168.11.13) - Network connectivity issue
2. r630-04 (192.168.11.14) - SSH password authentication failure

---

Issue 1: r630-03 Network Connectivity

Symptoms

• Host does not respond to ping
• "Destination Host Unreachable" error
• Cannot access via SSH or web interface

Diagnosis

# From your machine
ping -c 3 192.168.11.13
# Result: 100% packet loss, Destination Host Unreachable

Possible Causes

1. Server powered off

   • Check physical power status
   • Check power LED indicators
   • Verify power cable connections

2. Network cable disconnected

   • Check physical network cable
   • Verify cable is connected to correct switch port
   • Test with known-good cable

3. Network switch port issue

   • Check switch port status
   • Verify port is enabled
   • Check for port errors on switch

4. IP configuration issue

   • Server may have different IP
   • Network configuration may be incorrect
   • DHCP may have assigned different IP

5. Network routing issue

   • Check router/switch configuration
   • Verify VLAN configuration
   • Check firewall rules

Resolution Steps

Step 1: Physical Inspection

1. Check Power Status

   • Verify server is powered on
   • Check power LED indicators
   • Listen for fan noise

2. Check Network Cable

   • Verify cable is connected
   • Check for physical damage
   • Try different cable if available

3. Check Network Switch

   • Verify switch port is active
   • Check for port errors
   • Verify VLAN configuration

Step 2: Access via iDRAC (Dell R630)

If iDRAC is configured, access via:

• iDRAC IP: Check network configuration or use default
• Default credentials: root/calvin (if not changed)
• Access method: Web browser to iDRAC IP

Once in iDRAC:

1. Check server power status
2. Access console (KVM)
3. Check network configuration
4. Verify IP address assignment

Step 3: Console Access

If physical access is available:

1. Connect keyboard/monitor
2. Boot server if needed
3. Check network configuration:

   ip addr show
   cat /etc/network/interfaces
   

4. Verify IP address: 192.168.11.13
5. Test connectivity: ping 192.168.11.1

Step 4: Network Configuration Fix

If IP is incorrect or network is misconfigured:

# Edit network configuration
nano /etc/network/interfaces

# Ensure configuration matches:
# auto vmbr0
# iface vmbr0 inet static
#     address 192.168.11.13/24
#     gateway 192.168.11.1
#     bridge-ports <physical-interface>
#     bridge-vlan-aware yes

# Restart networking
systemctl restart networking

Step 5: Verification

Once accessible:

# Run verification script
./scripts/verify-r630-03-cluster-storage.sh

# Check cluster membership
ssh root@192.168.11.13 "pvecm status"

# Verify storage
ssh root@192.168.11.13 "pvesm status"

---

Issue 2: r630-04 SSH Password Authentication

Symptoms

• Host is reachable (ping works)
• SSH connection attempts fail with "Permission denied"
• All known passwords fail
• Web interface not accessible (port 8006)

Diagnosis

# Host is reachable
ping -c 2 192.168.11.14
# Result: Success

# SSH fails
ssh root@192.168.11.14
# Result: Permission denied (password authentication)

Tried Passwords

• ❌ L@kers2010
• ❌ password
• ❌ L@kers2010!
• ❌ L@kers2010@
• ❌ L@kers2010#
• ❌ All variations tested

Resolution Steps

Step 1: Console Access (Required)

Physical or iDRAC console access is required to reset password.

Option A: Physical Console

1. Connect keyboard/monitor to r630-04
2. Boot server if needed
3. Login with current password (if known)
4. Or boot into single-user mode

Option B: iDRAC Console

1. Access iDRAC web interface
2. Use Remote Console (KVM)
3. Access server console remotely

Step 2: Reset Root Password

Method 1: If you can login

# Login to console
# Then run:
passwd root
# Enter new password: L@kers2010
# Confirm password

Method 2: Single-User Mode (if password unknown)

1. Boot server
2. At GRUB menu, press 'e' to edit
3. Find line starting with "linux"
4. Add init=/bin/bash or single to end of line
5. Press Ctrl+X to boot
6. Mount filesystem: mount -o remount,rw /
7. Reset password: passwd root
8. Reboot: reboot -f

Step 3: Fix Proxmox Services

Once password is reset and SSH works:

# Run the complete fix script
./scripts/fix-r630-04-complete.sh L@kers2010

# Or manually:
ssh root@192.168.11.14

# Fix /etc/hosts
echo "192.168.11.14    r630-04 r630-04.sankofa.nexus" >> /etc/hosts

# Restart Proxmox services
systemctl restart pve-cluster
systemctl restart pvestatd pvedaemon pveproxy

# Verify
systemctl status pveproxy
ss -tlnp | grep 8006
curl -k https://localhost:8006

Step 4: Join Cluster (if not already)

If r630-04 is not in cluster:

# From ml110, get join information
ssh root@192.168.11.10 "pvecm create join-info"

# On r630-04, join cluster
ssh root@192.168.11.14
pvecm add <join-info-from-ml110>

Step 5: Verify Storage

# Check storage status
ssh root@192.168.11.14 "pvesm status"

# Activate storage if needed
# (Similar to r630-01 and r630-02)

---

Alternative Access Methods

iDRAC Access (Dell R630)

Both r630-03 and r630-04 are Dell R630 servers with iDRAC.

Default iDRAC Configuration:

• IP Range: Usually 192.168.0.x or DHCP assigned
• Default User: root
• Default Password: calvin (if not changed)

Finding iDRAC IP:

1. Check network documentation
2. Check DHCP leases on router
3. Check Omada controller for connected devices
4. Use default IP range: 192.168.0.100-200

Accessing iDRAC:

1. Open web browser
2. Navigate to iDRAC IP
3. Login with credentials
4. Use Remote Console (KVM) for full access

Network-Based Solutions

Option 1: Check from Working Nodes

# From ml110, r630-01, or r630-02
ssh root@192.168.11.10
ping 192.168.11.13  # r630-03
ping 192.168.11.14  # r630-04

# If reachable from cluster network, try SSH
ssh root@192.168.11.14

Option 2: Check Network Switch

• Access switch management interface
• Check port status for servers
• Verify VLAN configuration
• Check for port errors

Option 3: Check Router/Firewall

• Verify firewall rules
• Check routing tables
• Verify NAT configuration

---

Verification Checklist

r630-03 Checklist

• Server is powered on
• Network cable connected
• Network switch port active
• IP address correct (192.168.11.13)
• Ping works from cluster network
• SSH access works
• Proxmox services running
• Web interface accessible (port 8006)
• Cluster membership verified
• Storage configuration verified

r630-04 Checklist

• Console access obtained
• Root password reset
• SSH access works
• /etc/hosts configured correctly
• Proxmox services running
• Web interface accessible (port 8006)
• Cluster membership verified
• Storage configuration verified

---

Scripts Available

All scripts are in /home/intlc/projects/proxmox/scripts/:

1. verify-r630-03-cluster-storage.sh

   • Run once r630-03 is accessible
   • Verifies cluster membership and storage

2. fix-r630-04-complete.sh

   • Run once r630-04 password is reset
   • Complete fix for all r630-04 issues

3. verify-all-nodes-complete.sh

   • Run after both issues are resolved
   • Comprehensive verification of all nodes

---

Next Steps

Immediate Actions

1. Physical Access Required

   • Access r630-03 and r630-04 via console/iDRAC
   • Resolve network/password issues
   • Run verification scripts

2. Documentation Update

   • Update passwords in documentation
   • Update status reports
   • Document any configuration changes

After Resolution

1. Run Verification

   ./scripts/verify-all-nodes-complete.sh
   

2. Update Status Report

   • Mark issues as resolved
   • Update cluster membership
   • Update storage status

3. Cluster Health Check

   • Verify all 5 nodes in cluster
   • Check quorum status
   • Verify storage on all nodes

---

Support Resources

• Console Access Guide: R630-04-CONSOLE-ACCESS-GUIDE.md
• Troubleshooting Guide: R630-04-PROXMOX-TROUBLESHOOTING.md
• Status Report: docs/PROXMOX_CLUSTER_STORAGE_STATUS_REPORT.md
• Hardware Inventory: config/physical-hardware-inventory.md

---

Last Updated: 2026-01-03
Status: Awaiting Physical/Console Access