proxmox/docs/archive/VERIFICATION_SCRIPTS_GUIDE.md

# Verification Scripts Guide

**Last Updated**: 2025-01-11  
**Purpose**: Guide for using verification scripts before deployment

---

## Overview

Verification scripts help ensure your environment is properly configured for optimal deployment performance. Run these scripts before starting deployment to identify and fix issues early.

---

## Available Verification Scripts

### 1. Prerequisites Check

**Script**: `scripts/validation/check-prerequisites.sh`

**Purpose**: Validates that all required configuration files, keys, and directories exist in the source project.

**Usage**:
```bash
./scripts/validation/check-prerequisites.sh /path/to/smom-dbis-138
```

**What it checks**:
- Source project directory exists
- Required directories (config/, keys/validators/)
- Required files (genesis.json, permissions-nodes.toml, etc.)
- Optional files (config files for validators, sentries, RPC)
- Node-specific files (if config/nodes/ structure exists)
- CCIP configuration files
- Other service configuration files

**Output**:
- ✓ Green checkmarks for found items
- ✗ Red X for missing required items
- ⚠ Yellow warnings for optional missing items
- Summary with error and warning counts
- Recommendations for next steps

---

### 2. Storage Configuration Verification

**Script**: `scripts/validation/verify-storage-config.sh`

**Purpose**: Verifies storage configuration for optimal deployment performance.

**Usage**:
```bash
# Must be run on Proxmox host as root
sudo ./scripts/validation/verify-storage-config.sh
```

**What it checks**:
- Configured storage exists (`PROXMOX_STORAGE` from config)
- Storage type (local vs. network-based)
- Storage status and capacity
- Available space for deployment

**Output**:
- Storage type analysis
- Performance recommendations
- Local vs. network storage impact
- Estimated storage requirements

**Recommendations**:
- Use local storage (local-lvm, local, local-zfs) for best performance
- Ensures ~100GB per container available
- Can save 15-30 minutes deployment time

---

### 3. Network Configuration Verification

**Script**: `scripts/validation/verify-network-config.sh`

**Purpose**: Verifies network configuration and connectivity for optimal deployment performance.

**Usage**:
```bash
# Can be run from any machine with network access
./scripts/validation/verify-network-config.sh
```

**What it checks**:
- Network connectivity to Proxmox host
- Latency measurements
- Network interface speeds
- DNS resolution
- Internet connectivity
- Download speed estimation
- Proxmox bridge configuration

**Output**:
- Network interface status and speeds
- Latency measurements
- Download speed estimates
- Connectivity status
- Optimization recommendations

**Recommendations**:
- Use Gigabit Ethernet (1 Gbps) or faster
- Ensure low latency (<50ms)
- Use wired connection instead of wireless
- Can save 30-60 minutes deployment time

---

## Pre-Deployment Workflow

### Step 1: Verify Prerequisites

```bash
# Check all configuration files are ready
./scripts/validation/check-prerequisites.sh /home/intlc/projects/smom-dbis-138
```

**Expected Output**: All required files should show ✓ (green checkmarks)

**If errors occur**:
- Fix missing files
- Verify source project path is correct
- Check file permissions

---

### Step 2: Verify Storage (on Proxmox host)

```bash
# Verify storage configuration
sudo ./scripts/validation/verify-storage-config.sh
```

**Expected Output**: 
- ✓ Configured storage exists
- ✓ Storage is local (recommended)
- ✓ Sufficient capacity available

**If issues occur**:
- Update `PROXMOX_STORAGE` in `config/proxmox.conf`
- Ensure storage is local (not network-based)
- Free up space if needed

---

### Step 3: Verify Network

```bash
# Verify network configuration
./scripts/validation/verify-network-config.sh
```

**Expected Output**:
- ✓ Proxmox host is reachable
- ✓ Low latency (<50ms)
- ✓ Network interface is Gigabit or faster
- ✓ DNS resolution working
- ✓ Internet connectivity working
- ✓ Download speed >100 Mbps

**If issues occur**:
- Use wired connection instead of wireless
- Upgrade network connection if possible
- Check DNS configuration
- Verify Proxmox bridge is UP

---

### Step 4: Pre-cache OS Template (Optional but Recommended)

```bash
# Pre-cache OS template (saves 5-10 minutes)
sudo ./scripts/deployment/pre-cache-os-template.sh
```

**Expected Output**:
- ✓ Template downloaded or already cached
- Template details displayed

---

### Step 5: Start Deployment

Once all verifications pass, you're ready to deploy:

```bash
# Option 1: Phased deployment (recommended)
sudo ./scripts/deployment/deploy-phased.sh \
    --source-project /home/intlc/projects/smom-dbis-138

# Option 2: Full deployment
sudo ./scripts/deployment/deploy-validated-set.sh \
    --source-project /home/intlc/projects/smom-dbis-138
```

---

## Quick Verification Script

Create a combined verification script for convenience:

```bash
#!/usr/bin/env bash
# Quick verification - run all checks

echo "=== Running All Verification Checks ==="
echo ""

echo "1. Prerequisites Check..."
./scripts/validation/check-prerequisites.sh "$1" || exit 1

echo ""
echo "2. Storage Configuration (requires root on Proxmox host)..."
if [[ $EUID -eq 0 ]] && command_exists pvesm; then
    ./scripts/validation/verify-storage-config.sh || exit 1
else
    echo "  Skipping (not running as root on Proxmox host)"
fi

echo ""
echo "3. Network Configuration..."
./scripts/validation/verify-network-config.sh || exit 1

echo ""
echo "✅ All verification checks completed!"
```

Save as `scripts/validation/verify-all.sh` and run:
```bash
chmod +x scripts/validation/verify-all.sh
./scripts/validation/verify-all.sh /path/to/smom-dbis-138
```

---

## Expected Results

### Prerequisites Check

✅ **All Pass**: Ready for deployment
- No errors (red X marks)
- Warnings are acceptable (yellow ⚠)

❌ **Failures**: Fix before deployment
- Missing required files
- Missing required directories
- Configuration errors

### Storage Verification

✅ **Optimal**: Local storage configured
- Storage type: local-lvm, local, or local-zfs
- Sufficient capacity available
- Storage is UP and accessible

⚠️ **Acceptable**: Network storage configured
- Storage accessible
- May be slower than local storage
- Consider using local storage if possible

### Network Verification

✅ **Optimal**: Fast network configured
- Connection speed: 1 Gbps or faster
- Latency: <50ms
- Download speed: >100 Mbps

⚠️ **Acceptable**: Moderate network
- Connection speed: 100 Mbps
- Latency: <100ms
- Download speed: >10 Mbps
- Deployment will be slower but functional

---

## Troubleshooting

### Prerequisites Check Issues

**Problem**: Missing required files
- **Solution**: Ensure source project is correct and files are present
- **Check**: Run `ls -la /path/to/smom-dbis-138/config/`

**Problem**: Missing node-specific files
- **Solution**: Verify config/nodes/ structure or use flat config structure
- **Check**: Run `find /path/to/smom-dbis-138/config/nodes -type f`

### Storage Verification Issues

**Problem**: Configured storage not found
- **Solution**: Update `PROXMOX_STORAGE` in config/proxmox.conf
- **Check**: Run `pvesm status` to see available storage

**Problem**: Network storage detected
- **Solution**: Consider using local storage for better performance
- **Impact**: May add 15-30 minutes to deployment time

### Network Verification Issues

**Problem**: Cannot reach Proxmox host
- **Solution**: Check network connectivity, firewall rules
- **Check**: Run `ping <proxmox-host>`

**Problem**: Slow download speed
- **Solution**: Upgrade network connection or use local package mirrors
- **Impact**: May add 30-60 minutes to deployment time

---

## References

- **Storage and Network Guide**: `docs/STORAGE_NETWORK_VERIFICATION.md`
- **Deployment Time Estimate**: `docs/DEPLOYMENT_TIME_ESTIMATE.md`
- **Optimization Recommendations**: `docs/DEPLOYMENT_OPTIMIZATION_RECOMMENDATIONS.md`

---

## Summary

Always run verification scripts before deployment to:
1. ✅ Ensure all files are ready (prerequisites check)
2. ✅ Verify storage is optimal (storage verification)
3. ✅ Verify network is sufficient (network verification)

This helps prevent deployment failures and ensures optimal performance.