proxmox/docs/archive/STORAGE_NETWORK_VERIFICATION.md

# Storage and Network Configuration Verification

**Last Updated**: 2025-01-11  
**Purpose**: Guide for verifying storage and network configuration for optimal deployment performance

---

## Overview

Proper storage and network configuration significantly impacts deployment performance. This guide provides verification steps and recommendations for optimal deployment speed.

---

## Storage Configuration

### Why Storage Matters

- **Container Creation**: Faster I/O = faster container creation
- **OS Template Installation**: Local storage = ~15-30 minutes saved
- **Package Installation**: Local storage reduces I/O wait times
- **Overall Impact**: Local storage can save 15-30 minutes total

### Storage Types

| Storage Type | Performance | Recommendation |
|--------------|-------------|----------------|
| **local-lvm** | Excellent | ✅ Recommended (local SSD/HDD) |
| **local** (dir) | Good | ✅ Recommended for small deployments |
| **local-zfs** | Excellent | ✅ Recommended (ZFS pools) |
| **nfs** | Variable | ⚠️ Depends on network speed |
| **ceph** | Good | ⚠️ Network latency may impact |
| **glusterfs** | Variable | ⚠️ Network latency may impact |

### Verification Steps

#### 1. Run Storage Verification Script

```bash
# Verify storage configuration
./scripts/validation/verify-storage-config.sh
```

This script will:
- Check if configured storage exists
- Verify storage type (local vs. network)
- Check available capacity
- Provide recommendations

#### 2. Manual Verification

```bash
# List available storage
pvesm status

# Check storage details
pvesm status | grep local

# Verify configured storage exists
grep PROXMOX_STORAGE config/proxmox.conf
```

#### 3. Storage Configuration

Ensure `config/proxmox.conf` has:

```bash
# Use local storage for best performance
PROXMOX_STORAGE="local-lvm"  # Or "local" or "local-zfs"
```

### Recommendations

1. **Use Local Storage**: Prefer `local-lvm`, `local`, or `local-zfs`
2. **SSD Storage**: Use SSD-based storage for fastest performance
3. **Sufficient Capacity**: Ensure ~100GB per container (67 containers ≈ 6.7TB)
4. **Monitor I/O**: Watch storage I/O during deployment

---

## Network Configuration

### Why Network Matters

- **Package Downloads**: ~500 MB - 2 GB per container
- **OS Template Downloads**: ~200-500 MB (one-time)
- **Configuration Transfers**: Minimal but frequent
- **Overall Impact**: Good network can save 30-60 minutes total

### Network Requirements

| Connection Speed | Performance | Recommendation |
|------------------|-------------|----------------|
| **1 Gbps (Gigabit)** | Excellent | ✅ Recommended |
| **100 Mbps (Fast Ethernet)** | Good | ⚠️ Acceptable, may be slower |
| **10 Mbps** | Slow | ❌ Not recommended for large deployments |
| **Wireless** | Variable | ⚠️ Use wired if possible |

### Verification Steps

#### 1. Run Network Verification Script

```bash
# Verify network configuration
./scripts/validation/verify-network-config.sh
```

This script will:
- Check network connectivity to Proxmox host
- Test latency
- Verify network interface speeds
- Check DNS resolution
- Test internet connectivity and download speed
- Verify Proxmox bridge configuration

#### 2. Manual Verification

```bash
# Check network interfaces
ip link show

# Check bridge status
ip link show vmbr0

# Test connectivity
ping -c 3 <proxmox-host>

# Test download speed (rough estimate)
curl -o /dev/null -w "Speed: %{speed_download} bytes/s\n" https://github.com
```

#### 3. Network Configuration

Ensure `config/proxmox.conf` has:

```bash
# Configure bridge (usually vmbr0)
PROXMOX_BRIDGE="vmbr0"

# Configure Proxmox host (if remote)
PROXMOX_HOST="192.168.11.10"  # Or hostname
```

### Network Optimization Recommendations

1. **Use Wired Connection**: Avoid wireless for deployment
2. **Gigabit Ethernet**: Use 1 Gbps or faster connection
3. **Low Latency**: Ensure <50ms latency to Proxmox host
4. **Local Package Mirrors**: Consider apt mirrors for faster package downloads
5. **Pre-cache Templates**: Run `pre-cache-os-template.sh` before deployment
6. **Monitor Bandwidth**: Watch network usage during deployment

### Expected Network Usage

For complete deployment (67 containers):

| Operation | Data Transfer | Frequency |
|-----------|---------------|-----------|
| OS Template | ~200-500 MB | One-time (if not cached) |
| Package Downloads | ~500 MB - 2 GB per container | Per container |
| Configuration Files | ~1-10 MB per container | Per container |
| **Total** | **~35-135 GB** | Complete deployment |

**Breakdown**:
- Besu nodes (12): ~6-24 GB
- CCIP nodes (41-43): ~20-86 GB
- Other services (14): ~7-28 GB

---

## Pre-Deployment Checklist

### Storage Checklist

- [ ] Run `verify-storage-config.sh`
- [ ] Verify storage is local (not network-based)
- [ ] Check sufficient capacity (6.7TB+ recommended)
- [ ] Confirm storage is accessible
- [ ] Verify `PROXMOX_STORAGE` in config matches available storage

### Network Checklist

- [ ] Run `verify-network-config.sh`
- [ ] Verify network interface is Gigabit (1 Gbps) or faster
- [ ] Test latency to Proxmox host (<50ms recommended)
- [ ] Verify DNS resolution working
- [ ] Test internet connectivity
- [ ] Check download speed (>100 Mbps recommended)
- [ ] Verify Proxmox bridge is UP
- [ ] Confirm `PROXMOX_BRIDGE` in config matches actual bridge

### Optimization Steps

1. **Before Deployment**:
   ```bash
   # Pre-cache OS template (saves 5-10 minutes)
   ./scripts/deployment/pre-cache-os-template.sh
   
   # Verify storage
   ./scripts/validation/verify-storage-config.sh
   
   # Verify network
   ./scripts/validation/verify-network-config.sh
   ```

2. **During Deployment**:
   - Monitor storage I/O: `iostat -x 1`
   - Monitor network usage: `iftop` or `nload`
   - Watch resource usage: `htop`

3. **After Deployment**:
   - Review deployment logs for bottlenecks
   - Adjust parallel execution limits if needed
   - Consider optimizations for future deployments

---

## Troubleshooting

### Storage Issues

**Problem**: Slow container creation
- **Solution**: Use local storage instead of network storage
- **Check**: Run `pvesm status` to verify storage type

**Problem**: Insufficient storage space
- **Solution**: Free up space or expand storage
- **Check**: `pvesm status` shows available capacity

### Network Issues

**Problem**: Slow package downloads
- **Solution**: Use local package mirrors or upgrade network
- **Check**: Run network verification script

**Problem**: High latency
- **Solution**: Use wired connection, optimize network path
- **Check**: `ping <proxmox-host>` shows latency

**Problem**: DNS resolution failures
- **Solution**: Configure proper DNS servers
- **Check**: `nslookup github.com` should resolve

---

## Performance Impact

### Storage Impact

| Storage Type | Container Creation | OS Installation | Total Impact |
|--------------|-------------------|-----------------|--------------|
| Local SSD | Fast | Fast | Optimal |
| Local HDD | Good | Good | Good |
| Network (1 Gbps) | Moderate | Moderate | Acceptable |
| Network (<100 Mbps) | Slow | Slow | Significant delay |

**Time Savings**: Local storage saves ~15-30 minutes vs. network storage

### Network Impact

| Connection Speed | Package Downloads | Template Download | Total Impact |
|------------------|-------------------|-------------------|--------------|
| 1 Gbps | Fast | Fast | Optimal |
| 100 Mbps | Moderate | Moderate | Acceptable |
| 10 Mbps | Slow | Slow | Significant delay |

**Time Savings**: 1 Gbps saves ~30-60 minutes vs. 100 Mbps for complete deployment

---

## References

- **Deployment Time Estimate**: `docs/DEPLOYMENT_TIME_ESTIMATE.md`
- **Optimization Recommendations**: `docs/DEPLOYMENT_OPTIMIZATION_RECOMMENDATIONS.md`
- **Parallel Execution Limits**: `docs/PARALLEL_EXECUTION_LIMITS.md`
- **Proxmox Storage Documentation**: https://pve.proxmox.com/wiki/Storage

---

## Quick Verification

Run both verification scripts before deployment:

```bash
# Verify storage configuration
./scripts/validation/verify-storage-config.sh

# Verify network configuration
./scripts/validation/verify-network-config.sh
```

Both scripts will provide recommendations for optimal deployment performance.