- **RAM and drive specifications** for each R630 to support Ceph, VMs/containers, and growth.
**Scope:** All 13 R630s as Proxmox cluster nodes; optional separate management node (e.g. ml110) or integration of management on a subset of R630s. Design assumes **hyper-converged** (Proxmox + Ceph on same nodes) for shared storage and true HA.
**Extended inventory:** The same site includes 3× Dell R750 servers, 2× Dell Precision 7920 workstations, and 2× UniFi Dream Machine Pro (gateways). See [HARDWARE_INVENTORY_MASTER.md](../11-references/HARDWARE_INVENTORY_MASTER.md), [13_NODE_NETWORK_AND_CABLING_CHECKLIST.md](../11-references/13_NODE_NETWORK_AND_CABLING_CHECKLIST.md), and [13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md](../11-references/13_NODE_AND_ASSETS_BRING_ONLINE_CHECKLIST.md) for network topology, cabling, and bring-online order.
| **Quorum** | Majority = 7. With 13 nodes, up to 6 can be down and cluster still has quorum. |
| **Fencing** | Required for HA: failed node must be fenced (power off/reboot) so Ceph and HA manager can safely restart resources elsewhere. |
| **Qdevice** | Optional: add a quorum device (e.g. small VM or appliance) so quorum survives more node failures; not required with 13 nodes but improves resilience. |
### 2.2 Recommended node layout
| Role | Node count | Purpose |
|------|------------|---------|
| **Proxmox + Ceph MON/MGR/OSD** | 13 | Every R630 runs Proxmox and participates in Ceph (MON, MGR, OSD) for shared storage. |
| **Ceph OSD** | 13 | Each node contributes disk as Ceph OSD; replication (e.g. size=3, min_size=2) across nodes. |
| **Proxmox HA** | 13 | HA manager can restart VMs/containers on any node; VM disks on Ceph. |
| **Optional dedicated** | 0 | No dedicated “monitor-only” nodes required; MON/MGR run on all or a subset (e.g. 3–5 MONs). |
### 2.3 Network and addressing
- **Management:** One subnet (e.g. 192.168.11.0/24) for Proxmox API, SSH, Ceph public/cluster.
- **Ceph:** Separate VLAN or subnet for Ceph cluster network (recommended for DoD: isolate storage traffic).
- **VLANs:** Same VLAN-aware bridge (e.g. vmbr0) on all nodes so VMs/containers keep IPs when failed over.
- **IP plan for 13 R630s:** Reserve 13 consecutive IPs (e.g. 192.168.11.11–192.168.11.23 for r630-01 … r630-13). Document in `config/ip-addresses.conf` and DNS.
| **Minimum** | 128 GB | Ceph OSD + a few VMs; acceptable for lab or light production. |
| **Recommended** | 256 GB | Production: Ceph (OSD + MON/MGR) + many VMs/containers; headroom for failover and recovery. |
| **High** | 384–512 GB | Heavy workloads, large Ceph OSD count per node, or when consolidating from existing 503 GB nodes. |
**Ceph guidance:** Proxmox/Ceph recommend **≥ 8 GiB per OSD** for OSD memory. With 6–8 OSDs per node (see storage), **48–64 GiB** for Ceph plus Proxmox and guest overhead → **128 GB minimum**, **256 GB recommended**.
**DoD/MIL note:** Prefer **256 GB per node** for 13-node production so that (1) multiple node failures still leave enough capacity for HA migrations and (2) Ceph recovery and rebalancing do not cause OOM or instability.
### 3.3 RAM placement (if mixing sizes)
If not all nodes have the same RAM:
- Put **largest RAM** in nodes that run the most VMs or Ceph MON/MGR.
- Ensure **at least 128 GB** on every node that runs Ceph OSDs.
- Document exact DIMM layout per node (slot, size, speed) for change control and troubleshooting.
---
## 4. Drive Specifications — R630
### 4.1 R630 drive options (reference)
- **Internal bays:** Typically 8 × 2.5" SATA/SAS (or 10-bay with optional kit); some configs support NVMe (e.g. 4 × NVMe via PCIe).
- **Boot:** 2 drives in mirror (ZFS mirror or hardware RAID1) for Proxmox OS — **redundant, DoD-compliant**.
- **Data:** Remaining drives for Ceph OSD and/or local LVM (if hybrid).
### 4.2 Recommended drive layout per R630 (full Ceph)
**Cluster total:** 13 × 6 = 78 OSDs; with replication 3×, usable capacity ≈ (78 × 0.9 TB) / 3 ≈ **~23 TB** (before bluestore overhead; adjust for actual sizes).
### 4.3 DoD/MIL storage requirements
- **Encryption:** At-rest encryption for sensitive data. Options: Ceph encryption (e.g. dm-crypt for OSD), or encrypted VMs (LUKS inside guest). Document which layers are encrypted and key management.
- **Integrity:** ZFS for boot (checksum, scrub). Ceph provides replication and recovery; use **bluestore** with checksums.
- **Sanitization:** Follow DoD 5220.22-M or NIST SP 800-88 for decommissioning/destruction of drives.
- **Spare:** Maintain spare drives and document replacement and wipe procedures.
- **Target:** Size Ceph pool so that **used + 2 years growth** stays < 75% of usable. Example: 15–20 TB usable → ~5–7 TB used now + growth headroom.
---
## 5. Full HA and Failover Architecture
### 5.1 Components
| Component | Role |
|-----------|------|
| **Proxmox cluster** | 13 nodes; same cluster name; corosync for quorum. |
| **Ceph** | Shared storage: MON (3–5 nodes), MGR (2+), OSD on all 13. Replication size=3, min_size=2. |
| **Proxmox HA** | HA manager enabled; VMs/containers on Ceph added as HA resources; start/stop order and groups as needed. |
| **Fencing (STONITH)** | Mandatory: when a node is declared lost, fence device powers it off (or reboots) so Ceph and HA can safely reassign resources. Use Proxmox’s built-in fence agents (e.g. **fence_pve** with Proxmox API or IPMI/IDRAC). |
| **Network** | Redundant links where possible; same VLAN/bridge config on all nodes so failover does not change VM IPs. |
### 5.2 Ceph design (summary)
- **Pools:** At least one pool for VM/container disks (e.g. `ceph-vm`); optionally separate pool for backups or bulk data.
- **Replication:** size=3, min_size=2; tolerate 2 node failures without data loss (with 13 nodes).
- **Network:** Separate cluster network (e.g. 10.x or dedicated VLAN) for Ceph backend traffic; public for client (Proxmox) access.
- **MON/MGR:** 3 or 5 MONs (odd); 2 MGRs minimum. Spread across nodes for availability.
### 5.3 HA resource and failover behavior
- **HA resources:** Add each critical VM/CT as HA resource; define groups (e.g. “database first, then app”) and restart order.
- **Failure:** Node down → fencing → Ceph marks OSDs out → HA manager restarts VMs on other nodes using Ceph disks.
- **Maintenance:** Put node in maintenance → migrate VMs off (or let HA relocate) → fence not triggered; perform RAM/drive work.
### 5.4 What “full HA” gives you (DoD-relevant)
- **No single point of failure:** Storage replicated; compute can run on any node.
- **Automatic failover:** No manual migration for HA-managed guests.
- **Controlled maintenance:** Node can be taken down without losing services; documented procedures for patching and hardware changes.
---
## 6. DoD/MIL-Spec Compliance Framework
### 6.1 Alignment with DISA STIG / DoD requirements
DoD/MIL typically implies (summary; you must map to your exact ATO/contract):
| Area | Requirement | Implementation |
|------|-------------|----------------|
| **Hardening** | DISA STIG or equivalent for OS and applications | Apply STIG/CIS to Debian (Proxmox host) and guests; document exceptions. |
| **Authentication** | Strong auth, no default passwords, MFA where required | SSH key-only on Proxmox; no password SSH; RBAC in Proxmox; MFA for critical UIs if required. |
| **Access control** | Least privilege, RBAC, audit | Proxmox roles and permissions; separate admin vs operator; audit logs. |
| **Encryption** | TLS in transit; encryption at rest for sensitive data | TLS 1.2+ for API and Ceph; at-rest encryption (Ceph or LUKS) as required. |
| **Audit and logging** | Centralized, tamper-resistant, retention | rsyslog/syslog-ng to central log host; retention per policy; integrity (e.g. signed/hash). |
| **Change control** | Documented changes, rollback capability | Change tickets; config in Git; backups before changes; runbooks. |
| **Backup and recovery** | Regular backups, tested restore | Proxmox backups to separate storage; Ceph snapshots; DR runbook and tests. |
| **Physical and environmental** | Physical security, power, cooling | Out of scope for this doc; document in facility plan. |
### 6.2 Hardening checklist (Proxmox + Debian)
Use this as an operational checklist; align with your STIG version.
- [ ]**Audit:** auditd or equivalent for critical files and commands; logs to central host.
**Ceph:**
- [ ]**Auth:** Cephx enabled; key management per DoD key management policy.
- [ ]**Network:** Cluster network isolated; no Ceph ports exposed to user VLANs.
- [ ]**Encryption:** At-rest encryption for OSD if required; key escrow and rotation documented.
**Guests (VMs/containers):**
- [ ]**Per-guest hardening:** STIG/CIS per OS (e.g. Ubuntu, RHEL); documented baseline.
- [ ]**Secrets:** No secrets in configs in Git; use Vault or Proxmox secrets where applicable.
**Existing automation (this repo):** Use `scripts/security/run-security-on-proxmox-hosts.sh` (SSH key-only + firewall 8006), `scripts/security/setup-ssh-key-auth.sh`, and `scripts/security/firewall-proxmox-8006.sh`; extend to all 13 hosts and run with `--apply` after validating with `--dry-run`. Extend host list in scripts or via env (e.g. all R630 IPs).
### 6.3 Audit and documentation
- **Configuration baseline:** All Proxmox and Ceph configs in version control; changes via PR/ticket.
