d-bis/loc_az_hci
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
loc_az_hci/docs/architecture/pcie-allocation.md
defiQUG c39465c2bd
Some checks failed
Test / test (push) Has been cancelled
Details
Initial commit: loc_az_hci (smom-dbis-138 excluded via .gitignore) 
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-08 09:04:46 -08:00

8.3 KiB
Raw Permalink Blame History

PCIe Slot Allocation Map

Router/Switch/Storage Controller Server

This document provides the PCIe slot allocation map for the Router/Switch/Storage Controller server, ensuring optimal lane distribution and avoiding conflicts.

Slot Allocation

Visual Slot Map

┌─────────────────────────────────────────────────────────┐
│              Router Server PCIe Slots                    │
├─────────────────────────────────────────────────────────┤
│                                                           │
│  [x16_1]  Intel QAT 8970 (PCIe 3.0 x16)                 │
│           └─ Crypto acceleration (TLS/IPsec/compression)  │
│                                                           │
│  [x8_1]   Intel X550-T2 (2× 10GbE RJ45)                 │
│           └─ Future uplinks or direct server links      │
│                                                           │
│  [x8_2]   LSI 9207-8e (SAS HBA #1)                      │
│           └─ External storage shelves (2 shelves)        │
│                                                           │
│  [x8_3]   LSI 9207-8e (SAS HBA #2)                      │
│           └─ External storage shelves (2 shelves)        │
│                                                           │
│  [x4_1]   Intel i350-T4 (4× 1GbE WAN)                   │
│           └─ 4× Spectrum WAN connections                │
│                                                           │
│  [x4_2]   Intel i350-T8 (8× 1GbE LAN)                   │
│           └─ Remaining servers and appliances           │
│                                                           │
│  [x4_3]   Intel i225 Quad-Port (4× 2.5GbE LAN)         │
│           └─ Direct to ML110, R630, key services        │
│                                                           │
└─────────────────────────────────────────────────────────┘

Detailed Slot Configuration

Slot x16_1: Intel QAT 8970

  • Card: Intel QuickAssist Technology 8970
  • Interface: PCIe 3.0 x16
  • Lane Usage: x16 (full bandwidth)
  • Purpose: Crypto acceleration for TLS/IPsec/compression
  • Priority: High (ensures bandwidth and stability)
  • Thermal: High power consumption, ensure adequate cooling
  • Software: qatlib drivers, OpenSSL QAT engine

Rationale: QAT card requires maximum bandwidth for crypto operations. x16 slot ensures no bottlenecks.

Slot x8_1: Intel X550-T2

  • Card: Intel X550-T2 (2× 10GbE RJ45)
  • Interface: PCIe 3.0 x8
  • Lane Usage: x8 (full bandwidth)
  • Purpose: Future 10GbE uplinks or direct server links
  • Priority: High (future expansion)
  • Thermal: Moderate
  • Software: Intel PROSet drivers

Rationale: 10GbE requires x8 lanes for full bandwidth. CPU-connected slot preferred.

Slot x8_2: LSI 9207-8e (SAS HBA #1)

  • Card: LSI 9207-8e (SAS2308, IT mode)
  • Interface: PCIe 3.0 x8
  • Lane Usage: x8 (full bandwidth)
  • Purpose: External storage shelves (2 shelves)
  • Priority: High (storage performance)
  • Thermal: Moderate
  • Software: mpt3sas driver, IT mode firmware
  • Cables: 2× SFF-8644 Mini-SAS HD cables

Rationale: Storage HBAs require x8 lanes for optimal performance. CPU-connected slot preferred.

Slot x8_3: LSI 9207-8e (SAS HBA #2)

  • Card: LSI 9207-8e (SAS2308, IT mode)
  • Interface: PCIe 3.0 x8
  • Lane Usage: x8 (full bandwidth)
  • Purpose: External storage shelves (2 shelves)
  • Priority: High (storage performance)
  • Thermal: Moderate
  • Software: mpt3sas driver, IT mode firmware
  • Cables: 2× SFF-8644 Mini-SAS HD cables

Rationale: Second HBA for redundancy and additional storage capacity.

Slot x4_1: Intel i350-T4

  • Card: Intel i350-T4 (4× 1GbE ports)
  • Interface: PCIe 3.0 x4
  • Lane Usage: x4 (full bandwidth)
  • Purpose: 4× Spectrum WAN connections
  • Priority: High (WAN connectivity)
  • Thermal: Low
  • Software: Intel PROSet drivers, OpenWrt mwan3
  • Cables: 4× Cat6 Ethernet cables

Rationale: 4× 1GbE requires x4 lanes. WAN connectivity is critical.

Slot x4_2: Intel i350-T8

  • Card: Intel i350-T8 (8× 1GbE ports)
  • Interface: PCIe 3.0 x4
  • Lane Usage: x4 (full bandwidth)
  • Purpose: Remaining servers and appliances
  • Priority: Medium
  • Thermal: Low
  • Software: Intel PROSet drivers, OpenWrt firewall zones
  • Cables: 8× Cat6 Ethernet cables

Rationale: 8× 1GbE can operate on x4 lanes (2 Gbps per lane is sufficient).

Slot x4_3: Intel i225 Quad-Port

  • Card: Intel i225 Quad-Port (4× 2.5GbE ports)
  • Interface: PCIe 3.0 x4
  • Lane Usage: x4 (full bandwidth)
  • Purpose: Direct to ML110, R630, and two key services
  • Priority: High (key server connectivity)
  • Thermal: Low
  • Software: Intel PROSet drivers, OpenWrt firewall zones
  • Cables: 4× Cat6 Ethernet cables

Rationale: 4× 2.5GbE requires x4 lanes for full bandwidth.

Lane Budget Analysis

Total Lane Requirements

Slot Lanes Component Bandwidth
x16_1 16 Intel QAT 8970 ~16 GB/s
x8_1 8 Intel X550-T2 ~8 GB/s
x8_2 8 LSI 9207-8e #1 ~8 GB/s
x8_3 8 LSI 9207-8e #2 ~8 GB/s
x4_1 4 Intel i350-T4 ~4 GB/s
x4_2 4 Intel i350-T8 ~4 GB/s
x4_3 4 Intel i225 Quad ~4 GB/s
Total 52 ~52 GB/s

CPU Lane Availability

  • Typical Xeon E-2100: 16 PCIe 3.0 lanes from CPU
  • Chipset lanes: Additional lanes from PCH (varies by chipset)
  • Total available: Typically 24-40 lanes depending on chipset

Note: Ensure motherboard supports sufficient PCIe lanes. Most server motherboards provide adequate lane budget through CPU + chipset combination.

Thermal Considerations

High-Power Components

  1. Intel QAT 8970 (x16_1):

    • Power consumption: ~25-30W
    • Ensure adequate airflow
    • Consider slot spacing if possible

  2. LSI 9207-8e HBAs (x8_2, x8_3):

    • Power consumption: ~10-15W each
    • Moderate thermal load
    • Ensure proper cooling

Cooling Recommendations

  • Ensure adequate case airflow
  • Consider slot spacing for high-power cards
  • Monitor temperatures during operation
  • Use server-grade case with proper ventilation

Slot Priority and Conflict Resolution

Priority Order

  1. Critical (Must have):

    • x16_1: QAT 8970 (crypto acceleration)
    • x4_1: i350-T4 (WAN connectivity)
    • x8_2/x8_3: LSI HBAs (storage)

  2. High Priority:

    • x8_1: X550-T2 (future expansion)
    • x4_3: i225 Quad (key server connectivity)

  3. Medium Priority:

    • x4_2: i350-T8 (remaining servers)

Conflict Resolution

If lane budget is insufficient:

  1. Option 1: Use chipset-connected slots for lower-priority NICs
  2. Option 2: Reduce some x8 slots to x4 if card supports it
  3. Option 3: Use onboard NICs for some connections
  4. Option 4: Upgrade to CPU with more PCIe lanes

Physical Installation Notes

Installation Order

  1. Install QAT card first (x16_1) - highest priority
  2. Install storage HBAs (x8_2, x8_3) - critical for storage
  3. Install WAN NIC (x4_1) - critical for connectivity
  4. Install LAN NICs (x4_2, x4_3) - complete network setup
  5. Install 10GbE NIC (x8_1) - future expansion

Cable Management

  • Label all cables at both ends
  • Use cable management accessories
  • Document cable routing
  • Ensure cables don't obstruct airflow

Verification Checklist

  • All cards physically installed in correct slots
  • All cards detected in BIOS/UEFI
  • All cards detected in OS
  • Drivers installed and verified
  • All ports functional
  • Thermal monitoring active
  • Cable labeling complete
  • Documentation updated

Related Documentation

Reference in New Issue View Git Blame Copy Permalink