Hardware RAID vs IT-Mode HBA: Which Controller for ZFS, Ceph and Write-Safe Arrays

Quick answer: Choose by who owns redundancy. ZFS and Ceph need a true IT-mode HBA that passes raw disks, SMART data and flush commands straight through; VMware, Hyper-V and database arrays need hardware RAID with a healthy CacheVault or BBU behind write-back. Match the card's PCIe generation and port count to the slot and backplane, pay for 24G SAS only where the drives can use it, and on a pulled card check cache health, firmware and kit completeness before payment.

Every storage BOM has a line that says "RAID card" and a build that quietly assumes the wrong one. A cache-protected MegaRAID on a Ceph node presents virtual disks, so Ceph finds no OSDs; a bare HBA on a VMware host presents raw disks and no array. Both fit the same slot. Only one fits the workload.

Architecture first: cache-protected RAID or raw passthrough

• Hardware RAID owns redundancy. A ROC offloads parity, manages virtual drives and holds a DRAM cache. With write-back it acknowledges writes before media, and a BBU or CacheVault covers the gap on power loss. The OS sees one disk — right for VMware VMFS, Hyper-V CSV and database volumes. The cache module is not an accessory.
• An IT-mode HBA owns nothing. Firmware passes each drive through raw: no RAID metadata, no virtual disk, no hidden cache. SMART, serials and flush commands reach the OS, and redundancy moves up the stack to ZFS mirrors and raidz, or Ceph replication and erasure coding.
• Why ZFS needs it. ZFS manages vdevs, checksums, scrub and replacement by device, so it must see the physical disks. A controller hides the members: ZFS cannot heal a vdev or name the failed drive, and an unprotected write-back cache that acknowledges a sync write holds the only copy on power loss.
• Why Ceph needs it. One OSD per raw block device. BlueStore writes its own checksums and WAL, and Ceph replicates or erasure-codes above the disk. A RAID layer hides individual failures, blocks SMART prediction and makes the controller a single point of failure for every OSD behind it.
• "JBOD mode" is not IT mode. Many RAID cards expose drives as JBOD while the firmware stays RAID: metadata can persist, SMART may not pass through, and queue behaviour differs. True IT mode is a firmware personality or a dedicated model. The 9207 and 9300 are IT HBAs by design, but the same silicon ships as OEM RAID cards and older IR variants, so used cards are routinely crossflashed — trust the firmware, not the label. The 9400, 9500 and 9600 are IT by model. Do not run ZFS on RAID 0 members.
• Write-safe means cache-safe. If the workload needs hardware RAID, write-back is the performance reason and a healthy module is the safety condition. A dead module should force write-through — MegaRAID's default policy ends "No Write Cache if Bad BBU" — so verify nobody changed it. Parity RAID has a write hole; protected cache narrows it, ZFS's copy-on-write avoids it, and Ceph avoids it above the disk.

PCIe generation and port count: match the host, not the brochure

The controller cannot move more data than the slot and backplane carry. A 12G lane carries 12 Gb/s; a 4-lane SFF-8643 connector carries 48 Gb/s, about 4.8 GB/s after encoding. 24G doubles both. PCIe 3.0 x8 delivers about 7.9 GB/s and PCIe 4.0 x8 about 15.8 GB/s. A 16-port 12G card has 16 SAS lanes but only 7.9 GB/s to the host — fine, because SAS is switched and 16 HDDs never stream at full rate together. A 24G card needs PCIe 4.0 to be meaningful, which is why the 9500 and 9600 are PCIe 4.0 x8.

Ports are lanes, not bays: a 16i card feeds 16 drives directly, an 8i card needs an expander for 16 bays, and a 24-bay chassis wants a 24i card or an expander sharing its uplink. Connectors are generation-bound — SFF-8643 and SFF-8654 do not interchange, so quote card and cable kit together. And x8 means x8 electrical: a physical x16 slot wired as x8 trains at x8 — fine for 12G, a waste of a 24G card.

12G vs 24G SAS: where the extra bandwidth pays

• HDD arrays do not need 24G. A 12G SAS HDD streams at 200–300 MB/s; even 24 drives behind a 12G expander are limited by the drives, not the link. 12G gear is cheaper and the supply is deeper.
• 24G pays at the flash end. 24G SAS SSDs, all-flash shelves and Tri-Mode NVMe benefit, and the headroom helps rebuilds. In a mixed SAS/SATA/NVMe chassis, Tri-Mode matters more than the link rate.
• Every link must match. A 24G HBA on a 12G backplane trains at 12G; 24G drives on a 12G expander run at 12G. 24G uses SFF-8654 on the 9500 and 9600, SFF-8643 on the 9400 and older — plan cables and backplane together.
• Backward compatibility is real, not free. 24G ports accept 12G and 6G drives at their own rate, and 24G drives fall back on 12G ports; a mixed array runs at the slowest link.
• Spend the difference where it protects data. For a 16-bay HDD ZFS box, a 12G IT-mode HBA plus a verified spare beats a 24G card that never leaves 12G.

Three checks that decide a pulled controller

New cards are a purchase order. Pulled cards are an inspection, and the failures concentrate in three places.

1. Cache-protection health

On a RAID card the cache module is the only part that genuinely wears, and a dead module is quiet: write-back may be disabled, or — if someone changed the policy — the card may keep acknowledging writes it cannot protect. On ZFS or Ceph the module is irrelevant, which is why a pulled HBA is simpler to qualify. Read it with storcli /c0/cv show all (state, temperature, capacitance) and show status; on BBU cards storcli /c0/bbu show all adds voltage, next learn cycle and remaining capacity. Accept only "Optimal" — "Failed", "Missing" or the wrong module all mean card-only pricing. BBU packs contain lithium-ion cells, so cross-border air shipments often arrive without one; CacheVault supercapacitors are not lithium and normally travel with the card.

2. Firmware identity

storcli /c0 show returns product name, serial, SAS address, firmware, NVDATA, BIOS and the OEM field — the card's passport. OEM firmware is platform-locked: Dell PERC and HPE Smart Array variants use the same silicon but update through OEM tools, so a PERC can work in a third-party board and still refuse retail firmware. Quote it to its platform, or not at all. "IT mode" must be firmware, not a setting: on older LSI cards it is a crossflash — verify with sas2flash -list on the 9207 and sas3flash -list on the 9300 and later, and check for foreign configuration; on the 9400, 9500 and 9600 it is the HBA model. Unify the batch: one firmware and NVDATA pair per order.

3. Bracket, cables and the rest of the kit

Full-height and low-profile brackets are different parts; a 2U chassis needs the low-profile one, and pulls often arrive with only the full-height. SFF-8643 for 12G and SFF-8654 for 24G do not interchange, and 4i and 8i cables carry different lane counts — the cable is part of the bill of materials. The CacheVault module may mount remotely and connect by cable; without both the card has no protection, so ask for a photo of the module connected to the exact card. These are passive, full-length cards: they need server airflow and chassis depth.

"A pulled RAID card is not one product. It is a card, a cache module, a bracket and a cable kit — and most listings price only the first one."

— Huazhe incoming-inspection procedure

Condition grades: new, pulled and refurbished

For ZFS and Ceph, a pulled IT-mode HBA is usually the best-value line in the bill of materials: there is no cache module to age and no RAID metadata to clean. For a production hardware-RAID array, the cache module is the one line item worth buying new.

Sourcing RAID and HBA controllers

The RAID/HBA controller stock list carries IT-mode 9207 and 9300 HBAs, Tri-Mode 9400/9500/9600 cards and MegaRAID controllers with CacheVault; every card is flashed to the firmware you specify, burn-tested on an OEM platform, and shipped with its bracket and cable kit confirmed. Tell us the server model and backplane in the RFQ — firmware, bracket and cable are matched before quoting. The enterprise SSD and SAS HDD sourcing guide covers the drive side of the same build, and the used AI GPU verification guide applies the same recorded-data-first discipline to accelerator batches.

Notes from the Zhongshan desk

• "Tested" usually means POST. Most pulled cards are tested with no backplane, no drives and no cache module: the fan spins, the BIOS sees the card, the listing says tested. Ask for a disk-attached burn test and the matching storcli output.
• The module is the margin. CacheVault modules are a separate SKU and sell on their own; a card priced 30% under the market has often had its module removed. Compare kit to kit, not card to card.
• Air freight changes the kit. A BBU card without its battery may have shipped legally rather than been stripped; a CacheVault card has no such excuse.
• Most "DOA" cards are cable mismatches. SFF-8643 does not fit an SFF-8654 port, and a 4i cable does not feed an 8i connector. Confirm the cable part number before the card ships.
• Firmware follows the platform. A Dell PERC in a Supermicro chassis may POST and still be the wrong purchase: no retail update path and resale value tied to Dell.

FAQ

Can I run ZFS on a RAID card in JBOD mode? Not reliably. JBOD on RAID firmware can still hide SMART data, keep metadata and alter queue behaviour — ZFS needs a true IT-mode HBA with raw disks. Use an older LSI card crossflashed to IT, or an IT-mode 9400/9500/9600 model; do not build a pool on RAID 0 members, because the controller then owns the failure domain.

Do I need a battery or CacheVault for a write-safe array? For write-back cache, yes. Without a healthy module the controller either falls back to write-through, losing the performance case, or keeps acknowledging writes it cannot protect, which is worse. Check the cache policy and module state together before trusting a pulled card; ZFS avoids the question by not using controller RAID at all.

Is 24G SAS worth paying for over 12G? Only when the drives, backplane and expander are 24G, or the shelf is all-flash or Tri-Mode NVMe. A 12G HDD array sees no benefit, and 24G hardware runs at the slowest link in the chain. Put the difference into cache protection, a spare HBA or a second controller.

What should I check first on a used RAID card? Cache-protection health: run storcli /c0/cv show all or /c0/bbu show all and accept only "Optimal" with the correct module for the card. Then check firmware, NVDATA and the OEM field, then the bracket, cables and module mount. Ask for the output per card, not a batch-level promise from the seller.