ConnectX-7 vs Intel E810 — choosing server NICs for AI clusters and RoCEv2 fabrics

Quick answer: Choose the NIC family by the fabric you are growing into: ConnectX-6 Dx / ConnectX-7 for RoCEv2 at scale, GPUDirect RDMA and 200/400G east-west traffic; Intel E810 for mixed environments that need iWARP and RoCE in one part, strong virtualisation drivers and 25/100G density. Then match form factor (OCP 3.0 vs PCIe), slot generation (400G NDR needs PCIe 5.0 x16) and optics before comparing prices. On used stock, three checks decide the card: a loopback test at rated speed, unified firmware, and — on ConnectX — offload licence state.

A cluster BOM arrives with a line that just says "100G NIC × 32". The switches are Spectrum, the storage stack expects RoCE, the hypervisor team wants SR-IOV — and the purchase order goes out for whichever 100G card is cheapest that week. Six months later the fabric runs at 60% of expected bisection bandwidth and nobody can say why. NIC selection is a fabric decision, not a component decision, and the wrong family is expensive in ways that never show on the invoice.

1. Size for the fabric, not the server

Start with where the traffic actually flows:

• 10/25 GbE — general compute, management, and north-south traffic. Cheap, deep supply, and the right answer for most non-AI nodes.
• 100/200 GbE with RoCE v2 — AI and HPC east-west traffic, disaggregated storage, NVMe-oF. This is where the family choice becomes binding.
• 400G NDR — training-fabric spine connections and GPU-dense nodes; it needs a PCIe 5.0 x16 host slot to reach line rate and trains down in a Gen4 slot.

The rule that saves the most money: buy one generation ahead of your switches, not two. A 400G card behind 200G switches is stranded silicon; a 100G card on a 200G migration plan is a forklift you scheduled yourself.

2. RoCE v2 vs iWARP: the protocol decides the family

RDMA over Ethernet comes in two dialects, and they are not interchangeable inside one fabric:

• RoCE v2 runs over UDP and assumes a well-configured, mostly lossless fabric — PFC and ECN on the switches, DCQCN on the hosts. It dominates AI and HPC clusters because NVIDIA's ConnectX line and Spectrum switches were built around it, and because GPUDirect RDMA — NIC-to-GPU memory without the CPU — lives on the ConnectX side.
• iWARP rides on TCP and tolerates ordinary, lossy networks without switch surgery. It trades a little latency for a much wider deployment envelope.

ConnectX-5 and later speak RoCE v2. Intel E810 speaks both iWARP and RoCE v2. If your organisation already runs either dialect at the switch layer, the NIC family is decided for you; buying against the fabric means reconfiguring the network, not the card. For a greenfield AI cluster, the path of least resistance is ConnectX plus an RoCE-capable spine; for a mixed VMware-and-storage estate that cannot justify a lossless fabric, E810's dual personality is the whole argument.

3. The families side by side

Both families handle SR-IOV, PXE and iSCSI boot, so a mixed-grade BOM carries no compatibility penalty. The differences that move a purchase are offloads: ConnectX's ASAP2 and GPUDirect matter when the card feeds GPUs or heavy overlay networks — the same reasoning as our used AI GPU verification guide, one layer down the stack — while E810's driver maturity in VMware and Windows estates is the quieter win for virtualisation fleets.

4. Form factor and optics are the real checklist

• OCP 3.0 SFF and standard PCIe are not mechanically interchangeable. Confirm the cage or riser against your chassis before the quote, not after the delivery.
• Connector generations matter. QSFP56 ports accept QSFP28 optics at reduced rate — useful when migrating switch generations — but QSFP112 / OSFP on 400G parts is a new cage with new optics, not an upgrade path.
• DAC for in-rack, AOC for adjacent rows, transceivers beyond that. Mixing them without a plan is how fabrics end up with three spares of the wrong kind.
• Coded optics are the weak link in most bundles. Switch vendors lock ports to EEPROM coding; an uncoded or mis-coded optic passes link-up and then flaps under load. We kit matched, coded optics and pre-verify them with the card at rated speed for exactly this reason.

5. Buying used NICs: three checks decide the card

Adapters have no wear-out mechanics, so a verified pull is low-risk value — typically 30–50% below new. But "tested" on a listing usually means the link LED came on. Insist on:

1. A loopback test at rated speed, with the result attached. Not negotiated link-up — full rate, sustained, per port. A 100G card that trains at 25G in the test has a damaged lane, a dirty optic seat or the wrong firmware, and none of those appear in a photo.
2. Unified firmware per batch, and on ConnectX parts, verified offload licence state. Some ConnectX features are licence-gated at the firmware level; a used card without the licence your design assumes is a different product than the one on the BOM.
3. Bracket height and optics kit documented. Low-profile and full-height brackets arrive mixed in recycled stock, and "card only" pricing quietly excludes the optics that make it usable. Compare kit to kit.

"A used NIC is not one product. It is a card, a licence state, a bracket and a set of optics — and most listings price only the first."

— Huazhe incoming-inspection procedure

Where to buy tested server adapters in bulk

Our server network adapters stock list carries ConnectX-5 through ConnectX-7, Intel E810, X710 and X550 in new and pulled-and-tested grades — firmware unified per batch, offload licences verified on used ConnectX parts, loopback-tested at rated speed, with matched coded optics kitted on request. Tell us the workload, switch generation and host platform, and the exact variant, optics and cabling come back spec'd with the quote. Building the rest of the cluster? The GPU verification guide covers the accelerator batches the fabric feeds.

Notes from the Zhongshan desk

The most expensive NIC mistake we see is invisible in photos: offload licence state on used ConnectX cards. Two cards with the same part number can differ in what their firmware allows, and the listing price reflects whichever one it is. We read the licence state on every used ConnectX before grading — ask any supplier to do the same, in writing, per card.

Second, coded optics really are where bundles go wrong. A batch of third-party optics with the wrong EEPROM coding will link up on the bench and then flap under load at 2 a.m.; the fix costs more in engineer-hours than the optics saved. This is why we re-test or replace the optics in every used bundle and pre-verify them with the card.

And a quiet one: QSFP28 optics in QSFP56 cages genuinely do work at reduced rate, and they are a legitimate migration bridge — but label them in the rack map, because the next engineer will "fix" the mismatch by swapping in a cable that does not fit the budgeted design.

FAQ

Is RoCE v2 worth it without NVIDIA switches? RoCE v2 needs PFC and ECN on the switching path, and most enterprise switches from major vendors support both — you are not locked to Spectrum. The honest trade-off is operational: non-reference fabrics need careful buffer configuration, so for a first RDMA deployment, a reference-validated stack is cheaper in engineer time than in hardware.

Can the Intel E810 do 200G? No — E810 tops out at 2× 100G (CQDA2) or 4× 25G (XXVDA4). If the design calls for 200G per port, the part is ConnectX-6 HDR or ConnectX-7. E810's advantage is the dual iWARP/RoCE personality and driver maturity, not raw port speed.

Do I need GPUDirect RDMA for an inference cluster? For single-node inference, no — the GPU and NIC communicate fine through host memory. GPUDirect pays on multi-node training and tightly-coupled inference where east-west latency compounds across every step; it is a ConnectX-side feature, which is why AI fabric designs start from that family.

What should I check first on a used network adapter? A loopback test at rated speed, per port, with the result in writing. Then unified firmware per batch and — on ConnectX — the offload licence state; then bracket height and whether the optics are included, coded and tested. A card that cannot show a rated-speed loopback result has not been tested at all.