InfiniBand vs Ethernet: Are AI Data Centres Paying More for Performance They Don't Need?

Sometimes, but less often than the "InfiniBand premium" argument suggests. At public list prices, NVIDIA's Quantum InfiniBand and Spectrum-X Ethernet switches cost roughly the same per terabit of port capacity. Both are used in some of the largest AI training clusters ever built: Meta has run 24,576-GPU clusters on each. The real choice is less about raw speed and more about scale, how much tuning your team can do, which tools it already knows, and whether the cluster will train large models or mostly serve inference. For a few nodes running inference, a well-configured Ethernet fabric is usually enough. For tightly coupled training across thousands of GPUs, InfiniBand's built-in features can save tuning effort, and that is the part of the premium worth paying for.

How do current InfiniBand and Ethernet switches compare on paper?

Source: NVIDIA Quantum-2 and Quantum-X800 pages (updated 27 Aug 2026), NVIDIA XDR switch manual (21 Aug 2026) and SN5000 manual (7 Dec 2025). Prices are web list prices fetched 6 Nov 2026 (FiberMall for QM9700 and SN5600, Viperatech for Q3400); they are illustrative and exclude optics, licences and support. Per-Tb/s figures are our arithmetic.

Two things stand out. First, NVIDIA itself uses two counting conventions: Quantum-2 is quoted bidirectionally, the other two one way. Always recompute ports × port speed before comparing. Second, once you do, the per-port list cost of the newest InfiniBand and Ethernet switches is close. The older 400G InfiniBand switch is the expensive one per terabit, simply because it is a generation behind.

So where does the InfiniBand premium come from?

Mostly from the rest of the system rather than the switch. InfiniBand fabrics use InfiniBand host adapters, InfiniBand-specific optics and cables, a subnet manager (NVIDIA UFM in most deployments), and engineers who know how to run them. Ethernet fabrics can reuse the tools, monitoring and skills a data centre team already has for its front-end network. That operational difference, not the box price, is usually what people mean by "paying more".

How big is the network share overall? Estimates vary widely. Writing in The Next Platform in May 2024, Timothy Prickett Morgan paraphrased NVIDIA's argument that InfiniBand is about 20% of cluster cost, and said 10% or lower is normal for Ethernet-based clusters. In June 2025 he argued in The Register that the real figure may be as high as 30%. LightCounting puts optics alone at 2.7% of the top five cloud providers' capital spending in 2025, rising to 3.1% in 2026. Our post on the hidden cost of AI networking works through cables and optics in detail.

What does InfiniBand do that standard Ethernet doesn't?

• In-network reductions (SHARP): the switches perform part of an all-reduce themselves, which NVIDIA says halves the data transferred. SHARP v2 "doubled AllReduce bandwidth" on HDR; v4 ships with Quantum-X800 without published figures.
• Lossless by design: InfiniBand was built as a lossless fabric. Ethernet RDMA (RoCEv2) traditionally relies on priority flow control and ECN tuning to avoid drops, which is where much of the configuration effort goes.
• Adaptive routing, congestion control and quality of service are part of the InfiniBand stack. Spectrum-X adds NVIDIA's own adaptive routing and congestion control to Ethernet, but only when paired with BlueField-3 or ConnectX-8/9 SuperNICs.

Ethernet is closing the gap. The Ultra Ethernet Consortium released Specification 1.0 on 11 June 2025 and version 1.1 on 12 October 2026, which adds programmable congestion management, in-network collectives and a scale-up profile. Broadcom's Tomahawk Ultra adds credit-based flow control and link-layer retry, features long associated with InfiniBand. Compliance with UEC is by vendor self-attestation, so check what a specific product actually implements.

Convert every figure to the same basis before comparing fabrics: per port, per direction, in gigabytes or gigabits. NVIDIA's own switch pages use both one-way and bidirectional totals.

Does it matter for training?

It can, because training at scale is tightly coupled: every GPU waits for the slowest exchange. Meta built two 24,576-GPU H100 clusters in 2024, one on RoCE Ethernet and one on Quantum-2 InfiniBand, and reported it "successfully used both". The catch was tuning: Meta said large-cluster communication performance on RoCE was "initially poor and inconsistent", varying from about 10% to 90% of expected, until topology-aware scheduling, routing changes and NCCL tuning brought it above 90%. Llama 3 405B was then trained on the RoCE cluster.

The lesson is not that one fabric wins. It is that Ethernet at very large scale needs engineering effort that InfiniBand builds in, and InfiniBand needs specialist operations that Ethernet teams may not have. NVIDIA claims Spectrum-X delivers "1.6x" the AI network performance of off-the-shelf Ethernet, and that xAI's 100,000-GPU Colossus reached 95% data throughput versus 60% for standard Ethernet. Both are vendor claims without independent test detail.

And for inference?

Inference usually needs far less scale-out bandwidth. Token generation is memory-bound: each GPU spends most of its time reading weights and KV cache from its own HBM. Tensor and expert parallelism, which are communication-heavy, are normally kept inside a node or an NVLink domain such as a GB300 NVL72 rack, where GPUs exchange data at 900 GB/s per direction rather than the 100 GB/s per direction of an 800 Gb/s NIC. The scale-out fabric then carries requests, KV-cache transfers in disaggregated serving, and storage traffic. For a few inference nodes, a well-run RoCE Ethernet fabric is generally sufficient, and an InfiniBand fabric may be capacity you don't use. See NVLink vs InfiniBand inside an AI factory for where each link sits.

What is the market actually choosing?

Both, with Ethernet growing faster overall. Dell'Oro reported in March 2026 that Ethernet in AI back-end (scale-out) networks was more than double the size of InfiniBand in 2025. In the first quarter of 2026 Ethernet AI back-end sales more than doubled again, while InfiniBand "more than tripled" on 800G Blackwell Ultra switches, much of it upgrades to existing InfiniBand clusters. NVIDIA, which sells both, said on its May 2026 earnings call that InfiniBand was growing more than four times year over year, and that Spectrum-X is "now larger than all Ethernet network peers combined" (a vendor claim).

How should a buyer decide?

• A few nodes, mostly inference or fine-tuning: Ethernet with RoCEv2. It reuses existing skills and can share tooling with the front-end network.
• Large, tightly coupled training with a dedicated cluster team: InfiniBand reduces the tuning burden and adds SHARP. Spectrum-X is the Ethernet alternative if you prefer Ethernet operations and accept NVIDIA SuperNICs.
• Existing estate: extending an InfiniBand cluster is usually simpler than mixing fabrics, which is what Dell'Oro saw in the 2026 InfiniBand rebound.
• Multi-vendor or multi-tenant needs: Ethernet and UEC give more supplier choice; check UEC feature support product by product.
• Whatever you choose: budget for monitoring, NCCL testing and someone who can read the results. Our guide to reading 800 Gb/s and 130 TB/s figures helps with the numbers.

For background on host adapters, see NVIDIA ConnectX-7 vs Ordinary Ethernet, and for switch sizing, 400Gbps Networking for AI: Do You Really Need Such an Expensive Switch? If you are still at the budgeting stage, Could Your S$1 Million AI Cluster Be Wasting Money Because of Poor Networking? puts the fabric in context.

Buying in Singapore

SourceIT lists Ethernet switching for smaller AI and lab deployments: the NVIDIA Spectrum-2 SN3700 32-port 200GbE switch at S$28,523.25, the D-Link DQS-5000-32Q28 32-port 100G switch at S$20,186.40, and the 4-port 400G MikroTik CRS804-4DDQ at S$2,255.00 for linking a pair of systems (all listed 6 Nov 2026). The SN3700 is a 200G switch, not a 400G or 800G fabric switch. We do not currently list InfiniBand switches, ConnectX adapters or 400G/800G optics; ask us to source them for your design. For deskside AI that links over ConnectX-8 Ethernet, see our AI workstations and servers. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

Is InfiniBand faster than Ethernet? Not in raw port speed: both now run 800 Gb/s per port. InfiniBand's advantages are built-in lossless operation, SHARP in-network reductions and mature congestion control, which matter most for large, tightly coupled training.

Is InfiniBand more expensive? Per terabit at list price, the latest switches are close: about US$1,120 for the Q3400 and US$1,130 for the SN5600. Higher costs usually come from InfiniBand-specific adapters, optics, management software and skills.

Can large AI models be trained on Ethernet? Yes. Meta trained Llama 3 405B on a 24,576-GPU RoCE Ethernet cluster, after significant tuning of scheduling, routing and NCCL.

What is Spectrum-X? NVIDIA's Ethernet platform for AI, pairing Spectrum switches with BlueField-3 or ConnectX SuperNICs for adaptive routing and congestion control. NVIDIA claims 1.6x the AI network performance of off-the-shelf Ethernet.

Do inference clusters need InfiniBand? Usually not. Inference is mostly limited by GPU memory bandwidth, and communication-heavy parallelism is kept inside a node or NVLink domain. Ethernet with RoCEv2 is generally enough for a few inference nodes.