Because they do different jobs, and neither can do the other's economically. NVLink is the scale-up fabric. In a GB300 NVL72 rack it joins 72 Blackwell Ultra GPUs into one domain at 1.8 TB/s bidirectional per GPU, over copper, with a single switch hop between any two GPUs. InfiniBand is the scale-out fabric. Through one ConnectX-8 port per GPU at 800 Gb/s, or 100 GB/s per direction, it links racks into clusters of thousands of GPUs over optics across a data hall. Per GPU, InfiniBand has about a ninth of NVLink's bandwidth, but it reaches far further. In an AI factory, NVLink carries the traffic that has to be fast, and InfiniBand carries the traffic that has to travel. In many AI factories, Spectrum-X Ethernet takes InfiniBand's place in that second role.
How do NVLink and InfiniBand compare in a GB300 AI factory?
| Attribute | NVLink 5 (scale-up) | Quantum-X800 InfiniBand (scale-out) |
|---|---|---|
| Role | Joins GPUs inside one rack into a single domain | Joins racks and nodes into a cluster |
| Per-GPU bandwidth | 1.8 TB/s bidirectional (900 GB/s per direction) | 800 Gb/s per GPU, or 100 GB/s per direction (via ConnectX-8) |
| Domain size | 72 GPUs per NVL72 rack | 576 GPUs per GB300 SuperPOD scalable unit (8 racks), and more by adding units |
| Switch | NVLink 5 Switch: 72 ports, 7.2 TB/s bidirectional per chip, 18 chips per rack | Q3400: 144 × 800 Gb/s ports, 115.2 Tb/s (one-way port sum) |
| Media | Copper spine inside the rack | Mostly optical transceivers and fibre between racks |
| In-network compute | SHARP engines in the NVLink Switch | SHARP v4 |
| Typical traffic | Tensor-parallel all-reduce, expert-parallel all-to-all | Data-parallel gradients, pipeline activations, cross-rack collectives |
Source: NVIDIA GB300 NVL72 page (accessed 6 Nov 2026), NVIDIA NVLink page (updated 15 Sep 2026), NVIDIA HOTI 2025 NVLink talk (21 Aug 2025), NVIDIA XDR switch user manual (updated 21 Aug 2026), NVIDIA DGX GB300 SuperPOD reference architecture and Quantum-X800 page (updated 27 Aug 2026). The 100 GB/s figure is our conversion of 800 Gb/s ÷ 8.
What does NVLink handle inside the rack?
The traffic that happens most often and can least afford to wait. When a model is too large for one GPU, tensor parallelism splits each layer across GPUs and combines the partial results with an all-reduce at every layer. NVIDIA's Megatron documentation recommends keeping this inside the NVLink domain. Mixture-of-experts models add expert parallelism, which sends tokens to experts on other GPUs through all-to-all exchanges in every transformer block. In NVIDIA's words, these exchanges "can quickly saturate an already memory-bound decode phase". The NVL72's single-hop reach to all 72 GPUs is what makes wide expert parallelism practical. NVIDIA claims up to 1.8 times more throughput per GPU on DeepSeek-R1 when expert parallelism widens from 8 to 32 GPUs (a vendor claim). Our post GB300 NVL72: 72 GPUs, NVLink and NVSwitch covers the rack itself.
What does InfiniBand handle between racks?
The traffic that is less frequent but has to cross the building. Data parallelism runs copies of the model on different racks and synchronises gradients once per step. Pipeline parallelism passes activations from stage to stage, point to point. NVIDIA's SuperPOD designs build this fabric as a "rail-optimized, non-blocking" fat tree. Each GPU's NIC connects to a leaf switch dedicated to its rail, so same-rail traffic is one hop away and cross-rail traffic goes through the spine. In NVIDIA's B300 XDR SuperPOD reference design, one 576-GPU unit uses 8 leaf and 4 spine Q3400 switches. SHARP v4 performs reductions inside the switches, which, NVIDIA says, halves the data transferred for an all-reduce.
An AI factory runs more networks than these two. NVIDIA's reference designs add a front-end Ethernet network for logins, orchestration and user storage, and a storage fabric sized at 90 GB/s read and 45 GB/s write per 576-GPU unit at the standard tier. They also add a 1 GbE out-of-band network for BMC management, with BlueField-3 DPUs handling front-end and storage traffic.
Why not run NVLink across the whole data centre?
Reach, power and cost. NVLink in an NVL72 runs over a copper spine, and copper at these speeds reaches only a few metres. For the GB200 NVL72, Jensen Huang said that copper saved about 20 kW per rack compared with the optical transceivers and retimers needed to drive the same links over fibre. NVIDIA's architecture can scale an NVLink domain to 576 GPUs, and the Rubin Ultra roadmap from GTC 2025 showed a 576-die rack. Even so, every NVLink domain announced so far stops at a rack or a group of racks, not a building. Beyond that, InfiniBand or Ethernet takes over.
Why not run InfiniBand inside the rack too?
Because per GPU it has about a ninth of the bandwidth. A ConnectX-8 port gives a GPU 100 GB/s per direction, against NVLink 5's 900 GB/s. DeepSeek's V3 report shows what happens when heavy traffic is pushed onto the slower fabric. On H800 GPUs, where NVLink gave 160 GB/s against 50 GB/s for InfiniBand (3.2 times), cross-node expert parallelism ran at roughly a 1:1 computation-to-communication ratio. The team dedicated 20 of 132 streaming multiprocessors per GPU to communication and asked hardware vendors to "unify the IB (scale-out) and NVLink (scale-up) networks". With NVL72, the busiest expert traffic can stay inside one rack. That is exactly the split NVIDIA designed for.
NVLink makes 72 GPUs behave like one. InfiniBand makes many racks behave like one cluster.
Does it have to be InfiniBand, or will Ethernet do?
Ethernet is now the larger scale-out fabric by sales. Dell'Oro reported that Ethernet was more than double InfiniBand in AI back-end networks in 2025. InfiniBand sales then more than tripled in the first quarter of 2026, on 800G Blackwell Ultra upgrades. NVIDIA sells both: Quantum-X800 InfiniBand, and Spectrum-X Ethernet built on SN5600 switches (64 × 800 GbE) with SuperNICs. It claims Spectrum-X is 1.6 times faster for AI than off-the-shelf Ethernet (a vendor claim). Meta has trained on both, with one 24,576-GPU cluster on RoCE Ethernet and one on InfiniBand. Whichever fabric is chosen, the NVLink domain inside the rack stays the same. See InfiniBand vs Ethernet for AI and ConnectX-7 vs Ordinary Ethernet.
Where does a DGX Station fit in this picture?
A DGX Station GB300 has the same Blackwell Ultra GPU and the same 900 GB/s NVLink-C2C as a GB300 NVL72 compute node, but no NVLink to other GPUs. Two Stations link over ConnectX-8 at 2× 400 Gb/s with RoCEv2, a small scale-out network on a desk. NVIDIA positions the Station for work that is "developed and debugged locally" and "then brought to massive scale on the cloud", or deployed "to the data center". Our posts From DGX Station to GB300 NVL72 and 800Gbps InfiniBand vs 130TB/s NVLink explain how the numbers carry over, and DGX Station's networking covers the cabling.
Buying in Singapore
SourceIT does not currently list NVL72 racks, InfiniBand switches, 800G switches, ConnectX NICs or optical transceivers. For a data-centre project, ask us to source them. For smaller Ethernet fabrics we list the NVIDIA Spectrum-2 SN3700, a 32-port 200 GbE switch, at S$28,523.25 (listed 6 Nov 2026). Each 200 GbE port carries 25 GB/s per direction. For deskside development we list six DGX Station GB300 systems, all price on request, including the HP ZGX Fury G1n, Supermicro ARS-511GD-NB-LCC and Dell Pro Max with GB300. For GPU servers we list the Lenovo ThinkSystem SR680a V3 (B200), also price on request. See the AI Workstations and AI Servers collection. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
What is the difference between scale-up and scale-out? Scale-up joins GPUs into one tightly coupled domain, as NVLink does for 72 GPUs in an NVL72 rack. Scale-out joins many such domains into a cluster over a network, such as InfiniBand or Ethernet.
How much faster is NVLink than InfiniBand per GPU? About 9 times. NVLink 5 gives 900 GB/s per direction per GPU. An 800 Gb/s InfiniBand port through ConnectX-8 gives 100 GB/s per direction.
Does every GPU in a GB300 NVL72 have its own InfiniBand link? Yes. NVIDIA specifies ConnectX-8 at 800 Gb/s per GPU, which is 72 scale-out ports per rack, plus BlueField-3 DPUs for front-end and storage traffic.
Can Ethernet replace InfiniBand in an NVIDIA AI factory? Yes, for scale-out. NVIDIA offers Spectrum-X Ethernet as an alternative, and Dell'Oro reports that Ethernet was more than double InfiniBand in AI back-end networks in 2025. NVLink remains the scale-up fabric either way.
Can two DGX Stations be joined with NVLink? No. DGX Stations connect over ConnectX-8 networking at up to 2× 400 Gb/s using RoCEv2. NVLink between GPUs is a rack-scale feature of systems like the NVL72.
