NVLink vs InfiniBand vs PCIe: Is Your AI Supercomputer Only as Fast as Its Weakest Connection?
Not quite. An AI system is a ladder of links. Each step down is a lot slower than the one above it, and none of the steps can be skipped. On an NVIDIA Blackwell Ultra GPU, NVLink 5 moves 900 GB/s per direction to other GPUs. InfiniBand through a ConnectX-8 800 Gb/s port moves 100 GB/s per direction to other racks. A PCIe Gen6 x16 host link moves about 121 GB/s per direction. So the slowest link is roughly a ninth the speed of the fastest one. That matters only when the busiest traffic has to cross the slow link. A well-designed system keeps the chattiest traffic on NVLink and sends the less frequent traffic over the network. A badly mapped job does the opposite, and then the weakest connection really does set the pace.
How fast is each link in a Blackwell Ultra system?
| Link | What it connects | Per direction | Bidirectional | NVLink 5 is faster by (per direction) |
|---|---|---|---|---|
| HBM3e (B300 GPU) | GPU to its own memory | 8 TB/s total memory bandwidth (not quoted by direction) | n/a (HBM is about 8.9x NVLink 5's per-direction rate) | |
| NVLink 5 via NVSwitch | GPU to GPU, up to 72 GPUs in an NVL72 rack | 900 GB/s per GPU | 1.8 TB/s per GPU | 1x |
| NVLink-C2C | Grace CPU to GPU on the same superchip | 450 GB/s | 900 GB/s | 2x |
| PCIe Gen6 x16 (B300 host interface) | GPU to host, NIC or other PCIe devices | about 121 GB/s usable | 256 GB/s raw | about 7.4x |
| PCIe Gen5 x16 | Typical x86 server: CPU to GPU card, NIC or SSD | about 63 GB/s usable | 128 GB/s raw | about 14x |
| ConnectX-8, 800 Gb/s (InfiniBand or Ethernet) | GPU to other nodes and racks | 100 GB/s per GPU | 200 GB/s per GPU | 9x |
Source: NVIDIA "Inside Blackwell Ultra" technical blog (22 Aug 2025), NVIDIA NVLink page (updated 15 Sep 2026), NVIDIA Grace Hopper architecture blog (NVLink-C2C), PCI-SIG PCIe 6.0 release (11 Jan 2022), and the GB300 NVL72 page (accessed 6 Nov 2026). Usable PCIe figures are our calculation after encoding or FLIT overhead. 800 Gb/s ÷ 8 = 100 GB/s per direction. All are peak link rates, not measured application throughput.
The units in that table are the point of this post and the rest of the series. NVIDIA quotes NVLink as a bidirectional figure per GPU. Network ports are quoted in bits per second per direction. PCI-SIG headlines are bidirectional and raw. Our companion post, 800Gbps InfiniBand vs 130TB/s NVLink, walks through the conversions step by step.
Why are AI systems built as a ladder of links?
The short answer is distance and power. HBM sits on the GPU package, millimetres from the compute die, which is why it is the fastest. NVLink in an NVL72 rack runs over copper. For the GB200 NVL72, Jensen Huang described a spine of about 5,000 copper NVLink cables, roughly 2 miles in total, and said that copper avoided around 20 kW of optical transceivers and retimers. Those are GB200 figures, but the GB300 uses the same copper-spine design. Copper keeps power down, but it only reaches across a rack, a couple of metres at these speeds.
To leave the rack, traffic goes through ConnectX-8 NICs and optical cables to InfiniBand or Ethernet switches. These reach across a data hall and scale to thousands of GPUs, at a ninth of NVLink's bandwidth per GPU. Each step down trades bandwidth for reach and scale.
When does the weakest link actually slow you down?
When a job's busiest traffic is mapped onto it. Large-model training and inference split the work in several ways, and each one creates a different traffic pattern:
- Tensor parallelism splits each layer across GPUs, so partial results have to be combined with an all-reduce at every layer. NVIDIA's Megatron documentation treats this as traffic to keep inside the NVLink domain.
- Expert parallelism in mixture-of-experts models sends tokens to experts on other GPUs through all-to-all exchanges. These also favour the NVLink domain.
- Pipeline parallelism passes activations from one stage to the next point to point. This is a lighter, steadier load that can cross the network.
- Data parallelism synchronises gradients once per step, usually across nodes over InfiniBand or Ethernet.
Published measurements show what happens when the mapping strains a slower link. Microsoft's DeepSpeed team (Domino, arXiv 2409.15241) found that tensor-parallel communication took 17% to 43% of the GPT-3-13B training iteration time on one to four DGX-H100 nodes, and 22% to 47% across GPT-3 and Llama-2 sizes. In the DeepSeek-V3 technical report, cross-node expert parallelism on H800 GPUs ran at a computation-to-communication ratio of about 1:1. The team gave 20 of each GPU's 132 streaming multiprocessors to communication. Meta reported that its large untuned clusters achieved roughly 10% to 90% of expected communication performance. After topology-aware scheduling, routing changes and NCCL tuning, that rose above 90%.
An AI cluster isn't as fast as its weakest link. It's as fast as the slowest link that its busiest traffic has to cross.
What does the gap look like in seconds?
Take a simple, idealised transfer: 140 GB in one direction, about the size of a 70-billion-parameter model's weights at 16-bit precision. At peak link rates and before any protocol overhead:
- NVLink 5 at 900 GB/s: about 0.16 seconds
- ConnectX-8 at 100 GB/s: about 1.4 seconds
- PCIe Gen6 x16 at about 121 GB/s: about 1.2 seconds
- PCIe Gen5 x16 at about 63 GB/s: about 2.2 seconds
Repeated every step across thousands of iterations, that gap decides whether GPUs compute or wait. This is also why bandwidth isn't the whole story. Small messages are limited by latency. NVIDIA does not publish a raw NVLink hop latency. A July 2026 study (arXiv 2607.16100, with NVIDIA NCCL co-authors) measured a floor of about 1.4 µs for a small all-reduce on GB200 NVL72, and network latencies vary with the fabric and the software stack. Our post 20 Petaflops GPU, Slow Model? covers the other bottlenecks, from memory to batching.
Where does PCIe still matter?
Almost everywhere outside the GPU-to-GPU path. PCIe connects the NICs and the NVMe SSDs that feed training data and hold checkpoints. Blackwell Ultra's host interface is PCIe Gen6 x16, and ConnectX-8 includes a 48-lane PCIe Gen6 switch. On a conventional x86 server with PCIe GPU cards, every CPU-to-GPU copy also crosses PCIe. That path runs at about 63 GB/s per direction on Gen5, and CPU and GPU keep separate page tables, so the program has to copy data across explicitly. Grace-based systems replace that path with coherent NVLink-C2C. Our posts PCIe 6.0 vs NVIDIA NVLink and Why NVIDIA's Most Powerful AI Systems Don't Rely on PCIe Alone go into the detail.
What does this mean for a DGX Station on a desk?
A DGX Station GB300 has one Blackwell Ultra GPU, so there is no GPU-to-GPU NVLink inside it. Its fast internal link is NVLink-C2C between Grace and the GPU, at 900 GB/s bidirectional; the coherent memory explainer covers what that allows. Two Stations connect over ConnectX-8 at 2× 400 Gb/s with RoCEv2. That is 100 GB/s per direction in total, which is network-class bandwidth, not NVLink. So a job split across two Stations should use the network-friendly traffic patterns (pipeline or data parallel), not per-layer tensor parallel. See DGX Station's networking for the cabling.
How do you check a design for a weak link?
- List the traffic each parallelism strategy creates, and on which link it will run.
- Convert every figure to GB/s, per GPU, per direction.
- Check the scale-out fabric's oversubscription. Meta's Llama 3 cluster was non-blocking inside each 3,072-GPU pod but 1:7 oversubscribed between pods.
- Measure with nccl-tests and compare its "bus bandwidth" with the link's rated speed.
- Check storage and PCIe for data loading and checkpointing, which run outside the GPU fabric.
For the scale-out side, our posts NVLink vs InfiniBand: Why NVIDIA Uses Both and ConnectX-7 vs Ordinary Ethernet explain how the network layer is built.
Buying in Singapore
SourceIT does not currently list NVL72 racks, InfiniBand switches or ConnectX NICs. If your project needs them, ask us to source them. For deskside work we list six DGX Station GB300 systems, all price on request: the HP ZGX Fury G1n, MSI XpertStation WS300, ASUS ExpertCenter Pro ET900N G3, GIGABYTE W775-V10-L01, Supermicro ARS-511GD-NB-LCC and Dell Pro Max with GB300. For multi-GPU servers we list the Lenovo ThinkSystem SR680a V3 (B200) and HPE ProLiant DL380a Gen11, also price on request. For the storage tier, the Micron 7600 PRO 3.84 TB PCIe Gen5 SSD is S$5,183.00 (listed 6 Nov 2026). Browse the AI Workstations and AI Servers collection for more. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
Which is fastest: NVLink, InfiniBand or PCIe? NVLink. On Blackwell Ultra it carries 900 GB/s per direction per GPU (1.8 TB/s bidirectional). An 800 Gb/s InfiniBand port carries 100 GB/s per direction, and PCIe Gen6 x16 about 121 GB/s per direction after overhead.
Does a slow link always bottleneck the whole system? No. It bottlenecks the system when frequent, latency-sensitive traffic such as tensor-parallel all-reduces crosses it. Data-parallel and pipeline traffic can run over InfiniBand or Ethernet with much less penalty.
Why not use NVLink for everything? NVLink in an NVL72 runs over copper inside one rack. Reaching across a data hall needs optics, which add cost and power. For the GB200 NVL72, NVIDIA's CEO said copper saved about 20 kW per rack compared with optics.
Where is PCIe still used in NVIDIA AI systems? For NICs, NVMe storage, the GPU's host interface (PCIe Gen6 x16 on Blackwell Ultra) and add-in cards. Grace-based systems use NVLink-C2C instead of PCIe between CPU and GPU.
How do I compare interconnect figures fairly? Convert each one to GB/s, per GPU, per direction. Divide bit rates by 8, divide rack totals by the number of GPUs, and halve bidirectional figures.
