Because PCIe can't do the jobs these systems need most, which are moving data between GPUs fast enough, sharing memory coherently with the CPU, and joining dozens of GPUs into one domain. So NVIDIA puts three other links on top of PCIe. NVLink 5 joins GPU to GPU at 1.8 TB/s bidirectional per GPU, which NVIDIA describes as 14 times PCIe Gen5 x16. NVLink-C2C joins CPU to GPU at 900 GB/s bidirectional, with hardware coherence. ConnectX networking over InfiniBand or Ethernet joins rack to rack. PCIe has not disappeared. It still connects the NICs, the SSDs and the GPU's host interface, which is PCIe Gen6 on Blackwell Ultra. It is simply no longer the main road.
Which link does what in a GB300 NVL72 rack?
| Link | Connects | Rated bandwidth | What PCIe would lack in this role |
|---|---|---|---|
| NVLink 5 + NVLink Switch | All 72 Blackwell Ultra GPUs in the rack | 1.8 TB/s bidirectional per GPU (900 GB/s per direction). 130 TB/s aggregate across 72 GPUs. | About 7x to 14x the bandwidth, switching across 72 GPUs, and in-network reductions |
| NVLink-C2C | Each Grace CPU to its GPUs | 900 GB/s bidirectional (450 GB/s per direction) | Hardware coherence and a shared page table |
| PCIe Gen6 x16 | GPU host interface, plus the ConnectX-8's built-in 48-lane switch | 256 GB/s bidirectional raw (about 121 GB/s per direction usable) | Nothing. This is PCIe doing its normal I/O job. |
| ConnectX-8 SuperNIC | Each GPU to the scale-out InfiniBand or Ethernet fabric | 800 Gb/s per GPU, or 100 GB/s per direction | Reach. PCIe does not leave the server. |
| BlueField-3 DPU | Front-end network, storage and management traffic | Depends on configuration | Offload and isolation from the GPU fabric |
Source: NVIDIA GB300 NVL72 page (accessed 6 Nov 2026), NVIDIA "Inside Blackwell Ultra" blog (22 Aug 2025), NVIDIA Grace Hopper architecture blog (NVLink-C2C), NVIDIA NVL72 AI Factory reference architecture (updated 15 Sep 2026) and NVIDIA ConnectX-8 blog (18 May 2025). The bandwidth multiples are our calculation per direction against PCIe Gen6 and Gen5 x16.
Reason one: is PCIe simply too slow between GPUs?
For this job, yes. NVIDIA's "14x PCIe Gen5" claim compares like with like: 1,800 GB/s bidirectional for NVLink 5 against 128 GB/s bidirectional for PCIe Gen5 x16, which works out to 1,800 ÷ 128 ≈ 14. Against the PCIe Gen6 x16 link that Blackwell Ultra itself has, the multiple is about 7. For Vera Rubin, NVIDIA says NVLink 6 is "12x PCIe Gen6", which is 3,000 GB/s ÷ 256 GB/s ≈ 11.7, both bidirectional. Large-model training and inference split each layer across GPUs and combine the partial results every layer, so GPU-to-GPU bandwidth is used continuously, not occasionally. Microsoft's DeepSpeed team measured tensor-parallel communication at 17% to 47% of iteration time on DGX-H100 systems. That is exactly the kind of overhead a faster link is meant to reduce.
Reason two: why does coherence matter?
NVLink-C2C is a hardware-coherent link. With Address Translation Services, the Grace CPU and the GPU share one page table per process. The CPU can cache GPU memory a cache line at a time, and both can access CPU-resident and GPU-resident memory "concurrently and transparently", in NVIDIA's words. On a conventional x86 server with PCIe GPUs, CPU and GPU keep separate page tables. Data in system memory has to be copied across explicitly, at about 63 GB/s per direction on PCIe Gen5 x16. C2C runs at 450 GB/s per direction, about seven times faster, and removes much of the copying. This is what lets a GB300 treat CPU memory as a slower extension of GPU memory. Our posts DGX Station coherent memory explained and Inside NVIDIA GB300's 748GB coherent memory cover the practical effect.
Reason three: why does NVIDIA need its own switch?
PCIe is a tree rooted at a host CPU, and devices behind one switch share its uplinks. NVIDIA needed something that behaves like a flat, non-blocking crossbar for 72 GPUs. Each NVLink 5 Switch chip has 72 ports and 7.2 TB/s of bidirectional all-to-all bandwidth. Two chips sit in each switch tray, and nine trays make 18 chips per NVL72 rack. Every GPU has 18 NVLink links, one to each switch chip, so any GPU reaches any other in a single switch hop. The switches also run SHARP engines that perform reductions and multicast inside the network. That cuts the data GPUs have to send for collective operations (NVIDIA's description). Mixture-of-experts models, which send tokens to experts on other GPUs, depend on this all-to-all reach. NVIDIA claims up to 1.8 times more throughput per GPU on DeepSeek-R1 when expert parallelism widens from 8 to 32 GPUs on NVL72.
PCIe was designed to attach devices to a computer. NVLink was designed to make many GPUs behave like one computer.
Reason four: what about power and cabling?
Moving terabytes per second costs energy. NVIDIA rates NVLink-C2C at 1.3 picojoules per bit and claims it is more than five times as energy efficient as PCIe Gen5. At its full 900 GB/s, that rating works out to roughly 9 W for the link itself (our calculation). In the rack, NVIDIA chose copper over optics for the NVLink spine. For the GB200 NVL72, Jensen Huang described about 5,000 copper NVLink cables and said that optics would have needed roughly 20 kW of transceivers and retimers. The GB300 NVL72 uses the same copper-spine design. The trade-off is reach: copper keeps NVLink inside one rack, which is why scale-out still runs over InfiniBand or Ethernet. See NVLink vs InfiniBand: Why NVIDIA Uses Both.
Where does PCIe still matter in NVIDIA's systems?
- Host and I/O. Blackwell Ultra's host interface is PCIe Gen6 x16. ConnectX-8 includes a 48-lane PCIe Gen6 switch, so NICs, GPUs and SSDs can talk without going through the CPU.
- Storage. Every NVMe drive that feeds datasets and stores checkpoints is a PCIe device.
- Add-in cards. In a DGX Station GB300, the add-in card slots are PCIe Gen5 (one x16 and two more x16 slots wired x8). Two M.2 slots run at Gen6 x4 off the ConnectX-8. Adding an RTX card lowers the GB300 superchip's power cap to 1,250 W, or to 950 W for a card above 300 W such as the RTX PRO 6000, which is a trade-off to plan for.
- PCIe GPU servers and workstations. RTX PRO cards connect over PCIe. That works well when each card runs its own job, and less well when one model is split across cards. See PCIe 6.0 vs NVIDIA NVLink.
Is this also a lock-in strategy?
That is the critics' view, and it deserves a fair hearing. NVLink is proprietary. ABI Research called UALink "an open alternative to NVLink's single-vendor proprietary solution" and also said it was "years behind". NVIDIA's answer is NVLink Fusion, announced on 18 May 2025, which licenses NVLink to third-party CPUs and accelerators. Partners include MediaTek, Marvell, Fujitsu and Arm, and AWS is designing Trainium4 to integrate with NVLink 6. Open efforts are moving too. UALink 2.0 was published on 7 April 2026, and Ultra Ethernet 1.1 added a scale-up profile on 12 October 2026. Dell'Oro expects Ethernet to become the leading scale-up technology by the end of the decade. Our posts Is NVLink NVIDIA's Biggest Competitive Advantage? and NVLink vs InfiniBand vs PCIe take this further.
What should buyers take from this?
If your models fit on one GPU, PCIe is not your constraint and a well-configured PCIe workstation or server is a sound choice. If one model has to span several GPUs at high speed, as in large training runs, big mixture-of-experts models or high-volume serving, the NVLink domain is what you are paying for. A DGX Station sits in between. It has one GB300 with coherent C2C to 748 GB of memory, but no NVLink to other units, so two Stations work together over ConnectX-8 networking instead.
Buying in Singapore
SourceIT lists six DGX Station GB300 systems, all price on request: the HP ZGX Fury G1n, MSI XpertStation WS300, ASUS ExpertCenter Pro ET900N G3, GIGABYTE W775-V10-L01, Supermicro ARS-511GD-NB-LCC and Dell Pro Max with GB300. For PCIe-based builds, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition is S$29,100.00 (listed 6 Nov 2026). For multi-GPU servers we list the Lenovo ThinkSystem SR780a V3, price on request. We do not currently list NVL72 racks or ConnectX NICs; ask us to source them. See the AI Workstations and AI Servers collection. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
Does NVIDIA still use PCIe at all? Yes. Blackwell Ultra's host interface is PCIe Gen6 x16, ConnectX-8 includes a PCIe Gen6 switch, and SSDs and add-in cards are PCIe devices. GPU-to-GPU and CPU-to-GPU traffic in Grace systems goes over NVLink and NVLink-C2C instead.
Where does "14x PCIe" come from? NVLink 5's 1.8 TB/s bidirectional per GPU divided by PCIe Gen5 x16's 128 GB/s bidirectional is about 14. Against PCIe Gen6 x16 (256 GB/s), the multiple is about 7.
What does NVLink-C2C add over PCIe? About seven times the bandwidth (450 GB/s against about 63 GB/s per direction for Gen5 x16), plus hardware coherence. The CPU and GPU share a page table, so the GPU can use CPU memory without explicit copies.
Is NVLink open to other chipmakers? Partly. Through NVLink Fusion, announced in May 2025, partners such as MediaTek, Marvell, Fujitsu and Arm can build CPUs or accelerators that join an NVLink domain. AWS is designing Trainium4 to use it. UALink and Ultra Ethernet are the open alternatives.
Do I need NVLink for a workstation? Only if a single model must span several GPUs at high speed. For models that fit on one GPU, or for several independent jobs on separate cards, PCIe-connected cards work well.
