Seventy-two GPUs in one NVIDIA GB300 NVL72 rack and 72 GPUs spread across nine servers can have the same peak compute on paper. What changes is how fast each GPU can reach the other 71. In an NVL72, every GPU talks to every other GPU over NVLink at 900 GB/s in each direction, through a single switch hop. In nine 8-GPU servers, each GPU gets that speed only to the seven GPUs in its own server. Traffic to the other 64 goes over a network card: about 100 GB/s per direction with an 800 Gb/s ConnectX-8, roughly a ninth of the NVLink figure. With 72 PCIe cards the gap widens further. For workloads that split one model across many GPUs, that difference decides whether the GPUs compute or wait.
How do the three ways of connecting 72 GPUs compare?
| Measure | GB300 NVL72 (one rack) | 9 x 8-GPU NVLink servers (e.g. HGX B300) on an 800G fabric | 72 PCIe GPUs (e.g. 9 x 8-card servers) |
|---|---|---|---|
| Largest NVLink domain | 72 GPUs | 8 GPUs | None; cards connect over PCIe |
| GPU-to-GPU bandwidth inside the domain or server | 900 GB/s per direction to any of 71 peers | 900 GB/s per direction to 7 peers | About 63 GB/s per direction (PCIe Gen5 x16), often shared through switches or the CPU |
| Bandwidth to GPUs in other servers | Not applicable inside the rack | About 100 GB/s per direction per GPU (ConnectX-8, 800 Gb/s) | Depends on NICs; a 400 Gb/s NIC is 50 GB/s per direction, often shared by several GPUs |
| Aggregate GPU-to-GPU injection, per direction (calc) | 72 x 900 GB/s ≈ 64.8 TB/s | 72 x 100 GB/s ≈ 7.2 TB/s across servers | Lower still |
| Theoretical minimum time to all-reduce 1 GB across 72 GPUs (calc, link peak) | About 2.2 ms | About 19.7 ms on a flat ring at 100 GB/s | About 39 ms at 50 GB/s |
| Cables between GPUs | Copper NVLink spine inside the rack | One network port per GPU, plus leaf and spine switches and optics | As for the middle column, usually with fewer ports |
| Power and cooling | About 132–142 kW in one liquid-cooled rack (OEM figures) | About 126 kW across three racks for nine DGX B300 systems (calc from NVIDIA's ~56 kW per four-system rack) | Varies widely with the card |
Source: NVIDIA NVLink page (updated 15 Sep 2026) and GB300 NVL72 page (accessed 6 Nov 2026); PCI-SIG specifications; NVIDIA ConnectX-8 documentation; NVIDIA B300 XDR SuperPOD reference architecture (updated 2 Sep 2026); Supermicro and Vertiv rack figures. All-reduce times are our arithmetic: 1 GB x 2(n−1)/n ÷ link bandwidth, with n = 72. They are lower bounds at peak link speed, not measurements. NCCL's hierarchical algorithms reduce inside each server first, which narrows the gap for the middle column but does not close it.
Why does connection speed matter more than GPU count?
Because large models are split across GPUs, and every split creates traffic. NVIDIA's parallelism guidance describes the patterns. Tensor parallelism combines partial results with an all-reduce at every layer. Expert parallelism for mixture-of-experts models runs an all-to-all exchange in every transformer block. Pipeline parallelism passes activations from stage to stage, and data parallelism exchanges gradients once per step. The first two are frequent and latency-sensitive, so they need the fastest link available. In nine separate servers, tensor parallelism is in practice capped at eight GPUs, and wide expert parallelism has to cross the network.
Published measurements show how big that cost can be. DeepSeek reported that cross-node expert parallelism on H800 servers gave a computation-to-communication ratio of about 1:1. In other words, roughly as much time went to moving tokens as to computing on them. They gave 20 of each GPU's 132 SMs to communication. Microsoft's DeepSpeed team measured communication at 17% to 47% of training iteration time on one to four DGX H100 nodes. An NVL72 does not remove this traffic, but it moves far more of it onto a link about nine times faster per GPU than the network.
Seventy-two GPUs in separate servers are 72 fast processors on a slower network. Seventy-two GPUs in an NVL72 behave much more like one very large processor, because the slow hop has been moved outside the rack.
Does the bandwidth number tell the whole story?
No. Three other things change when you connect GPUs in one rack.
- Latency for small messages. During inference, many collectives are small and frequent. A July 2026 study co-authored by NVIDIA's NCCL engineers measured a small all-reduce floor of about 1.4 µs on GB200 NVL72, regardless of how many GPUs took part. They estimated that each microsecond saved cuts the cost per million output tokens by about 0.9% on a 4-GPU setup. NVIDIA publishes no comparable end-to-end figure for InfiniBand or Ethernet, and network latency depends heavily on the fabric and its tuning.
- Fewer network parts. Inside the rack, NVL72 uses a copper NVLink spine. Nine separate servers need a network port per GPU, plus leaf and spine switches, plus optics and cables between racks. Meta's Llama 3 405B paper, covering training on up to 16,384 H100 GPUs, reported 419 unexpected interruptions in a 54-day period. Of those, 35 were network switch or cable faults and 7 were NIC faults, about 10% of the total. Fewer optical links means fewer of those failure points, though an NVL72 rack is still a large single unit to keep healthy.
- Memory reach. In an NVL72, a GPU can access the memory of any other GPU in the rack at NVLink speed. That makes the rack's roughly 20 TB of HBM3E behave far more like one pool. Our post on 37 TB of memory and 130 TB/s NVLink explores what that enables.
When are 72 separate GPUs good enough?
Often. Many workloads do not split one model across dozens of GPUs:
- Many independent inference replicas, each on one GPU or one 8-GPU server, behind a load balancer.
- Models that fit inside one server. If an 8-GPU NVLink server holds the model and its KV cache, tensor parallelism never leaves the server.
- Data-parallel training of small and mid-size models, where gradients are exchanged once per step and overlap with compute.
- Batch jobs, rendering and simulation that run per GPU.
Separate servers are also easier to buy, house and cool in stages. NVL72 pays off for the largest dense models, for wide mixture-of-experts serving at high interactivity, and for training runs where communication dominates. To see where the network fits around the rack, read why NVIDIA uses both NVLink and InfiniBand. For how the 18 NVSwitch chips make the single-hop domain, read 72 GPUs, one AI supercomputer.
How should a buyer frame the decision?
- Profile the workload first. Measure how much of a training step or serving request is spent in communication on your current system. If it is small, more GPUs on a good network may be enough.
- Check the largest model you will run against one 8-GPU server's memory. If it fits with room for KV cache, the NVL72's main advantage shrinks.
- Count the whole fabric. Nine servers need switches, optics and cabling that an NVL72 rack partly replaces. Our post on the hidden cost of AI networking breaks this down.
- Check the facility. A 132–142 kW liquid-cooled rack needs a data hall built for it; separate air-cooled servers may fit an existing colocation contract.
Buying in Singapore
SourceIT does not list GB300 NVL72 racks or HGX B300 systems. For those projects, ask us to source through our partners; we will confirm availability and lead times rather than assume stock. For multi-GPU servers we list the Lenovo ThinkSystem SR780a V3 (B200 or H100 with NVLink switches), the Lenovo ThinkSystem SR680a V3 (B200) and the HPE ProLiant DL380a Gen11, all price on request. For PCIe GPU builds, the RTX PRO 6000 Blackwell Workstation Edition is S$29,100.00 (listed 6 Nov 2026). For smaller fabrics, the NVIDIA Spectrum-2 SN3700 32-port 200GbE switch is S$28,523.25 (listed 6 Nov 2026). It is a 200GbE switch, not an 800G AI fabric. For deskside work on the same GB300 superchip, see the DGX Spark and DGX Station collection, price on request. For InfiniBand, ConnectX-8 or 800G optics, which we do not list, ask us to source. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
Is an NVL72 faster than 72 GPUs in separate servers? For workloads that split one model across many GPUs, usually yes, because each GPU reaches the other 71 at 900 GB/s per direction instead of about 100 GB/s over an 800 Gb/s NIC. For independent jobs that run per GPU or per server, the difference is small.
Why can't tensor parallelism span nine 8-GPU servers efficiently? Tensor parallelism needs an all-reduce at every layer. Across servers, that traffic runs at network speed, about a ninth of NVLink per GPU, so most deployments keep tensor parallelism within the eight GPUs of one server.
How much bandwidth does a PCIe GPU have to its neighbours? A PCIe Gen5 x16 slot carries about 63 GB/s per direction after encoding overhead, and traffic between cards often shares switches or the CPU. That is about a fourteenth of NVLink 5's 900 GB/s per direction.
Does NVL72 remove the need for InfiniBand or Ethernet? No. NVLink connects the 72 GPUs inside the rack. Each GPU still has an 800 Gb/s ConnectX-8 for traffic between racks, storage and users.
Are the all-reduce times in the table measured? No. They are theoretical minimums at peak link bandwidth, using the standard all-reduce factor of 2(n−1)/n. Real collectives add latency and protocol overhead, and NCCL's hierarchical algorithms change the picture for multi-server setups.
