Your GPU Has 20 Petaflops of Power. So Why Is Your AI Model Still Slow?

Because petaflops measure only how fast the GPU can do arithmetic, not how fast it can be fed. The 20 PFLOPS on an NVIDIA DGX Station GB300 is NVIDIA's FP4 rating with sparsity (about 15 PFLOPS dense), and very few AI workloads run at that rate. Most of the time, a model waits on something else: memory bandwidth, the link between CPU and GPU, storage, the software stack, or the network. Each of those has its own speed limit, and the slowest one sets the pace. This guide takes them one at a time, so you can work out which one is holding your model back before you spend money on more compute.

Where does the data actually flow on a GB300 workstation?

Every token a model generates and every training step it takes moves data through a chain of links. On a DGX Station-class system, the chain looks like this:

Source: NVIDIA DGX Station and Grace Hopper/Blackwell Ultra technical material; measured figures from StorageReview (MSI XpertStation WS300, 24 Aug 2026) and ServeTheHome. PCIe per-direction figure calculated from the PCI-SIG rate after 128b/130b encoding. Peak ratings are not measured throughput.

The gap from top to bottom is enormous. HBM3e is about 18 times faster than the LPDDR5X tier, the CPU–GPU link is about 7 times faster than a PCIe Gen5 x16 slot, and the network delivers about 1/71 of HBM bandwidth. A model that touches the slower stages often will run at their speed, not the GPU's.

Bottleneck 1: is memory bandwidth starving the GPU?

For chatbots and agents, this is usually the first wall. NVIDIA's own inference guidance describes the decode phase as "a memory-bound operation": to generate each token, the GPU must read every active weight from memory, and the arithmetic per byte is tiny. A simple ceiling makes the point. A dense 70-billion-parameter model at 8-bit is about 70 GB of weights. At 7.1 TB/s, the GPU can read those weights about 101 times a second, so a single user can never see more than roughly 101 tokens per second, however many petaflops sit idle. At 4-bit the ceiling doubles to about 203. If the same weights sat in the 396 GB/s LPDDR5X tier, the ceiling would be about 6 tokens per second at 8-bit.

These are theoretical upper bounds that ignore KV cache reads and software overhead, but they explain why tokens per second track memory bandwidth more than petaflops. Batching many users together is what turns idle compute into throughput. StorageReview's 9,280 output tokens per second for gpt-oss-120b came from 128 concurrent requests, not one.

Bottleneck 2: is data crossing a slow CPU–GPU link?

On a conventional workstation, the GPU sits on a PCIe slot. Anything the CPU prepares, such as tokenised text, images, a dataframe, or a slice of a model that doesn't fit in VRAM, has to be copied across roughly 63 GB/s each way on PCIe Gen5 x16. Copying 70 GB takes over a second, every time. On GB300, the same transfer uses NVLink-C2C at 450 GB/s each way, about 0.16 seconds, and the GPU can read CPU memory directly instead of waiting for an explicit copy. Our post on why NVIDIA doesn't rely on PCIe alone covers that design choice. The catch is that C2C only moves data as fast as the LPDDR5X behind it, about 390 GB/s measured.

Bottleneck 3: is storage slowing loading and checkpoints?

Storage rarely limits token generation once a model is loaded, but it dominates start-up, dataset streaming and checkpointing. At the 13.1 GiB/s StorageReview measured with GPUDirect Storage, loading 140 GB of FP16 weights takes about 10 seconds in the best case. Ordinary file reads through the CPU, slower drives or a network share can stretch that to minutes. Training adds checkpoints, which hold weights plus optimizer state and can be several times the model size. Our SSD storage architecture guide explains the two Gen5 and two Gen6 M.2 slots and why drive choice matters.

Bottleneck 4: is the software leaving performance on the table?

The same hardware can behave very differently depending on the inference engine, precision and settings. Common causes of a slow model include:

• No batching or a small batch: one request at a time leaves most compute idle.
• The wrong precision: FP4 and FP8 halve or quarter the bytes per weight compared with FP16, which raises the decode ceiling directly. Not every model or engine supports them well.
• Kernels that don't use the hardware: an engine without Blackwell-optimised kernels can't reach the FP4 tensor-core rate.
• Memory placement: on a coherent system, data lands in HBM or LPDDR5X on first touch. A badly placed KV cache can quietly run at LPDDR5X speed.
• CPU-side preprocessing: tokenisation, image decoding or Python glue code that can't keep up with the GPU.

This is why two DGX Stations with the same 748 GB can still perform differently: reviewers found the hardware within about 2% across OEMs, so most of the variation comes from configuration.

A GPU is only as fast as the slowest thing feeding it. Find that link before you buy more petaflops.

Bottleneck 5: is the network the slowest link?

A single DGX Station has no NVLink to other GPUs. Two stations can be linked over ConnectX-8 at 2x 400 Gb/s, which is 100 GB/s each way in total, or roughly a ninth of what one Blackwell GPU gets over NVLink in a rack (900 GB/s each way). Splitting one model's layers across two machines (tensor parallelism) puts network traffic inside every layer, which is why that technique is normally kept inside an NVLink domain. In the data centre, published research shows the cost: Microsoft's DeepSpeed team found tensor-parallel communication took 17% to 47% of iteration time on DGX H100 nodes, and DeepSeek reported a compute-to-communication ratio of about 1:1 for cross-node expert parallelism. See 800 Gbps InfiniBand vs 130 TB/s NVLink for how those numbers compare, and DGX Station's networking for what the two QSFP112 ports are good for.

How do you work out which bottleneck you have?

• Check GPU utilisation and memory bandwidth together. Low tensor-core use with high memory traffic during generation means you're memory-bound. Batch more requests or use a lower-precision model.
• Time the first token separately from the rest. A slow first token points to prefill compute, model loading or long prompts. Slow later tokens point to memory bandwidth.
• Watch where memory is allocated. If the working set exceeds 252 GB, part of it lives in LPDDR5X at about 396 GB/s.
• Profile the input pipeline. If the GPU waits between batches, the CPU, storage or data loader is behind.
• Measure collective time on multi-GPU jobs. NCCL tests report bus bandwidth (for all-reduce, algorithm bandwidth × 2(n−1)/n), which can be compared directly with the link's rated speed.

For the peak figures themselves and what they can and can't tell you, read What Can 20 Petaflops Actually Do? and NVLink vs InfiniBand vs PCIe: where the weakest connection is.

Buying in Singapore

SourceIT lists six DGX Station GB300 systems, all price on request: the HP ZGX Fury G1n, MSI XpertStation WS300, ASUS ExpertCenter Pro ET900N G3, GIGABYTE W775-V10-L01, Supermicro ARS-511GD-NB-LCC and Dell Pro Max with GB300. All are in our NVIDIA DGX collection. If storage is your bottleneck, ask about drive options at quotation; for shared datasets on a server, the Micron 9550 PRO 7.68TB Gen5 E3.S (S$10,456, listed 6 Nov 2026) is available here. To size a system around your actual bottleneck, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

Why doesn't 20 petaflops mean 20 times faster AI? The figure is NVIDIA's FP4 peak with sparsity. Real workloads are usually limited by memory bandwidth, data movement or software long before they reach peak arithmetic. Measured matrix-multiply results on a DGX Station are well below the rating.

Is token generation limited by compute or memory? Mostly memory. Each generated token requires reading the model's active weights, so single-user speed is capped by memory bandwidth divided by model size. Batching many users is how compute gets used.

Does a faster SSD make my chatbot respond faster? Usually not once the model is loaded. Storage speed matters for loading models, streaming training data and saving checkpoints.

Why is LPDDR5X slower than HBM3e if both are coherent? Coherence means the GPU can address both, not that both run at the same speed. HBM3e is rated at 7.1 TB/s and LPDDR5X at 396 GB/s, about 18 times slower.

Will linking two DGX Stations double my speed? Only for work that splits cleanly, such as running separate jobs or data-parallel training. Workloads that split one model across both machines are limited by the 2x 400 Gb/s network, which is far slower than NVLink inside a rack.