Disaggregated inference splits the two phases of large language model inference, prefill and decode, onto separate groups of GPUs. Prefill reads the whole prompt at once and is limited by raw compute. Decode then generates the answer one token at a time and is limited by memory bandwidth and KV cache capacity. Running both on the same GPU means long prompts interrupt everyone else's answers. Separating them lets each pool be sized and tuned for its own job. NVIDIA's Dynamo inference framework is the best-known implementation, and vLLM, SGLang and TensorRT-LLM support it.
It is a technique for serving many users on multi-GPU systems. For a single desktop AI system, it usually doesn't apply. Knowing why helps you size hardware correctly at both ends.
What are prefill and decode?
| Prefill | Decode | |
|---|---|---|
| What happens | The model reads the entire prompt and builds the KV cache | The model generates the reply token by token |
| Work pattern | Many tokens processed in parallel | One token per user per step |
| Main bottleneck | Compute (FLOPS) | Memory bandwidth and KV cache capacity |
| What users notice | Time to first token (TTFT) | Speed of the streaming answer (inter-token latency) |
| Gets heavier with | Long documents, RAG context, big system prompts | Long answers, reasoning models, many users |
The KV cache links the two. Prefill creates it; decode reads it for every new token.
Why split them onto different GPUs?
When prefill and decode share a GPU, they compete. A user who pastes a 50-page contract triggers a heavy prefill, and every other user's streaming answer stutters while it runs. Serving software softens this with "chunked prefill", slicing long prompts into pieces between decode steps, but NVIDIA's Dynamo documentation notes the trade-off: larger chunks cause fewer but longer interruptions, smaller chunks cause more frequent short ones.
Disaggregation removes the contention. A prefill pool can be tuned for large, compute-heavy batches, and a decode pool for many concurrent sequences and a big KV cache. You can also scale them separately. A legal or research team with long documents needs more prefill; a customer-service bot producing long answers needs more decode.
How does disaggregated inference work?
- A router receives the request and picks a prefill worker, ideally one that already holds part of the prompt in cache.
- The prefill worker processes the prompt and computes the KV cache.
- The KV cache is transferred directly from the prefill GPU's memory to a decode GPU. In Dynamo this is done by the NIXL library over the fastest path available, such as NVLink inside a server or InfiniBand between servers.
- The decode worker generates and streams the answer.
Dynamo calls a layout with x prefill workers and y decode workers "xPyD", and lets workers be added or removed while the system runs. Its router can also decide to run short prompts locally on the decode worker, because sending a tiny KV cache across the network isn't worth it.
When does disaggregation pay off, and when doesn't it?
NVIDIA's own documentation is candid about this.
- At low concurrency, a single combined worker often performs better, because it avoids transfer overhead and splitting the GPUs into pools.
- Under sustained load, where prefill and decode compete, separate pools improve latency isolation.
- When decode KV capacity is the bottleneck, moving GPUs into a prefill pool can reduce total serving capacity, unless the latency gain is worth it.
Source: NVIDIA Dynamo documentation, disaggregated serving design notes (2026). Compiled 11 Nov 2026.
Disaggregation is how large AI services stop long prompts from slowing everyone else down. It needs several GPUs and a fast link between them. On one desktop box, the better tools are chunked prefill and good scheduling.
What does this mean for hardware buyers?
- Single desktop systems such as DGX Spark, GB10 desktops or one RTX PRO 6000 run prefill and decode together. Prefill speed, which depends on compute, and decode speed, which depends on memory bandwidth, are both worth checking. A system that streams answers quickly can still take a long time to start on a long document.
- Multi-GPU servers and DGX Station can split GPUs or partitions into prefill and decode pools. Inside one chassis, fast GPU-to-GPU links make KV transfer cheap.
- Clusters need high-bandwidth, low-latency networking between nodes, because KV caches for long prompts can be gigabytes. See ConnectX-7 vs ordinary Ethernet and whether you need a 400Gbps switch.
- Software maturity matters. Disaggregation adds moving parts: routers, transfer libraries and two pools to monitor. Our vLLM deployment guide covers the single-system baseline to start from.
Related techniques that also tackle decode speed include speculative decoding and FP4 quantization.
Buying in Singapore
For single-system serving, the NVIDIA DGX Spark 4TB (S$9,450) and NVIDIA RTX PRO 6000 Blackwell Workstation Edition (S$29,100). For multi-GPU servers where prefill and decode can be split, the RTX PRO 6000 Blackwell Server Edition (S$32,999) and, for clustering GB10 systems, the HP ZGX 0.4m QSFP112 DAC cable (S$365) (listed 11 Nov 2026; prices change; ask for a current quotation). DGX Station GB300 systems such as the ASUS ExpertCenter Pro ET900N G3 and GPU servers such as the Lenovo ThinkSystem SR675 V3 are price on request. To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
What is disaggregated inference? Disaggregated inference runs the prefill phase (reading the prompt) and the decode phase (generating the answer) of LLM inference on separate pools of GPUs, so each can be sized and tuned for its own bottleneck.
What is the difference between prefill and decode? Prefill processes the whole prompt in parallel and is compute-bound; it sets time to first token. Decode generates tokens one at a time and is limited by memory bandwidth and KV cache capacity.
What is NVIDIA Dynamo? NVIDIA Dynamo is an open inference serving framework that supports disaggregated serving, KV-cache-aware routing and KV transfer between GPUs through the NIXL library, with vLLM, SGLang and TensorRT-LLM backends.
Do I need disaggregated inference on a DGX Spark? Usually not. On a single system, prefill and decode share the GPU, and NVIDIA notes that at low concurrency a combined worker often performs better. Disaggregation suits multi-GPU servers and clusters under heavy load.
What hardware does disaggregated inference need? Several GPUs or GPU partitions plus a fast link to move KV caches between them, such as NVLink inside a server or InfiniBand or high-speed Ethernet between servers.
SourceIT Approved
Researched and verified by the SourceIT team for Singapore buyers.
