Speculative decoding is a technique that makes a large language model generate text faster without changing what it writes. A small, fast "draft" model guesses the next few tokens, and the large model checks all of those guesses in a single pass. Correct guesses are kept, and the first wrong one is replaced by the large model's own choice. Because the large model verifies every token, the output follows the same distribution as normal generation. Published results with the EAGLE family of methods show speedups of about 2 to 3 times in common serving software, with the biggest gains when only a few users are active.
For anyone running AI on their own hardware, it is one of the cheapest upgrades available. It is a software setting, not a new GPU.
Why is normal AI text generation slow?
Language models write one token at a time, and each token depends on the ones before it. For every token, the system reads the model's active weights from memory. With one user, the GPU spends most of its time waiting for memory rather than computing; it is memory-bound. That spare compute is what speculative decoding puts to work. Checking five proposed tokens in one pass costs little more than generating one, because the weights are read only once.
How does speculative decoding work, step by step?
- Draft. A small model, or a lightweight prediction head attached to the main model, proposes the next few tokens, for example five.
- Verify. The large model processes all five in one forward pass and works out what it would have chosen at each position.
- Accept. Tokens are accepted up to the first one the large model disagrees with.
- Correct. At the first mismatch, the large model's own token is used instead, and drafting starts again from there.
If three of five guesses are accepted, the system produces four tokens (three accepted plus the correction) for roughly the cost of one large-model step plus the cheap drafting. The share of guesses accepted, called the acceptance rate, decides almost everything.
What are the main types of speculative decoding?
| Method | How it drafts | Pros | Cons |
|---|---|---|---|
| Separate draft model | A smaller model from the same family, e.g. a 1B drafting for a 70B | Simple to understand; no training needed if a matching small model exists | Uses extra memory; must share the same tokenizer |
| EAGLE / EAGLE-3 | A small head trained to predict ahead from the main model's internal state | High acceptance rates; widely supported | Needs a trained head for each target model |
| Medusa-style heads | Several extra output heads predicting tokens further ahead | No separate model | Requires training the heads |
| N-gram / prompt lookup | Copies likely continuations from the prompt itself | Free; great for editing and code where output repeats input | Little gain on new content |
| Parallel drafting (P-EAGLE) | Drafts all guesses in one pass instead of one by one | Removes the drafter's own delay | Newer; fewer ready-made heads |
How much faster is speculative decoding in practice?
- Red Hat reported that EAGLE-3 in vLLM gave up to 2.5 times faster inference across a range of scenarios (July 2025).
- AWS describes EAGLE as achieving 2 to 3 times speedups and being widely deployed in vLLM, SGLang and TensorRT-LLM. Its parallel-drafting variant, P-EAGLE, added 55 to 69 percent more throughput than EAGLE-3 with one user on an NVIDIA B200, but only 5 to 25 percent at 64 concurrent users (March 2026).
- NVIDIA lists speculative decoding as one of the techniques Jetson Thor uses to speed up generative AI on robots.
Source: Red Hat Developer, Jul 2025; AWS Machine Learning Blog and vLLM blog, Mar 2026; NVIDIA developer blog, Aug 2025. Compiled 11 Nov 2026.
The pattern is consistent: big gains for a few users, small gains under heavy load. Once a server is busy, the GPU is already doing useful work in its spare compute by batching many users together, so there is less idle capacity for verification. Our test of 20 employees sharing one GB10 shows that side of the trade-off.
Speculative decoding helps most exactly where small businesses run AI: one person or a handful of users on a single desktop AI system. It is a free speed-up for a single-user box and a minor one for a busy server.
When does speculative decoding not help?
- High concurrency. With many simultaneous users, gains shrink and can disappear.
- Poorly matched drafts. If the draft model often guesses wrong, for example on specialised jargon it wasn't trained on, the extra work isn't repaid. Training a draft head on your own outputs raises acceptance.
- Creative, high-temperature output. More random sampling means fewer guesses match.
- Tight memory. A separate draft model needs its own memory and KV cache. On a system that is already full, it may not fit.
How do you turn it on?
In vLLM, speculative decoding is configured with the --speculative-config option, choosing a method such as eagle3, a draft model or head, and the number of tokens to speculate. TensorRT-LLM and SGLang offer equivalent settings, and NVIDIA publishes ready-made EAGLE-3 heads for several popular models. Start with three to five speculative tokens, measure tokens per second and the acceptance rate on your own prompts, then adjust. Our vLLM deployment guide for DGX Station covers the serving setup.
Does speculative decoding change which hardware to buy?
It shifts the balance a little. Because it uses spare compute to beat the memory-bandwidth limit, it helps bandwidth-limited systems such as 128 GB GB10 desktops make better use of their GPU. It doesn't remove the need for enough memory to hold the main model, the draft and the KV cache. Combined with FP4 quantization and MoE models, it is part of why a desktop system can now run models that needed a server two years ago.
Buying in Singapore
For single-user and small-team systems where speculative decoding helps most, the NVIDIA DGX Spark 4TB (S$9,450), ASUS Ascent GX10 128GB/1TB (S$10,350) and HP ZGX Nano G1n 4TB (S$11,499). For workstation builds, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition (S$29,100) and the ASUS TUF Gaming GeForce RTX 5090 32GB (S$7,467.50). For robots, the Jetson AGX Thor 128GB Developer Kit (S$9,075.60) (listed 11 Nov 2026; prices change; ask for a current quotation). To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
What is speculative decoding? Speculative decoding speeds up LLM text generation by having a small draft model guess several tokens ahead and the large model verify them in one pass. Accepted tokens are kept, so output quality is unchanged.
Does speculative decoding reduce accuracy? No. The large model checks every token and replaces any wrong guess with its own choice, so the output follows the same distribution as normal generation.
How much faster is speculative decoding? Published results with EAGLE-3 show about 2 to 3 times faster generation at low concurrency in vLLM, SGLang and TensorRT-LLM. Gains shrink as the number of simultaneous users rises.
What is EAGLE-3? EAGLE-3 is a speculative decoding method that attaches a lightweight prediction head to the main model to draft tokens. It achieves high acceptance rates and is supported by major inference engines.
When should I not use speculative decoding? When a server already handles many concurrent users, when memory is too tight for a draft model, or when output is highly random or specialised and guesses are rarely accepted.
SourceIT Approved
Researched and verified by the SourceIT team for Singapore buyers.
