What Is KV Cache? Why Long AI Conversations Need More GPU Memory

The KV cache is the memory a language model uses to remember the conversation so far. As a model reads a prompt, it computes a "key" and a "value" for every token at every layer, and stores them so it doesn't have to recompute them for each new word it writes. The cache grows with every token of context, and every active user needs their own. That is why long conversations, long documents and many simultaneous users can run a GPU out of memory even when the model itself fits comfortably. For Llama 3.1 70B, a single 128,000-token conversation needs about 40 GB of KV cache at 16-bit precision, more than the 35 GB the model's weights take at 4-bit.

The context window is the maximum number of tokens a model can consider at once. The KV cache is what that window actually costs in memory.

Context window vs KV cache vs model weights

A model that advertises a 128K or 1M-token context window can accept that much text. Whether your hardware can hold the KV cache for it is a separate question.

How do you calculate KV cache size?

The standard formula for a transformer is:

KV cache per token = 2 × layers × KV heads × head dimension × bytes per value

The 2 is for keys and values. Most modern models use grouped-query attention, where several attention heads share one set of keys and values, which cuts the KV heads, and the cache, considerably.

CALC from the published Llama 3.1 architecture: 2 × layers × 8 × 128 × 2 bytes. One sequence; multiply by the number of concurrent users. FP8 KV cache halves these figures.

How many users can share one GPU?

Take a 70B model at 4-bit (about 35 GB of weights) on a 96 GB GPU. After weights and roughly 6 GB of runtime overhead, about 55 GB (roughly 51 GiB) is left for KV cache.

CALC; illustrative only. Real capacity depends on the serving software, average rather than maximum context, and prefix sharing. Overhead figure is an assumption.

That table explains a common surprise. A system that served 20 staff comfortably for short chats struggles when the same people start pasting long reports. Our test of 20 employees on one GB10 shows how context length changes the picture.

Model size decides whether an AI system starts. KV cache decides how many people can use it, and how much they can paste in, before it slows down or runs out of memory.

What makes the KV cache grow fastest?

• RAG and document chat. Retrieval adds pages of context to every question.
• Reasoning models. Thousands of thinking tokens per answer also go into the cache. See test-time compute.
• Images and video. Each image becomes many tokens. See multimodal AI.
• AI agents. Tool results and long histories accumulate over many steps.
• Concurrency. Every active user multiplies everything above.

How can you reduce KV cache memory?

1. Quantize the cache. FP8 KV cache halves memory with little quality loss on most tasks, and lower-bit formats are emerging. See quantization formats.
2. Use paged attention. Engines such as vLLM allocate cache in small pages, so memory isn't reserved for context that is never used.
3. Reuse prefixes. If every request starts with the same system prompt or document, prefix caching stores it once.
4. Cap context sensibly. Set the maximum context to what users actually need, not the model's advertised maximum.
5. Retrieve less, better. Five well-chosen passages beat fifty loosely related ones for both accuracy and memory.
6. Choose efficient architectures. Models with fewer KV heads, sliding-window attention or compressed attention use far less cache per token.
7. Offload. Some serving frameworks can move inactive caches to CPU memory or SSD and bring them back when a user returns.

Which hardware handles long context best?

Capacity matters most. A 128 GB unified-memory GB10 system has room for a large model plus long contexts for a few users, although it reads the cache more slowly than a high-bandwidth GPU. An RTX PRO 6000 Blackwell offers 96 GB with much higher bandwidth, so it answers faster. For long contexts across many users, a DGX Station's large coherent memory or a multi-GPU server is the step up; see what DGX Station's 748 GB really means and why 128 GB matters for local AI.

Buying in Singapore

For long context with one or a few users, the NVIDIA DGX Spark 4TB (S$9,450), ASUS Ascent GX10 128GB/4TB (S$13,335) and HP ZGX Nano G1n 4TB (S$11,499). For faster long-context serving to a team, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition (S$29,100) or Server Edition (S$32,999), with the RTX PRO 5000 Blackwell (S$14,971.20) as a lower-cost step (listed 11 Nov 2026; prices change; ask for a current quotation). For many users with long documents, DGX Station GB300 systems such as the MSI XpertStation WS300 are price on request. To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

What is the KV cache in an LLM? The KV cache stores the keys and values a language model computes for each token it has already processed, so it doesn't recompute them when generating each new token. It grows with context length and is kept separately for each user.

How much memory does the KV cache need? Per token it is 2 × layers × KV heads × head dimension × bytes per value. For Llama 3.1 70B at FP16 that is about 320 KiB per token, or about 40 GB for a 128,000-token context.

What is the difference between context window and KV cache? The context window is the maximum number of tokens a model can consider. The KV cache is the memory those tokens actually occupy while the model runs.

Why does my AI server slow down with long documents? Long prompts and answers enlarge each user's KV cache, which uses up GPU memory and bandwidth, so fewer requests fit at once and each step reads more data.

How do I reduce KV cache memory? Use an FP8 KV cache, paged attention, prefix caching and sensible context limits, retrieve fewer but better passages, and choose models with efficient attention designs.