How Many GPUs Does Your AI Model Really Need? A Sizing Guide from 7B to 400B
Ask a vendor how many GPUs you need and the answer tends to be "more". Ask an engineer and you get a formula. The formula is short, and it suggests that many organisations are shopping one or two sizes too big.
This guide covers the first question only: will the model fit? Speed and user numbers come after, and we flag where they change the answer.
The rule of thumb
A language model is mostly a very large table of numbers, called parameters. The memory it needs depends on how many there are and how precisely each one is stored. A widely used estimate is:
GPU memory (GB) = parameters in billions × (bits ÷ 8) × 1.2
The 1.2 adds about 20% for working memory. At 16-bit precision that is roughly 2 GB per billion parameters before the overhead. Storing the numbers at lower precision, called quantisation, shrinks the model. 8-bit halves it. 4-bit halves it again, with some loss of quality.
Source: Modal: How much VRAM do I need for LLM inference?
Memory needed, by model size
| Model size | 16-bit (full quality) | 8-bit | 4-bit |
|---|---|---|---|
| 7B | 17 GB |
8 GB | 4 GB |
| 13B | 31 GB |
16 GB | 8 GB |
| 34B | 82 GB |
41 GB | 20 GB |
| 70B | 168 GB |
84 GB | 42 GB |
| 120B | 288 GB |
144 GB | 72 GB |
| 400B | 960 GB |
480 GB | 240 GB |
Our arithmetic using the formula above, rounded. Real figures vary with the model's design and the software that runs it.
What each GPU holds
| GPU | Memory per GPU | Eight in one server |
|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell Server Edition | 96 GB |
768 GB (our arithmetic) |
| NVIDIA H200 | 141 GB |
1.1 TB |
| NVIDIA B300 (Blackwell Ultra) | 288 GB |
2.3 TB |
Sources: PNY: NVIDIA RTX PRO 6000 Blackwell Server Edition, NVIDIA H200, The Register on Blackwell Ultra
Putting the two together
| You want to run | Memory at 16-bit | Smallest hardware that fits |
|---|---|---|
| 💬 A 7B to 13B assistant | 17 to 31 GB | 1 GPU One RTX PRO 6000, with room to spare |
| 🧑💻 A 34B coding model | 82 GB | 1 GPU One RTX PRO 6000 is tight; one H200 is comfortable |
| 🏢 A 70B general model | 168 GB | 1 to 2 GPUs Two RTX PRO 6000, two H200, or one B300 |
| 🧠 A 120B model | 288 GB | 2 to 4 GPUs Four RTX PRO 6000, three H200, or two B300. A single B300 has no headroom. |
| 🏛️ A 400B frontier-class open model | 960 GB | 8 GPUs An 8-GPU H200 server is tight; an 8-GPU B300 server fits with room |
Quantisation moves every row down a size. At 4-bit, the 70B model needs about 42 GB and fits on a single 96 GB card.
Four things the formula leaves out
1. Long documents and many users
A model keeps a working record of each conversation, known as the KV cache, and it grows with the length of the text. One published worked example puts it at about 2.6 GB for an 8,000-token context on a 70B model. That is for one conversation. Fifty people using the system at once is a different sum.
Source: Flavio Copes: How much VRAM do you need to run an LLM locally?
2. Fitting is not the same as fast
The formula tells you the minimum. More GPUs serve more people at once and answer faster. If the model fits on two cards and you buy eight, you have bought throughput, which is a legitimate purchase as long as you meant to make it.
3. Some models are bigger than they look
"Mixture of experts" models use only part of themselves for each word they produce, but the whole model still has to be loaded. One guide gives the example of a 671-billion-parameter model that activates 37 billion parameters per token and still needs roughly 382 GB at 4-bit.
Source: Techsy: LLM VRAM requirements guide
4. Training is a different sum
Everything above is for running a model. Fine-tuning and training need considerably more memory than inference, and this formula does not cover them.
The uncomfortable conclusion
For a company running a 70B-class model for its own staff, the fit question is answered by two to four GPUs, not by a rack. There are good reasons to buy more: many simultaneous users, long documents, several models side by side, or fine-tuning. They should be written down and counted, not assumed.
We made the same argument at rack scale in One NVL72 Rack or Nine 8-GPU Servers? and at strategy level in AI Servers Before AI Strategy.
Related on SourceIT
- NVIDIA DGX Spark personal AI supercomputer for desk-side development
- HPE Compute XD690 HGX B300 8-GPU server
- Lenovo ThinkSystem SR680a V4 HGX B300 8-GPU server
- HPE DL380a Gen12 vs DL384 Gen12 for inference servers
- All NVIDIA GB300 AI supercomputers
SourceIT Approved
Researched and verified by the SourceIT team for Singapore buyers.




