Rising component costs may adjust prices on selected items without prior notice. Supply constraints may add shipping time — we’ll notify you if affected. OEM costs and supply tightening — contact us to secure stock and pricing. For projects or bulk orders, choose “Request for Quotation” on any item.
Call our specialists(65) 6978 3502 Email our sales teamsales@sourceit.com.sg

AI & Workstations

How Many GPUs Does Your AI Model Really Need? A Sizing Guide from 7B to 400B

Ask a vendor how many GPUs you need and the answer tends to be "more". Ask an engineer and you get a formula. The formula is short, and it suggests that many organisations are shopping one or two sizes too big.

This guide covers the first question only: will the model fit? Speed and user numbers come after, and we flag where they change the answer.

The rule of thumb

A language model is mostly a very large table of numbers, called parameters. The memory it needs depends on how many there are and how precisely each one is stored. A widely used estimate is:

GPU memory (GB) = parameters in billions × (bits ÷ 8) × 1.2

The 1.2 adds about 20% for working memory. At 16-bit precision that is roughly 2 GB per billion parameters before the overhead. Storing the numbers at lower precision, called quantisation, shrinks the model. 8-bit halves it. 4-bit halves it again, with some loss of quality.

Source: Modal: How much VRAM do I need for LLM inference?

Memory needed, by model size

Model size 16-bit (full quality) 8-bit 4-bit
7B 17 GB
8 GB 4 GB
13B 31 GB
16 GB 8 GB
34B 82 GB
41 GB 20 GB
70B 168 GB
84 GB 42 GB
120B 288 GB
144 GB 72 GB
400B 960 GB
480 GB 240 GB

Our arithmetic using the formula above, rounded. Real figures vary with the model's design and the software that runs it.

What each GPU holds

GPU Memory per GPU Eight in one server
NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB
768 GB (our arithmetic)
NVIDIA H200 141 GB
1.1 TB
NVIDIA B300 (Blackwell Ultra) 288 GB
2.3 TB

Sources: PNY: NVIDIA RTX PRO 6000 Blackwell Server Edition, NVIDIA H200, The Register on Blackwell Ultra

Putting the two together

You want to run Memory at 16-bit Smallest hardware that fits
💬 A 7B to 13B assistant 17 to 31 GB 1 GPU One RTX PRO 6000, with room to spare
🧑‍💻 A 34B coding model 82 GB 1 GPU One RTX PRO 6000 is tight; one H200 is comfortable
🏢 A 70B general model 168 GB 1 to 2 GPUs Two RTX PRO 6000, two H200, or one B300
🧠 A 120B model 288 GB 2 to 4 GPUs Four RTX PRO 6000, three H200, or two B300. A single B300 has no headroom.
🏛️ A 400B frontier-class open model 960 GB 8 GPUs An 8-GPU H200 server is tight; an 8-GPU B300 server fits with room

Quantisation moves every row down a size. At 4-bit, the 70B model needs about 42 GB and fits on a single 96 GB card.

Four things the formula leaves out

1. Long documents and many users

A model keeps a working record of each conversation, known as the KV cache, and it grows with the length of the text. One published worked example puts it at about 2.6 GB for an 8,000-token context on a 70B model. That is for one conversation. Fifty people using the system at once is a different sum.

Source: Flavio Copes: How much VRAM do you need to run an LLM locally?

2. Fitting is not the same as fast

The formula tells you the minimum. More GPUs serve more people at once and answer faster. If the model fits on two cards and you buy eight, you have bought throughput, which is a legitimate purchase as long as you meant to make it.

3. Some models are bigger than they look

"Mixture of experts" models use only part of themselves for each word they produce, but the whole model still has to be loaded. One guide gives the example of a 671-billion-parameter model that activates 37 billion parameters per token and still needs roughly 382 GB at 4-bit.

Source: Techsy: LLM VRAM requirements guide

4. Training is a different sum

Everything above is for running a model. Fine-tuning and training need considerably more memory than inference, and this formula does not cover them.

The uncomfortable conclusion

For a company running a 70B-class model for its own staff, the fit question is answered by two to four GPUs, not by a rack. There are good reasons to buy more: many simultaneous users, long documents, several models side by side, or fine-tuning. They should be written down and counted, not assumed.

We made the same argument at rack scale in One NVL72 Rack or Nine 8-GPU Servers? and at strategy level in AI Servers Before AI Strategy.

Related on SourceIT

SourceIT Approved

SourceIT Approved

Researched and verified by the SourceIT team for Singapore buyers.

Share This Article
Facebook

More from our blog


← Older Post Newer Post →