Rising component costs may adjust prices on selected items without prior notice. Supply constraints may add shipping time — we’ll notify you if affected. OEM costs and supply tightening — contact us to secure stock and pricing. For projects or bulk orders, choose “Request for Quotation” on any item.
Call our specialists(65) 6978 3502 Email our sales teamsales@sourceit.com.sg

AI & Workstations

Small Models Are Getting Good Enough. What Does That Do to the Case for Rack-Scale AI?

The AI hardware story of the last three years has had one plot: models get bigger, so machines must get bigger. Racks went from 10 kW to 130 kW on the strength of it.

In June 2025 a research paper argued the opposite, and the awkward part is where it came from. The case that small models are the future was made by researchers at NVIDIA, the company that sells the big machines.

What the paper says

The paper is titled "Small Language Models are the Future of Agentic AI", written by a team from NVIDIA Research and the Georgia Institute of Technology. Its position is that small models are powerful enough, better suited and more economical for AI agents, which are systems that carry out multi-step tasks by calling tools and other software.

The reasoning is practical. Most of what an agent does is narrow and repetitive: classify a request, fill in a form, call a function, format an answer. The authors argue those jobs do not need a model that can also write poetry and discuss philosophy.

Claim What is reported
📏 What counts as small A model that fits on an ordinary consumer device. The authors say they would treat most models under 10 billion parameters as small.
💰 Cost to run Serving a 7-billion-parameter model is put at 10 to 30 times cheaper than serving a much larger one
🧩 Share of agent work One report of the paper says models under 10 billion parameters can handle 60 to 80% of agent tasks now given to models above 70 billion

Sources: Philipp Dubach's summary quoting the paper, Analytics Vidhya, PPC Land. This is a position paper by researchers, not NVIDIA product guidance, and the figures are the authors' own.

Why it matters for hardware

Memory decides hardware, and model size decides memory. By the rule of thumb in our GPU sizing guide, a 7-billion-parameter model needs roughly 17 GB at full precision. A 70-billion one needs roughly ten times that.

So if a business process can run on small models, the hardware question changes character. It stops being "which 8-GPU server" and becomes "how many modest GPUs, or do we need GPUs at all". We look at the second half of that in CPU-Only AI Inference.

The argument that big hardware is safe anyway

It would be a mistake to read the paper as the end of large systems. There are four good counter-arguments.

1. Cheaper usually means more, not less

When something useful gets ten times cheaper, people tend to use far more of it. A company that could afford one AI assistant may deploy fifty automated agents. Total demand for computing can rise even as each task gets cheaper.

2. The paper keeps the large model in the picture

The proposal is a mixed system: small specialists for routine steps, with a large model called in when a problem needs it. Somebody still runs the large model.

3. Small models come from large ones

Many small models are made by distilling or fine-tuning from bigger models. Building and training them is heavy work, even if running them is not.

4. Some work is simply hard

Long documents, complex reasoning and open-ended research still favour large models. That gap has narrowed. It has not closed.

Who should change their plans

If you are What this trend means
🏢 An enterprise automating defined processes Test small first You may need much less hardware than you were quoted
🧑‍💻 A team building many agents Plan a mix Many small GPUs for routine steps, access to a large model for the rest
🔬 A research group or model builder Little change Training and frontier work still need large systems
☁️ A provider serving many customers Little change Volume keeps large systems busy whatever the model size

Our view

The honest reading is not "big hardware is finished". It is that the default has moved. Two years ago, the safe assumption was that a serious AI project needed the largest model it could afford. Today the safe assumption is the reverse: start with the smallest model that might work, measure it, and move up only when it fails.

That order protects the buyer. Hardware bought for a model you turned out not to need is the most expensive kind. And it is worth remembering who pointed this out. A supplier's own researchers saying "you may need less" is not something to wave away.

Related on SourceIT

Share This Article
Facebook

More from our blog


← Older Post Newer Post →