Rising component costs may adjust prices on selected items without prior notice. Supply constraints may add shipping time — we’ll notify you if affected. OEM costs and supply tightening — contact us to secure stock and pricing. For projects or bulk orders, choose “Request for Quotation” on any item.
Call our specialists(65) 6978 3502 Email our sales teamsales@sourceit.com.sg

AI & Workstations

CPU-Only AI Inference: When a Server Without a GPU Is the Honest Answer

The standard assumption in AI planning is that running a language model means buying GPUs. For large models and busy services, that is correct.

For a surprising number of real projects it is not. The server you already own may be able to do the job, and the industry has little incentive to mention it.

What changed

Two things. Models got smaller, as we discuss in Small Models Are Getting Good Enough. And processors gained instructions built for AI arithmetic. Intel's version is called Advanced Matrix Extensions, or AMX, and is present in the fourth, fifth and sixth generations of Xeon server processors.

Both major processor makers now publish guidance for running models without a GPU, and both draw the line in about the same place: models of up to roughly 20 billion parameters.

Sources: OpenMetal: Intel AMX and AI inference, Intel: CPU-optimised LLM deployment on Xeon 6, GeekChamp: choosing a server CPU for AI inference

How fast is fast enough?

Speed is measured in tokens per second, a token being roughly three-quarters of a word. What matters is how it feels to the person waiting.

Speed per user How it feels
3 to 5 tokens a second Painfully slow Works, but nobody will enjoy it
Under 10 Too slow for comfort Acceptable for background jobs
15 to 30 Responsive Fine for everyday interactive use

Sources: It's FOSS: testing local LLMs without a GPU, Arcee: is running language models on CPU viable?

Published tests show small models clearing the top band on a processor. One hosting provider reports a small quantised Llama model reaching up to 57 tokens per second on an AMX-enabled processor, against 28 without AMX. That is one configuration and one small model, but it shows the effect of the new instructions.

Where a CPU works, and where it does not

Workload Our read
📄 Document search and embedding CPU A natural fit
🤖 A handful of internal agents on small models CPU Vendors describe this as viable for a small to medium number of agents
🌙 Overnight batch work: summarising, classifying CPU Speed per answer matters less when nobody is waiting
💬 A chatbot for a few dozen staff on a model under 20B Test it Depends on how many people ask at once
👥 Hundreds of simultaneous users GPU Parallelism is what GPUs are for
🧠 Models of 70B and above GPU
🔧 Training or fine-tuning GPU

Source for the agents row: Lenovo Press: implementing AI agents without GPUs. The rest of the table is our own assessment.

The practical advantages

  • You may already own it. A recent dual-processor server can be tried tonight, at no cost.
  • Memory is plentiful. Servers take hundreds of gigabytes of ordinary memory, at a fraction of the price of GPU memory.
  • No special facilities. No extra power feeds, no cooling upgrade, no change to the room.
  • Your team knows how to run it. It is a server like any other.
  • No waiting list. See why AI server lead times are so long.

The honest limits

A processor handles one conversation well and many conversations poorly. As Red Hat's engineers put it, during high-concurrency batched inference a GPU's parallelism translates directly into throughput and into more requests served per dollar. Past a certain number of simultaneous users, a GPU is not only faster but cheaper per answer.

There is also a ceiling on model size. Above roughly 20 billion parameters, responses on a processor become slow enough that people stop using the tool. If your use case needs a large model, this is not your route.

The controversial part

The numbers come from people selling processors. One server maker's paper claims that on-premises CPU inference can become more economical than cloud rental in as little as 4.5 months. That may be right for the workload they tested. It is still a vendor's calculation, and the comparison chosen was cloud, not a GPU server.

The same caution applies in the other direction. GPU vendors are not going to tell you that your existing servers will do. The only neutral evidence is your own test.

A one-week test before any purchase

  1. Pick the smallest model that might meet the need.
  2. Run it on a recent server you already have, using a quantised version.
  3. Measure tokens per second with one user, then with the number you expect at peak.
  4. Ask real users whether it is fast enough.
  5. If it is, stop. If it is not, you now know how much more you need, which is the question a GPU quote should answer.

That last point is the real value. Even when the test fails, you go into the GPU conversation with a measured requirement in place of a guess.

Related on SourceIT

SourceIT Approved

SourceIT Approved

Researched and verified by the SourceIT team for Singapore buyers.

Share This Article
Facebook

More from our blog


← Older Post Newer Post →