Rising component costs may adjust prices on selected items without prior notice. Supply constraints may add shipping time — we’ll notify you if affected. OEM costs and supply tightening — contact us to secure stock and pricing. For projects or bulk orders, choose “Request for Quotation” on any item.
Call our specialists(65) 6978 3502 Email our sales teamsales@sourceit.com.sg

AI & Workstations

How to Read MLPerf: What AI Benchmark Claims Do and Do Not Tell You

Sooner or later, every AI hardware presentation reaches a slide that says "number one in MLPerf". Often two competing vendors show that slide in the same week. Both can be telling the truth.

MLPerf is the closest thing the industry has to a fair test. It is also a menu from which vendors order the dishes that suit them. Knowing how to read it is the difference between using it and being used by it.

What MLPerf is

MLPerf is a family of benchmarks run by MLCommons, an industry consortium. The inference benchmarks were designed, in the words of the original paper, to be architecturally neutral, representative and reproducible, with input from researchers and engineers at more than 30 organisations.

Each benchmark fixes the things that would otherwise make comparison impossible: the exact model version, the input data, the processing steps and the accuracy that must be reached. Vendors run the test on their systems and submit results, which are published together.

Sources: MLPerf Inference Benchmark paper, AIwire on MLPerf Inference v5.0

The two words that matter most: closed and open

Compare 🔒 Closed division 🔓 Open division
Rules Same model, same precision, same processing for everyone Almost any optimisation allowed, including changed models
Comparable across vendors? Yes Not directly
What it shows How systems compare on equal terms What a vendor can achieve with its best effort
Use it for Purchasing comparisons Seeing where the technology is heading

Sources: HPCwire on MLPerf Inference v5.0, Spheron: MLPerf Inference v6.0 explained

The first check on any MLPerf claim is which division it came from. A headline figure from the open division is a demonstration, not a comparison.

The second thing to check: the scenario

The same hardware is tested in different ways, because real workloads differ.

Scenario What it imitates Closest to
💬 Server Requests arriving unpredictably, each needing an answer within a time limit A chatbot or assistant with live users
📦 Offline All the work available at once; only total throughput counts Overnight batch jobs, document processing

Offline numbers are always the larger ones, because the system can work at full tilt without worrying about response time. If you are buying for an interactive service and the slide shows an offline figure, you are looking at the wrong number.

What MLPerf does not tell you

What it cost

The results list performance. They do not list price. A system that is 20% faster and twice as expensive is not a better purchase, and nothing in the table will tell you that. You have to bring the price yourself.

Whether your model behaves the same way

The benchmark uses a fixed set of models. If yours is a different size, uses longer documents, or is run with different software, the ranking may not hold.

How much is hardware and how much is software

This is the detail vendors rarely highlight. In the v5.0 round, MLCommons reported that the median score on one large language model test doubled in a year, and the best result improved 3.3 times. Hardware did not get that much faster in twelve months. A large share of the gain was software tuning.

That cuts two ways. It means the system you buy may get faster after you buy it. It also means a benchmark result reflects an expert team tuning for one test, which is not what you will have on day one.

Who did not turn up

Submission is voluntary. Vendors enter the tests where they expect to do well and skip the ones where they do not. An empty row is information too.

Five checks before you trust a claim

Ask Why
1. Closed or open division? Only closed results compare like with like
2. Server or offline scenario? Match it to how you will use the system
3. Which model, and how close is it to mine? A win on one model does not transfer automatically
4. How many GPUs were in the tested system? Per-system and per-GPU results tell different stories
5. Is that exact configuration on my quote? The benchmarked system and the one you are offered may differ

The fair conclusion

It would be easy to end by saying benchmarks are marketing. That would be wrong. MLPerf is audited, its rules are public, and it is far better than the alternative, which is each vendor inventing its own test. Used properly it will stop you buying something clearly unsuitable.

What it cannot do is make the decision. It measures speed on standard tests. Your decision also depends on price, power, the room you have, and the people who will run it. We set out a different yardstick in Tokens per Watt Explained, and the sizing question that comes before any benchmark in How Many GPUs Does Your AI Model Really Need?

Related on SourceIT

SourceIT Approved

SourceIT Approved

Researched and verified by the SourceIT team for Singapore buyers.

Share This Article
Facebook

More from our blog


← Older Post Newer Post →