How to Read MLPerf: What AI Benchmark Claims Do and Do Not Tell You
Sooner or later, every AI hardware presentation reaches a slide that says "number one in MLPerf". Often two competing vendors show that slide in the same week. Both can be telling the truth.
MLPerf is the closest thing the industry has to a fair test. It is also a menu from which vendors order the dishes that suit them. Knowing how to read it is the difference between using it and being used by it.
What MLPerf is
MLPerf is a family of benchmarks run by MLCommons, an industry consortium. The inference benchmarks were designed, in the words of the original paper, to be architecturally neutral, representative and reproducible, with input from researchers and engineers at more than 30 organisations.
Each benchmark fixes the things that would otherwise make comparison impossible: the exact model version, the input data, the processing steps and the accuracy that must be reached. Vendors run the test on their systems and submit results, which are published together.
Sources: MLPerf Inference Benchmark paper, AIwire on MLPerf Inference v5.0
The two words that matter most: closed and open
| Compare | 🔒 Closed division | 🔓 Open division |
|---|---|---|
| Rules | Same model, same precision, same processing for everyone | Almost any optimisation allowed, including changed models |
| Comparable across vendors? | Yes | Not directly |
| What it shows | How systems compare on equal terms | What a vendor can achieve with its best effort |
| Use it for | Purchasing comparisons | Seeing where the technology is heading |
Sources: HPCwire on MLPerf Inference v5.0, Spheron: MLPerf Inference v6.0 explained
The first check on any MLPerf claim is which division it came from. A headline figure from the open division is a demonstration, not a comparison.
The second thing to check: the scenario
The same hardware is tested in different ways, because real workloads differ.
| Scenario | What it imitates | Closest to |
|---|---|---|
| 💬 Server | Requests arriving unpredictably, each needing an answer within a time limit | A chatbot or assistant with live users |
| 📦 Offline | All the work available at once; only total throughput counts | Overnight batch jobs, document processing |
Offline numbers are always the larger ones, because the system can work at full tilt without worrying about response time. If you are buying for an interactive service and the slide shows an offline figure, you are looking at the wrong number.
What MLPerf does not tell you
What it cost
The results list performance. They do not list price. A system that is 20% faster and twice as expensive is not a better purchase, and nothing in the table will tell you that. You have to bring the price yourself.
Whether your model behaves the same way
The benchmark uses a fixed set of models. If yours is a different size, uses longer documents, or is run with different software, the ranking may not hold.
How much is hardware and how much is software
This is the detail vendors rarely highlight. In the v5.0 round, MLCommons reported that the median score on one large language model test doubled in a year, and the best result improved 3.3 times. Hardware did not get that much faster in twelve months. A large share of the gain was software tuning.
That cuts two ways. It means the system you buy may get faster after you buy it. It also means a benchmark result reflects an expert team tuning for one test, which is not what you will have on day one.
Who did not turn up
Submission is voluntary. Vendors enter the tests where they expect to do well and skip the ones where they do not. An empty row is information too.
Five checks before you trust a claim
| Ask | Why |
|---|---|
| 1. Closed or open division? | Only closed results compare like with like |
| 2. Server or offline scenario? | Match it to how you will use the system |
| 3. Which model, and how close is it to mine? | A win on one model does not transfer automatically |
| 4. How many GPUs were in the tested system? | Per-system and per-GPU results tell different stories |
| 5. Is that exact configuration on my quote? | The benchmarked system and the one you are offered may differ |
The fair conclusion
It would be easy to end by saying benchmarks are marketing. That would be wrong. MLPerf is audited, its rules are public, and it is far better than the alternative, which is each vendor inventing its own test. Used properly it will stop you buying something clearly unsuitable.
What it cannot do is make the decision. It measures speed on standard tests. Your decision also depends on price, power, the room you have, and the people who will run it. We set out a different yardstick in Tokens per Watt Explained, and the sizing question that comes before any benchmark in How Many GPUs Does Your AI Model Really Need?
Related on SourceIT
SourceIT Approved
Researched and verified by the SourceIT team for Singapore buyers.




