Rising component costs may adjust prices on selected items without prior notice. Supply constraints may add shipping time — we’ll notify you if affected. OEM costs and supply tightening — contact us to secure stock and pricing. For projects or bulk orders, choose “Request for Quotation” on any item.
Call our specialists(65) 6978 3502 Email our sales teamsales@sourceit.com.sg

AI & Workstations

Tokens per Watt: The AI Benchmark Your CFO Should Care About, and Three Ways It Misleads

For years, AI hardware was compared on speed: how many calculations per second a chip could manage. That is changing. The industry's new favourite number is tokens per watt, how much AI output you get for the electricity you put in.

In Singapore, where data centre power is rationed, this is exactly the right thing to measure. It is also a metric promoted hardest by the company that scores best on it, and it can mislead in ways a finance team should understand before it appears in a business case.

What it measures

Metric Question it answers Who cares most
🚀 FLOPS (raw speed) How fast is the chip in theory? Engineers
⚡ Tokens per watt or per megawatt How much AI output from a fixed power supply? Anyone with a power cap
💰 Cost per million tokens What does a unit of AI output cost, all in? Finance
🗣️ Tokens per second per user How fast does it feel to the person waiting? Users

A token is a small piece of text, the unit an AI model reads and writes. Tokens per megawatt asks how much AI output a data centre can produce from a fixed amount of available power. As the benchmark firm SemiAnalysis puts it, power availability is often the binding constraint on new AI deployments, so a system that produces more tokens per megawatt serves more demand from the same electricity allocation.

Source: SemiAnalysis InferenceX glossary

Why it suddenly matters

NVIDIA has argued publicly that buyers use the wrong yardsticks. Instead of cost per GPU-hour or raw speed per dollar, it says the real benchmark is cost per token. On that measure its own figures are dramatic.

NVIDIA's claim: Blackwell vs the previous Hopper generation Figure
⚡ Tokens per megawatt 50 times more
💰 Cost per million tokens 35 times lower

Source: ITdaily on NVIDIA's cost-per-token argument. These are NVIDIA's figures.

The trade press noticed the obvious. One outlet's headline asked whether the idea was smart, or a plea for NVIDIA's own shop. Both can be true. A metric can be self-serving and correct.

Why it is the right question in Singapore

If power were unlimited, you could fix slow hardware by buying more of it. Here you cannot. New data centre capacity is released in small, tightly controlled allocations, and a colocation contract gives you a fixed number of kilowatts. Inside that cap, the only way to produce more is to get more out of each watt.

So for a Singapore buyer, tokens per watt is not a marketing abstraction. It is the ceiling on what your power allocation can ever deliver.

Three ways the number misleads

1. It hides how slow the tokens are

The highest tokens-per-megawatt figures come from running hardware flat out with many requests batched together. That is efficient, and it is slow for each individual user. The Register, analysing independent benchmark data, noted that chips can reach enormous throughput per megawatt but trade away responsiveness to do it, producing tokens that are cheap to serve and painfully slow to arrive.

A chatbot that makes users wait is not a bargain. Always ask for efficiency at the response speed your application needs.

Source: The Register on tokenomics

2. It assumes the hardware is always busy

Benchmarks measure a system at full load. Real systems are not at full load. The Uptime Institute worked through an example: at a benchmark-style throughput, energy per token came to 0.77 joules. At a more realistic throughput and 50% lifetime utilisation, the same hardware used about 3.5 joules per token, roughly five times as much.

Scenario (Uptime Institute example) Energy per token
🏁 Benchmark conditions 0.77 joules
🏢 Realistic load, 50% utilisation About 3.5 joules

Uptime's conclusion is the one to remember: the metric misleads when benchmark figures are read as fixed properties of hardware instead of reflections of specific operating conditions.

Source: Uptime Institute Intelligence

3. It is a moving target

The same hardware can score very differently from one month to the next. NVIDIA itself says software advances improved performance per watt on one model by up to five times in a single month. That is good news for owners. It also means any published figure is a snapshot of a particular model, software version and configuration, and may not match what you will run.

Source: IT Brief on NVIDIA's efficiency claims

And one thing it leaves out entirely

Tokens per watt at the chip is not tokens per watt at the meter. Cooling, power conversion and the building's overheads all take their share before electricity reaches a GPU. A facility with efficient cooling delivers more of its power allocation to computing. That is one of the real arguments for liquid cooling, and it is why the building matters as much as the rack.

How to use the metric properly

When someone quotes tokens per watt, ask Why
🧩 Which model, and how large? Results vary enormously between models
🗣️ At what speed per user? Efficiency at unusable latency is meaningless
📊 At what utilisation? Idle time multiplies real energy per token
💻 Which software stack and version? Tuning changes the answer by multiples
🏭 Measured at the chip or at the facility? Cooling and power losses are extra
🔍 Who ran the test? Vendor figures and independent ones differ

Independent, repeatable benchmarks now exist. SemiAnalysis's open InferenceX benchmark tracks throughput, cost per million tokens and tokens per megawatt across hardware and software, and shows the trade-off between efficiency and responsiveness instead of a single flattering point. It is a better starting place than any vendor slide.

The verdict

Tokens per watt is a real improvement on how AI hardware used to be judged. It ties the purchase to the two things that limit an AI project in Singapore, power and money, and it pushes buyers to think about output instead of specifications.

But a single number is always a simplification. The honest version of the metric has four parts: output, per watt, at an acceptable speed, at your real utilisation. Miss any one and the figure tells you about a benchmark, not about your business. We work through the utilisation part in Buy a Rack, Rent GPUs or Colocate?

Related on SourceIT

SourceIT Approved

SourceIT Approved

Researched and verified by the SourceIT team for Singapore buyers.

Share This Article
Facebook

More from our blog


← Older Post Newer Post →