Mixture of Experts (MoE) Models: Why Bigger Doesn't Always Mean More Expensive to Run

A mixture-of-experts (MoE) model is a large language model split into many smaller sub-networks, called experts, with a router that sends each token to only a few of them. The model stores the knowledge of a very large network, but each token uses only a fraction of it. OpenAI's gpt-oss-120b, for example, has 116.8 billion parameters in total but activates only about 5.1 billion per token. That is why MoE models can be huge and still answer quickly: compute per token follows the active parameters, while memory follows the total.

The catch is memory. Every expert must sit in memory even though only a few are used for each token. MoE models trade compute for capacity, and that changes what hardware you need.

How does a mixture-of-experts model work?

In a standard "dense" transformer, every layer runs every token through the same large feed-forward block. In an MoE model, that block is replaced by many experts plus a small router.

1. A token arrives at an MoE layer.
2. The router scores all the experts and picks the top few for that token. gpt-oss-120b has 128 experts per layer and uses the top 4.
3. Only those experts process the token, and their outputs are combined.
4. The next token may go to a completely different set of experts.

Experts don't map neatly to human topics such as "law" or "maths". They specialise in patterns the router learns during training. The result is a model that behaves like a large one but computes like a much smaller one.

Total parameters vs active parameters: why does it matter?

Source: OpenAI gpt-oss model card (Aug 2025); DeepSeek-V3 technical report; Meta Llama 3.3 model card. Compiled 11 Nov 2026.

Total parameters decide how much memory the model needs. Active parameters decide how much computation, and how much memory bandwidth, each token uses. A dense 70B model reads about 70 billion parameters for every token. gpt-oss-120b reads about 5 billion. On the same hardware, that is the difference between a model that types slowly and one that keeps up with reading speed.

With MoE, "how big is the model?" becomes two questions. Will it fit? That depends on total parameters. Will it be fast? That depends on active parameters.

What does MoE mean for hardware?

MoE favours hardware with a lot of memory, even if that memory is slower. This is exactly where 128 GB unified-memory systems such as NVIDIA's GB10 shine.

CALC: theoretical decode ceiling ≈ memory bandwidth ÷ active weight bytes per token, single user; 273 GB/s is GB10's quoted bandwidth. Real speeds are lower because of attention, KV cache reads and software overhead.

So a GB10 desktop, which is relatively slow for a dense 70B model, is a comfortable home for a 120B MoE model. A GPU with very fast memory but limited capacity, such as a 32 GB RTX 5090, has the opposite problem: it is quick, but the biggest MoE models don't fit unless experts are offloaded to system RAM, which slows things down. We compare those trade-offs in GB10 vs RTX 5090 and why 128 GB matters for local AI.

Are MoE models cheaper to run?

Per token, usually yes, because each token uses less computation and energy. But there are costs that dense models don't have.

• Memory. You pay for capacity to hold every expert. The largest MoE models, such as DeepSeek's 671B, need hundreds of gigabytes even at low precision, which means a DGX Station or multi-GPU server.
• Batching is less efficient. When many users are served at once, different tokens hit different experts, so more of the model is read per batch and the speed advantage narrows.
• Multi-GPU communication. Spreading experts across GPUs ("expert parallelism") means tokens travel between GPUs, so interconnect speed matters. See ConnectX-7 vs ordinary Ethernet.
• Fine-tuning is harder. Training MoE models evenly across experts takes more care than training dense ones.

Should a business choose an MoE model?

For running models privately on a single system, MoE models are often the best value today. They give large-model quality at small-model speed, provided you have the memory. They pair especially well with reasoning models, which generate thousands of tokens and benefit most from fast generation. Dense models still make sense when memory is tight, when you plan heavy fine-tuning, or for very small models where MoE adds little.

Buying in Singapore

For 120B-class MoE models in 128 GB, the NVIDIA DGX Spark 4TB (S$9,450), ASUS Ascent GX10 128GB/2TB (S$10,500), HP ZGX Nano G1n 4TB (S$11,499), the MSI Prestige N16 Flip AI+ 128GB laptop (S$10,854.12) and the Jetson AGX Thor 128GB for robots (S$9,075.60). For faster generation where the model fits in 96 GB, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition (S$29,100) (listed 11 Nov 2026; prices change; ask for a current quotation). For DeepSeek-class models, DGX Station GB300 systems such as the HP ZGX Fury G1n and MSI XpertStation WS300 are price on request. To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

What is a mixture-of-experts model? A mixture-of-experts (MoE) model splits a large neural network into many expert sub-networks and uses a router to send each token to only a few of them, so it computes like a smaller model while storing the knowledge of a larger one.

What is the difference between total and active parameters? Total parameters are all the weights stored in the model and decide memory needs. Active parameters are the ones used for each token and decide computation and speed. gpt-oss-120b has 116.8B total and about 5.1B active.

Do MoE models need less memory? No. Every expert must be loaded in memory, so memory follows total parameters. MoE models save computation and bandwidth per token, not capacity.

Why do MoE models run well on 128 GB systems like DGX Spark? These systems have plenty of memory but moderate bandwidth. MoE models need capacity for all experts but read only a small active portion per token, which matches that balance.

Are DeepSeek and gpt-oss MoE models? Yes. DeepSeek-V3 and R1 have 671B total and 37B active parameters, and OpenAI's gpt-oss-20b and gpt-oss-120b are also mixture-of-experts models.