AI Inference
10 articles
-
What Is Test-Time Compute and Why Does It Matter for AI Hardware?
Test-time compute explained: how reasoning models spend extra computation at inference, the effect on tokens, latency and cost, and hardware sizing.
Wei Fei
·
-
What Is Continuous Batching in AI Inference?
Continuous batching explained: how AI servers share one GPU among many users, static vs dynamic vs continuous batching, trade-offs and sizing.
Wei Fei
·
-
What Is KV Cache? Why Long AI Conversations Need More GPU Memory
KV cache and context windows explained with worked memory maths for 8B and 70B models, multi-user sizing, and ways to cut KV memory.
Wei Fei
·
-
What Is Disaggregated AI Inference? Separating Prefill and Decode
Disaggregated inference explained: prefill vs decode, KV cache transfer, NVIDIA Dynamo, when splitting phases across GPUs pays off and when it doesn't.
Wei Fei
·
-
AI Model Quantization Explained: FP4 vs FP8 vs INT8 vs INT4
Quantization formats compared: FP16, FP8, INT8, INT4, MXFP4 and NVFP4, with memory savings, accuracy trade-offs and GPU compatibility.
Wei Fei
·
-
What Is Speculative Decoding? How AI Servers Generate Responses Faster
How speculative decoding speeds up LLM responses with draft-and-verify, EAGLE-3 results, when it helps, when it doesn't, and what it means for hardware.
Wei Fei
·
-
Mixture of Experts (MoE) Models: Why Bigger Doesn't Always Mean More Expensive to Run
Mixture of experts explained: total vs active parameters, why MoE models need lots of memory but little compute per token, and hardware sizing.
Wei Fei
·
-
AI Model Serving Explained: From Notebook to Production
AI model serving explained: what changes between a notebook demo and a production service, the layers of a serving stack, key metrics and hardware.
Wei Fei
·
-
TensorRT-LLM vs vLLM: Which Inference Engine Should You Use?
TensorRT-LLM vs vLLM compared: performance, model support, hardware lock-in and operational effort, with a way to benchmark both on your own workload.
Wei Fei
·
-
NVIDIA NIM vs Ollama: Enterprise AI Deployment Compared
NVIDIA NIM vs Ollama compared: what each is, licensing and cost, performance, support and security, and when a business should move from one to the other.
Wei Fei
·
Wei Fei
