Quantization stores an AI model's numbers with fewer bits so the model takes less memory and runs faster. Most models are trained in 16-bit formats (FP16 or BF16). Converting them to 8 bits halves the memory; converting to 4 bits cuts it to roughly a quarter. FP8 and INT8 are 8-bit formats; FP4 and INT4 are 4-bit formats. The "FP" formats store floating-point numbers with an exponent, which handles a wide range of values well, while "INT" formats store evenly spaced integers. NVIDIA's Blackwell GPUs add NVFP4, a 4-bit format that NVIDIA reports loses 1 percent or less accuracy compared with FP8 on DeepSeek-R1 benchmarks.
The right format depends on two things: how much accuracy your task can afford to lose, and which formats your GPU can actually accelerate.
What do FP16, FP8, INT8 and FP4 actually mean?
| Format | Bits | Type | Memory vs FP16 | Typical use |
|---|---|---|---|---|
| FP32 | 32 | Floating point | 2× | Some training maths; rarely for inference |
| FP16 / BF16 | 16 | Floating point | 1× (baseline) | Training and full-quality inference |
| FP8 (E4M3, E5M2) | 8 | Floating point | About ½ | High-quality inference and training on Hopper, Ada and Blackwell |
| INT8 | 8 | Integer | About ½ | Mature inference format, older GPUs, vision models |
| INT4 (GPTQ, AWQ, GGUF Q4) | 4 | Integer, grouped scales | About ¼ | Local LLMs on consumer GPUs and Macs |
| MXFP4 | About 4.25 | Floating point, 32-value blocks | About ¼ | Open standard; gpt-oss ships in it |
| NVFP4 | About 4.5 | Floating point, 16-value blocks | About 3.5× smaller | Blackwell-native high-accuracy 4-bit |
Source: NVIDIA developer blog on NVFP4 (2025); NVIDIA Research, "Pushing Intelligence to 4-bit" (2026); OpenAI gpt-oss model card. Compiled 11 Nov 2026.
Why do NVFP4 and MXFP4 beat plain 4-bit?
A 4-bit number can only represent 16 values. To make that work, quantized formats split weights into small blocks and give each block a shared scale factor that stretches the 16 values to fit the block's range.
- MXFP4, the Open Compute Project standard, uses blocks of 32 values with a power-of-two scale. It is simple, but one large outlier in a block forces the other 31 values to be rounded coarsely.
- NVFP4 uses smaller blocks of 16 values, each with a more precise FP8 scale, plus a second scale for the whole tensor. That fits awkward value distributions more tightly. NVIDIA says it stores about 4.5 bits per value, cuts memory about 3.5 times compared with FP16 and 1.8 times compared with FP8, and its research team reports up to about 3 times FP8's peak inference throughput on Blackwell.
The trade-off is compatibility. NVFP4 is accelerated only on Blackwell. MXFP4 checkpoints are more portable across vendors.
How much accuracy do you lose?
Less than most people expect, if the format is chosen well.
- 8-bit (FP8, INT8): usually very close to the original on most tasks. A safe default for production.
- Good 4-bit (NVFP4, well-calibrated INT4 such as AWQ or GPTQ): small losses on most tasks; NVIDIA reports 1 percent or less for DeepSeek-R1-0528 in NVFP4 versus FP8.
- Aggressive 3-bit or 2-bit: noticeable losses, especially on reasoning, maths and code. Use only when nothing else fits.
Losses are uneven. Larger models tolerate quantization better than small ones, and some layers are far more sensitive than others; research on Qwen2.5 models found that sensitive components stay the same across model sizes. Test on your own tasks. Long documents, maths and multilingual text, including Chinese, Malay and Tamil, can degrade before general chat does.
Quantization is the reason a 120B model now fits on a desktop. But the bit count on the label matters less than whether your GPU accelerates that format natively.
Which NVIDIA GPUs support which formats?
| Architecture | Examples | INT8 | FP8 | FP4 / NVFP4 |
|---|---|---|---|---|
| Ampere | RTX A2000, A1000, RTX 30-series | Yes | No | No |
| Ada Lovelace | RTX 40-series, RTX 2000/4000 Ada | Yes | Yes | No |
| Hopper | H100, H200 | Yes | Yes | No |
| Blackwell | RTX 50-series, RTX PRO Blackwell, GB10, Jetson Thor, GB300 | Yes | Yes | Yes |
"No" here means no native tensor-core acceleration. Older GPUs can still load a 4-bit model to save memory, by unpacking the weights to a higher precision for each calculation, but they don't get the full speed benefit. That is why the same FP4 model can be dramatically faster on a Blackwell card than on an Ada card with similar memory.
Why do AI specs quote FP4 numbers?
Because FP4 produces the biggest figure. Jetson Thor's 2,070 TFLOPS, for example, is a sparse FP4 number; its dense FP8 figure is about a quarter of that. When comparing systems, make sure the precision and sparsity match. We explain how to read these figures in why 1 petaflop doesn't tell you how fast your AI will run.
How do you choose a format?
- Start from memory. Model parameters × bytes per weight must fit with room for the KV cache. Our SLM vs LLM sizing table has the maths.
- Match the hardware. On Blackwell, prefer NVFP4 or FP8. On Ada or Hopper, FP8. On older or non-NVIDIA hardware, INT8 or INT4 formats such as AWQ, GPTQ or GGUF.
- Prefer official checkpoints. Model makers increasingly publish FP8 or FP4 versions that are already calibrated, and gpt-oss ships natively in MXFP4.
- Test your own prompts. Compare answers against the full-precision model on 50 to 100 real examples before going live.
Buying in Singapore
For native FP4 and FP8 on a workstation, the NVIDIA RTX PRO Blackwell range covers the RTX PRO 2000 (S$3,039.60), RTX PRO 4000 (S$5,640), RTX PRO 4500 (S$8,880) and RTX PRO 6000 Workstation Edition (S$29,100), plus the ASUS ProArt GeForce RTX 5090 (S$7,770). For 128 GB Blackwell desktops, the NVIDIA DGX Spark 4TB (S$9,450) and ASUS Ascent GX10 128GB/1TB (S$10,350). For INT8 inference on a budget, the Ampere NVIDIA RTX A2000 6GB (S$885) suits vision models (listed 11 Nov 2026; prices change; ask for a current quotation). To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
What is AI model quantization? Quantization stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16. This cuts memory use and speeds up inference, with some loss of accuracy.
What is the difference between FP8 and INT8? Both use 8 bits. FP8 is a floating-point format with an exponent, so it handles a wide range of values well; INT8 uses evenly spaced integers and relies more on careful scaling. FP8 needs Ada, Hopper or Blackwell GPUs.
What is NVFP4? NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. It uses 16-value blocks with an FP8 scale plus a per-tensor scale, storing about 4.5 bits per value with accuracy close to FP8.
Does a 4-bit model lose much accuracy? Well-calibrated 4-bit formats usually lose only a little on most tasks, and NVIDIA reports 1 percent or less for DeepSeek-R1 in NVFP4 versus FP8. Reasoning, maths and long-context tasks should be tested.
Can older GPUs run FP4 models? They can often load 4-bit models to save memory, but only Blackwell GPUs accelerate FP4 natively. On Ampere or Ada cards, 4-bit models run without the full speed benefit.
