NVIDIA DGX Station's 748GB Memory: Can It Really Handle a Trillion-Parameter AI Model?

Yes, but only on specific terms. A trillion-parameter model needs about 2,000 GB at 16-bit precision and about 1,000 GB at 8-bit, and neither fits in the NVIDIA DGX Station GB300's 748 GB of coherent memory. At 4-bit precision the weights shrink to about 500 GB, which does fit, with roughly 248 GB left for the KV cache, the runtime and the operating system. That is how NVIDIA's "up to 1 trillion parameter" claim works. In practice it means a 4-bit, mixture-of-experts model, moderate context lengths, a limited number of simultaneous users, and accepting that most of the model sits in the slower LPDDR5X tier rather than in the 252 GB of fast HBM3e.

How much memory does a model of each size need?

The starting point is simple arithmetic: parameters multiplied by bytes per parameter. At 16-bit (BF16) each parameter takes 2 bytes, at 8-bit (FP8) 1 byte, and at 4-bit (NVFP4 or MXFP4) about half a byte, plus a small overhead for scaling factors that we ignore here.

Source: parameter counts from each model's published model card (Moonshot AI, DeepSeek, Meta, OpenAI); memory figures are SourceIT's arithmetic for weights only, excluding KV cache and runtime overhead. DGX Station memory from NVIDIA's datasheet (Aug 2026). Illustrative; actual file sizes vary with format.

The 671B row shows how tight "fits" can be. DeepSeek-R1's downloaded FP8 checkpoint folder is about 689 GB, which leaves almost nothing of the 748 GB for anything else. A 4-bit version is the practical choice on one system.

What is the KV cache, and why does context length matter so much?

When a model reads a prompt and writes an answer, it stores intermediate results for every token so far: the key-value (KV) cache. This memory grows with context length and with every simultaneous user, and it sits on top of the weights.

• A conventional dense model: Llama 3.1 405B has 126 layers, 8 key-value heads and a head dimension of 128. At 16-bit that is about 0.52 MB of KV cache per token, or about 68 GB for one 128,000-token conversation. Storing the cache at 8-bit halves that to about 34 GB.
• A model with compressed attention: DeepSeek's V3 and R1 models, and Kimi K2, which uses the same multi-head latent attention design, store a compressed cache. At 61 layers and 576 values per layer, that is about 70 KB per token at 16-bit, or about 9.2 GB for one 128,000-token conversation.

Now multiply by users. Ten people each working with 32,000-token documents on the dense 405B model would need about 169 GB of KV cache at 16-bit, on top of the weights. That is why "the model fits" and "the model is useful for your team" are different questions. With a 500 GB trillion-parameter model loaded, the remaining 248 GB has to cover the cache for every active session, the inference engine's working memory and the operating system.

Fitting the weights is the entry ticket. The KV cache decides how many people can actually use the model, and for how long a conversation.

What happens to speed when the model doesn't fit in HBM3e?

A 500 GB model cannot sit entirely in the 252 GB of HBM3e, so part of it lives in the Grace CPU's 496 GB of LPDDR5X. The GPU can read that memory directly over NVLink-C2C, but StorageReview measured only about 390 GB/s for that path, against about 6,875 GB/s from HBM3e. Every token the model writes needs its active weights read once, so the slower tier sets the pace for whatever lives there.

Mixture-of-experts (MoE) models soften the blow, because each token uses only a fraction of the parameters. Kimi K2 activates about 32 billion of its trillion parameters per token, about 16 GB at 4-bit. As a theoretical upper bound, reading 16 GB per token would allow about 444 tokens per second if everything came from HBM3e, and about 25 per second if everything came from LPDDR5X. Real systems land below these ceilings, and where between them depends on how many of the active experts sit in fast memory.

Published tests show the effect. In StorageReview's ASUS ExpertCenter Pro ET900N G3 review, GLM-5.2 at NVFP4, with about 215 GB of experts offloaded to Grace memory, produced 35 output tokens per second for one user and 139 across 32 users. Nemotron 3 Ultra 550B at NVFP4 peaked at 159 tokens per second. Compare that with gpt-oss-120b, which fits entirely in HBM3e and reached 9,280 output tokens per second at 128 concurrent requests in the same review. Our post on how the two memory tiers work explains the bandwidth maths, and 748GB Is Not All GPU VRAM covers the basics.

Does 4-bit quantisation hurt answer quality?

Sometimes, and it depends on the model and the task. Some models, such as gpt-oss-120b, are released in a 4-bit format by their developers, so 4-bit is their intended form. Others are released at 8- or 16-bit and quantised afterwards, and the quality loss varies by model, method and task. Formats such as NVFP4 use fine-grained scaling to limit that loss, but no vendor figure substitutes for testing on your own documents and questions. Run a set of real prompts at 8-bit and 4-bit (on cloud capacity if necessary) and compare the answers before you size hardware around a quantised trillion-parameter model.

When does a trillion-parameter model make sense on one DGX Station?

• It can make sense for research teams evaluating the largest open-weight models privately, for low-concurrency expert use where answer quality matters more than speed, and for overnight batch work where tokens per second matter less than finishing the job on-premises.
• It usually doesn't for serving a whole company interactively. A model in the 100B to 250B range that fits in HBM3e will typically serve far more users, far faster.
• Two systems can be linked over ConnectX-8 for "over 1.4 TB of coherent memory", in NVIDIA's words, which relieves the capacity squeeze but adds networking and software complexity. See DGX Station's networking.

The capacity question feeds directly into cost. A model that serves fewer tokens per hour costs more per token to own, which matters when you compare a DGX Station with cloud APIs. How Many AI Tokens Can DGX Station Actually Generate? sets out measured throughput by model size.

Buying in Singapore

SourceIT lists five DGX Station GB300 systems, all price on request: the HP ZGX Fury G1n, MSI XpertStation WS300, ASUS ExpertCenter Pro ET900N G3, GIGABYTE W775-V10-L01 and Supermicro ARS-511GD-NB-LCC. All five have the same 252 GB HBM3e plus 496 GB LPDDR5X memory, so the model-sizing maths above applies to each. Tell us which models and context lengths you plan to run, and we can help you check the fit before you order. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

Can a DGX Station run a 1-trillion-parameter model? At 4-bit precision, yes. The weights need about 500 GB of the 748 GB coherent memory. At 8-bit (about 1,000 GB) or 16-bit (about 2,000 GB) a single system cannot hold the model.

Will a trillion-parameter model run fast on it? Not compared with smaller models. Because 500 GB exceeds the 252 GB of HBM3e, part of the model sits in LPDDR5X, which StorageReview measured at about 390 GB/s into the GPU, against about 6,875 GB/s for HBM3e. A comparable large MoE test produced 35 tokens per second for one user.

How much memory does the KV cache need? It depends on the model's attention design, the context length and the number of users. Llama 3.1 405B needs about 0.52 MB per token at 16-bit, or about 68 GB for one 128,000-token conversation. Models with compressed attention, such as DeepSeek-R1, need about 70 KB per token.

Can DeepSeek-R1 run at FP8 on one DGX Station? Only just. The FP8 checkpoint is about 689 GB, leaving little of the 748 GB for KV cache and the system. A 4-bit version is the practical option on one unit.

Which model size suits a DGX Station best for a team? Models that fit entirely in the 252 GB of HBM3e, such as gpt-oss-120b, which StorageReview measured at 9,280 output tokens per second across 128 concurrent requests. Larger models fit, but serve far fewer users at a time.