Could Your S$1 Million AI Cluster Be Wasting Money Because of Poor Networking?

It could, and the loss would show up as idle GPUs rather than as a line on an invoice. In an illustrative build priced from public US list prices, S$1 million does not quite buy two 8-GPU Blackwell servers with a basic InfiniBand fabric: that comes to about S$1.1 million, of which the network is only about 7%. The hardware cost of the network is small. The cost of a poor network is larger: on our illustrative three-year straight-line basis, every 10% of GPU time lost to communication stalls, congestion or flapping links is worth about S$110,000, more than the whole network in this build. Published studies show losses of that size are common in clusters that have not been designed and tuned for AI traffic. All prices here are illustrative.

What does S$1 million buy in AI hardware?

Source: illustrative only. HGX B200 (Supermicro SYS-A22GA-NBRT-G1) at US$401,609 as configured by AVADirect; QM9700 at US$35,000 (FiberMall); 400G SR4 optic US$1,299 and 800G SR8 optic US$779 (NADDOD), all fetched 6 Nov 2026 and converted at USD/SGD 1.2774. SN3700 at SourceIT's listed price of S$28,523.25 (6 Nov 2026). Excludes GST, import, local margin, storage, racks, power distribution, cooling, colocation, software and support.

So a S$1 million budget realistically buys one 8-GPU node and its fabric, with about S$425,000 left for storage, a front-end network, racks, power, cooling, hosting and support, or two nodes if the budget stretches by about 10%. Reseller prices vary widely: one HGX B300 server was listed at US$522,559 by one reseller and US$760,000 by another. Treat these as orders of magnitude, not quotes.

How does poor networking waste GPU time?

By making GPUs wait for each other. Distributed training synchronises after every step, so the slowest exchange sets the pace for the whole job. Published measurements show how much time that can take:

• Communication share: Microsoft's DeepSpeed team measured tensor-parallel communication at 22–47% of training iteration time across GPT-3 and Llama 2 sizes on one to four DGX H100 nodes.
• Mixture-of-experts models: DeepSeek reported a computation-to-communication ratio of about 1:1 for cross-node expert parallelism and dedicated 20 of each GPU's 132 streaming multiprocessors to communication.
• Untuned fabrics: Meta said communication performance on its large RoCE Ethernet clusters was initially "poor and inconsistent", between about 10% and 90% of expected, before scheduling, routing and NCCL tuning brought it above 90%.
• Congestion: in an Alibaba study of more than 10,000 GPUs, 16 of 27 large jobs hit "fail-slow" episodes with an average 1.34 times delay, and network congestion caused 42 of 43 communication slowdowns seen in small-scale probes.
• Link faults: network switches, cables and adapters accounted for 42 of 419 unexpected interruptions, about 10%, during 54 days of Meta's Llama 3 405B training.

Not every slowdown is the network. ByteDance found that 42.5% of the jobs it traced were slowed by stragglers, wasting about 10% of allocated GPU-hours, but mostly because of uneven computation rather than the network. A good network removes one major cause, not all of them. Our post Your GPU Has 20 Petaflops of Power. So Why Is Your AI Model Still Slow? covers the other bottlenecks.

What is idle GPU time worth in dollars?

Take build B at S$1,105,250 and spread it over three years. That is about S$30,700 of hardware per month before power, hosting or staff. If 10% of GPU time is lost to communication stalls, the lost value is about S$3,070 a month, or about S$110,500 over three years. That is more than the S$79,219 the network cost in the first place. Put another way, in this build a network problem that wastes more than about 7% of GPU time costs more than the network. Adding hosting makes it worse: each 8-GPU server draws roughly 14 kW, and at CBRE's Singapore colocation range of US$330–475 per kW per month (via W.Media, July 2026), two servers cost about US$9,200–13,300 a month to host whether they compute or wait.

The network is often the smallest line in an AI hardware budget and the biggest lever on how much of the rest is actually used.

What networking mistakes waste the most money?

• Too few ports. An HGX server has eight 400G network adapters, one per GPU. A 4-port 400G switch such as the MikroTik CRS804 can link two systems on a single rail, but it is not a fabric for an 8-GPU server.
• Saving on speed without checking the workload. Build B′ saves about S$50,700 by using a 200GbE switch, but it halves per-GPU network bandwidth compared with 400G. That suits inference or data-parallel fine-tuning; it can slow communication-heavy multi-node training.
• Ignoring rails. NVIDIA's reference designs connect GPU n in every server to the same leaf switch ("rail-optimized"), so most traffic is one hop. Random cabling sends more traffic through the spine.
• Oversubscribing the wrong layer. Meta's Llama 3 cluster is non-blocking within 3,072-GPU pods and 1:7 oversubscribed between them, with jobs placed to match. Oversubscription without topology-aware scheduling causes congestion.
• Forgetting the other networks. Storage, front-end and management traffic need their own ports. NVIDIA's B300 reference design gives each node 200 Gb/s of front-end Ethernet and requires at least 100 Gb/s to user storage.
• Cheap, unqualified cables. ByteDance traced link flapping to poor adapter-cable-switch link quality. See the hidden cost of AI networking.
• No measurement. Run NCCL tests at installation and after changes. For all-reduce, bus bandwidth = algorithm bandwidth × 2(n−1)/n; an 8-GPU result of 400 GB/s algorithm bandwidth equals 700 GB/s bus bandwidth, which is the figure to compare with link speed.

Is the fabric choice itself a big cost?

Less than the design. At third-party list prices, the newest InfiniBand and Ethernet switches cost about the same per terabit of capacity, and both are used in very large training clusters. The difference is mostly operational: tuning effort, tools and skills. InfiniBand vs Ethernet compares the two, and why your next AI data centre needs more than GPUs covers storage, power and cooling.

Is a group of deskside systems an alternative?

For some teams. Using the S$150,000 ballpark from this series for a DGX Station GB300 (public US pricing of about US$93,000–125,000 is roughly S$119,000–160,000 before GST), S$1 million covers about six units. Each has one Blackwell Ultra GPU with 748 GB of coherent memory and two 400 Gb/s ConnectX-8 ports, and needs no data hall. Units connect over Ethernet, not NVLink, so six Stations suit six teams or users working independently far better than one large training job. See DGX Station's networking, from DGX Station to GB300 NVL72 and DGX Station vs traditional GPU servers.

How should a S$1 million project be planned?

• Define the workload: inference, fine-tuning or multi-node training, and model sizes.
• Decide what must run as one job; that sets the scale-up and scale-out design.
• Budget the network, storage, power, cooling and hosting alongside compute, not after it.
• Plan rails, port counts and cable lengths before ordering.
• Reserve time and budget for burn-in, NCCL testing and monitoring.
• Compare the three-year cost with renting; see Buy DGX Station or Keep Paying for Cloud GPUs?

Buying in Singapore

SourceIT can help scope departmental AI infrastructure. Listed networking includes the NVIDIA SN3700 32 × 200GbE switch at S$28,523.25, the MikroTik CRS804-4DDQ 4 × 400G switch at S$2,255.00 and the HP ZGX QSFP112 400G DAC at S$365.00 (listed 6 Nov 2026). GPU servers such as the Lenovo ThinkSystem SR680a V3 and HPE ProLiant DL380a Gen11 are quoted on request, as are DGX Station GB300 systems such as the MSI XpertStation WS300 and Dell Pro Max with GB300 (price on request). We do not list HGX servers, InfiniBand switches or 400G/800G optics; ask us to source them. Browse our AI workstations and servers. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

How many Blackwell GPUs does S$1 million buy? At the illustrative US list prices used here, one 8-GPU HGX B200 server with a basic fabric costs about S$575,000. Two servers with 16 GPUs come to about S$1.1 million before GST, storage, power and hosting.

What share of an AI cluster's cost is networking? In the small illustrative builds here, 3–11%. Published estimates for large clusters range from about 10% for Ethernet to 20% for InfiniBand, and one industry editor argues it can reach 30%.

How much GPU time can poor networking waste? It varies. Meta saw untuned large-cluster communication perform at 10–90% of expected, and Alibaba found network congestion behind most communication slowdowns it studied. Well-tuned clusters recover most of that.

Is 200G networking enough for AI? For inference and data-parallel fine-tuning on a few nodes, often yes. For communication-heavy multi-node training, it halves per-GPU bandwidth compared with 400G and can slow jobs.

How do I check whether my network is the bottleneck? Run NCCL tests and compare bus bandwidth with the link speed, monitor link error counters, and check how much of each training step is spent in communication.