NVIDIA ConnectX-7 vs Ordinary Ethernet: Why AI Clusters Need More Than Fast Internet

Because an AI cluster is not moving web pages or files; it is moving pieces of a model's working memory between GPUs, thousands of times a second, while the GPUs wait. Ordinary office Ethernet, even at 10 Gb/s, sends that data through the operating system's network stack and the CPU, which adds delay and caps throughput. The NVIDIA ConnectX-7 network chip inside every GB10 system, such as the NVIDIA DGX Spark (S$8,800) and ASUS Ascent GX10 (from S$8,100), adds two 200 Gb/s ports with RDMA over Converged Ethernet (RoCE), which lets one machine write into another's memory with far less overhead. It is still Ethernet underneath, but used in a very different way. Prices at SourceIT, listed 28 Oct 2026.

What is the difference between the two kinds of port?

From NVIDIA's DGX Spark user guide. The QSFP ports are Ethernet-only; the NIC connects to the GB10 chip over two PCIe Gen5 x4 links, so each physical port appears as two network interfaces in Linux, each with a matching RoCE device.

What is RDMA, and why does AI need it?

Remote Direct Memory Access lets one computer read from or write to another's memory without involving either machine's operating system for every packet. In normal networking, data is copied through the kernel's network stack and handled by the CPU on both ends. That is fine for email and file shares. For AI it is a problem, because when a model is split across machines, every generated token requires many small exchanges between them, and each one waits for the slowest step.

RoCE carries RDMA over Ethernet. NVIDIA's NCCL library, which handles GPU-to-GPU communication in PyTorch, vLLM and similar tools, uses the ConnectX-7's RoCE devices. Its October 2026 eight-unit work added a new all-to-all communication backend, called ROCEnante, that runs on switched topologies. None of that runs over the 10 GbE port.

Fast internet moves data to you. An AI fabric moves data between GPUs that are waiting for it.

Why isn't a faster office network enough?

Three reasons. Speed: 200 Gb/s per port is 20 times the 10 GbE port. Overhead: RDMA avoids the CPU and kernel work that ordinary TCP traffic needs. And the measurements show how much that matters. Users on NVIDIA's developer forum report that a single ordinary TCP stream (iperf3) across the GB10 link reaches only about 27 to 33 Gb/s, rising to about 106 Gb/s with four streams, while RDMA tests (ib_write_bw) reach about 111 Gb/s on each of the port's two PCIe halves. NCCL's all-reduce test shows about 27 GB/s of bus bandwidth. The same cable carries far more useful data when the software uses RDMA.

One limitation has been reported: according to a post on NVIDIA's forum, GPUDirect RDMA, which lets the network card read GPU memory directly on data-centre systems, is not supported on GB10, and applications use host memory buffers instead. That is one reason a GB10 cluster does not scale like a data-centre GPU server. See Can Eight NVIDIA GB10 Systems Compete With Enterprise AI Infrastructure?

What cables and switches does it need?

• Two units: one QSFP112 direct-attach copper cable. NVIDIA's guide names approved part numbers from Amphenol and Luxshare. SourceIT lists the NVIDIA QSFP112 400G DAC, 40 cm at S$155 and the HP ZGX QSFP112 DAC at S$365.
• Four or more units: a switch with a 200 Gb/s Ethernet port for each unit, such as the MikroTik CRS804 DDQ (S$2,255) or NVIDIA Spectrum-2 SN3700 (S$28,523.25). See 400Gbps Networking for AI: Do You Really Need Such an Expensive Switch?
• No faster cables. NVIDIA states: "Faster cables don't help because the port is capped at 200 Gb/s."
• One cable per link. NVIDIA says a second cable between two units adds no performance.

Prices listed 28 Oct 2026.

Should ConnectX-7 traffic go on the office network?

No. Keep the QSFP links as a private network between the AI units, cabled directly or through their own switch. Keep each unit's 10 GbE port on your office network for users, remote access and updates. Mixing them gains nothing: office switches rarely have 200G ports, and RoCE traffic is sensitive to congestion from other traffic. If your office network itself is slow, the causes are usually elsewhere; see Your Internet Is 10Gbps. So Why Is Your Office Network Still Slow?

Do you need it at all?

Only if you link GB10 units together or connect them to very fast storage. A single GB10 serving staff over the office network never uses its ConnectX-7 ports. Two units linked for a model that needs 256 GB is the most common reason to use them; the benefits and limits are in Two NVIDIA GB10 Systems, One AI Model: Does Performance Actually Double?, and the setup steps in How to Cluster Two ASUS Ascent GX10 Units.

Buying in Singapore

SourceIT supplies GB10 systems from ASUS, NVIDIA, HP, Dell, Lenovo, GIGABYTE and MSI, QSFP cables and 200G-capable switches, with local warranty and a GST invoice, and can confirm cable compatibility and commission a cluster. Browse the NVIDIA DGX Spark collection. For a quotation, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

What is ConnectX-7 on the DGX Spark? NVIDIA's network chip that provides two QSFP ports at up to 200 Gb/s each with RDMA over Converged Ethernet (RoCE), used to link GB10 units into a cluster. It is separate from the 10 GbE office network port.

Why can't GB10 units cluster over normal 10 GbE? AI clustering needs high bandwidth and low overhead for constant GPU-to-GPU exchanges. The 10 GbE port lacks the speed and RDMA support, and NVIDIA's cluster tools use the QSFP ports.

What is RoCE? RDMA over Converged Ethernet: a way for one machine to read or write another's memory over Ethernet without the operating system handling every packet, which cuts delay and CPU load.

Does GB10 support GPUDirect RDMA? According to a post on NVIDIA's developer forum, no. Applications use host memory buffers registered for RDMA instead.

Will a 400G cable make the link faster? No. NVIDIA states the GB10's ports are capped at 200 Gb/s, so faster cables do not help.