MikroTik CRS812 400G breakout to DGX Spark: 200G link caps at 106 Gb/s and NCCL falls back to Socket
We are standing up four DGX Spark nodes for disttributed training and hung them off a MikroTik CRS812-8DS-2DQ-2DDQ. The plan was one 400G QSFP-DD port on the switch feeding two nodes at 200G each, so a couple of breakout cables cover the whole cluster.
- MikroTik CRS812-8DS-2DQ-2DDQ, QSFP-DD ports used in breakout mode
- NADDOD Q2Q56-400G-CU2, QSFP-DD 400G to 2x QSFP56 200G passive DAC
- DGX Spark nodes with onboard ConnectX-7
- validation with iperf3 and nccl-tests all_reduce_perf
Both ends report a clean 200GbE link, but throughput sits at a bit over half of that:
# iperf3 -c <peer> -P 8
[SUM] 0.00-10.00 sec ... 106 Gbits/sec
single stream: ~30 Gbits/sec
# ip -br link
enp1s0f1np1 UP
enP2p1s0f1np1 UPNCCL looks worse than that: all_reduce_perf reports Socket transport and roughly 2 GB/s bus bandwidth across the four nodes, so RDMA is clearly not being used at all.
What we have already checked:
- reseated and swapped the breakout cable between switch ports, identical numbers
- raised the stream count, the aggregate stays pinned near 106 Gb/s
- confirmed the switch reports the port at 200G, not 100G
The part I keep coming back to is that one physical QSFP56 port shows up as two logical interfaces on the node. Is the breakout only delivering half the lanes to each node, or is a 200G port on this hardware supposed to look like that?
Comments 7
That is exactly what it is, and the breakout cable is innocent. On this platform the ConnectX-7 port is attached over PCIe x4 and gets exposed as two logical halves, each carrying about 100 Gb/s. The full 200G only shows up when both halves are loaded at the same time, so a single-address iperf3 run stopping just past 100 Gb/s is the expected result rather than a fault.
What made it work here:
With both halves carrying traffic you should land close to line rate on the pair, and nccl-tests all_reduce_perf will report a completely different class of bus bandwidth once it stops going through sockets.
Before you blame the cable, how exactly are you driving those two interfaces? You pasted enp1s0f1np1 and enP2p1s0f1np1 as both UP, but is iperf3 hitting both of them, or only whichever one carries an address?
Worth posting as well: the MTU on the nodes and on the CRS812 ports, because 1500 hurts badly at this rate, and the contents of /etc/nccl.conf. When NCCL picks Socket transport it is almost always because something in that file told it not to touch the IB path, not because the fabric is broken.
Fair questions. Only enp1s0f1np1 has an address, the enP2p1s0f1np1 half is up but unconfigured, and every iperf3 run so far went to that single address. MTU is 1500 on the nodes and I have not touched the L2 MTU on the switch side either. And yes, there it is:
That was already in the image and I never questioned it. So the 106 Gb/s is possibly just one half of the port plus a little headroom?
Different stack, same shape of surprise. My side was a ConnectX-6 (MT28908) under RHEL 8.4, MLNX_OFED 5.7, firmware 20.32.2004, MFT 4.21; the far end a Switch-IB 2 SB7800, and between the two a LinkX MCP7H50-H002R26 splitter taking 200G down to 2x100G. The port came up as
and neither mlxlink -d mlx5_0 -p 1 --speeds edr nor --speeds hdr changed a thing. Turned out to be a ceiling in the silicon rather than a config error: EDR gear forms links only 1x or 4x wide, and a splitter of that kind leans on 2x grouping, which arrived with HDR. So either a plain 4x EDR cable into that switch, or move to HDR if the split is a must.
Moral for breakouts in general: work out what lane grouping each end can actually form before you measure anything.
One thing from the recipe above deserves spelling out, because it is where people lose the gains again: the MTU has to match on the switch too. Setting 9000 on the hosts while the ports still pass 1500-byte frames buys you drops rather than throughput. Set the L2 MTU on the CRS812 ports and verify with a large-payload ping before you rerun any test.
The IPv6 point is the same story. With both halves addressed and IPv6 left on you get extra GID entries per port, and the index your test was told to use is not necessarily the one you think it is. Turning IPv6 off on the CX7 interfaces keeps that table small and predictable across reboots, which matters more than it sounds when you are comparing runs.
Worth keeping in mind that a breakout port which stays down is a completely different animal from yours. On a second-hand Arista DCS-7060CX-32S (EOS 4.16.8FX-7060X) my four sub-interfaces showed no errors at all and simply never linked. The box was sitting at transceiver qsfp default-mode 4x10G while the far end presented 25G, and Et17/1 stayed errdisabled until the rate was set by hand:
In the same session a Mellanox cable was refused as an unqualified transceiver and had to be swapped for an Arista-compatible 100GBASE-CR4 QSFP28 DAC. At the other extreme, a colleague never got a QDD-4X100G-2P5M breakout up on a QFX5220-32CD running Junos 23.2R1-S1.8-EVO toward a Mellanox switch, with a port profile the Port Checker validated and every FEC variant tried. Yours at least negotiates and passes traffic.
Confirmed, and the port was never broken. I addressed enP2p1s0f1np1 as its own interface, set MTU 9000 on the nodes and on the switch ports, turned IPv6 off on both CX7 interfaces and remved NCCL_IB_DISABLE=1 frrom /etc/nccl.conf.
Two parallel iperf3 sessions, one per half, now give 196-198 Gb/s aggregate. all_reduce_perf across the four nodes reports 23.76 GB/s bus bandwidth and NCCL is on the RDMA path, no Socket transport in the log any more. The NADDOD Q2Q56-400G-CU2 had been doing its job the whole time.