ConnectX-7 400G OSFP to a QSFP-DD switch port: is a passive DAC enough, or do the lane rates get in the way
Bringing up a pair of GPU boxes against our existing 400G fabric, and I have already managed to buy the wrong cable once, so I would rather get the theory straight before ordering again.
- NIC: ConnectX-7 MCX715105AS-WEAT, single 400G port, OSFP cage
- Switch: 400G QSFP-DD ports
- In hand: a passive 400G QSFP-DD to OSFP DAC with a finned OSFP end
switch port : QSFP-DD 400G
NIC cage : OSFP, flat top
cable : QSFP-DD to OSFP passive DAC, finned OSFP endThe finned end does not seat in the NIC cage at all - that housing is built for a switch cage with room for the fins, and the card has none. Beyond the mehcanics, everything I have read so far says the OSFP port on this card runs 100G lanes while 400G on QSFP-DD is built from 50G lanes, which suggests a passive cable cannot bridge the two ends whatever the connectors look like.
What I checked before posting:
- went through the validated cable list and found no QSFP-DD to OSFP entry matching this pairing;
- confirmed the flat-top requirement for the NIC cage with the card documentation;
- asked the supplier for a flat-top version of the same DAC, which they can supply, but that does nothing about the lane question.
So is there any passive DAC that genuinely connects a QSFP-DD 400G switch port to an OSFP ConnectX-7, or is transceivers plus fiber the only honest answer here?
Comments 5
Which switch, though? You named the port type but not the model, and that is what decides whether thee is a usable breakout profile on that side at all.
Also say whether you need one honest 400G link end to end, or whether 4x100G into the card would close the task just as well. Those are two rather different shopping lists, and the second one is a good deal shorter.
You have two separate problems and only one of them is about the cable you bought.
Mechanical: the CX-7 cage takes flat-top OSFP modules. Most OSFP DACs on the market ship in the finned housing meant for switch cages, and that is exactly what you have in hand, so it will never seat properly in the card.
Electrical: the 400G OSFP port on that card runs four lanes of 100G, while 400G on QSFP-DD is eight lanes of 50G. A passive DAC is copper and nothing else, it gears nothing, so even with a flat-top end on the NIC side the two ends never agree on lanes. That is the part people tend to discover after the cable has already arrived.
The combination pointed at for this exact pairing is optical, not copper: a 400G DR4 QSFP-DD transceiver (MMS1V00-WM) in the switch, MPO-12/APC crossover fiber from the MFP7E30 family, and a flat-top OSFP module (MMS4X00-NS400) in the NIC. Three line items for one link, which is annoying, but it is the set that is actually meant to work together.
NIC to NIC is the easy case for what it is worth: flat-top OSFP800 DACs do that without any of the above.
Copper is not entirely off the table, it just depends on which card is in the box.
The dual-port 200G CX-7 with QSFP112 cages takes QSFP56 DACs off a QSFP-DD switch port through a fanout without drama, because the lane rates line up there. And on the 400G OSFP part someone reported running an OSFP to QSFP28 adapter with ordinary 100G DACs into an existing fabric, so the mechanical sde can be worked around.
The caveat is that the adapter route has no backing whatsoever. Asked point blank about passive OSFP to QSFP28 adapters for exactly that case, NVIDIA sent people off to read which cables are validated in the ConnectX-7 firmware docs, and added that adapters are neither tested nor guaranteed. Someone who asked the same thing about an SN4700 on QSFP-DD and an SN3700 on QSFP56 got nothing better. It may well work on your bench; nobody will help you the day it stops workig.
Data point from the generation above, because it is the same family of surprises.
An Arista 7060X6 with an ACC OSFP 800G to 2x400G QSFP112 splitter would not link to a ConnectX-8 at all, while a ConnectX-7 on the same port with the same cable came up immediately. NIC diagnostics returned a PHY FW group status carrying
Recommendation: Negotiation failure. Forcing the port speed to 400G instead of letting it auto-negotiate brought the physical link straight up. AEC cables of the same shape worked with the ConnectX-8 unchanged, and the same behaviour turned up against other Tomahawk 5 based switches, which points at an auto-negotiation and link-training interop issue rather than a bad cable.Worth filing away for when your optics do arrive: if a link stays down, take auto-negotiation out of the picture before blaming the transceiver.
The lane-rate point deserves underlining, because it bites on the InfiniBand side just as hard.
We had a Quantum-2 QM9700 that needed to reach an old Switch-IB 2 SB7800. NDR drives every OSFP cage as four PAM4 lanes of 100G; the SB7800 is EDR, four NRZ lanes of 25G. A direct link between those two was never validated and is not supported at all - the blessed path puts an HDR switch in the middle. What exists as an unsupported alternative is a passive splitter: MCP7Y60-H001, MCP7Y60-H01A, MCP7Y60-H002 in 1 m, 1.5 m and 2 m. One twin-port cage on the NDR side fans out to a pair of QSFP28 ports on the old switch, each half dropping down to EDR rates on the way.
Read the strings attached twice, though. Copper EDR out of the QM9700 tops out at 2 m, so the splitter is a same-rack trick and nothing more. And you spend a cage: what used to be one OSFP port now counts as two in the topology. Different gear, same lesson as yours - whn the two ends disagree on lane rate, plain copper is not the thing that reconciles them.