ConnectX-4 MCX456A-ECAT will not link to a Cisco NCS on 100GBASE-LR4 while loopback passes on both ends
We run a pair of racks that hand off to a carrier-facing Cisco NCS, and one of the 100G server uplinks has never come up since the build. Same result after moving the server to another cabinet with a different patch panel, so I have stopped treating it as a one-off.
- Supermicro server, NVIDIA/Mellanox ConnectX-4 MCX456A-ECAT, both ports free
- generic 100GBASE-LR4 QSFP28, single mode, 10 km reach, one at each end
- Cisco NCS on the far side, dark fibre between the two rooms
The part that keeps me stuck: each module passes a loopback on its own device. With the fibre looped straight back into the same QSFP28 the NIC reports a clean 100G link, and the NCS does the same on its side. Put the real span in between and there is nothing.
# module looped back on the NIC itself
Speed: 100000Mb/s
Link detected: yes
# same module, real span to the NCS
Speed: Unknown!
Link detected: noWhat we have already tried:
- swapped both modules for spares out of the same batch, no change
- moved the server and re-patched through a different panel
- opened a case with the NIC vendor, and the answer was a pointer to the validated transceiver list in the firmware release notes, which does not explain why loopback works
Is there something about LR4 on the ConnectX-4 that would let it link locally but never across a real span, or am I chasing the wrong end of this?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 3
This reads like a dirty path, not a compatibility problem.
Everything you have replaced so far sits on the side that already tested good, which is why nothing changed - the span itself is the one thing still untouched. So work the path:
The validated transceiver list you were pointed at is worth a glance, but a module that comes up cleanly in loopback is already being driven correctly by the NIC. Compatibility lists explain modules that get refused outright, not modules that link locally and die across a span.
If cleaning does not do it, the next step is a light source and power meter on the dark fibre, or an OTDR if you can borrow one, before you buy another NIC or another pair of optics.
A loopback only proves that one port can hear itself - laser, receiver, rate settings. It says nothing about the glass between your two rooms, and that is the one piece you have not tested. So before the NIC gets blamed again, get numbers off both ends with the real span patched in: what is Rx power on the NCS port, and what is it on the NIC? Rx sitting under the Low Warn threshold with the span in place is the classic pointer at the far end or at the path rather than at the local port. On the Linux side
ethtool -mshould give you the same reading, and if it comes back withCannot get module EEPROM information: Input/output errordo not read that as a dead module - on mlx5 that is usually firmware-side module access, andmst start,mst cable addand thenmlxcableswill get you the values anyway.Cleaning was the answer. We put a scope on the end faces and both modules plus both patch cords were contaminated; the cord running through the inter-room panel was the worse of the two. Cleaned everything in the path, re-seated, and the 100G link to the NCS came up on the first attempt and has stayed up since.
Mildly annoyed at myself for spending that long on the compatibility angle when the loopback result was telling me all along that the modules were fine and the path was not. For anyone who lands here later: loopback proves the port, not the fibre.