Fortinet 25G DAC links between identical FortiSwitch units but not FS2048 to FS648
We are collapsing two aggregation rows onto FortiSwitch, and the last piece is a 25G interconnect between an FS2048 and an FS648 in adjacent racks. Everything else in the design came up on the first attempt; this one link refuses to.
- FortiSwitch 2048, 25G front-panel port
- FortiSwitch 648, 25G front-panel port
- Fortinet FN-CABLE-SFP28-5 passive DAC, vendor cable, not third party
- Both ports otherwise untouched apart from the VLAN config
What I get:
FS2048 port: down, no rx/tx counters moving
FS648 port: down, no rx/tx counters moving
same FN-CABLE-SFP28-5 between two FS648 units: up at 25G, stableAlready done:
- swapped in a second FN-CABLE-SFP28-5 from the same box, no change
- moved both ends onto different 25G ports on each chassis, no change
- proved the cable is good by patching it between two identical FS648 units, where it comes up immediately
So the cable is fine and the ports are fine, but the combination is not. Is there something on the 25G ports that has to match between the two models before the link will train?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 6
That is exactly it. The two chassis are built on different ASIC and PHY generations, and the error correction each one settles on by itself at 25G is not the same on both sides, so the link never finishes training. You are left with a clean down/down and nothing in the logs to chew on.
Pin the same FEC mode by hand on both ports:
Do it on the FS2048 and on the FS648, using each side's own port name. CL91 is the Reed-Solomon variant and cleans up a good deal more than the CL74 firecode option, but which of the two you choose matters far less than choosing the same one twice: both PHYs have to encode and decode with an identical scheme or the training never completes, and "auto" on two different PHY families is not an identical scheme.
The port should come up as soon as the second side is committed. If you later mix a third model into this, set it explicitly there too rather than assuming the default carried over.
A cable that links between identical units and dies between different models is the physical layeer failing to agree on something, and on 25G copper that is nearly always FEC.
Before anything else, post what fec-state is currently configured on both ports. The default is not the same across FortiSwitch generations, and the two models you are joining are not the same ASIC/PHY family, so "factory settings on both ends" does not mean "the same setting on both ends".
If the two ports come back with different values there, you have your answer before touching anything else.
Nothing was touched on either side, so both ports are running whatever the image sets by default. The FS2048 side is bare:
Same on the FS648. Speed and auto-negotiation untouched as well, I only put the ports into the right VLAN. If the defaults differ per model, that would explain why the same cable is perfectly happy between two identical boxes.
Same class of problem well outside Fortinet, for what it is worth. I had a 25G SFP28 passive DAC that linked happily between a UniFi USW-Pro-Aggregation and a server with an Intel SFP28 card, and gave nothing at all on sfp28-2 of a MikroTik CCR2004-1G-12S+2XS - no error on either side, just no link. Tried a Ubiquiti UACC-DAC-SFP28-3M and a Lenovo 7Z57A03558, same result. At one point the port did come up and dropped again after about two seconds, which was the hint that something was failing to train rather than the cable being dead.
FEC again: the Ubiquiti side keeps FEC on with no supported way to change it, and RouterOS moved the default from fec91 to no FEC in 6.49. What worked here was going to RouterOS 7.4, where the FEC options are exposed, running
and then setting the port to fec74 with auto-negotiation off, flow control off in both directions, 25 Gbps full duplex, plus a port profile override on the UniFi side pinning 25G FDX. That is my hardware and my firmware, so treat the exact recipe as a starting point and verify on yours.
Worth adding that the same knob has a different name depending on whose CLI you are standing in, which is what makes this painful once a rack holds more than one vendor. On Cisco 25G links between a Catalyst 9300 stack and a Catalyst 9500 pair, fec cl108 on both ends was what brought them up; on the 100G between that 9500 pair and a Nexus 9000, what worked was fec off on both sides. Same decision, different keywords.
And FEC is not always something you turn on. On a Nexus 93180YC-EX with SFP-H25GB-SR into a Cavium 25G adapter, the switch port sat at FEC auto and expected FEC because of the optic, while the NIC reported no FEC capability at all, so the ends never agreed and the interfaces stayed down with the modules recognised. There, fec off on the switch interface was the fix, and show interface confirms the mode moving from Auto to Off.
So the rule is not "use cl91", it is "decide the mode, then set it explicitly at both ends".
Confirmed. set fec-state cl91 on the FS2048 port changed nothing on its own, then the same on the FS648 port and the link came up within a couple of seconds. Counters moving on both sides, and it survived a reboot of each chassis.
From now on the setting stays explicit on every 25G port in this pair rather than trusting a default. Two evenings of swapping perfectly good cables for one line of config.