Accton AS9716-32D at 100G: INNOLIGHT QSFP28 LR4 links to Arista but never to a Fortigate 3810D CFP2-LR4
We are bringing up a leaf on an Accton AS9716-32D running SONiC master (build master.207726). Four of the 400G-capable ports have to run at 100G towards a firewall pair, and not one of them will link.
- Accton AS9716-32D, SONiC master.207726
- INNOLIGHT TR-FC13R-N00 QSFP28 LR4 in the switch cage
- Fortigate 3810D with a CFP2-LR4 on the far end
- ports forced to 100G, autoneg off, RS FEC, MTU 9100
sudo config interface speed Ethernet0 100000
show interfaces status
Interface Speed MTU FEC Admin Oper
Ethernet0 100G 9100 rs up downWhat I tried:
- the port as 2 lanes with PAM4 and as 4 lanes with NRZ, no difference in either direction
sudo config interface fec Ethernet0 noneinstead of RS FEC- the same INNOLIGHT modules and the same fibre against an Arista QSFP28 port, where the link comes up immediately
So the optics and the fibre are cleared. Something about how this 400G port is driven at 100G, or about what the CFP2 side expects on the lanes, keeps it dark. Has anyone got a 400G cage on this platform talking 100G to a CFP2-LR4, and what did the port profile look like?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 7
Two questions before anyone starts on lane arithmetic. What does the Fortigate say on its side, does the CFP2 see light and stay down, or does it not register the module at all? And what FEC is configured on the 3810D port?
On an LR4 pair, RS FEC enabled on exactly one end is enough to keep the link down forever, and CFP2-LR4 ports usually have no RS FEC to turn on in the first place.
The Fortigate shows the interface down and nothing useful in the counters. The module is recognised there and that port carried a 100G link to another box before we touched anything. I cannot find a FEC setting on that side at all, exposed or hidden.
On our side RS FEC comes off cleanly: after
sudo config interface fec Ethernet0 nonethe FEC column readsnone, the port bounces and comes back admin up, oper down in exactly the same way. So dropping RS FEC on the switch alone buys us nothing.The two profiles you tried are not equivalent, and one of them cannot work by construction. A CFP2-LR4 is a four-lane NRZ interface: four 25G electrical lanes towards the host, four wavelengths on the fibre. If your port profile puts the cage on 2 lanes of PAM4, the switch is driving 2x50G into a peer that has to hand four NRZ lanes to its line side, and nothing trains regardless of the FEC setting. The only candidate profile in this pairing is 4 lanes of NRZ.
With that fixed, the remaining variable is FEC. 100GBASE-LR4 does not carry RS FEC, and a CFP2-LR4 port typically offers no way to switch it on, which matches what you found on the Fortigate. If your side still has
rson that port, that alone is enough to explain a permanently dark link.So: 4-lane NRZ, autoneg off,
sudo config interface fec Ethernet0 none, bounce the port, then readshow interfaces statusback to check that both settings actually stuck.Adding the boring standards angle to that. Nothing negotiates FEC here, it is a static property of both ends and it has to match on every lane of the link. The vendor guidance for channelised 100G says the same thing in one sentence: when a 100G port is broken into 25G lanes, both ends of every lane must run the same FEC mode or the lanes never train.
It is also worth dumping what the TR-FC13R-N00 EEPROM actually claims. These modules report 100GBASE-LR4 and 25GBASE-LR compliance, and some platforms pick their default lane mode from that field rather than from what you typed into the config.
Different vendors, same shape. I had a 100G link between a MikroTik CCR2216 and an Arista over QSFP28 SR4 optics that had been up for months and then simply stopped coming up. RouterOS had gone from 7.8 to 7.12.1 in that window. Both ends forced: auto-negotiation=no, fec-mode=fec91, MTU 9000 and 100G-baseSR4-LR4 on the CCR, 100gfull with reed-solomon error correction on the Arista. Replacement optics changed nothing, received power on all fur lanes was healthy, and splitting the port into 4x25G only gave me flapping lanes.
Never foound the answer there either, but the pattern is the same as yours: the optics are innocent and one side's port driver changed underneath you. If you are sitting on a master snapshot, that is the first variable I would move.
Careful with the last step. Setting FEC to none and seeing
nonein the output is not proof that the port is running without RS FEC. On several of these platforms the value lands in the config database and the CLI prints it back happily, while the serdes keeps the mode it was programmed with until the port is really re-created. A speed change on a 400G-capable cage re-creates it anyway, so the two knobs interact and the order matters.Same caution for the lane profile: on a breakout-capable cage that comes from the platform port configuration, not from
sudo config interface speedalone. If the profile was never regenerated, both of your attempts may well have run on the same underlying lane map, which would explain why PAM4 and NRZ looked identical. Reload the box after the change and check from a cold start.Worked through all of it. Port re-created as 4 lanes of NRZ, FEC none on our side, autoneg off, config saved, full reload, checked again from a cold start. The CFP2 pair is stiill down. RS FEC on the Fortigate is genuinely not exposed, so we cannot even create a matching mismatch from that end.
Meanwhile the same INNOLIGHT TR-FC13R-N00 in the same cage links against an Arista QSFP28 port within seconds, so the modules and the fibre stay cleared.
I wanted to attach the full technical dump but it is over the size limit here. For now the four ports stay down and the firewall uplinks moved to another box. If anyone has a 400G port on this platform actually running 100G into a CFP2, I would like to know which profile you ended up with.