IBM Flex System EN4093 drops SFP+ trunk ports into ERRDISABLE at boot and during operation
Two Flex System enclosures, each with an EN4093R 10Gb Scalable Switch (option 49Y4270) as the network module. Ports come back error-disabled after a chassis power event, and less often drop into the same state while everything is running. It is not always the same ports, which is what makes this so tiresome to chase.
Setup:
- IBM Flex System EN4093 10Gb Scalable Switch, option 49Y4270
- uplink trunk of four ports to the core, IBM optics plus one DAC that was added later
- QSFP+ port broken out towards the second enclosure
- MSTP running on our side, core is a different vendor
What the port list shows after a boot:
port 5 ERRDISABLE reason: link flap detect threshold exceeded
port 17 ERRDISABLE reason: mismatched link capabilities
port 19 ERRDISABLE reason: mismatched link capabilitiesTried:
- shutdown / no shutdown on the affected ports brings most of them back until the next event
- cleaned and reseated every fibre in the trunk, no change to the pattern
- compared the far-end port configuration, speeds match on paper
What is actually pushing these ports into errdisable, and is there a way to stop it happening at every boot instead of clearing it by hand each time?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 5
This switch has seven documented states that will disable a port, and they have very little to do with one another:
Yours is the fourth one, and it is the one most people meet, because it is the one you build for yourself: mix module speeds or types inside a single trunk and the switch objects. One DAC sitting beside three optical modules is enough. Take the DAC out of there and fill the slot with a module matching the other three.
For the port on the flap detector, bouncing it is still the documented recovery:
After that, read back the spanning tree configuration on that link and go over the copper and the fibre by hand. Follow any configuration change with a reload, or the running state quietly stops matching what you think you set.
Two things not to expect. Two of those seven states keep the port down after the timeout expires and want a hand on the port. And no firmware release fixes this - the vendor's line is that you configure your way around it, so identical modules across every trunk member plus clean fibre is as far as prevention goes.
Two different reasons in the same paste is where I would start. Does a given port always come back disabled for the same reason, or does one that fell over on capabilitise this boot trip the flap detector the next? A reason that stays put per port and a reason that wanders are two separate investigations, and only one of them ends with you buying hardware.
The second thing worth nailing down is what the core runs for spanning tree. You are on MSTP; if the far side puts Cisco-flavoured PVST BPDUs onto those uplinks, this switch has a protection mechanism that reacts to exactly that and takes the port down, and from the outside it looks like the failure you are already chasing. Can you tell whether any of the drops line up with a topology change on the core rather than with your own boots?
Per port it is consistent, across the box it is not. The trunk members that fall over always come back with mismatched link capabilities, and the access port on 5 only ever trips the flap detector, on no interval I can find a pattern in. So it does look like two faults wearing the same jacket.
Spanning tree on the far side I cannot answer yet - the core belongs to another team and I have asked them what it actually emits on those uplinks. Nothing in our log ties a drop to a topology change over there so far, but I was reading it for link events rather than for that, so I would not call it ruled out.
Different vendor, same shape. We had a FortiGate 201F hanging off a FortiSwitch 548D on SFP+ with Fortinet's own DAC between them, and the 10Gbps link simply would not stay up - it dropped whatever we did to speed and duplex, and rolling both ends back from FortiOS 7.4 to 7.2.5 changed nothing.
What held in the end was a shorter Fortinet DAC with STP switched off on that one link, and it has been up ever since. The part worth carrying over is the reasoning that came afterwards: the longer the passive copper run, the more the signal has degradd by the time it arrives, so at 10G any long or borderline DAC belongs on the suspect list whatever label is printed on it. With three optics and one DAC in an errdisable trunk, I would be looking hard at the odd one out.
One thing to do before you touch any hardware: write down the exact cadence of the flaps against the log timestamps.
On a completely unrelated switch, a TL-SG3452X, every populated SFP+ port went down and back up every ten to fifteen minutes with STP messages in the log at each flap, and the obvious conclusion was bad optics - until a first-party TL-SM5220-1M DAC flapped in exactly the same rhythm. That killed the optics theory in a single test and pointed at a firmware regression instead; the only working answer there was staying on the older build.
A regular interval means something is timing out on a schedule. A random one means something physical. Cheap test, and it saves you buying modules you do not need. It also pays to keep the previous firmware image around so a downgrade stays on the table.