OCe14000 LoM reports link up with the cable removed, so ESXi teaming never fails over
Small vSphere cluster, two 10G uplinks per host into a pair of top-of-rack switches. After the ToR was rebooted for a firmware update, part of the VMs on one host went silent and vMotion failed on both uplinks - yet ESXi never marked anything down and the adapter LEDs stayed lit the whole time.
- Fujitsu Primergy RX2540 M1
- Emulex OneConnect OCe14000 LAN-on-motherboard adapter (VID 10df DID 0720 SVID 1734 SSID 120e)
- ESXi 6.0 U3
- SFP+ optics into the ToR, plain active/standby teaming on the vSwitch
What convinced me this is not the switch: I pulled the fibre out of the adapter completeely and it still looks alive.
esxcli network nic get -n vmnic2 (cable physically removed)
Link Status: Up
Firmware Version: 11.2.1194.36
esxcli software vib list | grep elxnet
elxnet 10.2.309.6vAlready tried:
- reseated the optic and the fibre, swapped the patch lead
- moved the uplink to the other ToR switch, whose port goes down as expected
- restarted the management agents on the host
Because the host believes the uplink is live, the teaming policy has no reason to move anything and the VMs stay pinned to a dead port. Is this a known driver-versus-firmware problem on OneConnect, or should I be looking at the LoM hardware?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 6
That combination is your fault, and it is not listed as a supported pairing for ESXi 6.0. The adapter ends up half alive: it stops moving traffic but carries on advertising the port as connected, so the teaming policy never gets the down event it needs in order to act. That is why this reached you as a partial isolation with vMotion dead on both uplinks instead of an honest NIC failure - a card that dies properly is far easier to survive than one that lies.
Take the driver up to 11.2.1149.0. That is the elxnet level qualified against firmware 11.2.1194.36 in the VMware compatibility list, so you are moving the driver to meet the firmware rather than rolling the card back. Verify afterwards with the two commands you already have:
The vib line should show the new version, and Link Status should follow the cable again once the host is back up. Test it by pulling the fibre with a VM running on that uplink before you trust the cluster with it.
The habit worth taking away: on OneConnect the driver and the firmware move as a pair, so schedule the two together instead of letting a maintenance bundle drag one of them forward on its own. Comparing those two lines takes one command, and it belongs long before you start pulling optics or blaming the switch.
The two lines you posted are the interesting part: firmware 11.2.1194.36 running underneath elxnet 10.2.309.6v. Did that firmware arrive with a server maintenance bundle at some point, separately from the driver?
Check that pairing against the compatibility list for ESXi 6.0, not against "newest must be fine". OneConnect is one of the families where driver and firmware are qualified as a pair, and a mismatched pair does not necessarily fail loudly - it half works, which is much worse.
Also worth saying whether the adapter's second port behaves the same way with its cable out.
Yes, the firmware went on with a server maintenance bundle; the driver has not been touched since the host was built.
Both LoM ports behave identically: cable out, esxcli network nic get still reports Link Status: Up, the LEDs stay on, and the vSwitch keeps the uplink in the active list. The switch side is clean and its port drops the moment I unplug.
So the only thing out of step on this host is the elxnet version against firmware 11.2.1194.36.
Different family, same lesson. A pair of Emulex LPe31000/LPe32000 FC ports that had run untouched for ages stopped seeing any LUNs after a Proxmox kernel moved to 5.15.64 and later 5.15.74. Cabling and optics never touched, and the log said:
That is an lpfc regression on the host side, not an optical fault; it showed up in kernels aftre 5.15.60. Pinning the boot kernel back was the workaaround that held in production:
The opt-in 5.19 kernel also worked for people who did not mind leaving the 5.15 branch. The repair was supposed to land in 5.15.77, but I never got round to running that build myself, so take it as hearsay. The point stands: when a link that used to work goes dead right after something changed on the host, read the host change log before you go anywhere near the transceivers.
Adding the mirror image of this, because it trains the same reflex. Indicators are software, and software gets it wrong in both directions.
On EX3400 and EX2300 there is a Junos defect, PR1428703, where the SFP+ and SFP port LEDs stay dark while the link is genuinely up and passing traffic. People hit it moving off 15.1X53 onto the 18.1 and 19.x trains, mostly with DAC. The CLI disagrees with the panel:
reports the LED as Green while the physical one is off. Some builds were reported as fixed and dark LEDs kept being reported on others, so I would not call it cleanly closed.
Yours is lit with no link, that one is dark with a link. Either way, trust the far end and the counters, never the indicator.
Driver is at 11.2.1149.0 on both hosts now. With the cable pulled, the LEDs go out, esxcli reports the link down, and the standby uplink takes over the way it always should have - vMotion ran clean over each uplink separately as a test.
The driver-and-firmware pair is going into our server maintenance checklist so the next bundle does not quietly split them again.