Brocade 200e between Proxmox and FreeNAS: Buffer I/O error and isp0: Receive Error while all ports show online
Running a small virtualization setup: two Proxmox 6 nodes and a target on FreeNAS 11.2. While the HBAs were connected to the target directly, everything ran for months without a single error. Put a used Brocade 200e between them to avoid crossing cables, and that's when it started.
- two Proxmox 6 nodes, HBA QLogic QLE2462 and QLE2432
- target FreeNAS 11.2
- switch Brocade 200e, Fabric OS 6.1.0a
- SFPs and LC patch cords from old stock, unlabeled
On the nodes, in the log:
Buffer I/O error on dev dm-5followed by constant device resets. On the target side - firmware command timeouts (CTIO7), and roughly once a minute:
isp0: Receive ErrorAfter that the target drops off from both initiators at once, and the only fix is rebooting the node.
What I've already tried:
- went back to direct connection - no errors at all, so the HBAs, disks, and the target itself aren't the issue
switchshowshows all three ports online, minimal zoning, one zone- reseated the patch cords, rebooted the switch
Where should I look on the switch itself? online in switchshow is clearly misleading me, but I don't yet know how to check it properly.
Show original (Русский)
Also available in: Русский
العربية
Deutsch
English
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Türkçe
Tiếng Việt
简体中文
Comments 4
You've already found it yourself: rising crc_err and enc_out are frame corruption on the line, which further up the stack turns into firmware timeouts, device resets, and
isp0: Receive Error. Neither the kernel on the nodes nor the target is at fault here, they're honestly reporting what's arriving.What you've already captured reads like this:
sfpshowon those same two ports with identically-length cords is a comparison between symmetric links, not against some number out of your head. The third, clean port serves as your referenceWhat's left is small. Run
fabriclog -s- it shows ports flapping even when switchshow is drawing online at that moment. And replace SFPs together with the LC patch cords on the suspect ports, not separately. That's exactly how a similar case of mine ended: replacing the modules and cords on the two problem ports, after which porterrshow stayed at zero for a full day under load, and the fabric stopped falling apart.The logic is simple: with a direct connection there are two connectors on the run, through a switch there are four, plus two extra modules. A borderline-power SFP or a dusty cord that the direct connection could still tolerate no longer holds up over that run. So online in switchshow isn't a diagnosis, it's just the fact of a login.
switchshow tells you exactly one thing: the port saw light and logged into the fabric. It knows nothing about signal quality, so trusting it in this situation is pointless.
Run
portstatsclearon all three ports, put load through, then checkporterrshow- you want crc_err and enc_out, whether they're rising and on which ports exactly. Alsosfpshowon each port: receive power and voltage, worth comparing between ports. And show whatsysctl dev.isp.0returns on the FreeNAS side at the moment the target drops.Cleared the counters, put load through, took a look. Here's the picture: on two ports crc_err and enc_out climb in bursts, exactly at the moments when Buffer I/O error shows up on the nodes, while the third port stays at zero.
sfpshowon those same two ports shows a noticeably lower receive level than the neighboring one, with cords of the same length.sysctl dev.isp.0during a drop shows the HBA reinitializing, meaning it's reacting to a break rather than causing one. Looks like it's physical, not Proxmox and not the target.A similar trap shows up outside FC too, so the counters are worth checking regardless. There was a setup with an Intel X520-2 with 10Gtek SR modules at 850nm and a Brocade FastIron CX 648S-PoE with an FCX-2XG module and Brocade XFPs, five meters of fiber between them.
The server honestly brought up 10GbE and transmitted, but there was no receive at all, and the switch port sat Up at speed None. Checked
show media, compared wavelength and reach on both sides, disabled trunk negotiation on the port - none of it went anywhere, you can't pin down the speed on the XFP. The same fiber ran fine at gigabit on SFP+ ports. Same moral as yours: Up on the port doesn't mean frames are actually arriving.