AFBR-89BDDZ QSFP28 vendor field reads 0905000000000000 after the SP/FPGA interface moved to a FIFO

Asked 24 days ago Active 19 days ago Viewed 38 Original language: English

We look after the switch management firmware on our own hardware, and ever since the SP/FPGA interface was changed from memory mapped buffers to a FIFO, the transceiver inventory has been coming back as junk on some ports.

QSFP28 optics, all of them AFBR-89BDDZ out of the same batch
reads go over the service processor and FPGA path to the module EEPROM
the host side helper is get_i2c_status_and_read_buffer: check status, read the whole buffer, check status again

What we get instead of vendor data:

one port, vendor field: 0905000000000000
the next port, vendor field: all ones
on another machine: vendor field all zeros, elsewhere repeated digit characters

What we have tried so far:

re-read the same port several times in a row, and the garbage is stable per port rather than random noise
power cycled the modules, no difference
confirmed every module in the chassis is the same part number, so this is not us mis-decoding some exotic vendor block

Before we start pulling optics out of production, is this more likely to be the modules, the I2C path, or our own read helper?

Also available in: English العربية Deutsch Español Français हिन्दी Bahasa Indonesia Italiano 日本語 한국어 Nederlands Polski Português Русский Türkçe Tiếng Việt 简体中文

rackhub KZ

Comments 7

Accepted answer

That points squarely at the read path, and the FIFO change is the reason. A memory mapped buffer returns the same contents however often you ask; a FIFO gives up each byte exactly once and then it is gone. The sequence get_i2c_status_and_read_buffer inherited, status check, full buffer read, status check again, made sense with memory behind it and does not with a FIFO: it empties the queue while the module's EEPROM read is still on the wire. You get whatever was sitting there at that instant, and the bytes that land late stay behind and turn up on the next read.

That is exactly the fingerprint you describe. Stable garbage per port, corruption walking along with the sequence, all zeros on one machine and repeated digit patterns on another depending on how the timing falls out. Nothing there requires a bad module, and your swap result rules the optics out anyway.

The fix is to stop making the caller responsible for the completion check. Move that responsibility inside the read routine of the transceiver driver itself: it waits until the I2C transaction reports done and only then touches the buffer, so no caller is able to drain it early by construction. Patching a single call site would just push the race somewhere else.

Fair warning that this is the proposed shape of the fix rather than something with years behind it, so validate it on your own platform before you trust the inventory again. Cheap lesson to carry away though: mangled vendor strings deserve a module-versus-port test first, because read order turns out to be the culprit far more often than the optics do.

linkeng21US 22 days ago Original (English)

The one measurement that splits this in half: does the corruption stay with the module or with the port? Take the module out of the port returning 0905000000000000, swap it with a module from a port that reads correctly, and re-read both. If the bad string follows the physical module, go and look at the optics. If it stays on the port number, or worse, moves along to whatever module gets read next, the modules are innocent and you have a host read problem.

While you are in there, dump the raw EEPROM bytes alongside the decoded fields. Junk in a decoded vendor string with sane bytes underneath is a completely different bug from junk in the bytes themselves.

optictechES 24 days ago Original (English)

Ran that swap on a pair of ports. The bad data did not move with the module: the port that returned 0905000000000000 kept returning it with a different module seated, and the module we pulled out read perfectly in its new slot.

Better than that, when we changed the order the ports are read in, the corruption moved with the sequence. The garbage lands on whichever module is read after the one that misbehaves. So it tracks read order, not the physical part. Raw bytes are wrong too, so it is not a decode problem on our side.

rackhubKZ 23 days ago Original (English)

Different root cause, same trap, from the driver side. On an Intel E810-C with the out of tree ice 1.15.4 I got wrong and incomplete pages out of ethtool -m on QSFP28 optics: page 1 and page 3 data, thresholds and per lane monitors, did not match what the module actually holds. I never got a proper root cause statement out of it, the thread was closed as solved without much detail, so treat this as anecdote rather than gospel.

What I ended up doing was updating the ice driver and the E810 NVM with nvmupdate64e, cross checking against a read from the in kernel ice driver on another host, and pulling specific pages with

ethtool -m <iface> hex on

plus explicit offset and length, instead of trusting the decoded output. If your platform can do a raw dump, compare raw against decoded before believing either one.

lambdapilot72IT 21 days ago Original (English)

Worth spelling out the layering, since it makes this kind of hunt much shorter. On Linux ethtool -m decodes the module EEPROM (vendor name, OUI, part number, serial, date code, and DDM values when the module has them), ethtool -e dumps raw bytes, and where the I2C bus is exposed i2cdump -y 1 0x50 reads A0h while i2cdump -y 1 0x51 reads A2h.

In A0h the vendor name lives in bytes 20-35 and PN, rev and SN in 40-59. So if the vendor field is mangled and the part number two dozen bytes further along is intact, that on its own says the read is timing dependent rather than the EEPROM being bad, which fits what you are seeing.

One failure not to confuse with this one: ethtool -m returning Input/output error is usually just a module with no DDM. Byte 92 bit 6 of A0h is the flag for whether A2h is there at all, and that test was put into the in-kernel ixgbe and bnx2x drivers long ago so they would stop reaching for another 256 bytes that do not exist.

sfpsmith28RU 21 days ago Original (English)

Same genre of problem on SONiC boxes, for anyone arriving from that side. sfputil show eeprom says Cannot get Module EEPROM data: Invalid argument for certain modules, or quietly disagrees with show interfaces transceiver eeprom on the same port.

From what I have run into it is mostly platform and driver gaps rather than optics: the two commands used inconsistent key names in the 202012 branch and that was fixed in 202205, some platforms lose QSFP modules from sfputil after a power cycle until a driver fix lands, and on others get_transceiver_info is simply not implemented. When I need something I can trust for scripting I go to the optoe kernel driver and do raw reads of the SFP, QSFP or CMIS EEPROM myself. Check it on your own platform though, the behaviour varies a lot between them.

gigengIN 20 days ago Original (English)

Update from our side. We moved the completion check into the driver read function as suggested, and the vendor strings have been correct on every port across a few hundred inventory passes, including the two ports that used to trade garbage between them. Raw bytes match the decoded fields now.

I am calling this partial rather than closed for the moment: we are carrying it as a patch that has not landed in our tree yet, and there is one more platform with a different FPGA build to verify before we trust it everywhere. But the optics were fine all along, which is the part I would have got wrong on my own.

rackhubKZ 19 days ago Original (English)