One port of an X520-DA2 does only 1 Gbit/s in iperf3 after the move from TrueNAS Core to SCALE 24.10
The storage box went from Core 13.0-U6.7 to SCALE 24.04 and then on to 24.10, and ever since one port is crawling while its twin on the same card is perfectly happy. The 25G card behaves the same way, which is what makes me doubt my own eyes.
- Intel X520-DA2 with Intel multimode SFP+ optics, 10G to the switch
- Intel XXV710-DA2 with Intel SFP28 optics, 25G to the same switch
- TrueNAS SCALE 24.10 on the NAS (was Core 13.0-U6.7, then 24.04)
- iperf3 between the NAS and a client as the yardstick
Link state on the slow port looks exactly like it should:
$ ethtool enp1s0f0 | grep -E 'Speed|Duplex|Link detected'
Speed: 10000Mb/s
Duplex: Full
Link detected: yes
$ iperf3 -c 192.168.3.2
# parks itself around 1 Gbit/s for the whole run
# the second port of the same card does line rate against its own clientAlready done:
- swapped the optics between the two ports of the card; the slow port stayed slow, the fast one stayed fast
- compared sysctl net.ipv4.tcp_congestion_control on both ends, same value
- reseated both ends and cleaned the ferrules
Everything measurable at the link layer says 10G and 25G full duplex, and payload is still stuck at roughly gigabit. Are the modules degrading, is the NIC on its way out, or am I looking at the wrong layer?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 3
You already ruled the optics out yourself: you swapped them between ports and the slowness stayed with the port. So leave the modules alone and separate link-layer facts from throughput numbers, because they are answering different questions.
Two things normally differ between a fast and a slow port on the same card. Look at them together:
If one sits at MTU 1500 and the other at 9014, and they are in different VLANs, then your slow test is not going out of the NIC and back, it is going through the host's routing path. Put a client in the same VLAN and subnet as the slow port and run iperf3 -c against that. No router in the path, no MTU mismatch, nothing to argue about.
If the single-VLAN test gives you line rate, the port and the transceiver are fine and what you have actually measured is inter-VLAN routing performance on the host, which is where a fair number of Core to SCALE moves end up: forwarding between VLANs is visibly worse than it was under Core.
That is a diagnosis and not really a cure. You get most of it back by keeping the heavy flows inside one VLAN, or by handing the routing to the switch instead of the NAS. At minimum it stops you from returning two perfectly good SFP+ modules.
Before anyone starts pulling optics out: what does the switch say about those two ports? Negotiated speed, duplex and the error counters on both. And which end runs the iperf3 server in the slow test - does the ceiling stay put when you turn the run around and push from the other side?
Then post the MTU and the VLAN of the slow port and of the fast one, side by side. A port that negotiates 10G and only moves a gigabit of payload is a forwarding or path problem in almost every case, not an optical one. If the two ports do not sit in the same VLAN, iperf3 is grading your router, not your link.
Different hardware, same shape. Two boxes back to back on X520-DA cards over an SFP+ DAC, Hyper-V Server Core 2012 R2 on one side and a NAS4Free 9.1 storage box on the other. Straight through, that pair did 8-9 Gbit/s reading and writing. The moment the port was bound to a Hyper-V virtual switch it dropped to something like 500 Mbit/s, and retesting the cards under Windows Server 2012 R2 and Windows 8.1 changed nothing at all.
It was never proven. The one reply I got asked what disks and what RAID level sat behind each end, which is a fair question, because the storage can be the ceiling long before the 10G path is. The habit I took from it: measure the same link with the extra layer attached and with it detached, and against a RAM disk if you can arrange one. The cable and the cards were not the problem there either.