MCX516A-CCAT mesh on 100G DAC: lshw says 40Gbit/s and one iperf3 stream tops out at 21 Gbit/s

Asked 20 days ago Active 19 days ago Viewed 49 Original language: English

We run a three-node cluster with the nodes cabled directly to each other over 100G DAC, no switch in the path, and the interfaces put into a broadcast bond so replication traffic has its own fabric. Before I put real load on it I wanted a baseline, and the numbers do not agree with each other.

3x Mellanox ConnectX-5 EN, MCX516A-CCAT, dual-port QSFP28
one 100G DAC between every pair of nodes
AMD EPYC hosts, Proxmox on all three

ethtool is perfectly happy:

# ethtool ens1
Settings for ens1:
        Supported link modes:   100000baseCR4/Full
        Advertised link modes:  100000baseCR4/Full
        Speed: 100000Mb/s
        Duplex: Full
        Link detected: yes

lshw is not:

# lshw -class network
  *-network
       description: Ethernet interface
       vendor: Mellanox Technologies
       capacity: 40Gbit/s

And iperf3 between two of the nodes sits at roughly 21 Gbit/s, which is nowhere near either figure.

Already done:

moved the cable to the second port on both cards, no change
swapped in another DAC of the same type, no change
link stays up the whole time, no errors climbing on the counters

So which of the two tools is lying to me, and is it the cable, the card or the driver I should be going after?

Also available in: English العربية Deutsch Español Français हिन्दी Bahasa Indonesia Italiano 日本語 한국어 Nederlands Polski Português Русский Türkçe Tiếng Việt 简体中文

coaxpilot98 GB

Comments 5

Accepted answer

Nothing you posted points at the DAC. There are two unrelated thhings happening here.

First, the disagreement. lshw -class network prints a capability figure it works out on its own, and on these cards it will cheerfully say capacity: 40Gbit/s about a link that came up at 100G. The negotiated rate is what ethtool reports, and yours says Speed: 100000Mb/s with 100000baseCR4/Full advertised. That side of it is fine, there is nothing to fix.

Second, the throughput. Check the slot before you touch anything else:

# lspci -vv
        LnkCap: ... Speed 8GT/s ...
        LnkSta: ... Speed 2.5GT/s ... (downgraded)

If LnkSta trained at 2.5GT/s while LnkCap says 8GT/s, you are capped well below the wire and no amount of cable swapping will help. Reseat the card and make sure it sits in a slot that is actually wired for the full width.

Then stop measuring with a single stream:

# iperf3 -P 8 -c <peer>

Around 21 Gbit/s is about what one core will give you on this class of host, so that number on its own tells you very little. Watch CPU during the run and look at what the idle states are doing as well: cores dropping into deep C-states between bursts cost you real bandwidth at this rate.

linkeng56TW 20 days ago Original (English)

Before you order replacement anything, post the LnkSta line from lspci -vv for that card and the exact iperf3 command line you used. One stream at 100G measures a single CPU core, not the link, and people burn days on this. And confirm the second port is really carrying the other leg of the mesh while you test, rather than sitting idle: one slot feeding two live 100G ports is a different budget froom one. I would park lshw for now, it is not the tool for this question.

lambdahawk88AE 20 days ago Original (English)

One correction on the C-state part: if the hosts are EPYC, the intel_idle knobs that get pasted into every one of these threads do nothing for you, that driver is not in the path on AMD at all. The lever that worked for me was processor.max_cstate=2 on the kernel command line. Same idea, different platform. The rest of that post stands, in particular not reading a negotiated rate out of lshw.

optictechES 19 days ago Original (English)

Slightly different failure, same family of hardware, worth ruling out once the slot is sorted: a direct mesh of ConnectX-5 QSFP28 ports is very easy to get wrong at layer 3. I had three nodes on MCX516A-CCA_Ax, firmware 16.35.4030 with the DOCA 2.8.0 driver, wired with MCP1600-C003E30L 3 m copper DAC. Every link reported active at 100 Gbps and not one ping crossed. All six mesh interfaces had addresses out of a single 10.5.5.x subnet with no switch anywhere in the path, so the kernel had no way to decide which physical port a given destination belonged to. One subnet per node pair, 10.5.5.x, 10.5.6.x and 10.5.7.x, and it started working. Worth running ip a and ip route on all three boxes before anybody blames copper.

rxnode71UA 19 days ago Original (English)

Both points landed. lspci -vv showed the card trained at 2.5GT/s against a LnkCap of 8GT/s, so that was suspect number one. I moved the cards into other slots on all three boxes, LnkSta now comes up at 8GT/s, and with iperf3 -P 8 the same pair of nodes immediately went past the single-stream figure.

I also gave up on the broadcast bond and rebuilt the mesh on Open vSwitch with RSTP. With iperf spread across three CPU threads and both ports I am measuring about 95 Gbit/s now, which is close enough to line rate for what this cluster does. lshw still insists on 40Gbit/s and I have stopped looking at it.

coaxpilot98GB 19 days ago Original (English)