Chelsio T580 under pfSense: how do I put a QSFP+ port into 4x10G for a breakout cable
I run pfSense on a small box with a Chelsio T580 and I want one of the QSFP+ ports feeding four 10G hosts instead of a single 40G link. The breakout cable is still on its way, so I would rather have the card in the right mode before it lands.
- pfSense box, Chelsio T580, one QSFP+ port sitting on a 40G DAC today, the second cage empty
- cxgbe driver, cxgbetool available from the shell
- QSFP+ to 4xSFP+ breakout DAC, OCTTAPCABLE3M (an OCTCABLE5M-SR AOC as the backup plan)
- 10G hosts on the far end, plain SFP+ cages
Right now the card only ever shows two 40G interfaces:
cxgbe0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
media: Ethernet 40Gbase-CR4 <full-duplex>
cxgbe1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect
status: no carrierWhat I have already done:
- walked the cxgbetool subcommands looking for anything resembling a port-mode setting, found nothing
- looked for a loader tunable for cxgbe that would split the port, also nothing
- read enough to be fairly sure the cable on its own will not do it, the adapter has to be in 4x10G mode first
So is the 40G / 4x10G split something the FreeBSD driver can switch at all, or does it live in the adapter configuration? And if it is the configuration, what is the supported way to change it from a FreeBSD-based box?
Also available in: English
العربية
Deutsch
Español
Français
हिन्दी
Bahasa Indonesia
Italiano
日本語
한국어
Nederlands
Polski
Português
Русский
Türkçe
Tiếng Việt
简体中文
Comments 5
Short version: the driver cannot do it. The port layout comes from the configuration loaded into the adapter, which is exactly why neither cxgbe nor cxgbetool exposes a mode switch. You change it with Chelsio's own configuration utility and the card comes back in the new personality.
Get the utility onto the box (a USB stick mounted somewhere is enough) and run it:
It asks which adapter you mean and then which mode you want. The breakout one is listed as Spider (4x10G), that is the entry that turns the QSFP+ port into four 10G lanes.
On a clean install the first run tends to fall over straight away:
Nothing exotic, just a missing library. Install flashrom and it comes along as a dependency:
That brings libpci 3.5.6_1 with it, after which the script starts and the mode change reports success.
One caveat, since you are doing this before the cable arrives: all you get at that point is the utility telling you the mode was applied. Whether the four lanes actually come up is only proven with the breakout cable in the cage, so do not tear down the 40G side of the config until you have tested it.
Before you keep hunting for driver knobs: which configuration is the card actually running, and can you get python2.7 onto that box? The thing that changes this is a Chelsio utility, not part of the FreeBSD tooling, and on a stock install it needs a couple of pieces that are not there out of the box.
One more thing, given that cxgbe0 is linked at 40G right now: is there anything riding on that port you cannot afford to drop? The mode comes from the configuration loaded into the adapter, so the card comes back in a different personality and both ports disappear while that happens. Do it from the console rather than across the 40G link.
Confirmed, and in exactly that order. First run died with
then pkg install flashrom pulled libpci 3.5.6_1 in and the script came up fine. It listed the adapter, I picked Spider (4x10G), and it reported the mode change as successful.
Cable is still not here, so I cannot say anything about the four ports themselves yet. I will report back once OCTTAPCABLE3M is actually in the cage.
Same idea on switch silicon, with a lot more ceremony. On an ICX 7750 the 40G port will not split until the box is out of cut-through mode:
Then you strip any configuration off the 40G port and enable breakout on it:
After the second reload the four sub-ports show up as 1/2/1:1 through 1/2/1:4 and behave as ordinary independent 10G interfaces. Two reloads to plug in one cable, but at least it is documented behaviour rather than a hidden adapter configuration you have to know exists.
Worth adding that the mode is only half the battle, the far end also has to accept the cable itself. On MX204 and MX10003 a plain 40G DAC linked fine under older Junos; once 100G DACs picked up official support, the 40G ones quietly fell off the list, and today they are not supported on those boxes at all. What is supported is 4x10G breakout optics, QSFPP-4X10GE-LR or SFPP-4X10GE-SR, and third-party parts generally have to be coded to the matching SKU before the port will look at them.
Your Chelsio is far less fussy than a router linecard, but if the four legs are landing on vendor gear, check what that gear does with a breakout DAC before you go back to blaming the adapter mode.