LACP Link Aggregation with FreeNAS

[eugene]

Hi guys,

I have a FreeNAS box to be used as a storage appliance. Two of its 10GBASE-T ports are connected to a OS6900-T20 switch, for which I intend to do linkagg.

Following the example in the Network Configuration Guide, I configured the switch as such.

Code: Select all

linkagg lacp agg 2 size 2 actor admin-key 5 name "FreeNAS"
linkagg lacp port 1/1 actor admin-key 5
linkagg lacp port 1/2 actor admin-key 5

In FreeNAS, a LACP-based link aggregation was created between the interfaces bxe0 and bxe1

Code: Select all

bxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 20:67:7c:01:3a:c8
        hwaddr 20:67:7c:01:3a:c8
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active

bxe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 20:67:7c:01:3a:c8
        hwaddr 20:67:7c:01:3a:cc
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 20:67:7c:01:3a:c8
        inet 10.40.3.1 netmask 0xffff0000 broadcast 10.40.255.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: bxe0 flags=0<>
        laggport: bxe1 flags=0<>

However, after configuring everything, the link remains down. This is the status of the switch.

Code: Select all

-> show linkagg
Number  Aggregate     SNMP Id   Size Admin State  Oper State     Att/Sel Ports
-------+-------------+---------+----+------------+--------------+-------------
   2     Dynamic      40000002   2   ENABLED      DOWN            0   2

-> show vlan
 vlan    type   admin   oper    ip    mtu          name
------+-------+-------+------+------+------+------------------
1      std       Ena     Ena   Ena    1500    VLAN 1
4094   mcm       Ena     Dis   Dis    9198    MCM IPC

-> show vlan members
  vlan       port         type         status
--------+------------+------------+--------------
  1         1/3          default    inactive
  1         1/4          default    inactive
  1         1/5          default    inactive
  1         1/6          default    inactive
  1         1/7          default    inactive
  1         1/8          default    inactive
  1         1/9          default    inactive
  1         1/10         default    inactive
  1         1/11         default    forwarding
  1         1/12         default    forwarding
  1         1/13         default    forwarding
  1         1/14         default    forwarding
  1         1/15         default    forwarding
  1         1/16         default    forwarding
  1         1/17         default    forwarding
  1         1/18         default    forwarding
  1         1/19         default    inactive
  1         1/20         default    inactive
  1         2/1          default    inactive
  1         2/2          default    inactive
  1         2/3          default    inactive
  1         0/2          default    inactive

Has anyone had experience configuring LACP between Alcatel switches and *nix boxes? Any gotchas I should be aware of? I have a nagging feeling that the admin-ley has something to do with it.

Thanks in advance!

[pstolpe]

Hi, the configuration seems correct so check the admin status of the individual linkagg agg 2 interfaces on the OS6900.
If it's not working I'd look into FreeNAS settings.

[eugene]

Both interfaces 1/1 and 1/2 in the linkagg are up.

Code: Select all

-> show interfaces port 1/1
Slot/Port 1/1     :
 Operational Status     : up,
 Last Time Link Changed : Thu Jan 10 10:27:04 2019,
 Number of Status Change: 17,
 Type                   : Ethernet,
 SFP/XFP                : N/A,
 EPP                    : Disabled,
 Link-Quality           : GOOD,
 MAC address            : 2c:fa:a2:b0:90:88,
 BandWidth (Megabits)   :    10000,             Duplex           : Full,
 Autonegotiation        :   1  [ 10000-F 1000-F 100-F 100-H ],
 Long Frame Size(Bytes) : 9216,
 Inter Frame Gap(Bytes) : 12,
 Rx              :
 Bytes Received  :              1273440, Unicast Frames :                    0,
 Broadcast Frames:                    0, M-cast Frames  :                 9952,
 UnderSize Frames:                    0, OverSize Frames:                    0,
 Lost Frames     :                    0, Error Frames   :                    0,
 CRC Error Frames:                    0, Alignments Err :                    0,
 Tx              :
 Bytes Xmitted   :              8062070, Unicast Frames :                   34,
 Broadcast Frames:                 1323, M-cast Frames  :                72960,
 UnderSize Frames:                    0, OverSize Frames:                    0,
 Lost Frames     :                    0, Collided Frames:                    0,
 Error Frames    :                    0

-> show interfaces port 1/2
Slot/Port 1/2     :
 Operational Status     : up,
 Last Time Link Changed : Thu Jan 10 10:27:01 2019,
 Number of Status Change: 31,
 Type                   : Ethernet,
 SFP/XFP                : N/A,
 EPP                    : Disabled,
 Link-Quality           : GOOD,
 MAC address            : 2c:fa:a2:b0:90:89,
 BandWidth (Megabits)   :    10000,             Duplex           : Full,
 Autonegotiation        :   1  [ 10000-F 1000-F 100-F 100-H ],
 Long Frame Size(Bytes) : 9216,
 Inter Frame Gap(Bytes) : 12,
 Rx              :
 Bytes Received  :          40859264656, Unicast Frames :             37889649,
 Broadcast Frames:                 2006, M-cast Frames  :                 9982,
 UnderSize Frames:                    0, OverSize Frames:                    0,
 Lost Frames     :                    0, Error Frames   :                    0,
 CRC Error Frames:                    0, Alignments Err :                    0,
 Tx              :
 Bytes Xmitted   :           4903182912, Unicast Frames :             27947396,
 Broadcast Frames:               176720, M-cast Frames  :              3686601,
 UnderSize Frames:                    0, OverSize Frames:                    0,
 Lost Frames     :                    0, Collided Frames:                    0,
 Error Frames    :                    0

The linkagg's Operational State remains DOWN no matter what I configure on the switch.

Code: Select all

-> show linkagg agg 2

Dynamic Aggregate
  SNMP Id                  : 40000002,
  Aggregate Number         : 2,
  SNMP Descriptor          : Dynamic Aggregate Number 2 ref 40000002 size 2,
  Name                     : FreeNAS,
  Admin State              : ENABLED,
  Operational State        : DOWN,
  Aggregate Size           : 2,
  Number of Selected Ports : 2,
  Number of Reserved Ports : 0,
  Number of Attached Ports : 0,
  Primary Port             : NONE,
  Port Selection Hash      : Source Destination Ip,
  Wait To Restore Time     : 0 Minutes
LACP
  MACAddress               : [00:00:00:00:00:00],
  Actor System Id          : [00:00:00:00:00:00],
  Actor System Priority    : 0,
  Actor Admin Key          : 5,
  Actor Oper Key           : 5,
  Partner System Id        : [00:00:00:00:00:00],
  Partner System Priority  : 0,
  Partner Admin Key        : 0,
  Partner Oper Key         : 274

-> show linkagg agg 2 port

Slot/Port          Aggregate   SNMP Id   Status    Agg  Oper   Link Prim
-------------------+----------+--------+----------+----+-----+-----+----
            1/1     Dynamic      1001   RESERVED      2  DOWN UP    NO
            1/2     Dynamic      1002   RESERVED      2  DOWN UP    NO

I thought it had to do with the Partner Admin Key so I changed it to 5.

Code: Select all

-> linkagg lacp agg 2 partner admin-key 5

Dynamic Aggregate
  SNMP Id                  : 40000002,
  Aggregate Number         : 2,
...
  Partner Admin Key        : 5,
  Partner Oper Key         : 274

Partner Oper Key changed to 0 after a minute but I don't know what this means.

[silvio]

the partner admin key is wrong. You see at the other end is the key 274. So you has do configure 274 or nothing (prefered way) as partner admin key.
What is the output of show linkagg port 1/1 and for 1/2?
best regards
Silvio

[eugene]

> So you has do configure 274 or nothing (prefered way) as partner admin key.

In FreeBSD (the OS of FreeNAS), setting the partner's administrative key or operational key is not an option. Did you mean to configure it in FreeBSD or in the OS6900?

I tried configuring the linkagg again and this time round I used the admin-key of 274.

Code: Select all

linkagg lacp agg 10 size 2 actor admin-key 274
linkagg lacp port 1/1 actor admin-key 274
linkagg lacp port 1/2 actor admin-key 274

The results are as follows.

Code: Select all

-> show linkagg agg 10

Dynamic Aggregate
  SNMP Id                  : 40000010,
  Aggregate Number         : 10,
  SNMP Descriptor          : Dynamic Aggregate Number 10 ref 40000010 size 2,
  Name                     : FreeNAS,
  Admin State              : ENABLED,
  Operational State        : DOWN,
  Aggregate Size           : 2,
  Number of Selected Ports : 2,
  Number of Reserved Ports : 0,
  Number of Attached Ports : 0,
  Primary Port             : NONE,
  Port Selection Hash      : Source Destination Ip,
  Wait To Restore Time     : 0 Minutes
LACP
  MACAddress               : [00:00:00:00:00:00],
  Actor System Id          : [00:00:00:00:00:00],
  Actor System Priority    : 0,
  Actor Admin Key          : 274,
  Actor Oper Key           : 274,
  Partner System Id        : [00:00:00:00:00:00],
  Partner System Priority  : 0,
  Partner Admin Key        : 0,
  Partner Oper Key         : 274

-> show linkagg port 1/1

Dynamic Aggregable Port
  SNMP Id                       : 1001,
  Slot/Port                     : 1/1,
  Administrative State          : ENABLED,
  Operational State             : DOWN,
  Port State                    : SELECTED,
  Link State                    : UP,
  Selected Agg Number           : 10,
  Port position in the aggregate: 0,
  Primary port                  : NO
LACP
  Actor System Priority         : 0,
  Actor System Id               : [2c:fa:a2:b0:90:81],
  Actor Admin Key               : 274,
  Actor Oper Key                : 274,
  Partner Admin System Priority : 0,
  Partner Oper System Priority  : 0,
  Partner Admin System Id       : [00:00:00:00:00:00],
  Partner Oper System Id        : [00:00:00:00:00:00],
  Partner Admin Key             : 0,
  Partner Oper Key              : 0,
  Selected Agg Id               : 10,
  Attached Agg Id               : 0,
  Actor Port                    : 0,
  Actor Port Priority           : 0,
  Partner Admin Port            : 0,
  Partner Oper Port             : 0,
  Partner Admin Port Priority   : 0,
  Partner Oper Port Priority    : 0,
  Actor Admin State             : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act0.tim0.agg1.syn0.col1.dis1.def1.exp0

-> show linkagg port 1/2

Dynamic Aggregable Port
  SNMP Id                       : 1002,
  Slot/Port                     : 1/2,
  Administrative State          : ENABLED,
  Operational State             : DOWN,
  Port State                    : SELECTED,
  Link State                    : UP,
  Selected Agg Number           : 10,
  Port position in the aggregate: 1,
  Primary port                  : NO
LACP
  Actor System Priority         : 0,
  Actor System Id               : [2c:fa:a2:b0:90:81],
  Actor Admin Key               : 274,
  Actor Oper Key                : 274,
  Partner Admin System Priority : 0,
  Partner Oper System Priority  : 0,
  Partner Admin System Id       : [00:00:00:00:00:00],
  Partner Oper System Id        : [00:00:00:00:00:00],
  Partner Admin Key             : 0,
  Partner Oper Key              : 0,
  Selected Agg Id               : 10,
  Attached Agg Id               : 0,
  Actor Port                    : 1,
  Actor Port Priority           : 0,
  Partner Admin Port            : 0,
  Partner Oper Port             : 0,
  Partner Admin Port Priority   : 0,
  Partner Oper Port Priority    : 0,
  Actor Admin State             : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act0.tim0.agg1.syn0.col1.dis1.def1.exp0

No luck in getting it up. It seems like most FreeNAS users managed to get it up and working with a Cisco switch and an occasional Netgear. A Brocade user didn't manage to get it to work. I don't know if this is an implementation problem in FreeBSD or on the switch.

The config on a Cisco switch is slightly simplier too. The admin-key is absent from the config.

[eugene]

Holy cow it finally worked! This has given me a headache for 3 days... Silvio thankyouthankyouthankyou. There were 2 new steps that I performed so I don't exactly know which one did the trick:

1. Reboot FreeNAS: I've read that this is recommended after configuring the network interface (not necessarily LACP) but it's 2019 and I really cannot comprehend why this is so.
2. Config the partner's admin-key: I suspect this did the trick. I'll verify this on another FreeNAS+OS6900 LACP setup a few weeks later.

Here's the final config for the OS6900. FreeBSD (and thus FreeNAS) uses the administrative key of 274.

Code: Select all

linkagg lacp agg 10 size 2 actor admin-key 274 name "FreeNAS"
linkagg lacp agg 10 partner admin-key 274
linkagg lacp port 1/1 actor admin-key 274
linkagg lacp port 1/2 actor admin-key 274

On FreeNAS, just follow the tutorial to create the link aggregation.

That's it!

[eugene]

My happiness was short-lived. The interface died shortly after I wrote the post. The good news is that I found out what kinda activated the lagg0 interface on FreeBSD: tcpdump.

When tcpdump is run, the interface comes online and life is good for about 5 minutes. Then the interface goes back down. This is mind-boggling... More debugging to do...

[silvio]

Hi,
there are some missunderstanding with actor admin key and partner actor admin key at you.
actor admin key is local at your site and partner the other site.... (thats clear).

In the output of show linkagg port 1/1 and 1/2 and of show linkagg agg 10 you see:
# Actor Admin Key : 274, = configured at my site for me (with linkagg lacp port 1/1 actor admin-key 274)
# Actor Oper Key : 274, = learned in my own switch (both must be the same - so it is okay here)
# Partner Admin Key : 0, = configured at my site to only allow specific actor keys from the other site (... partner admin-key 274). If you don't configure this (=0) than all partner keys are allowed (this is the prefered config)
# Partner Oper Key : 0, = received from the other site via LACP (this must the same at all the commands).

So I suggest to never configure the partner key - only the own site:

Code: Select all

linkagg lacp agg 10 size 2 
linkagg lacp agg 10 actor admin-key 10
linkagg lacp port 1/1 actor admin-key 10
linkagg lacp port 1/2 actor admin-key 10

But your problem seams to be an other point:

Actor Admin State : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
Actor Oper State : act1.tim1.agg1.syn0.col0.dis0.def1.exp0,
Partner Admin State : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
Partner Oper State : act0.tim0.agg1.syn0.col1.dis1.def1.exp0

Most of the Oper flags (own and partner) are different:
F.e. the Partners Oper Timeout is not set (0 means long) but our own timeout is set (1 means short).

This flag you can disable with
> no linkagg lacp port 1/1 actor admin-state TIMEOUT
after the configuration you have to make a port admin down/up to takeover this admin setting to oper(ational).

Via this way you can change you own flags fitting to the partners (disable timeout, enable collector, enabel distributor).
But one site should be active (so keep act untouched).

More infos about this you find at http://www.ieee802.org/3/ad/public/mar9 ... 1_0399.pdf

If you find a solution please post it here. It will be usefull for us too.
best regards
Silvio

[eugene]

Thanks for the explanation. It's a lot clearer to me now.

The config in Alcatel's Network Configuration Guide works just fine:

Code: Select all

linkagg lacp agg 10 size 2 actor admin-key 10
linkagg lacp port 1/1 actor admin-key 10
linkagg lacp port 1/2 actor admin-key 10

After configuring LACP on both ends, and without doing anything else, this is the state of FreeNAS and on the switch. Note that "bxe0" and "bxe1" are not marked as active/collecting/distributing.

Code: Select all

## On FreeNAS
$ ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
	ether 20:67:7c:01:3a:c8
	inet 10.40.3.1 netmask 0xffff0000 broadcast 10.40.255.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	laggport: bxe0 flags=0<>
	laggport: bxe1 flags=0<>

## On the OS6900
-> show linkagg port 1/1
...
  Actor Admin State             : act1.tim0.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim0.agg1.syn1.col0.dis0.def0.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act1.tim0.agg1.syn0.col0.dis0.def1.exp0

-> show linkagg port 1/2
...
  Actor Admin State             : act1.tim0.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim0.agg1.syn1.col0.dis0.def0.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act1.tim0.agg1.syn0.col0.dis0.def1.exp0

Once I run tcpdump on FreeNAS, the LACP interface comes up. After the tcpdump process stops, the link returns to the Operational State of DOWN.

Code: Select all

## On FreeNAS
# tcpdump -c 20 -e ether proto 0x8809 -vvv -X -U -s0 -nnni lagg0 -w lagg0.pcap
$ ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
	ether 20:67:7c:01:3a:c8
	inet 10.40.3.1 netmask 0xffff0000 broadcast 10.40.255.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	laggport: bxe0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: bxe1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

## On the OS6900
-> show linkagg port 1/1
  Actor Admin State             : act1.tim0.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim0.agg1.syn1.col1.dis1.def0.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act1.tim0.agg1.syn1.col1.dis1.def0.exp0

-> show linkagg port 1/2
  Actor Admin State             : act1.tim0.agg1.syn0.col0.dis0.def1.exp0,
  Actor Oper State              : act1.tim0.agg1.syn1.col1.dis1.def0.exp0,
  Partner Admin State           : act0.tim0.agg1.syn0.col1.dis1.def1.exp0,
  Partner Oper State            : act1.tim0.agg1.syn1.col1.dis1.def0.exp0

This is queer behaviour. It could be a FreeBSD bug or a NIC issue.

[eugene]

TLDR: This is a Broadcom driver bug in FreeBSD 11.x. I'm using FreeNAS 11.2 so it's affected by the bug. The only solution is to put the interface in promiscuous mode.

Code: Select all

ifconfig lagg0 promisc

FreeBSD users have experienced similar issues when an interface's MAC address is changed. It's a known bug that affects Realtek NICs. I thought it was because the interfaces lagg0 and bxe1 assumed bxe0's h/w MAC when it joined the link agg, but this isn't the root of my problem.

My NIC is a 10GBASE-T Broadcom Limited NetXtreme II BCM57810. A bug exists in the Broadcom driver of FreeBSD 11.x in which ethernet multicast wouldn't work unless the interface is in promiscuous mode.

Putting the interface in promiscuous mode is the only possibility for the time being. Thank you Silvio for your help again!