Our potentially stacking issue is over 2 years with 2 switch stacks. We supposed that our stacking might not follow the best practice but it should be fine as it is in a 1 to 0 pattern (similar with the ones on 6450s A to B). Stacking cabling might be faulty but it is unlikely as there are 2 switch stacks affected. We think VFL might me the culprit or auto-fabric but this is disabled, I am not too familiar with these Alcatel features (in my opinion it should be disabled by default anyway when they come out of the box).
It looks that every 6 months or so stacking goes out of sync.
Please find some logs, it might tell more to you than it does to me, thank you.
Code: Select all
20/05/2019
20/05/2019 22:28 Event Type SWITCH-STACK-2 - Port 4/1/54 · Alcatel-Lucent OS6860 Stacking 4/1/54 Down
20/05/2019 22:28 Event Type SWITCH-STACK-2 - Port 2/1/53 · Alcatel-Lucent OS6860 Stacking 2/1/53 Down
May 20 17:12:38 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21235595.311)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 17:42:38 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21237395.720)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 17:42:42 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21237399.515)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 18:20:00 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21239637.644)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 18:20:05 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21239642.655)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 18:48:35 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21241352.79)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:0
May 20 18:48:39 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21241356.342)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 20:05:57 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21245994.977)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 20:06:01 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21245998.965)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 20:06:51 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21246048.275)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 20:06:56 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21246053.93)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:1
May 20 21:39:13 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21251590.346)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 21:39:17 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21251594.737)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 21:40:06 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21251643.438)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:DEL MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 21:40:11 SWITCH-STACK-2 swlogd: slNi EVENT info(5) NO_CONSOLE:(21251648.858)msgHandlerExt[1559]BCD_SRCLRN_RTRMAC_MSG Rcvd Action:ADD MAC 2c:fa:a2:5d:2c:f5 Vl 1027 gport -1 l3bit:
May 20 22:27:19 SWITCH-STACK-2 swlogd: portMgrNi main info(5) : [pmnHALLinkStatusCallback:208] LINKSTS 2/1/53 DOWN (gport 0x20034) Speed 0 Duplex HALF
May 20 22:27:19 SWITCH-STACK-2 swlogd: stpNi _SOKt info(5) stpnimsg_processMsgFromPM: PM_LINK_STATUS_MSGID gPort=x20034 linkStatus=0
May 20 22:27:19 SWITCH-STACK-2 swlogd: linkAggNi main warning(4) Sync Out port=2/1/53(131124) partner:Out actor:In Dist Coll Selected Attached
May 20 22:27:19 SWITCH-STACK-2 swlogd: linkAggNi main info(5) la_ni_mgt_rcv_pm_link_down_f: send link down state to cmm 2/1/53(131124)
May 20 22:27:19 SWITCH-STACK-2 swlogd: intfNi Drv info(5) niEsmHandleEvent: current NI state:RUNNING event:8
May 20 22:27:19 SWITCH-STACK-2 swlogd: intfNi Drv info(5) niEsmSendLinkStatusChgMsg(578): linkstatus DOWN sent on peerId=2
May 20 22:27:19 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) Leave Req agg:260 port:2/1/53(131124)
May 20 22:27:19 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) la_cmm_send_vc_vfl_aggregate_down 2/1 2/1/53(131124)
May 20 22:27:19 SWITCH-STACK-2 swlogd: linkAggNi main info(5) la_ni_send_vfl_aggregate_down_f 2 1 2/1/53(131124)
May 20 22:27:19 SWITCH-STACK-2 swlogd: stpNi _AGRt info(5) VC-VFL_AGG_DONW: vflId=1 lastPort=x20034
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi port_mgr info(5) NI:vcmNI_client_rx_pm@1343: VFL link 2/1 down (last 2/1/53)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_hal_vfl_primary_port_update@2518: Port data (vfl 1, ifIdx 101053, unit 0, port 54, mod 4, action 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_hal_vfl_trunk_port_active@2267: Active ifIdx=101053 0
May 20 22:27:19 SWITCH-STACK-2 swlogd: vfcn main info(5) [vfccQsHandleLinkEvents:459] process dcbx for gport 131124 LINK DOWN, Speed 0
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_bcm_trunk_table_update@1839: Data (mod 4, port 54, tid 1029)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_build_trunk_add_info@1812: Info (pri idx -1, ports 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vfcn main info(5) [vfccQsHandleVCVFLEvents:1514] PM_VC_VFL_AGG_DOWN_MSGID: vfl 2000104, port 131124
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_bcm_trunk_table_update@1854: Trunk info (mod 4, port 54, unit 0, ports 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vfcn main info(5) [vfccQsHandleVCVFLEvents:1522] DEBUG: chassisId 2, slot 1
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmNi bcm info(5) NI:vcmNI_bcm_trunk_table_update@1860: Trunk destroy/create (4, 54, 0, 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm ipc info(5) CMM:vcmCMN_tx_data_cb@3222: autoVflEnableAckPeerCMM -> (chassis 4, CMM-A, port 17654, len 48)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm config info(5) CMM:vcmCMM_send_auto_vfl_enable_ack_msg_2peercmm@11643: autoVflEnableAckPeerCMM msg (0x9901e5, dst 4/65 ret 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm config info(5) CMM:vcmCMN_cancel_activity_timer_cb@19035: Process (auto vfl port remote wait:19/301054/0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm config info(5) CMM:vcmCMN_cancel_activity_timer_cb@19075: No timer running (auto vfl port remote wait:19/301054/0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm vfl info(5) CMM:vcmCMM_remove_vfl_member_config@9452: Postpone delete vfl 4/1 (40000264) port 4/1/54 (301054)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm vfl info(5) CMM:vcmCMN_start_vfmember_timer_cb@7342: Process (1, 301054, 60)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm vfl info(5) CMM:vcmCMM_vflink_del_port@3033: Delete port [idx 0] (301054, size 0)
May 20 22:27:19 SWITCH-STACK-2 swlogd: stpCmm mVCM info(5) PM_VC_VFL_AGG_DOWN_MSGID chassis id =2 vfl id=1, last gport=131124
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm ipc info(5) CMM:vcmCMN_tx_data_cb@3190: regUpdateVfl -> (chassis 0, CMM-A, port 36024, len 184)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vfcm main info(5) [vfccQsHandleVCVFLEvents:1514] PM_VC_VFL_AGG_DOWN_MSGID: vfl 2000104, port 131124
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm regist info(5) CMM:send_status_update@2978: vfl (app esm, peer 0/65/36024, mask 'if' [0x1])
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm ipc info(5) CMM:vcmCMN_tx_data_cb@3190: regUpdateVfl -> (chassis 0, CMM-A, port 36030, len 184)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vfcm main info(5) [vfccQsHandleVCVFLEvents:1522] DEBUG: chassisId 2, slot 1
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm regist info(5) CMM:send_status_update@2978: vfl (app linkagg, peer 0/65/36030, mask 'if' [0x1])
May 20 22:27:19 SWITCH-STACK-2 swlogd: intfCmm Mgr info(5) esmVcmVflEventCallback: vfl data event received (mask 0x3)
May 20 22:27:19 SWITCH-STACK-2 swlogd: intfCmm Mgr info(5) esmVcmVflEventCallback:609 esmAllVflLinkScan disable, ignore msg
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm ipc info(5) CMM:vcmCMN_tx_data_cb@3190: regUpdateVfl -> (chassis 0, CMM-A, port 36102, len 184)
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm regist info(5) CMM:send_status_update@2978: vfl (app ipms, peer 0/65/36102, mask 'if' [0x1])
May 20 22:27:19 SWITCH-STACK-2 swlogd: vcmCmm vfl info(5) CMM:vcmCMN_process_linkscan_event_cb@20162: Process linkscan event hwLinksc--More--
15/11/2018
SWITCH-STACK-1
Port 4/1/54 down
Port 2/1/53 down
No historical logs
Jan 12 2018
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: portMgrNi main info(5) : [pmnHALLinkStatusCallback:208] LINKSTS 1/1/54 DOWN (gport 0x35) Speed 0 Duplex HALF
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: intfNi Drv info(5) niEsmHandleEvent: current NI state:RUNNING event:8
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: intfNi Drv info(5) niEsmSendLinkStatusChgMsg(578): linkstatus DOWN sent on peerId=2
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: stpNi _SOKt info(5) stpnimsg_processMsgFromPM: PM_LINK_STATUS_MSGID gPort=x35 linkStatus=0
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggNi main warning(4) Sync Out port=1/1/54(53) partner:Out actor:In Dist Coll Selected Attached
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggNi main info(5) la_ni_mgt_rcv_pm_link_down_f: send link down state to cmm 1/1/54(53)
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) Leave Req agg:258 port:1/1/54(53)
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) la_cmm_send_vc_vfl_aggregate_down 1/1 1/1/54(53)
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: vfcn main info(5) [vfccQsHandleLinkEvents:459] process dcbx for gport 53 LINK DOWN, Speed 0
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggNi main info(5) la_ni_send_vfl_aggregate_down_f 1 1 1/1/54(53)
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) VFL Aggregate Down . LACP port: 1/1/54(1054) agg: 258 link:DOWN agg_admin:UP timedout:TRUE
Jan 12 07:19:32 SWITCH-STACK-2 swlogd: linkAggCmm main info(5) UnReserve Req port:1/1/54(53)
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: intfCmm Mgr info(5) cmmEsmSendDynamicInfoAck: chassis 2 slot 1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: intfCmm Mgr info(5) cmmEsmHandleNiMsg: Rex NI_ESM_DYNAMIC_INFO_SYNC from chassisId 2 NI 1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _STPt info(5) chassis:2 slot 1 connection 1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _STPt info(5) Get the Vlan SnapShot for chassis =2 slot =1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _VLNt info(5) Chassis Connected 1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _VLNt info(5) get snapshot ch to read =2 start(zero based)=1 end(zero based)=1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _MSGt info(5) handle ni state response ch 2 slot 1 fresh ni:0
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _INIT info(5) NI 2/1 Recover
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm _INIT info(5) port_init chassis:2 slot:1
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm mMCM info(5) port 101053 is VC-VFL member
Jan 12 07:23:50 SWITCH-STACK-2 swlogd: stpCmm mMCM info(5) port 101054 is VC-VFL member
10/06/2017
SWITCH-STACK-2 down
Switch info:
CODE: SELECT ALL
SWITCH-STACK-2# show virtual-chassis topology
Local Chassis: 2
Oper Config Oper
Chas Role Status Chas ID Pri Group MAC-Address
-----+------------+-------------------+--------+-----+------+------------------
1 Slave Running 1 100 0 2c:fa:a2:5d:2c:f5
2 Master Running 2 100 0 2c:fa:a2:5d:1f:01
3 Slave Running 3 100 0 2c:fa:a2:5d:28:1d
4 Slave Running 4 100 0 2c:fa:a2:5d:17:fb
SWITCH-STACK-2# show virtual-chassis vf-link
VFLink mode: Auto
Primary Config Active Def Speed
Chassis/VFLink ID Oper Port Port Port Vlan Type
-------------------+----------+---------+-------+-------+---------+-----------
1/0 Up 1/1/53 1 1 1 21G
1/1 Up 1/1/54 1 1 1 21G
2/0 Up 2/1/54 1 1 1 21G
2/1 Up 2/1/53 1 1 1 21G
3/0 Up 3/1/53 1 1 1 21G
3/1 Up 3/1/54 1 1 1 21G
4/0 Up 4/1/53 1 1 1 21G
4/1 Up 4/1/54 1 1 1 21G
SWITCH-STACK-2# show virtual-chassis consistency
Legend: * - denotes mandatory consistency which will affect chassis status
licenses-info - A: Advanced; B: Data Center;
Config Oper Oper Config
Chas Chas Chas Hello Control Control
Chas* ID Status Type* Group* Interv Vlan* Vlan License*
------+------+---------+-------+------+-------+--------+--------+----------
1 1 OK OS6860 0 15 4094 4094 A
2 2 OK OS6860 0 15 4094 4094 A
3 3 OK OS6860 0 15 4094 4094 A
4 4 OK OS6860 0 15 4094 4094 A
SWITCH-STACK-2# show virtual-chassis vf-link
VFLink mode: Auto
Primary Config Active Def Speed
Chassis/VFLink ID Oper Port Port Port Vlan Type
-------------------+----------+---------+-------+-------+---------+-----------
1/0 Up 1/1/53 1 1 1 21G
1/1 Up 1/1/54 1 1 1 21G
2/0 Up 2/1/54 1 1 1 21G
2/1 Up 2/1/53 1 1 1 21G
3/0 Up 3/1/53 1 1 1 21G
3/1 Up 3/1/54 1 1 1 21G
4/0 Up 4/1/53 1 1 1 21G
4/1 Up 4/1/54 1 1 1 21G
SWITCH-STACK-2# show virtual-chassis topology
Local Chassis: 2
Oper Config Oper
Chas Role Status Chas ID Pri Group MAC-Address
-----+------------+-------------------+--------+-----+------+------------------
1 Slave Running 1 100 0 2c:fa:a2:5d:2c:f5
2 Master Running 2 100 0 2c:fa:a2:5d:1f:01
3 Slave Running 3 100 0 2c:fa:a2:5d:28:1d
4 Slave Running 4 100 0 2c:fa:a2:5d:17:fb
SWITCH-STACK-2# show virtual-chassis neighbors
Chas VFL VFL
ID 0 1
----+---+---
1 4 2
2 3 1
3 2 4
4 3 1
SWITCH-STACK-2# show microcode
/flash/vc_dir
Package Release Size Description
-----------------+-------------------------+--------+-----------------------------------
Uos.img 8.2.1.335.R01 247586880 Alcatel-Lucent OS
SWITCH-STACK-2# show hardware-info
Chassis 1
CPU Manufacturer : Broadcom
CPU Model : ARM
Compact Flash Manufacturer : Micron Technology
Compact Flash size : 1997967360 bytes
RAM Manufacturer : Other
RAM size : 2021064 kB
FPGA version : 0.9
U-Boot Version : 8.1.1.70.R01
Power Supplies Present : 1,2
NIs Present : 1,-
SWITCH-STACK-1# show virtual-chassis consistency
Legend: * - denotes mandatory consistency which will affect chassis status
licenses-info - A: Advanced; B: Data Center;
Config Oper Oper Config
Chas Chas Chas Hello Control Control
Chas* ID Status Type* Group* Interv Vlan* Vlan License*
------+------+---------+-------+------+-------+--------+--------+----------
1 1 OK OS6860 0 15 4094 4094 A
2 2 OK OS6860 0 15 4094 4094 A
3 3 OK OS6860 0 15 4094 4094 A
4 4 OK OS6860 0 15 4094 4094 A
SWITCH-STACK-1# show virtual-chassis neighbors
Chas VFL VFL
ID 0 1
----+---+---
1 2 4
2 1 3
3 2 4
4 1 3
SWITCH-STACK-1# show microcode
/flash/certified
Package Release Size Description
-----------------+-------------------------+--------+-----------------------------------
Uos.img 8.2.1.335.R01 247586880 Alcatel-Lucent OS
SWITCH-STACK-1# show hardware-info
Chassis 1
CPU Manufacturer : Broadcom
CPU Model : ARM
Compact Flash Manufacturer : Micron Technology
Compact Flash size : 1997967360 bytes
RAM Manufacturer : Other
RAM size : 2021064 kB
FPGA version : 0.9
U-Boot Version : 8.1.1.70.R01
Power Supplies Present : 1,2
NIs Present : 1,-
