6850E Lost Connectivity with 6509
Posted: 24 Mar 2012 23:46
Hello Everyone,
I encountered a loss of connectivity with my 6850E, whereby I couldn't ping or telnet the switch in question. The switch had been up for over 6 months without a problem until now.
The 6850E has two uplinks back to a Cisco 6509, aggregated using LACP. When the connectivity loss occurred, both constituent ports for the port channel were up and running on the 6509.
The only solution I had was disabling both port channels on the 6509, then brought them back up. Once this happened, connectivity was restored to the switch.
Here's the weird part: there were a number of 6850E's that encountered the same issue and fix, though not all of the ones deployed were affected. They service the same number of vlans, and are configured identically.
As there were no clients being serviced on the switch (I don't know what impact the initial disruption would have had).
Thanks in advance,
John
Anyhow, here's some of the configuration information:
interface Port-channel49
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
end
6509#sh run int g8/34
Building configuration...
Current configuration : 202 bytes
!
interface GigabitEthernet8/34
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
channel-group 49 mode active
end
6509#sh run int g9/34
Building configuration...
Current configuration : 202 bytes
!
interface GigabitEthernet9/34
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
channel-group 49 mode active
end
<the 6850>
show configuration snapshot 802.1q
! 802.1Q :
vlan 4 802.1q 1 "TAG AGGREGATE 1 VLAN 4"
show spantree mode
Spanning Tree Global Parameters
Current Running Mode : 1x1,
Current Protocol : N/A (Per VLAN),
Path Cost Mode : AUTO,
Auto Vlan Containment : N/A
Cisco PVST+ mode : Enabled
Vlan Consistency check: Disabled
show system
System:
Description: Alcatel-Lucent 6.4.4.342.R01 GA, April 18, 2011.,
Object ID: 1.3.6.1.4.1.6486.800.1.1.2.1.7.1.42,
Up Time: 189 days 3 hours 52 minutes and 56 seconds,
Contact: <redacted>
Name: <redacted>
Location: <redacted>
Services: 72,
Date & Time: SAT MAR 24 2012 22:16:33 (CDT)
Flash Space:
Primary CMM:
Available (bytes): 74522624,
Comments : None
show log swlog timestamp 03/24/2012 09:00
Displaying file contents for '/flash/swlog2.log'
FILEID: fileName[/flash/swlog2.log], endPtr[63997], configSize[64000], mode[2]
Time Stamp Application Level Log Message
------------------------+--------------+-------+--------------------------------
Displaying file contents for '/flash/swlog1.log'
FILEID: fileName[/flash/swlog1.log], endPtr[25931], configSize[64000], mode[1]
Time Stamp Application Level Log Message
------------------------+--------------+-------+--------------------------------
SAT MAR 24 09:49:53 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:49:53 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:49:57 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:49:57 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:49:59 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:50:01 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 09:55:49 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:55:49 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:55:51 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 09:55:54 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:55:54 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:55:59 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:56:01 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 09:56:01 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 09:59:44 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:59:49 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:59:51 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 10:12:19 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 10:12:29 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:36:30 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 11:36:30 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 11:36:31 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:36:31 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 11:36:31 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 11:36:39 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 11:36:41 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:36:41 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 11:41:05 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 11:41:06 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 11:41:09 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:41:11 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:41:21 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 11:41:24 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 11:41:29 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:41:29 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 11:41:31 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:41:31 2012 STP info Topology changed on VLAN/STP id 1
*note, the 6509 is the spanning-tree root bridge for both vlans 1 and 4.
I encountered a loss of connectivity with my 6850E, whereby I couldn't ping or telnet the switch in question. The switch had been up for over 6 months without a problem until now.
The 6850E has two uplinks back to a Cisco 6509, aggregated using LACP. When the connectivity loss occurred, both constituent ports for the port channel were up and running on the 6509.
The only solution I had was disabling both port channels on the 6509, then brought them back up. Once this happened, connectivity was restored to the switch.
Here's the weird part: there were a number of 6850E's that encountered the same issue and fix, though not all of the ones deployed were affected. They service the same number of vlans, and are configured identically.
As there were no clients being serviced on the switch (I don't know what impact the initial disruption would have had).
Thanks in advance,
John
Anyhow, here's some of the configuration information:
interface Port-channel49
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
end
6509#sh run int g8/34
Building configuration...
Current configuration : 202 bytes
!
interface GigabitEthernet8/34
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
channel-group 49 mode active
end
6509#sh run int g9/34
Building configuration...
Current configuration : 202 bytes
!
interface GigabitEthernet9/34
description <redacted>
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
no ip address
mls qos trust dscp
channel-group 49 mode active
end
<the 6850>
show configuration snapshot 802.1q
! 802.1Q :
vlan 4 802.1q 1 "TAG AGGREGATE 1 VLAN 4"
show spantree mode
Spanning Tree Global Parameters
Current Running Mode : 1x1,
Current Protocol : N/A (Per VLAN),
Path Cost Mode : AUTO,
Auto Vlan Containment : N/A
Cisco PVST+ mode : Enabled
Vlan Consistency check: Disabled
show system
System:
Description: Alcatel-Lucent 6.4.4.342.R01 GA, April 18, 2011.,
Object ID: 1.3.6.1.4.1.6486.800.1.1.2.1.7.1.42,
Up Time: 189 days 3 hours 52 minutes and 56 seconds,
Contact: <redacted>
Name: <redacted>
Location: <redacted>
Services: 72,
Date & Time: SAT MAR 24 2012 22:16:33 (CDT)
Flash Space:
Primary CMM:
Available (bytes): 74522624,
Comments : None
show log swlog timestamp 03/24/2012 09:00
Displaying file contents for '/flash/swlog2.log'
FILEID: fileName[/flash/swlog2.log], endPtr[63997], configSize[64000], mode[2]
Time Stamp Application Level Log Message
------------------------+--------------+-------+--------------------------------
Displaying file contents for '/flash/swlog1.log'
FILEID: fileName[/flash/swlog1.log], endPtr[25931], configSize[64000], mode[1]
Time Stamp Application Level Log Message
------------------------+--------------+-------+--------------------------------
SAT MAR 24 09:49:53 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:49:53 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:49:57 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:49:57 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:49:59 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:50:01 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 09:55:49 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:55:49 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:55:51 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 09:55:54 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:55:54 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:55:59 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:56:01 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 09:56:01 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 09:59:44 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 09:59:44 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 09:59:49 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 09:59:51 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 10:12:19 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 10:12:29 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:36:30 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 11:36:30 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 11:36:31 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:36:31 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 11:36:31 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 11:36:39 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 11:36:41 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:36:41 2012 STP info Topology changed on VLAN/STP id 1
SAT MAR 24 11:41:05 2012 LINKAGG info Port Leave . port: 1023 agg: 1
SAT MAR 24 11:41:06 2012 LINKAGG info Aggregate Down due to all port down: port: 1024 agg: 1
SAT MAR 24 11:41:09 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:41:11 2012 MIPGW info === stp === stpNewRoot trap generated
SAT MAR 24 11:41:21 2012 LINKAGG info Aggregate UP . port: 1023 agg: 1
SAT MAR 24 11:41:24 2012 LINKAGG info Port Join . port: 1024 agg: 1
SAT MAR 24 11:41:29 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:41:29 2012 STP info Topology changed on VLAN/STP id 4
SAT MAR 24 11:41:31 2012 MIPGW info === stp === stpRootPortChange trap generated
SAT MAR 24 11:41:31 2012 STP info Topology changed on VLAN/STP id 1
*note, the 6509 is the spanning-tree root bridge for both vlans 1 and 4.