6900 routing issue

[songstadw]

Hi all. I have an odd situation where one member of a "cluster" of 2 6900s cannot reliably ping a remote system that the other member can.

10.6.90 is our OSPF network although the remote system does not have OSPF enabled
There are static routes to 10.50.99.0 on both routers
OSPF built ospf routes to 10.50.99.0 on member 2 but not on member 1
Routing shouldn't really matter though for the 10.6.90 addresses since they are on the same subnet and are reachable from both systems at least part of the time.

Remote system pings both cluster members fine
member 2 pings the remote system by 10.50.99.1 and 10.6.90.50 fine.
member 1 pings the remote system by 10.50.99.1 fine, but fails on 4 of 6 tries to 10.6.90.50

All systems connect to a third 6900 that can ping all devices and interfaces fine.

Remote system 10.6.90.50 /24 (10.50.99.1 /24)

Cluster member 1 10.6.90.3 /24 (10.0.97.3 /24)
Cluster member 2 10.6.90.99 /24 (10.0.97.2 /24)
Cluster VRRP address 10.6.90.1
router in the middle 10.6.90.5 /24 (10.0.97.33 /24)

Can anyone shed any light on ping results like this? They failures are consistently in the same order. But only on the one member.
-> ping 10.6.90.50 count 20
PING 10.6.90.50 (10.6.90.50) 56(84) bytes of data.
64 bytes from 10.6.90.50: icmp_seq=1 ttl=255 time=0.817 ms
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
64 bytes from 10.6.90.50: icmp_seq=2 ttl=255 time=0.682 ms
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
64 bytes from 10.6.90.50: icmp_seq=3 ttl=255 time=0.698 ms
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
64 bytes from 10.6.90.50: icmp_seq=4 ttl=255 time=0.700 ms
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
64 bytes from 10.6.90.50: icmp_seq=5 ttl=255 time=0.800 ms
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable
From 10.6.90.50: icmp_seq=0 Destination Port Unreachable

--- 10.6.90.50 ping statistics ---
5 packets transmitted, 5 received, +15 errors, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.682/0.739/0.817/0.061 ms, pipe 6
->

[Gleylancer]

This is a really interesting problem, because you actually get the answer from the host you're calling and it's not a Gateway or Host reachability issue.

From RFC-1122: A host should generate Destination Unreachable messages with code 3 (Port Unreachable), when the designated transport protocol is unable to demultiplex the datagram but has no protocol mechanism to inform the sender.

Translation: Your destination host gets trash but can only answer via ICMP, because the source protocol is also ICMP.

I would definitely mirror a port here and look into it with Wireshark. It doesn't look like a routing issue to me because the host you're trying to reach answers itself.

[songstadw]

This turned out to be an issue with the static routes not playing nicely with the OSPF routes. When I enabled OSPF on all routers, the problem went away.