Before we get started in the actual configurations,
let's get grounded in what's going to happen.
I have a Windows host at 20.
1.
15 with especially configured MAC address of 0000.
2001.
0115 to mimic its IP.
I also have an Ubuntu Linux host at 20.
1.
16 also with a specially configured MAC address 0000.
2001.
0116 to mimic its IP address.
These are connected together on the LAN with the default-gateway at 20.
1.
First,
we'll make sure that the ARP cache is on the Windows host and the Cisco
router gateway are complete and accurate to the Ubuntu host.
Then I'll make a special change to the Linux host MAC address which
will not update the network through gratuitous ARPs.
At that point,
you'll see how the Windows host handles the problem and how
the Cisco router also handles the problem.
Then we'll take a look at some solutions.
Let's bring up my Windows command prompt and check the ARP cache.
I see my default-gateway there and another host at 20.
1.
100, of course,
our static entries for the broadcast and multicast addresses are there,
but I'm going to ping 20.
1.
16 to make sure I have its information in my ARP cache.
So I'll check my ARP cache, arp -a, and there I see 20.
1.
16 with that specially configured MAC address.
I'll also move over to the router's configuration.
Here we're going to check its ARP cache with show arp.
Now I do see the 20.
1.
16 with that specially configured MAC address, but I'm also going to ping 20.
1.
15 so that we have that in our ARP cache on the router.
Show arp should show me 20.
1.
15 at its specially configured MAC address as well.
I'll move over to the Linux host and I'll use ifconfig ens33,
which is my network card, to see our IP address there of 20.
1.
16, and then to verify that MAC address, I'll use ip link show.
And again, right there is my specially configured MAC address.
To be able to see the failure of the false,
stale ARP cache entries, I'll return to my Windows and start a Wireshark capture.
I'll also add an ARP display filter.
I'm going to return to my Ubuntu Linux and I'll change the MAC address on it
in such a way that it doesn't generate any gratuitous ARPs,
although I'm doing this through a special command-line,
this can happen in the wild in a couple of different ways.
In some multi home systems,
the software controlling multiple network cards can make a change to
the IP to MAC bindings without updating the network.
Also, in some server redundancy protocols,
when a failover happens,
the same IP address could be associated with the different systems MAC address.
And lastly, in some virtualization software, you find the same problem.
Switching a MAC address assignment for the virtual machines at the same
IP address will result in no updates to the network.
Now to set a new MAC address for use on my ens33 card,
I'm going to use the ip link set command and set dev ens33 to use the address,
and in this case, I'm going to make it a clearly unique address,
aa:bb:cc:16:16:16.
Now that I've done that,
the Windows machine and the Cisco router are unaware of this change.
Let's go back to my Windows host and see what affect that has in the network.
I'll return to my Windows command prompt and I'll show my ARP
cache to make sure that we're not up-to-date.
Notice the 20.
1.
16 is still at that old MAC address, so I'm going to ping 20.
1.
16 and my card should try to send pings to that IP address with the old
MAC address and we'll see how Windows handles it.
Two requests timed out and two successes,
it must have figured out the correct MAC address.
Let's go back to our Wireshark and see just how.
In my Wireshark,
I'm going to add ICMP to my ARP filter and here we can see what's going on.
My card 20.
1.
15 sent an ICMP ping to 20.
1.
16, and if we look at the ethernet header,
it's trying to do it to that MAC address it thought was assigned to 20.
1.
16.
Then, because it was getting no response,
it sent a directed ARP asking who has 20.
1.
16,
but notice that it's still coming from my network card's MAC address
to that false old MAC address of 00:00:20:01:01:16,
and of course,
our Ubuntu Linux machine is not responding because
it's no longer at that MAC address.
Again,
my system sends that second ping that fails because
it's going to the wrong MAC address.
And again, my card sends two more directed ARPs at that false MAC address,
but the good news is Windows wises up and says hey I better
send a broadcast out asking who has 20.
1.
16.
Immediately, our Ubuntu Linux system at 20.
1.
16 responds oh yeah, I'm now at aa:bb:cc:16:16:16.
Now that the Windows machine has the correct MAC address,
we're going to see ICMP Echo ping request and now reply from that system.
And again, ICMP Echo request and a successful reply from that system.
Windows has solved the problem of stale and false information in its ARP cache.
Go Windows!
And if we return to the Windows command prompt and check our ARP cache,
we now see that the ARP cache has been updated with that new MAC address.
Now even if Windows didn't adapt to this issue and fix it,
the Windows ARP cache timer is quite short.
From as little as a few seconds up to a few minutes,
that stale information would time out and Windows would have to
go through the ARP request and reply process.
Now let's see how the Cisco router does.
Let's move over to my Cisco router command prompt and check
its ARP cache to see if it figured it out.
In this case, 20.
1.
16 is still at that old MAC address in its ARP cache 0000.
2001.
0116.
I wonder what's going to happen when I ping 20.
1.
16.
Now in Cisco, the dots there are timeout so we are not getting any response.
It looks like success rate is 0%, 0 out of 5, that's pretty bad.
Let's check the ARP cache with show arp and see if it did any better.
Nope, it still believes that 20.
1.
16 is still at that old MAC address in its ARP cache.
Let's see why.
I'm going to show int f0/0, the 1 servicing that LAN,
and we'll take a look at the timer for that ARP cache.
If we look in this line, we can see the ARP Timeout is set to 4 hours.
Cisco routers and Layer-3 switches have made a tradeoff
between accurate information and ARP overhead.
Because most local area networks are fairly stable in
their IP to MAC bindings and the router communicates with
the same devices all day long,
Cisco has decided its more advantageous to have a long ARP cache timer
than to deal with higher overhead with more ARP traffic.
Clearly, it didn't work out in this case.
Now it is relatively rare that we see changes to
systems IP to MAC address bindings,
which don't create gratuitous ARPs and keep the network up-to-date.
Just as in this test case,
changes made to IP or MAC addresses in some failover and
virtualization technologies also do not properly update
network systems through gratuitous ARP, and consequently,
you may see this behavior.
It will appear as a loss of connectivity,
especially through the gateway routers and for some systems within the LAN.
Now the question is, what are we going to do about it.
At this point,
one of the things that we can do is we could clear the ARP cache
on all of our routers in Layer-3 switches.
This would force them to update their ARP cache on those interfaces.
I can do that with clear arp-cache interface f0/0 and
force it to learn new IP to MAC bindings.
Now if we show the ARP cache,
everything is cleared out or the timers have returned to 0,
and if I ping 20.
1.
16,
we notice 1 timeout because dynamic ARP is doing its job
and I can show the ARP cache to see that it's been updated
with the right MAC address.
Another thing that we can do is we can shorten that timer.
I'll move to global configuration mode and into the interface configuration
for f0/0 and I can set the ARP cache timer by using the ARP timeout command
and we can set that as a matter of seconds,
of course, you get to pick the number of seconds,
in this case, I'm going to set it to 5 minutes or 300 seconds.
Again,
this is going to increase the ARP overhead on both the router
and within the local area network as this router has to ARP for
everything it's communicating with.
In a big LAN, that can be significant overhead on the router,
which can create problems of throughput or handling other traffic elsewhere.
As a result, I would rather fix the problem at the other end.
What I mean by that is I would like to know why there are systems on the
LAN changing their MAC address in a way in which it doesn't automatically
update the network through gratuitous ARPs,
so I might lower the gateways ARP cache timer until I can
localize that system or systems and figure out how to make
their MAC addresses either more stable or make sure that
they're sending those gratuitous ARPs.
Then I can return to my gateway devices and I can set the
ARP cache timer back to its default.
If you don't want to convert 4 hours back into seconds,
for the command just use default arp timeout and
that'll set it back to the 4 hours.
To verify it,
I'll use the show int f0/0 and we'll see that ARP timer is back to 4 hours.
To summarize,
stale and false ARP cache entries rarely cause problems in
modern networks due to dynamic ARP's robustness and the relative
stability of our local area networks.
ARP cache timers and automatic updates through gratuitous ARP means
that thankfully we shouldn't see this very often.
However, it's great to have this knowledge in your back pocket.
Problems occur when lack of dynamic updates through gratuitous
ARP cross with those long ARP cache timers to create false and
stale information leading to the wrong IP and MAC address
bindings in our ARP caches.
Fixes for this include clearing your ARP caches.
Now you can do this once or twice,
but it's going to get repetitive really quickly.
Another option is you can change the timers,
especially on your default-gateways,
which could have up to four hours set as their ARP cache timer.
Remember however,
changing timers on your default-gateway or on service systems will
increase the ARP processing overhead on the device,
as well as increase ARP traffic in the network,
so only do this as a short-term fix until you can solve the real problem.
You can also potentially change the software or the people processes
which are failing to send the dynamic ARP updates.
If this is happening because redundancy failover of
virtualization software is not updating the network,
you may have little choice but to change the timers or
change out the software altogether.
Of course, your ultimate goal is to create stability in your LAN.
Try not to change dynamic ARPs processes or timers without
fully investigating what your options are and seeing what the consequences are in the network.