[Autogenerated] Now it's time to look at
those slow responses. When you're looking
for any application slowdown, you always
want to capture at the server first, to
eliminate any possibility of the network.
Once the server and application have been
exonerated, then we can start to look at
the network. It makes for a faster
investigation. If you eliminate some of
the possible suspects. Jest. We're here. A
tennis primary. We'll start a capture here
first, then move over to en is secondary
and started capture there as well. N s
stub is next, and then finally, we'll
start a capture on N S bind. That's our
Linux server, so we'll use TCP dump. The
command is pseudo TCP dump. I also want to
do a capture filter for Port 53 or ICMP.
Then I'll add a minus W. To write the file
to desk. Since Gabriella works with the
users she's heard plenty of complaints
about Sloan is she'll go ahead and act as
our user and will capture on her station
while she walks through the steps about
the complaints she's received. She'll open
a few example pages so we can get
measurements for response time are
examples will be Apple Del than Cisco
Azure and then VM Ware. And then, of
course, our own example. Globomantics dot
com. Now we're over at Sue Hani's
workstation. She's a security analyst for
Globomantics dot com. She'll be working
with Jaden a little bit later to implement
some security changes. She's going to go
to the same web page is some of those
DNA's responses might still be in cache
and better be fast, but some of them may
have aged out, depending on their time to
live. And so our servers will have to go
through the Rikers in process again. When
we look at the traces, we'll know which
are which. Now, all we have to do is save
all of the files and get them onto a
single computer in order to stop the
capture on N S find, all we have to do is
a control C to stop. Great. No packets
were dropped by the server while we were
capturing. Jayden will be the one to look
at. All of the packets are cool is to look
at commands and replies from a single
server perspective, but it would be more
convenient to have all of the packets in a
single peak app. We could use Merge cap at
the command line, but quite frankly, the
easiest thing to do is to shift. Click all
of the files and just drag and drop them
in to wire shark wire. Shark will go ahead
and put them in order by the time stamp of
the packet. If we were capturing in
multiple places to watch the same packet
travel through the network, this would not
be our approach. We would not merge them
together, but we're looking at a single
packet in a single place, and rather than
looking each file individually, we're just
putting them together to make it easier
for us. That's why I'm urging works
perfectly in order to filter for just the
traffic for our test, we'll use matches
again. Only this time we're going to take
advantage of those special characters or
medic characters. We'll have to double
quote it anytime. Use a medic character
with wire shark filters. You have to
double quote around the entire filter, and
we're going to be doing an or so I'll use
an open round bracket and put in
Globomantics pipe single pipe. That's the
or Apple pipe del pipe, Cisco Pipe, azure
pipe, VM ware and then close round
bracket. Don't forget your double-click a
haute. Much better. We're down to 20% of
the packets. It's easy to see with the
source and destination that the servers
performed recur versioning. What we are
most interested in are those responses
that were from cache. They're the ones
that should be fast. We'll start with
statistics for DNS to give ourselves a big
picture. This dialog box does not
automatically bring in the filter, so
we'll have to copy and paste it so that we
only get information specific to what we
care about there. Now. We can see that our
minimum response time was 480
microseconds. Let's listen like half a
millisecond. That is awesome. We love
that. In my opinion, a DNA is response
from cache when you're capturing, closest
to the server should be within one or two
milliseconds, so our average of 100
milliseconds is not good. However, there's
no way to filter for responses that were a
cache reply versus a reply that was recur
versioning. So looking at a statistic of
8.66 seconds as the maximum. It's kind of
misleading. It could be because the Ricker
version process took so long or it could
be already in a server. We'll have to
filter to find out. But at least now we
know our best case scenario, our worst
case scenario and our average. Let's
export the specified packets. So we can
work with just the filtered peak app
we can sort by Delta application to see
those worst case scenarios, this one
happens to be for the Dell web page. When
this packet was sent, there were 14
minutes and 52 seconds left of the time to
live. We have no way of knowing what it
was when it started, though, unless we
happened to capture the Rikers in process.
This packet is from our primary server.
Going back out to the user in Chicago will
filter for the transaction ID to see the
rest of the pack. It's involved in the
conversation. Our clients Suhani went to
the primary server who did not respond and
remember we were capturing at the primary
server. It's not like that packet
disappeared on the network. 168
milliseconds Later, she asked again. Only
this time she went to the secondary
server. The secondary server went ahead
and replied, However, almost eight seconds
later, the primary went ahead and sent the
answer back to the client. Hopefully that
eight packets was taken up with Ryker
versioning. Let's do a filter and see.
We'll start with an I P address filter for
the N s primary server. Then we'll add in
the query Name s s t dot dell dot com.
That way we'll be able to see if N. S
primary went out and asked the
authoritative server for the answer. Oh,
my we can tell that the client went to the
primary in the first packet in the list
asking for that name. Then our primary did
go to the authoritative server for Dell
and asked for that IP address. And that
server, the authoritative server for Dell,
answered in 46 milliseconds. Some of that
is network. Some of that is DNS. Then it
took almost eight seconds for N s primary
to take that response and create his own
response back to Gabriella. There is no
reason for that. That is such a bad thing.
That, my friends, is what we call in
packet World a smoking gun. Something is
wrong with Ennis primary. We don't know
yet. If it's Deen is or the server itself
Server could be experiencing load, it
could be under provisioned. There are many
reasons why it would take this delay.
Usually, the issue is it's doing some
other process. Maybe there was a CPU
spike. Maybe there was a hard Docker
spike, but definitely it was not the
network and a desire to the server or DNS
to see if the reason that it was such a
slow response was because Ennis Primary
was processing so many other requests will
do a filter for the time range. In between
the request and the reply. We can use
theorize vel time field. To create that
filter, we want an arrival time that is
greater than or equal to the request and
less than or equal to the response when
the filters get too long to display.
Sometimes I just copy and paste them into
note pad so that I can see the entire
filter at once. G E is for greater than or
equal to, and Ellie is for less than or
equal to. I don't have to take out the
parentheses but they're extraneous and I
can't help myself who paste that into our
filter. Let's see what other DNS queries
and replies was happening during that same
time frame. 45 packets Not exactly being
hammered with a ton of DNS. Requests will
have to give this information to the
system Admin for the DNS server. They can
check the log files for the CPU or hard
Docker spikes and to make sure it's not
under provisioned spoiler alert. I am that
sys admin, and it has been under
provisioned if we change the N s primary
to on IP source filter. Now we can focus
on the worst case scenarios for the Delta
application time. Predominantly, it's not
too hideous, but we should be worried
about one that is almost four seconds, and
the one we've been looking at that's
almost eight. That is a lifetime. Let's go
back and see the worst case response times
for the Bind Server. All of the servers
have the same provisioning specs, so the
slowest response time is just over a
second for the Bind server in Chicago.
When we filter based on the transaction
ID, we see something similar. Our client
0.4. That's actually that's Gabriella. The
0.6 was Suhani. Sorry about that. My
apologies to Suhani. Gabriella makes a
request to the bind server and receives
silence as the reply. After 634
milliseconds, she RETRANSMITS and tries
again. So far, we've been seeing varying
times for those retransmissions. When the
clients don't get a response, it's been as
much as two seconds, and I've seen it as
small as 25 milliseconds. But after the
second attempt are Bind Server sends the
reply. Let's see if it's because he had to
perform recur versioning. There's only two
packets in between 4, 63 466. So instead
of doing a filter, I'll just mark the
second request and clear the current
filter to see what packets are in between.
Very similar scenario. Only with faster
times Gabriella goes to the server. The
bind server goes out to the authoritative
server, gets a reply very quickly at 8.5
milliseconds, then seems to take a whole
second to translate the reply received
from the authoritative server to reply
they could send to the client, Jaden
checked some of the statistics on the Bind
server to see if it was experiencing any
CPU spikes or network spikes will not
experiencing any. But Jaden realizes that
for both his Microsoft and Bind servers,
he needs to enable logging that way. When
these things happen, we can check the logs
at the same time to see how all of those
statistics come together. Jaden is
interested in comparing the response times
for all four servers he wants to eliminate
any of those times. We had to wait for
Riker versioning and focus only on the
response times when our servers were the
authority servers. For example, when we go
to the sales web page, let's make a graph.
To make this easier to see, I'll put in
one line for each server IP address and
select the maximum for DNA's time for
each. Unfortunately, I copied and pasted
DNA's dot com into each of the entries. So
let me fix that really quick. We'll give
each server a color and zoom into the time
of the most activity definitely looks as
if the primary had the most issues, but
the secondary and the stub are far from
being innocents and the bind server, while
his bars are shorter is an innocent
either. The longer you collect the data,
the more accurate the visual becomes real.
I'll and Jayden had a lot on their plate,
but they and the rest of the team had wire
shark to show them how to isolate and
resolve issues as quickly as possible. The
goal is always the smallest impact on the
users and customers up next securing DNA's.