Tải bản đầy đủ - 0 (trang)
6 Using traceroute, tcptraceroute, and mtr to Pinpoint Network Problems

6 Using traceroute, tcptraceroute, and mtr to Pinpoint Network Problems

Tải bản đầy đủ - 0trang






* * *

lw-core1-ge2.rtr.liquidweb.com (

116.363 ms

115.567 ms


lw-dc1-dist1-ge1.rtr.liquidweb.com ( 129.055 ms 137.067 ms *

host6.miwebdns6.com ( [open] 130.926 ms 122.942 ms 125.739 ms

An excellent utility that combines ping and traceroute is mtr (My Traceroute). Use

this to capture combined latency, packet loss, and problem router statistics. Here is

an example that runs mtr 100 times, organizes the data in a report format, and stores

it in a text file:

$ mtr -r -c100 oreilly.com >> mtr.txt

The file looks like this:

HOST: xena


1. pyramid.alrac.net


2. gateway.foo.net


3. router.foo.net






6. gbr1.st6wa.ip.att.net


7. br1-a350s5.attga.ip.att.net


8. so0-3-0-2488M.scr1.SFO1.gblx 1.0%

9. sonic-gw.customer.gblx.net


10. 0.ge-0-1-0.gw.sr.sonic.net


11. gig50.dist1-1.sr.sonic.net


12. ora-demarc.customer.sonic.ne 5.0%

13. www.oreillynet.com

















Avg Best




23.5 23.1 21.6

23.4 24.4 21.9

52.8 57.9 44.5

61.9 62.4 50.1

61.4 76.2 46.2

57.2 60.0 44.4

73.9 83.4 64.0

72.6 79.9 69.3

71.5 78.2 67.6

81.1 84.3 73.1

69.1 82.9 69.1

75.4 81.0 69.8

Wrst StDev







127.3 10.3



307.8 48.8

107.1 11.6

265.9 27.6





169.1 12.1

144.6 10.2



This shows a reasonably clean run with low packet loss and low latency. When

you’re having problems, create a cron job to run mtr at regular intervals by using a

command like this (using your own domain and filenames, of course):

$ mtr -r -c100 oreillynet.com >> mtr.txt && date >> mtr.txt

This stores the results of every mtr run in a single file, with the date and time at the

end of each entry.

You can watch mtr in real time like this:

$ mtr -c100 oreillynet.com

You can skip DNS lookups with the -n switch.


If any of these consistently get hung up at the same router, or if mtr consistently

shows greater than 5 percent packet losses and long transit times on the same router,

then it’s safe to say that particular router has a problem. If it’s a router that you control, then for gosh sakes fix it. If it isn’t, use dig or whois to find out who it belongs

to, and nicely report the trouble to them.

Save your records so they can see the numbers with their own eyes.



Chapter 19: Troubleshooting Networks

There are a lot of web sites that let you run various network tools, such as ping and

traceroute, from their sites. This is a good way to get some additional information for


mtr can generate a lot of network traffic, so don’t run it all the time.

tcptraceroute sends TCP SYN packets instead of UDP or ICMP ECHO packets.

These are more likely to get through firewalls, and are not going to be ignored by

routers. When the host responds, tcptraceroute sends TCP RST to close the connection, so the TCP three-way handshake is never completed. This is the same as the

half-open (-sS) scan used by Nmap.

See Also

• man 8 traceroute

• man 1 tcptraceroute

• man 8 mtr

19.7 Using tcpdump to Capture and Analyze Traffic


You really need to see what’s going over the wires, and you know that tcpdump is

just the powerhouse packet sniffer you want. But, you don’t know how to filter all

those masses of traffic. How do you make it show only what you want to see?


tcpdump can filter your traffic as precisely as you like. Just follow these examples to

learn the more commonly used filters.

You should routinely use the -p switch to prevent the interface from going into promiscuous mode because promiscuous mode is pretty much useless on switched


Capture all traffic on a single host:

# tcpdump -pi eth0 host uberpc

Capture all traffic on more than one host:

# tcpdump -pi eth0 host uberpc and stinkpad and penguina

Capture all traffic on more than one host, except from a specified host:

# tcpdump -pi eth0 host uberpc and stinkpad and not penguina

Capture traffic going to a host:

# tcpdump -pi eth0 dst host uberpc


Using tcpdump to Capture and Analyze Traffic |


Capture traffic leaving a host:

# tcpdump -pi eth0 src host uberpc

Capture a single protocol:

# tcpdump -pi eth0 tcp

Capture more than one protocol:

# tcpdump -pi eth0 tcp or udp or icmp

Capture a specific port:

# tcpdump -pi eth0 port 110

Capture several ports:

# tcpdump -pi eth0 port 25 or port 80 or port 110

Capture a port range:

# tcpdump -pi eth0 portrange 3000-4000

Watch traffic leaving a port:

# tcpdump -pi eth0 src port 110

Watch traffic entering a port:

# tcpdump -pi eth0 dst port 110

Look for packets smaller than the specified size:

# tcpdump -pi eth0 less 512

Look for packets larger than the specified size:

# tcpdump -pi eth0 greater 512

Watch SSH connections from certain hosts:

# tcpdump -pi eth0 src host uberpc or stinkpad and dst port 22

Watch for traffic leaving one network and entering two other networks:

# tcpdump -pi eth0 src net and dst net or

The -X switch reads the data payload, but the default is to only read 68 bytes, so -s0

displays the whole data payload, as this example from an IRC conversation shows:

# tcpdump -X -s0 -pi eth0

10:40:14.683350 IP > P 1:65(64) ack 410 win


0x0000: 4500 0074 c43b 4000 4006 8157 c0a8 010a E..t.;@.@..W....

0x0010: 8cd3 a66b 8a3a 1a0b 420f ddd1 bb15 eb3b ...k.:..B......;

0x0020: 8018 3e96 4309 0000 0101 080a 0012 625e ..>.C.........b^

0x0030: dcbe 2c65 5052 4956 4d53 4720 236c 696e ..,ePRIVMSG.#lin

0x0040: 7578 6368 6978 203a 746f 2062 6520 6120 uxchix.:to.be.a.

0x0050: 7375 7065 722d 7365 6b72 6974 2073 7079 super-sekrit.spy

0x0060: 2c20 7573 6520 7468 6520 2d73 2073 7769 ,.use.the.-s.swi

0x0070: 7463 680a




Chapter 19: Troubleshooting Networks

This particular incantation:

# tcpdump -pXi eth0 -w tcpdumpfile -s0 host stinkpad

captures all traffic passing through Stinkpad, including data payload, and stores it in

the file tcpdumpfile. You can read this file with:

# tcpdump -r tcpdumpfile

Directing tcpdump output to a file lets you study it at leisure, or open it with Wireshark

to read it in a prettier interface. The -w switch creates a file format that Wireshark can

read. Figure 19-1 shows what it looks like in Wireshark.

Figure 19-1. Examining tcpdump output in Wireshark

This command lets you see the live capture and store it in a file. This doesn’t create a

file that Wireshark can read, but it does create a text file that you can parse with your

favorite text-searching utilities:

# tcpdump -pXi eth0 -s0 host stinkpad -l | tee tcpdumpfile

This is a good way to catch infected hosts that are sending out spam because nobody

should be sending anything from port 25 except your official mail servers:

# tcpdump -pni eth0 dst port 25 and not src host mailserver1

The -n switch turns off name resolution.

Finally, you might want to use the -c switch to limit the number of packets captured:

# tcpdump -c 1000 -pXi eth0 -w tcpdumpfile -s0

Otherwise, it will run until you hit Ctrl-C.


Using tcpdump to Capture and Analyze Traffic |



tcpdump should be your number one network troubleshooting tool because it shows

you exactly what is happening over your wires. Don’t guess—run tcpdump.

Let’s dissect some typical tcpdump output, using an excerpt from checking mail:

# tcpdump -pi eth0

14:23:02.983415 IP xena.alrac.net.58154 > host6.foo.com.pop3s: S 3100965180:

3100965180(0) win 5840 (DF)

• 14:23:02.983415 is the timestamp, in hh:mm:ss:fraction format.

• xena.alrac.net.58154 is the originating host and port.

• host6.foo.com.pop3s is the destination host and port.

• S is the first part of the three-way TCP handshake (SYN, SYN, ACK).

• 3100965180:3100965180 is the byte sequence/range. The initial sequence number

(ISN) is generated randomly. Then, sequence numbers for the rest of the bytes in

the connection are incremented by 1 from the ISN. Because no data are

exchanged at this stage, both numbers are the same.

• win 5840 is the window size, or the number of bytes of buffer space the host has

available for receiving data.

• mss 1460 is the maximum segment size, or maximum IP datagram size that can

be handled without using fragmentation. Both sides of the connection must

agree on a value; if they are different, the lower value is used. This is called path

MTU (Maximum Transmission Unit) discovery. MTU is the size of the total

frame, which includes the MSS plus TCP/IP headers, and any other headers that

are required by the sending protocol.

• sackOK means “selective acknowledgments,” which allows the receiver to

acknowledge packets out of sequence. Back in the olden days, packets could

only be acknowledged in sequence. So, if the third packet out of a hundred packets received went missing, the host could only acknowledge the receipt of the

first two packets, and the sender would have to resend all packets from number

3 through 1,000. sackOK allows only the missing packets to be resent.

• timestamp 4546985 0 measures the round-trip time. There are two fields: the

Timestamp Value and the Timestamp Echo Reply. On the first exchange, the

Echo Reply is set to 0. When the second host receives that packet, it transfers the

timestamp from the old packet’s Timestamp Value field to the new packet’s

Timestamp Echo Reply field. Then, it generates a new value for the Timestamp

Value field. So, the Timestamp Value field contains the latest timestamp, while

the Timestamp Echo Reply field contains the previous timestamp.

• nop, or “no operation,” is just padding. TCP options must be multiples of 4

bytes, so nop is used to pad undersized fields.



Chapter 19: Troubleshooting Networks

• wscale 0 is a nifty hack to get around the original window size limitation of

65,535 bytes. wscale provides for a full gigabyte of buffer. Both sides of the connection must support this and agree; otherwise, the window size does not


• (DF) means “don’t fragment.”

Sometimes, you need the correct physical placement to capture the type of information you want. For example, if you want to catch infected hosts sending out spam, or

want to watch traffic between networks, you’ll need to run tcpdump on a router. Or,

plug-in your handy network administrator laptop between the router and the switch,

if you have dumb switches. Smart switches have network monitoring ports.

Plug-in your handy network administrator laptop between the Internet and your firewall to get an unfiltered view of what’s trying to enter your network.

See Also

• man 8 tcpdump

19.8 Capturing TCP Flags with tcpdump


The syntax for tcpdump filters is pretty easy to understand, until you come to the

part about filtering on specific TCP flags, like SYN, ACK, RST, and so forth. Then, it

goes all bizarre. How do you know what to use?


The tcpdump manpage tells how to calculate the correct values for TCP flags. You are

welcome to study it and learn how to figure them out from scratch. Or, you can copy

them from here.

Capture all SYN packets:

# tcpdump 'tcp[13] & 2 != 0'

Capture all ACK packets:

# tcpdump 'tcp[13] & 16 != 0'

Capture all SYN-ACK packets:

# tcpdump 'tcp[13] = 18'

Capture all FIN packets:

# tcpdump 'tcp[13] & 1 != 0'

Capture all URG packets:

# tcpdump 'tcp[13] & 32 != 0'


Capturing TCP Flags with tcpdump |


Capture all PSH packets:

# tcpdump 'tcp[13] & 8 != 0'

Capture all RST packets:

# tcpdump 'tcp[13] & 4 != 0'

These may be combined with other filtering options such as ports, hosts, and networks, just like in the previous recipe.


There are several scenarios where you’ll want to look for certain TCP flags, such as

when you’re investigating suspicious activity, or having problems with misconfigured services sending the wrong responses. Another way to do this sort of filtering is

to capture a lot of data with minimal filtering and dump it to a file with the -w

switch, then examine the file in Wireshark. Then, you’ll be able to filter the same set

of data several different ways without having to get a new capture each time.

Using Wireshark to analyze and filter a tcpdump capture is probably the most flexible and powerful method available. Figure 19-2 shows my favorite feature, Follow

TCP Stream. This lets you pluck out a single TCP stream from all the masses of data

you’ve collected. Wireshark supports all the same filters as tcpdump, and has lots of

nice graphical menus to help you put them together.

Figure 19-2. Wireshark can highlight a single TCP stream



Chapter 19: Troubleshooting Networks

You may prefer to use Wireshark in place of tcpdump entirely. If you’re running any

headless boxes or servers without X Windows, you’ll still want to know how to use


See Also

• man 8 tcpdump

• Wireshark: http://www.wireshark.org/

• Wireshark’s included Help pages

19.9 Measuring Throughput, Jitter, and Packet Loss

with iperf


You want to measure throughput on your various network segments, and you want

to collect jitter and datagram loss statistics. You might want these just as a routine

part of periodically checking your network performance, or you’re running a VoIP

server like Asterisk, Trixbox, or PBXtra, so you need your network to be in extragood shape to have good call quality.


Use iperf, which is a nifty utility for measuring TCP and UDP performance between

two endpoints. It must be installed at both ends of the connection you’re measuring;

in this example, that is Xena and Penguina. We’ll call Xena the server and Penguina

the client. First, start iperf on Xena in server mode, then fire it up on Penguina. (The

easy way is to do all this on Xena in two X terminals via SSH.)

carla@xena:~$ iperf -s

-----------------------------------------------------------Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

-----------------------------------------------------------terry@penguina:~$ iperf -c xena

-----------------------------------------------------------Client connecting to xena, TCP port 5001

TCP window size: 16.0 KByte (default)

-----------------------------------------------------------[ 3] local port 49215 connected with port 5001

[ 3] 0.0-10.0 sec

111 MBytes 92.6 Mbits/sec

And it’s done. That’s a good clean run, and as fast as you’re going to see over Fast



Measuring Throughput, Jitter, and Packet Loss with iperf |


You can conduct a bidirectional test that runs both ways at once:

terry@penguina:~$ iperf -c xena -d

-----------------------------------------------------------Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

----------------------------------------------------------------------------------------------------------------------Client connecting to xena, TCP port 5001

TCP window size: 56.4 KByte (default)

-----------------------------------------------------------[ 5] local port 59823 connected with port 5001

[ 4] local port 5001 connected with port 58665

[ 5] 0.0-10.0 sec

109 MBytes 91.1 Mbits/sec

[ 4] 0.0-10.0 sec 96.0 MBytes 80.5 Mbits/sec

Or, one way at a time:

$ terry@uberpc:~$ iperf -c xena -r

Compare the two to get an idea of how efficient your Ethernet duplexing is.

Troubleshooting multicasting can drive a network administrator to drink, but fortunately, iperf can help. You’ll run iperf in server mode on all of your multicast hosts,

and then test all of them at once from a single client:

admin@host1:~$ iperf -sB

admin@host2:~$ iperf -sB

admin@host3:~$ iperf -sB

carla@xena:~$ iperf -c

If you’re using multicasting for video or audio streaming, you’ll want to test with

UDP instead of the default TCP, like this:

admin@host1:~$ iperf -sBu

admin@host2:~$ iperf -sBu

admin@host3:~$ iperf -sBu

carla@xena:~$ iperf -c -ub 512k

Adjust the -b (bits per second) value to suit your own network, or use -m for megabits. Testing with UDP will generate a number of useful and interesting statistics. If

the server is still running, stop it with Ctrl-C, then run this command:

carla@xena:~$ iperf -su

-----------------------------------------------------------Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size:

108 KByte (default)


Then, start the client:

terry@penguina:~$ iperf -c xena -ub 100m

-----------------------------------------------------------Client connecting to xena, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size:

108 KByte (default)



Chapter 19: Troubleshooting Networks

Tài liệu bạn tìm kiếm đã sẵn sàng tải về

6 Using traceroute, tcptraceroute, and mtr to Pinpoint Network Problems

Tải bản đầy đủ ngay(0 tr)