TCP/IP Troubleshooting: A Structured

Approach - Part 2: Troubleshooting

Routing Tables
• Launched: Oct 17, 2006
• Updated: Nov 15, 2006
• Section: Articles & Tutorials :: Network Troubleshooting
• Author: Mitch Tulloch

This article explains how routing tables work and how to troubleshoot routing
problems in Windows-based networks.

In the first article of this series, I outlined a structured approach for troubleshooting
TCP/IP networking issues on Windows-based networks. Key to this structured
approach was three things:

• Understanding the networking technologies and protocols that underlie the

problem.
• Determining the different elements of the problem and their attributes.
• Determining which troubleshooting steps and tools should be applied to
resolve the problem.

I displayed these items as a bullet list instead of a numbered list because network
troubleshooting generally isn't as easy as 1-2-3. In other words, it's often more of an
art (i.e. based on intuition) than a science (based on a methodology).

At the foundation of TCP/IP networking is the routing table, a data construct on each
host on a TCP/IP network. Routing tables serve the following three purposes:

• They are used to store the information about other subnets on the network and
how you can reach hosts on these networks.
• They are used to determine which host (called the next-hop IP address) each
packet should be forwarded to in order to reach the host this packet is
ultimately destined for.
• They are used to determine which network interface (called the next-hop
interface) should be used to forward this packet so it gets to its ultimate
destination.

Understanding routing tables is therefore essential if you want to be able to effectively

troubleshoot routing issues on a TCP/IP network. Let's look at how routing tables
work, what they look like in different scenarios, and what troubleshooting steps and
tools might be indicated in different situations. We'll begin by examining the routing
table on a single-homed server (a server with a single network interface) that has a
single IP address assigned to it. I've chosen this example because it's the easiest one to
understand, and in next month's follow up article we'll look at more complicated
scenarios including servers with multiple IP addresses (such as web servers) and
servers with multiple network interfaces (such as servers that are connected to both
the LAN and to a separate network used for running backup jobs).

Routing Table for Single-Homed Server with a Single

IP Address
The following routing table is for a server that has IP address 172.16.11.30 on the
172.16.11.0/24 network:

To display this routing table, you open a command prompt window and type route
print at the command line. Let's take this table apart so we can understand how it
works.

Each routing entry (or route) in the routing table is comprised of five fields:

• Network Destination. An IP address or subnet representing a possible target

destination to which IP packets may be forwarded.
• Netmask. A bitmask used for matching the destination field in a packet's IP
address to one of the possible network destinations above.
• Gateway. The next-hop IP address to which the packet must be forwarded to
reach the particular destination network.
• Interface. The next-hop interface which must be used to forward the packet to
reach the particular destination network.
• Metric. The cost of the route.

Example 1: Destination Host on Local Subnet

For our first example, let's say this particular server (172.16.11.30) has to send a
packet to another host with IP address 172.16.11.80, which is on the same subnet.
This packet will thus have source address 172.16.11.30 and destination address
172.16.11.80. Here is how Windows uses its routing table to decide which route to
use:

1. Windows first takes each route from the table in turn and performs a bitwise
AND between the destinations address in the packet (172.16.11.80) and the
bitmask (Netmask) of the selected route. Here are the results, where each route
in the table is identified by its network destination:

Route Netmask 172.16.11.80 AND Netmask

0.0.0.0 0.0.0.0 0.0.0.0
127.0.0.0 255.0.0.0 172.0.0.0
172.16.11.0 255.255.255.0 172.16.11.0
172.16.11.30 255.255.255.255 172.16.11.80
172.16.255.255 255.255.255.255 172.16.11.80
224.0.0.0 224.0.0.0 160.0.0.0
255.255.255.255 255.255.255.255 172.16.11.80

2. For each route, the result of this ANDing is then compared with the Network
Destination field of the route, and a match means the route can be used to
forward the packet to its destination address. If more than one match is found,
Windows uses the route with the longest match (the route whose Netmask has
the highest number of 1 bits). If this doesn't result in a unique match, Windows
uses the match that has the lowest cost (Metric). Finally, if more than one
match has the same lowest cost, Windows arbitrarily chooses one of them as
the route to use. From the table above, you can see that this ANDing process
results in two matches (routes 1 and 3) so Windows chooses the one that has
the longest match, which is row 3. The result of all this is that Windows now
knows which route to use to get this packet to its destination. Here’s what this
route looks like in the server's routing table:
Network Netmask Gateway Interface Metric
Destination
172.16.11.0 255.255.255.0 172.16.11.30 172.16.11.30 20

3. Windows now uses the following algorithm to decide what to do next:

A. If the route's Gateway field matches the address of one of the network
interfaces on the server (or if the Gateway is empty) then Windows
sends the packet directly to its destination address using the interface
specified in the route.
B. If the route's Gateway field does not match the address of any of the
network interfaces on the server, then Windows sends the packet to the
address in the route's Gateway field.

Clearly, condition A is the case here since the route's Gateway field (172.16.11.30) is
the address assigned to the server's single network card. Windows therefore
determines that the destination address is on the local subnet and that means Windows
can send the packet directly to that address without needing to forward it to any
routers. So in this case, Windows simply sends the packet to 172.16.11.80 using the
server's 172.16.11.30 network interface, and the receiving host gets it.
Example 2: Destination Host on Remote Subnet
Now let's go through the same process, but this time let's say the server is trying to
send the packet to a host on a different subnet, say a host with the address
172.16.10.200. In other words, the packet has a source address of 172.16.11.30 and a
destination address of 172.16.10.200. Here's how Windows uses its routing table to
decide which route to use this time:

1. Windows takes each route from the table and performs a bitwise AND
between the destination address in the packet (172.16.10.200) and the bitmask
(Netmask) of the route. The results this time are like this:

Route Netmask 172.16.11.200 AND Netmask

0.0.0.0 0.0.0.0 0.0.0.0
127.0.0.0 255.0.0.0 172.0.0.0
172.16.11.0 255.255.255.0 172.16.10.0
172.16.11.30 255.255.255.255 172.16.10.200
172.16.255.255 255.255.255.255 172.16.10.200
224.0.0.0 224.0.0.0 160.0.0.0
255.255.255.255 255.255.255.255 172.16.10.200

2. For each route, the result of ANDing is compared with the Network
Destination field of the route, and a match means the route can be used to
forward the packet to its destination address. From our second table above,
you can see this time that there is only one match i.e. row one where the
route's Network Destination field 0.0.0.0 matches the result of the AND
action. So the route that Windows will use to forward the packet to its
destination will be the following route:

Network Netmask Gateway Interface Metric

Destination
0.0.0.0 0.0.0.0 172.16.11.1 172.16.11.30 20

3. Windows then uses the previously-described algorithm to decide what to do

next, and this time condition B is the case here since the route's Gateway field
(172.16.11.1) does not match the address assigned to the server's single
network card, which is 172.16.11.30. Windows therefore determines that the
destination address is on a remote subnet and that means Windows cannot
send the packet directly to its destination but must instead forward the packet a
router, which will then take care of getting the packet to its destination by
forwarding it along further. So in this case, Windows sends the packet to the
address shown in the Gateway field of the selected route (172.16.11.1) using
the server's 172.16.11.30 network interface. Once the router at 172.16.11.1
receives the packet, it determines what actions need to be taken next to
forward the packet on to its final destination of 172.16.10.200, and of course
this depends on whether the 172.16.11.10/24 network is a neighboring subnet
to 172.16.11.11/24 (i.e. connected by a single router to it) or a remote network
(connected by a series of routers with intermediate networks in between them).
Troubleshooting Tips
What can fail in the above process? First, it's possible that Windows might be unable
to select a route whose Network Destination field matches the bitwise AND of the
Netmask field of the route and the packet's destination address. If this happens, you've
got a routing error, and this will probably be indicated to you by some network
application running on your server. What happens usually is that Windows uses TCP
to notify the upper layer of the network stack that the packet can't be sent, and an error
message of some kind usually results.

In this situation, you probably have either a corrupt routing table or an invalid
persistent route in your routing table. Persistent routes are routes you add manually to
the table using the route -p add command and which persist across reboots since their
values are stored in the registry. If you add routes that are invalid, they can produce
strange results, though most often they simply result in traffic being dropped
mysteriously.

On the other hand, if the destination host is on a remote subnet and Windows
forwards the packet to a router (the default gateway address) and this router can't
select a route, then what usually happens in this case is that the route returns an ICMP
message of "Destination Unreachable – Host Unreachable" to the host that sent the
packet. In this case, TCP will notify upper layers and some sort of error message will
be displayed.

In either situation, a useful way to proceed is to examine the routing tables on the
sending host and any intermediate routers along the way to the destination host, and
see if these routing tables are consistent or look corrupted. A corrupted routing table
can be restored (at least on Windows machines) by resetting the TCP/IP stack using
the netsh int ip reset command, see KB299357 for details. Note that this reset
operation does not remove persistent routes you've added to your routing table.

Conclusion
Now that you know a bit about how routing tables work and how to troubleshoot
them, in the next article we'll look at more complex examples such as servers with
multiple addresses and several network cards.