Network - Troubleshooting the routing table

To ensure that the routing table has been correctly configured.

The routing table contains information used by the network stack to decide where each packet should be sent next. This decision may result in the packet being delivered locally, or forwarded to another machine, or rejected as unroutable. For packets that are forwarded the routing table is concerned only with selection of the next ‘hop’, at which point another routing decision will be made.

Within the routing table is a list of routes. Each route specifies an address range (expressed as a network address and a netmask), the interface to which packets matching that address range should be sent, and (optionally) the address of a gateway machine.

The route that is used for a given packet is the most specific one with an address range that matches the ultimate destination address. If the route has a gateway then the packet is forwarded to that gateway, otherwise it is forwarded directly to its final destination.

If the network address and netmask of a route are both zero then it will match packets with any destination address, but only if there is no other route that matches. Most routing tables have such an entry, which is known as the 'default route'.

There are a number of different ways in which routes may be created:

• Whenever you bring up an interface using ifup, a route is automatically created for the address range that is directly reachable through that interface.
• You can specify further static routes to be created when an interface is brought up.
• If an interface uses an automatic configuration protocol such as DHCP and is able to obtain a suitable gateway address then it may create a default route to that gateway.
• If an interface uses a point-to-point protocol such as PPP then it may automatically create a default route to the far end of the link.

If your network is sufficiently complex to make use of a routing protocol such as BGP or RIP then these can automatically add any number of dynamic routes to the table.

A routing table error may result in:

• traffic being forwarded to the wrong interface, or
• traffic not being forwarded when it should have been.

Note that either of these issues could equally indicate a problem with iptables, and a failure to forward could indicate that forwarding has not been enabled.

In the unlikely event that you are using the NAT capabilities of iproute2 (as opposed to NAT in iptables) then a configuration error could additionally cause traffic to be given the wrong source or destination address. You should also be aware that iproute2 has the ability to support multiple routing tables, therefore the route taken by a packet may depend on more than just its destination address.

Suppose that a machine has been configured to act as a boundary router between a local area network (connected to interface eth0 with the address 192.168.0.1/24) and the public Internet (connected to interface ppp0 with the address 203.0.113.144/32). The default gateway is 203.0.113.1. Because the local area network uses a private address range, iptables on the boundary router has been configured to SNAT them to its public IP address.

In order to test this configuration you have attempted to ping a machine on the public Internet (198.51.100.1) from a machine on the local area network (192.168.0.2), but this has failed.

There are two possibilities to consider:

• that the routing table has not been populated with the routes that you intended, or
• that the table has been populated as intended but the routes do not have the desired effect.

It is usually worth quickly checking the first point before launching into any detailed investigation of the second.

Remember that routes created using the route and ip route commands are not persistent. Even if you are certain that the routing table was correct at some point in the past, that configuration will not necessarily survive a reboot.

The traditional method for viewing the content of the routing table is by means of the route command, however this predates iproute2 and the information it displays may be incomplete. The preferred method is to use the ip route command:

ip route show

The response is a list of routes, which for the scenario described above should be similar to:

203.0.113.1 dev ppp0  proto kernel  scope link  src 203.0.113.144
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1
default via 203.0.113.1 dev ppp0

These routes specify that:

• traffic to 203.0.113.1 (the gateway to the public Internet) should be forwarded directly to its destination through ppp0;
• traffic to anywhere within 192.168.0.0/24 (the local area network) should be forwarded directly to its destination through eth0; and
• traffic to anywhere else should be forwarded through ppp0 to the gateway at 203.0.113.1.

There is little benefit in attempting to predict the effect of the table on individual packets because this can be done more quickly and reliably using the automated method below. However you should aim to ensure that there are no spurious or missing routes, and that the destination addresses and device names are correct.

It is possible to test the effect of the routing table on a particular packet using the command ip route get. You need to know where the packet came from (IP address and interface name) and where it is supposed to go to (IP address).

For example, the outbound test packet in the scenario above originates from 192.168.0.2 and is received by the router on eth0. It is destined for 198.51.100.1. You can inquire how such packets are routed using the command:

ip route get to 198.51.100.1 from 192.168.0.2 iif eth0
<code>
 
which should give a response of the form:
 
<code>
198.51.100.1 from 192.168.0.2 via 203.0.113.1 dev ppp0  src 192.168.0.1
    cache <src-direct>  mtu 1500 advmss 1460 hoplimit 64 iif eth0

The main points to check are that the IP address of the next hop is correct (203.0.113.1 in this case) and that the packet is being sent onwards through the appropriate interface (ppp0).

You should also check the reverse path, because you will get no response to a ping unless it can be routed in both directions:

ip route get to 192.168.0.2 from 198.51.100.1 iif ppp0

This should give a response of the form:

192.168.0.2 from 198.51.100.1 dev eth0  src 203.0.113.144
    cache  mtu 1500 advmss 1460 hoplimit 64 iif ppp0

Points to note:

• The ip route get command is sensitive to whether forwarding has been enabled, so if it returns the error ‘No route to host’ then that may be the reason.
• Routing is performed before any change to the source address by the iptables SNAT target. You should therefore use the real address from which traffic originates, not the address from which it purports to originate after NATting.
• Routing is performed after any change to the destination address by the iptables DNAT target. You should therefore use the address to which the traffic is redirected after NATting, not the address to which it was originally sent.

Though deprecated, the route command it adequate for inspecting most of the configurations that are likely to be encountered in practice:

route -n

The -n option causes IP addresses to be displayed numerically. This is usually preferable to converting them into names.

For the scenario described above the output should look similar to the following:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
203.0.113.1     0.0.0.0         255.255.255.255 UH    0      0        0 ppp0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         203.0.113.1     0.0.0.0         UG    0      0        0 ppp0