BGP and asymmetric routing

What is Asymmetric Routing?

Asymmetric routing is the situation where packets from A to B follow a different path than packets from B to A. Asymmetric routing is very common with BGP, and completely avoiding it is impossible. Fortunately, under normal circumstances, asymmetric routing doesn’t cause any problems, as routers don’t care about this and obviously, the sending and receiving hosts see packets in both directions. Firewalls, on the other hand, often need to see the packets flowing in both directions to function properly. So it’s important to place firewalls somewhere before the packet flows in the incoming and outgoing directions diverge.

Figure 1: symmetric routing.

Figure 1 shows a situation where asymmetric routing is unlikely. There are two paths between AS100 and AS500: through AS200 and AS300 for a total of three hops or through AS400 for a total of two hops. So if the local preference attribute is the same, the shorter AS path will make packets flow through AS400 in both directions, as shown by the blue arrows for packets from AS100 to AS500, and the red arrows showing the direction from AS500 to AS100.

Figure 2: asymmetric routing.

In figure 2, however, the two available paths are both two AS hops, so the AS path length step in the BGP path selection algorithm won’t force a decision. This generally means that the path selection decision will come down to the tie breaker rules, so there’s about a 50% chance that traffic from AS100 to AS500 will flow through AS200 and a 50% chance that it will flow through AS400. The same is true in the opposite direction, making for the following possibilities:

So there’s a 50% probability that traffic is routed symmetrically and a 50% probability that traffic is routed asymmetrically.

Figure 3: (asymmetric) hot potato routing between two ASes

However, there’s another feature of BGP that all but guarantees asymmetric routing. Consider figure 3, which shows ASes 100 and 500 that interconnect in two locations: west and east. A customer of AS100 in location west communicates with a customer of AS500 in location east. The default behavior of BGP in this situation is to hand off the traffic to the neighboring AS at the first possible opportunity. (“Hot potato” or “early exit” routing.) So the traffic from AS100 to AS500 flows to AS500 in location west, and the return traffic from AS500 to AS100 is handed off in location east. The reasons for this behavior are the “prefer eBGP paths over iBGP paths” rule and the “prefer the path with the lowest IGP cost” rule in the BGP path selection algorithm.

In the situation in figure 2, the fact that routing is asymmetric is easily determined by looking at the AS paths: if the AS paths in the two directions show the same ASes, packets are routed symmetrically; if not, asymmetrically. But in the situation shown in figure 3, the AS paths don’t indicate asymmetric routing. But traceroutes in the two directions will show differences, assuming that the routers that show up in the traceroutes have a DNS name that indicates their location. (The addresses in the traceroutes will be different in both directions even with symmetric routing because the traceroutes hit different interfaces with different addresses in the two directions.)

We now know how asymmetric routing happens, but not why it can be problematic. For that, consider figure 4.

Figure 4. Asymmetric routing and firewalls

In this network, there are two routers that connect to different external networks. Each router then connects to a firewall in order to reach the internal network. And what happens in this network setup is that traffic that enters the network through router 2 is then handled by firewall 2. However, outgoing traffic is handled by firewall 1. Most firewalls are configured to observe traffic in the outgoing direction and then automatically allow through the incoming packets belonging to the same TCP session. So firewall 1 is looking out for the return traffic matching the earlier outgoing traffic, but that traffic never shows up at firewall 1. Firewall 2, on the other hand, doesn’t know that incoming packets match earlier outgoing packets, as it never saw those outgoing packets, and thus blocks those incoming packets. As a result, all destinations that are routed asymmetrically are unreachable.

The solution to this problem is to adjust the placement of the firewalls or internal routing such that traffic in both directions flows through the same firewall, even if incoming traffic enters the network through a different router than the router that handled the matching outgoing traffic. This is shown in figure 5.

Figure 5. Asymmetric routing is hidden from the firewalls

It’s important to realize that if multiple paths are used at the same time, it is not possible to make BGP always select symmetric paths. The only BGP configuration that avoids asymmetric paths is a primary/backup configuration, where under normal circumstances only the primary path is used and the backup path doesn’t handle any traffic at all. See our blog post on BGP communities for an example of how to set up a BGP primary/backup configuration. But simply placing firewalls closer to the systems they protect rather than at the edge of the network will usually avoid problems related to asymmetric routing.