Migrating to BGP

Here on the Noction blog we’ve covered many aspects of using BGP to connect your network to the internet. However, often implementing BGP isn’t as simple as coming up with the right configuration and then calling it a day. One of the trickiest aspects of BGP is migrating from a non-BGP setup to a BGP setup. That’s the topic we’ll be covering today.

The most common way to connect to the internet without BGP is to point a default route towards your ISP, and the ISP pointing a static route for your addresses towards your router. Most of the time, non-BGP customers get address space from their ISP. ISPs get large “provider aggregatable” (PA) address blocks from the Regional Internet Registry that serves the area (ARIN in North America) for this purpose. Alternatively, some customers have their own “provider independent” (PI) address space. If the customer doesn’t run BGP themselves, the ISP injects the customer’s PI block in BGP so it’s visible to the rest of the internet.

A BGP migration is simple when you, the customer, obtain new address space from the RIR, either PI or PA space. You can then setup BGP sessions to one or more ISPs and possibly peers over an internet exchange or private peering, and then test whether everything works as intended. After testing, you’ll have to renumber from the address space you were previously using to the new address block. The disadvantage of this type of migration is that it involves renumbering, which is time consuming and can be disrupting. But the upside is that you can simply add the new BGP configuration while your ISP maintains the static route for your existing address block, so adding BGP can happen without impact to network services.

It is of course also possible to advertise your new address block over BGP to a new ISP while the existing addresses are reachable through an existing ISP. In this situation, it’s important to think about how outgoing traffic will flow. If packets with a source address from the new address block are routed over the old ISP, or packets with a source address from the old address block over the new ISP, it’s likely that those packets will be filtered. Without further action, at least—it’s usually possible to ask an ISP to accept packets with “wrong” source addresses as long as you are the legitimate holder of those addresses.

Alternatively, if you have two routers, during the migration period, you can have each router only talk to one ISP, as shown in the following figure:

Router 1 handles the blue addresses routed through ISP A and router 2 the red addresses routed through ISP B. The blue hosts use blue addresses and have a default route pointed to router 1. Router 1 receives incoming packets towards these addresses from ISP A, and the blue hosts send outgoing packets towards router 1 which then forwards them over the link to ISP A. Similarly, router 2 handles the link to ISP B, and the hosts that have been renumbered in the new red address space point a default route to router 2 so their outgoing traffic flows through ISP B.

Once all hosts have been renumbered, router 1 can be reconfigured to announce the new address space towards ISP A. The default gateway address can then be made to be a VRRP address that is configured on both routers, as per the next figure:

In the case where you already have your own PI or PA block, this has the advantage that you don’t have to renumber. The disadvantage is that you’ll have to migrate addresses that are in production and risk a service interruption.

In this case, it’s best to find some address space you can use to test your BGP setup. If you have an IPv4 /24 or an IPv6 /48 that you’re not currently using for production traffic, you have announce that prefix over BGP and see if your BGP configuration works as intended. A good way to do this is using the various BGP looking glasses and traceroute servers that are available around the internet. However, if the test prefix overlaps with a larger address block (shorter prefix) that is currently advertised in BGP by a service provider, make sure you’re seeing your own BGP announcement rather than receive incoming packets through the larger advertisement by your ISP. Again, it can be helpful to test this using a new ISP. You should then see the new ISP and your link to that ISP in traceroutes.

If you don’t have at least a /24 or a /48, you can test with a smaller block, but this won’t give you reachability to the entire internet. However, unless your ISP(s) filter your test prefixes, at least you can test towards them. Or ask to borrow a prefix from an ISP for testing.

Once you’re satisfied that your BGP configuration is working, it’s time for the tricky parts: you’ll have to start advertising your prefix yourself, your ISP has to remove their static route for your address block and your ISP has to stop advertising the prefix. Obviously, you’ll want to do this during a time when relatively few users are using the network.

If currently ISP A is advertising your prefix to the world, and you start advertising that same prefix to ISP A, ISP A’s own advertisement will overrule yours, so it’s very hard to be sure that your advertisement will be propagated as intended by ISP A when they stop advertising it. Even if ISP A sees your prefix as advertised by you under your AS number in their BGP tables, that doesn’t guarantee success: it’s possible that ISP A’s filters or ISP A’s BGP neighbor’s filters allow your prefix when it’s advertised under ISP A’s AS number, but not when it’s advertised under your AS number. And you probably don’t want to find that out the hard way once ISP A stops advertising your prefix under their AS number.

There are two ways to work around this. The first is by advertising your prefix to a new ISP (ISP B) in addition to your existing ISP (ISP A). The new ISP never advertised your prefix under their AS number, so the filters should be correct. You’ll probably see your prefix showing up with your own AS number and ISP B’s AS number in the AS path in at least some looking glasses.

The other workaround is to split your prefix in half and advertise both sub-prefixes in addition to the full prefix. So if you have 172.16.0.0/16, you advertise 172.16.0.0/16, 172.16.0.0/17 and 172.16.128.0/17. ISP A’s advertisement of the /16 will overrule yours through ISP A, so it won’t show up. But as long as ISP A doesn’t advertise the /17s, ISP A will propagate your advertisement of those /17s to the rest of the world. You can check this in looking glasses to make sure there are no issues.

When you start announcing your address space yourself, this will propagate through most of the internet very quickly. However, it never hurts to give BGP ten minutes or so to settle. Once you advertise the prefix yourself, your ISP can remove their static route towards you. Then ask your ISP to stop announcing your prefix. With your announcement going out over a second ISP or with your prefix announced as two subprefixes, it’s very likely that the change doesn’t create any impact at all, or only leads to a few lost packets here and there due to minimal path hunting as the shorter AS path from ISP A is replaced by a slightly longer AS path through ISP B.

However, it’s possible that bad things happen, so make sure your ISP can restore the static route and resume advertising your prefix immediately when issues arise.

If you split your prefix into subprefixes, it’s best to remove those subprefixes at some point when you’re satisfied everything is stable. However, removing longer prefixes always leads to path hunting, so be prepared for a minute or two of instability when you do this.

With the above in mind, have a safe migration.