On August 8th, 2014 some ISPs experienced a phenomenon called the “512k Day”. The global BGP routing table, which consists of the global Internet routes and reflects how ISPs reach destination IPs, stretched to more than 512,000 records. Routers with the high-speed memory called Ternary Content Addressable Memory (TCAM) limited to 512,000 entries, were unable to store more than 512k routes. As a result, the affected routers crashed due to a higher CPU usage as they switched to slow software switching. This caused network outages around the globe as packets were lost on their way to a destination. The workaround from Cisco suggested using route filtering and a default route that helped decrease the number of routes in the affected devices. Prefix lists could be used as an alternative to access lists in many BGP route-filtering commands[1
Now, after almost 5 years, we might encounter a similar problem. The size of the global Internet routing table is reaching 768k routes so the possibility of the TCAM resource exhaustion at 768k routes is coming. Although many legacy routers with limited TCAM size have been replaced already, it is possible that there are still many devices remaining that may encounter the TCAM issues. The good news is that the TCAM size for IPV4 routes can be modified for some network equipment. For example, to increase TCAM on 3BXL modules for Cisco Catalyst 6500 and 7600 series routers and switches you can enter the command below. Afterward, save the configuration and reboot the device .
7600(config)# mls cef maximum-routes ip 1000
The command does not increase the overall size of the FIB TCAM, but it reduces the number of routing entries that are allocated to the IPv6 in order to increase the amount of TCAM space for IPv4. Maximum routes are set to 1024000 (1000k or 1000 * 1024 ) and the total number of available MPLS labels, IPv6 routes, and IPv4 multicast routes is reduced to only 8,000. See Picture 1.
Picture 1: Overall Size of the TCAM FIB
Picture 2 depicts a new TCAM allocation after the reboot. These new settings will help your network avoid any performance degradation, routing instability, or impact on availability. In other words, the proposed solution will help you deal with TCAM overflow at the 768k boundary.
Picture 2: New TCAM Allocation After Reboot
|Note: Cisco strongly recommends to check the number of MPLS, IPv6, and Multicast routes before you increase the allocation for IPv4 routes. Enter the show mls cef summary command in order to verify the total amount of routes per protocol (Picture 3).
Picture 3: Checking the Total Number of Routes per Protocol
So far, we have explained the effects of TCAM exhaustion and discussed a workaround that reduces the size of IPv6 TCAM in favor of IPv4 TCAM. The most common question, however, is when the global BGP table size will reach the 768,000 routes. The answer can be provided by a Twitter bot that tweets the size of IPv4 BGP table 4 times a day. The bot reported that the global Internet routing table had passed 768,000 routes on April 17th, 2019 (Picture 4). The prefixes smaller than /24 were filtered.
Picture 4: The Global BGP table Exceeds 768k Routes on Apr 17th, 2019
However, the Classless Inter-Domain Routing (CIDR) that also provides statistics on the global Internet routing table, reported that the global Internet routing table had passed 768,000 routes on March 5th, 2019. This illustrates an important principle in BGP, that there is no single authoritative view of the Internet’s inter-domain routing table – all views are in fact relative to the perspective of each BGP speaker .
Picture 5: The BGP table Exceeded 768k Routes on Match 5th, 2019
Another interesting factor that raises ambiguity in when and if the 768k boundary has been reached is the interpretation of the “k” value. Some sources suggest that “k” should be interpreted as 1024 for TCAM size. This gives us the value of 786432 global Internet routes that have not been reached yet.
You can simply check an actual size of the global Internet routing table connecting to any of the public route servers from the list. Open a terminal and run the command below. The username is ‘rviews’ with the password set to ‘rviews’.
$ telnet route-server.ip.att.net
email@example.com> show route summary
Picture 6: Checking the Size of the Global Internet Routing Table from AT&T Looking Glass Service
There are 735,386 active IPv4 BGP routes and 64,665 active IPv6 BGP routes as per AT&T Looking glass service (Picture 6).
Your networking equipment should be able to manage the ongoing growth of the Internet routing table. If your router might be affected by the 768k limit, you have a number of options here. For instance, you can reconfigure your router to decrease the number of TCAM entries allocated for IPv6 routes in favor of IPv4 routes before your table hits 768k or to filter some prefixes. An upgrade or a replacement of your legacy network equipment can also be an effective solution in order to cope with the outgoing growth of the table.