IRP Installation and Configuration Guide

1 Introduction

1.1 How to contact support

• Email us at support@noction.com
• Open a support request at https://helpdesk.noction.com

1.2 What is IRP

1.2.1 IRP Features

1.2.2 IRP Components

Figure 1.2.1 IRP Components

---

1.2.3 IRP Technical Requirements

1.2.3.1 Hardware requirements

In production, a dedicated server for each IRP instance is strongly recommended. The system can also be deployed on a Virtual Machine with matching specifications, provided that this is hardware- or paravirtualization (Xen, KVM, VMware). Os-level virtualization (OpenVZ/Virtuozzo or similar) is not supported.

1. CPU
   • Recommended Intel® Xeon® Processor E3/E5 family, for example:
     • 1x Intel® Xeon® Processor E3 family for up to 20 Gbps traffic;
     • 1x Intel® Xeon® Processor E5 family for 40 Gbps or more traffic.
2. RAM
   • if providing sFlow/NetFlow data at least 16 GB, recommended - 32 GB;
   • if providing raw traffic data by port mirroring:
     • minimum 16 GB for up to 10 Gbps traffic;
     • minimum 32 GB for 40 Gbps traffic.
     • Additional RAM would be required to maintain large amount of BGP & BMP sessions (for example, Bgpd occupies about 10G of RAM for 16 full view BGP sessions; the estimation may change due to growth of world’s BGP table and new IRP features).
3. HDD
   • At least 160GB of storage.
   • SAS disks are recommended (SSDs are required only for 40Gbps+ networks);
   • HDD partitioning:
     – LVM is recommended;
     – At least 100GB disk space usable for /var or separate partition;
     – At least 10GB disk space usable for /tmp or separate partition. This is required for big mysql tables manipulation. More disk space might be required under heavy workload.
4. NIC
   • if providing sFlow/NetFlow data - at least 1 x 1000Mbps NIC while two NICs are recommended (one will be dedicated to management purposes).
   • if providing raw traffic data by port mirroring - additional 10G interfaces are required for each of the configured SPAN ports (Myricom 10G network cards with Sniffer10G license are recommended to be used for high pps networks). When configuring multiple SPAN ports the same number of additional CPU cores are needed to analyze traffic.

In very large networks carrying hundreds or more of Gbps of traffic and in IRP configurations with very aggressive optimization settings configurations with 2x or 4x CPUs are recommended. The IRP servers in these cases should also allocate double the recommended RAM and use SSD storage.

A different supported OS can be installed on customer request.

1.2.3.2 Software requirements

IRP has a dependency on MySQL/MariaDB server and it expects the latest version from official OS repositories. In case the DBMS has been installed from a different repository it is strongly advised that the database instance and its configuration is purged before proceeding with IRP installation.
IRP requires root access to local database instance during first installation. In case the root access can’t be given, use the statements below to grant all necessary privileges to the ’irp’ user and database.:
GRANT ALL ON $dbdbname.* TO ’$dbusername’@’$dbourhost’ IDENTIFIED BY ’$dbpassword’ WITH GRANT OPTION
GRANT SELECT ON $dbdbname_fe.* TO ’$dbusername_fe’@’$dbourhost’ IDENTIFIED BY ’$dbpassword_fe’
where $dbdbname (4.1.1↓), $dbusername (4.1.1↓), $dbpassword (4.1.1↓), $dbourhost (4.1.1↓) are the corresponding parameters from /etc/noction/db.global.conf
$dbdbname_fe and $dbusername_fe from /etc/noction/db.frontend.conf

IRP’s Frontend is built on PHP 5.6 and Symfony 2.8 framework. When they are not part of mainstream OS they are retrieved from Noction’s repositories.

1.2.3.3 Network-related information and configuration

• Ownership of the AS for the network where IRP is deployed,
• BGP protocol is used for routing and,
• Network is multi-homed.

1. Prepare a network diagram with all the horizontal (own) as well as upstream (providers) and downstream (customers) routers included. Compare if your network topology is logically similar to one or more of the samples listed in section Collector Configuration↓ for example Flow export configuration ↓.
2. Identify the list of prefixes announced by your AS that must be analyzed and optimized by IRP.
3. Review the output of commands below (or similar) from all Edge Routers:
   • ‘sh ip bgp summary‘
   • ‘sh ip bgp neighbor [neighboor-address] received-routes‘
   • ‘sh ru‘ (or similar)
   The settings relating to BGP configuration, prefixes announced by your ASN, the route maps, routing policies, access control list, sFlow/NetFlow and related interfaces configurations are used to setup similar IRP settings or to determine what settings do not conflict with existing network policies.
   
4. Provide traffic data by:
   1. sFlow, NetFlow (v1, 5, 9) or jFlow and send it to the main server IP. Make sure the IRP server gets both inbound and outbound traffic info.
      Egress flow accounting should be enabled on the provider links, or, if this is not technically possible, ingress flow accounting should be enabled on all the interfaces facing the internal network.
      NetFlow is most suitable for high traffic volumes, or in the case of a sophisticated network infrastructure, where port mirroring is not technically possible.
      Recommended sampling rates:
      1. For traffic up to 1Gbps: 1024
      2. For traffic up to 10Gbps: 2048
   2. Or: configure port mirroring (a partial traffic copy will suffice). In this case, additional network interfaces on the server will be required - one for each mirrored port.
   See also: Collector Configuration↓
5. Setup Policy Based Routing (PBR) for IRP active probing.
   • Apart from the main server IP, please add an additional alias IP for each provider and configure PBR for traffic originating from each of these IPs to be routed over different providers.
   • No route maps should be enforced for the main server IP, traffic originating from it should pass the routers using the default routes.
   • Define Provider↔PBR IP routing map
   In specific complex scenarios, traffic from the IRP server should pass multiple routers before getting to the provider. If a separate probing Vlan cannot be configured across all routers, GRE tunnels from IRP to the Edge routers should be configured. The tunnels are mainly used to prevent additional overhead from route maps configured on the whole IRP⟷Edge routers path.

If network has Flowspec capabilities then alternatively Flowspec policies can be used instead of PBR. Refer for example Flowspec policies↓, global.flowspec.pbr↓.

1. Configure and provide SNMP for each provider link, and provide the following information:
   • SNMP interface name (or ifIndex)
   • SNMP IP (usually the router IP)
   • SNMP community
   This information is required for the report generation, Commit Control decision-making and prevention of overloading a specific provider with an excessive number of improvements.

The above is applicable in case of SNMP v2c. If SNMP v3 is used further details will be required depending on security services used.

1. To setup cost related settings as well as the Commit Control mechanism, provide the maximum allowed interface throughput for each provider link as well as the cost per Mbps for each provider.

1.2.4 IRP Operating modes

1.2.4.1 Non-intrusive mode

1.2.4.2 Intrusive mode

1.2.4.3 Going Intrusive

1. The optimizing component of IRP (Core) is taken offline and existing improvements are purged. The Core being offline guarantees IRP will not automatically insert new improvements into its Current Improvement table and hinder the Go Intrusive process.
   

   root@server ~ $ service core stop
   root@server ~ $ mysql irp -e ’delete from improvements;’
   

   Stop IRP Core and purge existing improvements
2. Enable Intrusive Mode and adjust IRP Core and BGPd parameters as follows:
   

   root@server ~ $ nano /etc/noction/irp.conf
   global.nonintrusive_bgp = 0
   core.improvements.max = 100
   bgpd.improvements.remove.next_hop_eq = 0
   bgpd.improvements.remove.withdraw = 0
   

   Switch to Intrusive Mode and adjust IRP Core and BGPd parameters
3. Improvements towards test networks are introduced manually so that client’s traffic is not affected. The improvements are chosen so that they cover all client’s providers. Any public networks could be used for test purposes. Just keep in mind that preferably your network shouldn’t have traffic with chosen test network in order to do not re-route the real traffic. Use the template below in order to insert the test improvements:
   

   mysql> insert into improvements 
     (ni_bgp, prefix, peer_new, ipv6, asn) 
     values 
       (0, ’10.10.10.0/24’, 1, 0, 48232),
       (0, ’10.10.11.0/24’, 2, 0, 48232),
       (0, ’10.10.12.0/24’, 3, 0, 48232);
   

   Inserting test improvements
4. Make sure that ’route-reflector-client’ is set for IRP BGP session.
5. Make sure that ’next-hop-self’ is not configured for IRP BGP session.
6. On iBGP sessions (between edge routers, route-reflectors; except session with IRP) where ’next-hop-self’ is configured, the following route-map should be applied:
   

   route-map RM-NHS
   set ip next-hop peer-address
   neighbor X.X.X.X route-map out RM-NHS
   

   Remove next-hop-self route-map (RM-NHS) example
   where X.X.X.X is the iBGP neighbor
   route-map contents should be integrated into existing route-map in case other route-map already configured on the iBGP session.
7. Use the commands below to restart IRP BGPd to use actual IRP configuration and establish BGP session(s) and verify if BGP updates are being announced:
   

   root@server ~ $ service bgpd restart
   root@server ~ $ tail -f /var/log/irp/bgpd.conf
   ​
   Wait for the following lines for each BGP session:
   NOTICE: Adding peer X
   NOTICE: BGP session established X
   INFO: N update(s) were sent to peer X
   

   Restart IRP BGPd
   where X is the router name and N is the number of the updates sent towards the X router.
8. Verify if IRP BGP announcements are properly propagated across all the network. Run the following commands on each router (the commands vary depends on the router brand):
   

   show ip bgp 10.10.10.1
   show ip bgp 10.10.11.1
   show ip bgp 10.10.12.1
   

   Show BGP information for specified IP address or prefix
   Analyze the output from all the routers. If the IRP BGP announcements are properly propagated, you should see /25 (refer to 4.1.4.31↓) announcements and the next-hop for each announcement should be the improved provider’s next-hop:
   10.10.10.1 - provider 1 next-hop
   10.10.11.1 - provider 2 next-hop
   10.10.12.1 - provider 3 next-hop
   (refer to 4.1.12.25↓, 4.1.12.33↓, 4.1.15.3↓).
   
   Run the following commands in order to check if IRP improvements are announced and applied:
   

   root@server ~ $ traceroute -nn 10.10.10.1
   root@server ~ $ traceroute -nn 10.10.11.1
   root@server ~ $ traceroute -nn 10.10.12.1
   

   Traceroute destination networks
   Again, you should see corresponding providers’ next-hops in the traces.
9. If the tests are successful perform the steps below:
   1. Delete test improvements
      

      root@server ~ $ mysql -e "delete from improvements where prefix like ’10.10.1%’;"
      

      Delete test improvements
   2. Configure at most 100 improvements and revert BGPd configuration
      

      root@server ~ $ nano /etc/noction/irp.conf
      core.improvements.max = 100
      bgpd.improvements.remove.next_hop_eq = 1
      bgpd.improvements.remove.withdraw = 1
      

      Configure the maximum improvements limit and revert BGPd configuration
   3. Restart IRP Core and BGPd
      

      root@server ~ $ service core restart
      root@server ~ $ service bgpd restart
      

      Restart IRP Core and BGPd
10. If everything goes well, after 1-2 hours the maximum number of improvements announced is increased to 1000 and after 24 hour to 10000.

1. Delete test improvements
   

   root@server ~ $ mysql -e "delete from improvements where prefix like ’10.10.1%’;"
   

   Delete test improvements
2. Switch the system to non-intrusive mode
   

   root@server ~ $ nano /etc/noction/irp.conf
   global.nonintrusive_bgp = 1
   

   Switch the system to non-intrusive mode
3. Restart IRP Core and BGPd
   

   root@server ~ $ service core restart
   root@server ~ $ service bgpd restart
   

   Restart IRP Core and BGPd

1.2.5 BGP Monitoring

1.2.5.1 Internal monitor

• the provider will be marked as FAILED,
• all the improvements towards this provider will be withdrawn from the routing tables to avoid creating black holes,
• new improvements towards this providers will not be made.

Internal monitors for Internet Exchange peering partners are not initialized until there is an improvement made towards it. When running in non-intrusive mode internal monitors for IX peers are not initialized at all.

1.2.5.2 External monitor

• the provider will be marked as FAILED,
• all the improvements towards this provider will be withdrawn from the routing table,
• new improvements towards this providers will not be made.

Figure 1.2 System Dashboard

Starting with version 1.8.5, IRP requires at least the Internal Monitor to be configured. Otherwise, the system Frontend will return an error as shown below.

Figure 1.3 Error: Internal BGP Monitor not configured

1.2.5.3 BMP monitoring station

• BMP lists both active and inactive routes advertised by peers on an Internet Exchange. The additional information is used by IRP to evaluate and identify the best candidate peers at all times. Without BMP data IRP has knowledge about active routes only which only point to a single peer on the IX while all the alternatives are hidden.
• route changes even for inactive routes are visible via BMP. This allows IRP the opportunity to revisit previously made probes and improvements not only at predefined re-probing intervals but also when route changes are detected for both active and inactive routes.
• prefix monitors for IX improvements consume significant router CPU resources in order to service the SNMP requests traversing the router’s relevant OIDs. More so this information is at times inaccurate and vendor dependent. When BMP data is available IRP uses this routing data to determine if IX peers still advertise the routes and no longer makes the SNMP requests for those prefixes thus significantly reducing the CPU overhead especially on routers servicing very large IX.
• IRP reconstructs the AS Path for candidate providers in order to make accurate iBGP announcements of improvements. Unfortunately network configuration practices might cause some errors during reconstruction of AS Paths using traceroute. BMP data makes the reconstruction of AS Path redundant and more accurate as this BGP attribute can be retrieved from actual (inactive) routes received from neighbors.
• improvements can be re-visited on AS Path changes. Both new and old provider AS Path attributes are monitored via BMP for changes. When changes are detected IRP re-probes the prefix to ensure the network uses the best available route. Note that re-probing can be triggered on any AS Path changes or only on major ones - when AS Path traverses a different set of autonomous systems.

As per BMP RFC requirements IRP BMP monitoring station never attempts to establish BMP or any other connections with the monitored router leaving the full scope of decisions regarding when and if BMP data is communicated in network’s responsibility.

1.2.6 Outage detection

Several improvements-related reports need to indicate the original route for a specific prefix. This value is taken from the last probing results for this prefix, even if the results are outdated (but not older than 24h). Since the outage-affected prefixes are rerouted in bulk by as-pattern, in some cases the reports can show the same provider for both the old and the new route.

1.2.7 VIP Improvements

• top-volume-autonomous-systems monitoring and optimizing traffic to commercial partners that should have access to your networks via the best performing routes
• monitoring and optimizing traffic to your remote locations, operating as separate networks
• monitoring and optimizing traffic to AS, which are known for the frequent accessibility issues due to geographical or technical reasons

If a prefix is being announced from multiple Autonomous Systems, you can see different ASNs in Current improvements↓ report in the prefixes translated from ASN

1.2.8 Retry Probing

• Performance and Cost improvements
  • An old provider has been removed from the system configuration.
    Example: “Old provider and performance metrics not known. New packet loss 55%, avg rtt 105 ms.”
• Commit Control improvements
  • An old provider’s has been removed from the system configuration.
    Example: “Previous provider not known. Rerouted 1 Mbps to Peer5[5] (250 Mbps, 50%)”
  • An old provider’s bandwidth statistics are not available.
    Example: “Rerouted 6 Mbps from Peer1[1] to Peer5[5] (250 Mbps, 50%)”
  • A new provider’s bandwidth statistics are not available.
    Example: “Rerouted 6 Mbps from Peer1[1] (250 Mbps, 50%) to Peer5[5]”
  • The old and new providers’ bandwidth statistics are not available.
    Example: “Rerouted 6 Mbps from Peer1[1] to Peer5[5]”

Commit control improvements are reconfirmed based on their average bandwidth usage (and not on current bandwidth usage). This way if performance characteristics allow it, even when current bandwidth usage is low but the average is still relevant, the improvement is preserved thus anticipating network usage cycles and reducing number of route changes.

• The Commit Control feature has been disabled.
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (Commit Control is disabled)“
• The low prefix traffic volume is less than the configured bandwidth limits (core.commit_control.agg_bw_min↓).
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (low traffic volume, irrelevant for the Commit Control algorithm)”
• The system has been switched from the Cost mode to the Performance mode (applied for cost improvements only).
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (Performance Improvements mode)”
• A prefix has been added to the ignored networks/ASN list.
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (added to ignored networks/ASN)”
• The improvement’s performance metrics are not the best ones anymore.
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (Performance Improvement not actual anymore)”
• The maximum number of improvements limits (core.improvements.max↓, core.improvements.max_ipv6↓) are exceeded.
  Example: “Prefix 1.0.2.0/24 withdrawn from Improvements (no more available Improvement slots)”

1.2.9 Routing Policies

Policies can be configured for networks (ASN) or aggregate prefixes. These are unpacked into existing specific sub-prefixes from the routing table before being processed by IRP components.

In version 3.9 IRP added support for policies by Country. Note that IRP maps individual prefixes to a country and does not use a transitive inference based on AS records. These prefix mappings allow IRP to more accurately work with large transcontinental AS.

Note that after upgrade all existing policies are assigned (implicitly) the lowest default priority of 0.

Routing policies are designed to control outgoing paths for destination networks that are outside your infrastructure. This feature should not be used to manipulate your infrastructure network behavior.

Avoid setting up of policies that point to same prefix. When improvement by aggregate is enabled, multiple prefixes can point to the same aggregate and this can cause unpredictable behaviour.

The use case of cascading is to apply a policy to a remote AS that transits a few other AS. Still, a cascading policy can cover a huge number of down-streams. This number is parameterized and can be set to values that best fit customer’s needs. Refer for example 4.1.4.15↓.

Do not alter the /etc/noction/policies.conf file manually because any modifications will be overwritten by the system. Any changes can only be performed from the Configuration -> Routing Policies section in the system Frontend.

The rule will be ignored by the platform if the Routing Policy contains a syntax/logical error.

1.2.10 Support for Centralized Route Reflectors

Figure 1.4 Support for Centralized Route Reflectors

• The Next-Hop-Self option should not be used. A direct iBGP session is required between IRP and each of the divergence points (Reflector1, Edge2) in case it is enabled. This restriction is not applicable between reflectors and core/distribution layers.
• Next-Hop addresses should be accessible (exist in the routing table) where next-hop-self is not applied (Either static routes or IGP is used).
• An Internal Monitor should be configured to retrieve the eBGP session state from the device where the corresponding eBGP session is terminated. For example, ISP1 should be monitored on Reflector1, ISP2 on Edge1, and ISP3 and ISP4 on Edge2.
• Injecting routes to reflector(s) can cause temporary routing loops.

1.2.11 Support for Internet Exchanges

Figure 1.5 IRP configuration in a multi-homed network connected to transit providers as well as and Internet Exchange

1.2.12 Optimization for multiple Routing Domains

Figure 1.6 City wide network

Figure 1.7 Regional network

Figure 1.8 Intercontinental network

Figure 1.9 Multiple routing domains

• Multiple locations belonging to the same network (AS) represented in the diagram by POP SJC, POP SFO and POP LAX (of course, more than 3 routing domains are supported).
• The locations are distinguished by the different Routing Domains within which they operate (depicted by RD SJC, RD SFO, and RD LAX)
• The Routing Domains are managed by edge routers belonging to different locations
• Nearby locations that process routing data differently should be split into different Routing Domains, even if they have the same upstream providers. In the diagram above RD SFO and RD SFO’ are depicted as part of a single Routing Domain. A decision to split or keep in the same routing domain should be made based on exact knowledge on how routing data is processed.
• Inter-datacenter loop interconnects the different locations (depicted by idc1, idc2 and idc3 segments)
• Data flows between locations take only the short path (in the example POP SJC can be reached from POP SFO via idc2 path (short) or idc3 + idc1 path (long))
• Each Routing Domain has different providers and different preferred routes to reach a specific destination (a1, b1, c1)
• A single IRP instance collects statistics about traffic (Irpflowd only), probes available destinations and makes improvements towards specific prefixes/networks on the Internet.
• IRP assumes RTT of zero and unlimited capacity to route traffic within a Routing Domain
• IRP assumes that Sites are not physically too far away. It is ok to have different sites in the same city or region as at this scale inter-datacenter links have predictable characteristics. When taking intercontinental links into consideration this is quite probably not the case.
• Distances between sites (idc1, idc2, idc3 delays) are measured in advance and specified in IRP’s configuration.

• private connections,
• L2 links with guaranteed service,
• MPLS links

VPNs via public Internet could be used with Multi Routing Domain feature but is a suboptimal choice. Under such conditions IRP MUST be prevented from making Global Improvements. This way IRP will do only local optimizations in each Routing Domain and will operate similarly to multiple IRP instances (while probing excessively because it probes destinations via remote locations too).

• IRP does not take measurements of inter-datacenter link delays (idc1, idc2 and idc3). This values are configurable.
• IRP does not monitor if inter-datacenter links are operating normally. In case such a link is broken it is expected IRP to loose BGP connectivity with routing domain routers and this will cause IRP improvements to be withdrawn till the link is restored.
• IRP does not try to detect if the traffic is following long or short paths on the inter-datacenter links. In the image above traffic from RD SJC can follow path idc1 (short) or idc2+idc3 (long). IRP always assumes the short path is being followed internally.
• IRP does not take measurements of inter-datacenter link capacity and current bandwidth usage. At this stage IRP assumes there is enough inter-datacenter link capacity to also carry the (few) global improvements. Also, IRP tries to minimize usage of inter-datacenter links.

rda:rdb:rtt

global.rd_rtt = 3:42:20 42:1:17 1:3:35

Round trip time between one routing domain and another is calculated by executing PING towards edge routers and taking average integer value:

$ ping X -c 10 -q 
PING X (X) 56(84) bytes of data. 
--- X ping statistics — 
10 packets transmitted, 10 received, 0% packet loss, time 9085ms 
rtt min/avg/max/mdev = 40.881/41.130/41.308/0.172 ms 

SPAN cannot be used because it does not carry attributes to distinguish traffic between different providers

Refer your router capabilities in order to produce the correct route map. The route map MUST be integrated into existing route maps. It is not sufficient to simply append them.

neighbor <neighbor-from-another-RD> send-community (should be configured for all iBGP sessions)
​
ip community-list standard CL-IRP permit 65535:1
route-map RM-IRP-RD deny 10
  match community CL-IRP
route-map RM-IRP-RD permit 20
​
router bgp AS
  neighbor <neighbor-from-another-RD> route-map RM-IRP-RD out

Refer your router capabilities in order to produce the correct route map. The route map MUST be integrated into existing route maps. It is not sufficient to simply append them.

policy-options{
	policy-statement IRP-CL {
		term 0 {
	        from {
	         	protocol bgp;
	         	community IRP-RD;
	        }
	        then reject;
	    }
	    term 1 {
	        then accept;
	    }
	}
	community IRP-RD members 65535:1;
}
protocols {
	bgp {
    	group ebgp {
        	type external;
        	neighbor 10.0.0.1 {
            	export IRP-CL;
        	}
    	}
    }
}

• (c2+idc1 - margin) is better than best local alternative a1 in RD SJC
• (c2+idc3 - margin) is better than best local alternative b2 in RD SFO

• a1, b2 and c2 represent roundtrip times determined by IRP during probing of a destination.
• idc values are configurable and are set as one entry of global.rd_rtt↓ parameter.
• margin is given by core.global.worst_ms↓.

Global improvements can degrade performance in some routing domains. If performance for some routing domains degrades, IRP announcements for the global improvement also carry designated BGP community attributes set by rd.X.community_worsening↓.

1.2.13 Improvements weight

1.2.14 Notifications and events

• SMS
• Email
• Slack (via Webhook)
• SNMP Traps

Only events for valid subscriptions using correctly configured channels will be delivered.

Events

• Abnormal correction: irpflowd
• Abnormal correction: irpspand
• Inbound traffic low: SPAN
• Inbound traffic low: Flow
• Inbound traffic normal: Flow
• Inbound traffic normal: SPAN
• Outbound traffic low: SPAN
• Outbound traffic low: Flow
• Outbound traffic normal: Flow
• Outbound traffic normal: SPAN

• Commit Control overload by X Mbps
• Commit Control overload by X%
• Commit Control provider X overloaded by Y Mbps
• Commit Control provider X overloaded by Y%

• Configuration Invalid: BGPd
• Configuration Invalid: Core
• Configuration Invalid: Explorer
• Configuration Invalid: irpapid
• Configuration Invalid: irpflowd
• Configuration Invalid: irpspand
• Configuration Invalid: irpstatd
• Configuration Ok: BGPd
• Configuration Ok: Core
• Configuration Ok: Explorer
• Configuration Ok: irpapid
• Configuration Ok: irpflowd
• Configuration Ok: irpspand
• Configuration Ok: irpstatd

• Congestion or Outage
• Outage: Confirmed and rerouted

• Failed PBR (IPv6) check for provider
• Failed PBR (IPv4) check for provider
• Successful PBR (IPv4) check for provider
• Successful PBR (IPv6) check for provider
• Explorer performance low
• High number of VIP prefixes degrades IRP performance

• ExternalMonitor (IPv4) Failed status for a provider. All improvements towards the provider will be withdrawn.
• ExternalMonitor (IPv4) OK status for a provider. All improvements towards the provider will be announced.
• ExternalMonitor (IPv6) Failed status for a provider. All improvements towards the provider will be withdrawn.
• ExternalMonitor (IPv6) OK status for a provider. All improvements towards the provider will be announced.
• InternalMonitor (IPv4) Failed status for a provider. All improvements towards the provider will be withdrawn.
• InternalMonitor (IPv4) OK status for a provider. All improvements towards the provider will be announced.
• InternalMonitor (IPv6) Failed status for a provider. All improvements towards the provider will be withdrawn.
• InternalMonitor (IPv6) OK status for a provider. All improvements towards the provider will be announced.

• Provider SNMP stats down: X
• Provider SNMP stats up: X

• IRP BGP session disconnected
• IRP BGP session established

• Excessive packet latency for prefix
• Excessive packet loss for prefix
• Improvements spike
• Low rate of announced IPv4 improvements
• Low rate of announced IPv6 improvements
• New improvement

• Service started: BGPd
• Service started: Core
• Service started: Explorer
• Service started: irpapid
• Service started: irpflowd
• Service started: irpspand
• Service started: irpstatd
• Service stopped: BGPd
• Service stopped: Core
• Service stopped: Explorer
• Service stopped: irpapid
• Service stopped: irpflowd
• Service stopped: irpspand
• Service stopped: irpstatd

SNMP Traps

1.2.15 IRP API

1.2.16 IRP Failover

Overview

A failover license is required for the second node. Check with Noction’s sales team for details.

• slave node running same version of IRP as the master node,
• MySQL Multi-Master replication of ’irp’ database,
• announcement of the replicated improvements with different LocalPref and/or communities by both nodes,
• monitoring by slave node of BGP announcements originating from master node based on higher precedence of master’s announced prefixes,
• activating/deactivating of slave IRP components in case of failure or resumed work by master,
• syncing master configuration to slave node.

For exact details about IRP failover solution refer to configuration guides (2.6↓, 3.12.13.5↓), template files, and (if available) working IRP configurations. For example, some ’irp’ database tables are not replicated, ’mysql’ system database is replicated too, some IRP components are stopped.

IRP versions 3.5 and earlier do no offer failover capabilities for Inbound improvements. It is advised that in these versions only one of the IRP instances is configured to perform inbound optimization in order to avoid contradictory decisions. In case of a failure of this instance inbound improvements are withdrawn.

Figure 1.10 Failover high level overview

• two IRP nodes - Master and Slave,
• grayed-out components are in stand-by mode - services are stopped or operating in limited ways. For example, the Frontend detects that it runs on the slave node and prohibits any changes to configuration while still offering access to reports, graphs or dashboards.
• configuration changes are pushed by master to slave during synchronization. SSH is used to connect to the slave.
• MySQL Multi-Master replication is setup for ’irp’ database between master and slave nodes. Existing MySQL Multi-Master replication functionality is used.
• master IRP node is fully functional and collects statistics, queues for probing, probes and eventually makes Improvements. All the intermediate and final results are stored in MySQL and due to replication will make it into slave’s database as well.
• BGPd works on both master and slave IRP nodes. They make the same announcements with different LocalPref/communities.
• BGPd on slave node monitors the number of master announcements from the router (master announcements have higher priority than slave’s)
• Timers are used to prevent flapping of failover-failback.

Requirements

1. second server to install the slave,
2. MySQL Multi-Master replication for the irp database.

MySQL replication is not configured by default. Configuration of MySQL Multi-Master replication is a mandatory requirement for a failover IRP configuration. Failover setup, and specifically MySQL Multi-Master replication should follow a provided failover script. Only a subset of tables in irp database are replicated. Replication requires extra storage space, depending on the overall traffic and platform activity, for replication logs on both failover nodes.

1. a second set of BGP sessions will be established,
2. a second set of PBR IP addresses are required to assign to the slave node in order to perform probing,
3. a second set of improvements will be announced to the router,
4. a failover license for the slave node,
5. Key-based SSH authentication from master to slave is required. It is used to synchronize IRP configuration from master to slave,
6. MySQL Multi-Master replication of ’irp’ database,
7. IRP setup in Intrusive mode on master node.

In case IRP failover is setup in a multiple Routing Domain configuration and IRP instances are hosted by different RDs this must be specified in IRP configuration too. Refer Optimization for multiple Routing Domains↑, global.master_rd↓, global.slave_rd↓.

Failover

In order for this mechanism to work IRP needs to operate in Intrusive mode and master’s node announcements must have higher priority then the slave’s.

• master synchronizes its configuration to slave. This uses a SSH channel to sync configuration files from master to slave and process necessary services restart.
• MySQL Multi-Master replication is configured on relevant irp database tables so that the data is available immediately in case of emergency,
• components of IRP such as Core, Explorer, Irppushd are stopped or standing by on slave to prevent split-brain or duplicate probing and notifications,
• slave node runs BGPd and makes exactly the same announcements with a lower BGP LocalPref and/or other communities thus replicating Improvements too.

It is imperative that master’s LocalPref value is greater than slave’s value. This ensures that master’s announcements are preferred and enables slave to also observe them as part of monitoring.

Slave node only considers that master is down and takes over only if master’s Improvements are withdrawn from all edge routers in case of networks with multiple edge routers.

This is true only if LocalPref and/or communities assigned to slave node are preferred. If other most preferable announcements are sent by other network elements , no longer announcements from slave node will be best. This defeats the purpose of using IRP failover.

Failback

During failback it is recommended that both IRP nodes are monitored by network administrators to confirm the system is stable.

Recovery of failed node

Recovery speed is constrained by restoring replication of MySQL databases. On 1Gbps non-congested links replication for a full day of downtime takes approximately 30-45 minutes with 200-250Mbps network bandwidth utilization between the two IRP nodes. During this time the operational node continues running IRP services too.

Upgrades

It is imperative that master and slave nodes are not upgraded at the same time. Update one node first, give the system some time to stabilize and only after that update the second node.

1.2.17 Inbound optimization

• advises your edge routers to advertise different prefixes of your network with different counts of BGP prepends to each of your upstream providers;
• monitors inbound and outbound traffic on your network interfaces using standard protocols (SNMP, NetFlow, sFlow) to determine if there is need for action;
• continuously assesses network health and performance to ensure that when overloads are possible it knows good and reliable alternative routes to use;
• uses a proprietary inferring mechanism that takes into account the inertial nature of the Internet to react to inbound preference changes and thus dampens the urge to “act now” and destabilize the network;
• provides you with configuration, monitoring and visualization tools that allow you to cross-check and validate its actions.

1. Segmenting your network into prefixes that are controlled by IRP
2. Coordinating communication over BGP between IRP and routers
3. Setting network conditions for making Inbound Improvements
4. Reviewing Inbound optimization results

Starting with version 3.7 IRP has the capability to also monitor and improve transit routes. Refer Optimization of transiting traffic ↓ for details.

Figure 1.11 Unbalanced inbound/outbound peering

‘

Figure 1.12 Fluctuating inbound traffic

‘

1.2.18 Flowspec policies

Flowspec policies can be used only in conjunction with Flowspec capable routers.

As specified by BGP, when a session disconnects all the announcements received from that session are discarded. The same is generally true for Flowspec policies. Still, since some vendors recognize Flowspec policies but implement them using capabilities other than BGP a confirmation is needed whether on session disconnect the specific router model indeed removes all the underlying constructs and reverts to a known state.

• controlling bandwidth usage of your low priority traffic towards external services, for example throttling bandwidth usage originating on your backup systems towards off-premises services.
• anticipating inbound traffic towards your services and shaping bandwidth use in advance, for example anticipating low numbers of legitimate customers from Russia, China or India on your e-commerce services and setting high but controllable rate limits on packets originating in those networks.
• reacting on a packet flooding incident by dropping specific packets, for example dropping all packets targeting port 53.
• redirecting some traffic for scrutiny or cleansing, for example forwarding port 80 packets through an intelligent device capable of detecting RUDY, slow read or other low-bandwidth/amplification attacks.

• Source or destination IP address specified as either CIDR format prefix or direct IP address
• Traffic protocols, for example TCP, UDP or ICMP
• Source or destination TCP/UDP ports
• Throttle, drop and redirect actions.

It is important to note that IRP does not cross-validate Flowspec policies with improvements. While it is possible that for example a Flowspec redirect action pushes some traffic a different way to what an improvement advises, usually improvements cover many prefixes and while there will be a contradiction for one prefix there will be many other prefixes that IRP improves to compensate for these unitary abnormalities. It is recommended that Flowspec policies take precedence over improvements in order to benefit from this compensating nature of improvements.

1.2.19 Throttling excessive bandwidth use with Flowspec

In case thresholds for excessive bandwidth use are set to very aggressive levels IRP can create large numbers of Flowspec policies.

• configure excess threshold and throttling multipliers
• periodically determine current and average prefix bandwidth usage for this hour of the day
• verify if current usage exceeds the average by a larger factor then the threshold multiplier
• rate-limit excessive bandwidth usage prefixes at their average use times throttling multiplier.

1.2.20 Maintenance windows

Setting a maintenance window by router sets each of the providers on the router with a maintenance window of their own.

• a maintenance window is configured in advance and can be removed/revised at any time,
• a maintenance window sets details for single provider. If needed multiple maintenance windows can be setup and even overlapping maintenance windows are OK.
• IRP highlights maintenance windows in IRP’s Frontend sidebar so that it is easy to spot current maintenance window status.
• optionally IRP can preserve existing improvements so that once the maintenance window ends improvements are reimplemented. It is advised that this feature is used only when the maintenance window is very short (a few minutes).
• an unloading period can be setup. During unloading IRP actively re-routes outbound prefixes through other available providers. While IRP is able to make most of the unloading improvements fast consideration shall be given to the announcement rate limitations setup in BGPd in order for all the improvements to reach network routers in time for maintenance window starting time.
• a prepend time can be setup. This is only applicable if Inbound optimization is operational. If this time is setup then IRP will prepend configured inbound prefixes with the maximum allowed number of prepends through the provider link under maintenance in order to deflect inbound traffic towards other providers. Refer to Inbound optimization↑ for more details about Inbound optimization.

1.2.21 Optimization of transiting traffic

• The number of potential routes is very large and only some will/should be targeted for optimization. For this IRP uses filters by ASN/prefix allowing network administrators to configure what segments of the Internet to focus IRP’s attention to.
• The improvements are visible on the Internet and excessive route changes can be flagged by external monitoring services as flapping or route instability. IRP protects against this by rate limiting number of route changes through a specific provider.
• Once a route has been improved all its traffic might be diverted away from this network and no new statistics will be available to make further inferences. In such cases IRP reverts old transit improvements during network’s off-peak hour by either decreasing the number of advertised prepends or withdrawing the improvement altogether (configurable).
• Transit improvements apply to the same prefixes as do outbound improvements. The outbound improvements are withdrawn in order to avoid the risk of contradictory routing decisions.

• Potential blackholing of traffic when all alternative routes are withdrawn. IRP implements a protection against this by traversing all RIB-in entries for improved transit prefixes and confirming that the route is still being announced by other providers besides IRP. Still there can be a short time period between confirmations when all alternative routes have been withdrawn and IRP did not yet get a chance to re-confirm this. Attempting to reduce this time period by setting up a higher frequency of confirmations leads to increased load on the router and a tradeoff needs to be made for this. For example when the number of providers is quite large the probability that a route will be withdrawn through all of them is quite small and thus the frequency of confirmations can be reduced too.
• Additional CPU load on edge router(s) for servicing mandatory SNMP requests that check alternative route presence and BGP Best Path selection inside IRP.
• Working with missing BGP attributes that is not available over SNMP.
• Strict upper limits on the number of possible Inbound improvements are imposed by the trade-off required to reduce excessive router’s CPU load.

1.2.22 Circuit issues detection

Circuit issues detection feature is available for transit providers only.

• provider is marked with an issue badge and excluded from being considered a candidate for performance improvements,
• reprobing is performed for destination prefixes routed through affected provider,
• outbound improvements through affected provider are withdrawn,
• max prepends are announced for inbound and transit prefixes through affected provider,
• FlowSpec rules to induce drop of BGP session towards affected provider are added,

Note that this is only possible if FlowSpec is enabled and the network is capable of processing FlowSpec rules. Dropping the BGP session with the affected provider causes all outbound and inbound traffic through this provider to be re-routed through known good providers.

• affected provider is monitored to detect if the issue was temporary and if loss averages return to normal restore it to a good state.

1.3 IRP Optimization modes

1.3.1 Performance optimization

Figure 1.13 Performance optimization algorithm

1.3.2 Cost optimization

Figure 1.14 Cost optimization algorithm

Note that the diagram does not highlight higher cost cases. IRP operating in cost optimization mode can make improvements towards higher cost providers only when current routes suffer from loss.

1.3.3 Commit Control

IRP usually allows CC improvements when the candidate providers have better or equal loss to current route. This can be configured under core.commit_control.loss_override↓.

1.3.3.1 Flexible aggressiveness of Commit algorithm based on past overloads

Figure 1.15 More aggressive when actual exceeds schedule

1.3.3.2 Trigger commit improvements by collector

1.3.3.3 Commit Control improvements on disable and re-enable

• when Commit Control is disabled for a provider (peer.X.cc_disable↓ = 1), this Provider’s Commit Control improvements are deleted;
• when Commit Control is disabled globally (core.commit_control↓ = 0), ALL Commit Control improvements are deleted.

1.3.3.4 Provider load balancing

Figure 1.16 Provider load balancing

1.3.3.5 Commit control of aggregated groups

Figure 1.17 Commit control of aggregated groups

It is important to note that in order for this feature to be effective there must be providers configured in IRP that are not part of this group. This way when candidate improvements are considered there are alternative routes via those providers that the traffic can be rerouted to.

1.3.3.6 95th calculation modes

• Separate 95th for in/out: The 95th value for inbound and outbound traffic are independent and consequently bandwidth control for each is performed independently of each other. For this 95th calculation modes IRP monitors two different 95th for each inbound and outbound traffic levels.
• 95th from greater of in, out: At each time-point the greater of inbound or outbound bandwidth usage value is used to determine 95th.
• Greater of separate in/out 95th: 95th are determined separately for inbound and outbound traffic and the larger value is used to verify if commitments have been met.

1.3.3.7 Other commit control features

Figure 1.18 Sorted Commit Control improvements