2.13.1 Initial failover configuration

Prerequisites #

Before proceeding with failover configuration the following prerequisites must be met:

One IRP node is configured and fully functional. We will refer to this node as $IRPMASTER.
Second IRP node is installed with the same version of IRP as on $IRPMASTER. We will refer to this node as $IRPSLAVE.

Second IRP node MUST run the same operating system as $IRPMASTER.

When troubleshooting problems, besides checking matching IRP versions ensure the same versions of irp MySQL databases are installed on both failover nodes.

IRP services, MySQL and HTTP daemons are stopped on $IRPSLAVE node.
Network operator can SSH to both $IRPMASTER and $IRPSLAVE and subsequent commands are assumed to be run from a $IRPMASTER console.

$IRPMASTER and $IRPSLAVE must have different hostnames.

Configure communication channel from $IRPMASTER to $IRPSLAVE #

This channel is used during failover configuration and subsequently $IRPMASTER uses it to synchronize IRP configuration changes when these occur. It uses key-based authentication without a passphrase to allow automated logins on $IRPSLAVE by $IRPMASTER processes.

Adjust firewalls if any so that $IRPMASTER node can access $IRPSLAVE via SSH.

Generate SSH keys pair WITHOUT passphrase:

Listing 2.56: Generate keys on $IRPMASTER

root@IRPMASTER ~ # ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -C "failover@noction"

Default keys files are used. In case your system needs additional keys for other purposes we advise that those keys are assigned a different name. If this is not possible then keys file name designated for failover use should be also specified in IRP configuration parameter global.failover_identity_file.

Copy public SSH key from master to slave instance:

Listing 2.57: Install public key on $IRPSLAVE

root@IRPMASTER ~ # cat ~/.ssh/id_rsa.pub | while read key; do ssh $IRPSLAVE "echo $key >> ~/.ssh/authorized_keys"; done

Check if $IRPMASTER can login to $IRPSLAVE without using a password:

Listing 2.58: Check SSH certificate-based authentication works

root@IRPMASTER ~ # ssh $IRPSLAVE

Install certificate and keys for MySQL Multi-Master replication between $IRPMASTER and $IRPSLAVE #

MySQL Multi-Master replication uses separate communication channels that also require authentication. IRP failover uses key-based authentication for these channels too.

Adjust firewalls if any so that $IRPMASTER and $IRPSLAVE can communicate with each other bidirectionally.

Create Certificate Authority and server/client certificates and keys. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:

Listing 2.59: Generate CA and certificates

# cd && rm -rvf irp-certs && mkdir -p irp-certs && cd irp-certs

# openssl genrsa 2048 > $(hostname -s)-ca-key.pem

# openssl req -new -x509 -nodes -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) CA/emailAddress=support@noction.com" -key $(hostname -s)-ca-key.pem -out $(hostname -s)-ca-cert.pem

# openssl req -newkey rsa:2048 -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) server/emailAddress=support@noction.com" -nodes -keyout $(hostname -s)-server-key.pem -out $(hostname -s)-server-req.pem

# openssl rsa -in $(hostname -s)-server-key.pem -out $(hostname -s)-server-key.pem

# openssl x509 -req -in $(hostname -s)-server-req.pem -days 3600 -CA $(hostname -s)-ca-cert.pem -CAkey $(hostname -s)-ca-key.pem -set_serial 01 -out $(hostname -s)-server-cert.pem

# openssl req -newkey rsa:2048 -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) client/emailAddress=support@noction.com" -nodes -keyout $(hostname -s)-client-key.pem -out $(hostname -s)-client-req.pem

# openssl rsa -in $(hostname -s)-client-key.pem -out $(hostname -s)-client-key.pem

# openssl x509 -req -in $(hostname -s)-client-req.pem -days 3600 -CA $(hostname -s)-ca-cert.pem -CAkey $(hostname -s)-ca-key.pem -set_serial 01 -out $(hostname -s)-client-cert.pem

Verify certificates. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:

Listing 2.60: Verify certificates

# openssl verify -CAfile $(hostname -s)-ca-cert.pem $(hostname -s)-server-cert.pem $(hostname -s)-client-cert.pem

server-cert.pem: OK
client-cert.pem: OK

Install certificates in designated directories. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:Cross copy client key and certificates:

Listing 2.61: Install certificates in designated directories

# mkdir -p /etc/pki/tls/certs/mysql/server/ /etc/pki/tls/certs/mysql/client/ /etc/pki/tls/private/mysql/server/ /etc/pki/tls/private/mysql/client/

# cp $(hostname -s)-ca-cert.pem $(hostname -s)-server-cert.pem /etc/pki/tls/certs/mysql/server/
# cp $(hostname -s)-ca-key.pem $(hostname -s)-server-key.pem /etc/pki/tls/private/mysql/server/
# cp $(hostname -s)-client-cert.pem /etc/pki/tls/certs/mysql/client/
# cp $(hostname -s)-client-key.pem /etc/pki/tls/private/mysql/client/

# cd && rm -rvf irp-certs

Listing 2.62: Cross copy client key and certificates

root@IRPMASTER ~# scp "/etc/pki/tls/certs/mysql/server/$IRPMASTER-ca-cert.pem" "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/"
root@IRPMASTER ~# scp "/etc/pki/tls/certs/mysql/client/$IRPMASTER-client-cert.pem" "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/"
root@IRPMASTER ~# scp "/etc/pki/tls/private/mysql/client/$IRPMASTER-client-key.pem" "$IRPSLAVE:/etc/pki/tls/private/mysql/client/"

root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/certs/mysql/server/$IRPSLAVE-ca-cert.pem" "/etc/pki/tls/certs/mysql/client/"
root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/$IRPSLAVE-client-cert.pem" "/etc/pki/tls/certs/mysql/client/"
root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/private/mysql/client/$IRPSLAVE-client-key.pem" "/etc/pki/tls/private/mysql/client/"

Adjust file permissions. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:

Listing 2.63: Set file permissions for keys and certificates

# chown -R mysql:mysql /etc/pki/tls/certs/mysql/ /etc/pki/tls/private/mysql/ 
# chmod 0600 /etc/pki/tls/private/mysql/server/* /etc/pki/tls/private/mysql/client/*

Configure MySQL replication on $IRPSLAVE #

Each node of MySQL Multi-Master replication is assigned its own identifier and references previously configured certificates. There are also configuration parameters such as binary/relay log file names and auto-increment and auto-increment-offset values.

IRP includes a template config file /usr/share/doc/irp/irp.my_repl_slave.cnf.template. The template designates $IRPSLAVE as second server of the Multi-Master replication and includes references to $(hostname -s) that need to be replaced with the actual hostname of $IRPSLAVE before installing. Apply the changes and review the configuration file. Alternatively a command like in the below example can be used to create $IRPSLAVE config file from template. Ensure using actual short host name instead of the provided variable:

Listing 2.64: Example $IRPSLAVE configuration from template

# Ubuntu 
root@IRPSLAVE ~# sed 's|$(hostname -s)|$IRPSLAVE|' < /usr/share/doc/irp/irp.my_repl_slave.cnf.template  > /etc/mysql/conf.d/irp.my_repl_slave.cnf 
# RedHat 
root@IRPSLAVE ~# sed 's|$(hostname -s)|$IRPSLAVE|' < /usr/share/doc/irp/irp.my_repl_slave.cnf.template  > /etc/my.cnf.d/irp.my_repl_slave.cnf

The config file created above must be included into $IRPSLAVE node’s MySQL config my.cnf. It is recommended to store these files inside OS-specific directories for MariaDB configuration files (Ubuntu: /etc/mysql/conf.d, RedHat: /etc/my.cnf.d/, otherwise it should be included via !include /path/to/file from main MariaDB config.

Check Multi-Master configuration on $IRPSLAVE:

Listing 2.65: Check MySQL on $IRPSLAVE works correctly

root@IRPSLAVE ~# systemctl start mariadb 
root@IRPSLAVE ~# tail -f /var/log/mysqld.log 
root@IRPSLAVE ~# mysql irp -e "show master status \G" 
root@IRPSLAVE ~# systemctl stop mariadb

Configure MySQL replication on $IRPMASTER #

Configuring $IRPMASTER is done with running services.

Configuring MySQL Multi-Master replication on $IRPMASTER should only be done after confirming it works on $IRPSLAVE.

Similarly to $IRPSLAVE above IRP comes with a template configuration file for $IRPMASTER – /usr/share/doc/irp/irp.my_repl_master.cnf.template. The template designates $IRPMASTER as first server of the Multi-Master replication and includes references to $(hostname -s) that need to be replaced with the actual hostname of $IRPMASTER before installing. Apply the changes and review the resulting configuration file.

Alternatively a command like the example below can be used to create $IRPMASTER config file from template. Ensure using actual short host name instead of the provided variable:

Listing 2.66: Set $IRPMASTER as a first node for Multi-Master replication

#Ubuntu
root@IRPMASTER ~# sed 's|$(hostname -s)|$IRPMASTER|' < /usr/share/doc/irp/irp.my_repl_master.cnf.template > /etc/mysql/conf.d/irp.my_repl_master.cnf
#Redhat
root@IRPMASTER ~# sed ’s|$(hostname -s)|$IRPMASTER|’ < /usr/share/doc/irp/irp.my_repl_master.cnf.template > /etc/my.cnf.d/irp.my_repl_master.cnf

Again, the config file created above must be included into $IRPMASTER node’s MySQL config my.cnf. It is recommended to store these files inside OS-specific directories for MariaDB configuration files (Ubuntu: /etc/mysql/conf.d, RedHat: /etc/my.cnf.d/, otherwise it should be included via !include /path/to/file from main MariaDB config.

Check MySQL Multi-Master configuration on $IRPMASTER:

Listing 2.67: Check MySQL on $IRPMASTER works correctly

root@IRPMASTER ~# systemctl restart mariadb
root@IRPMASTER ~# tail -f /var/log/mysqld.log
root@IRPMASTER ~# mysql irp -e "show master status \G"

If Multi-Master configuration on $IRPMASTER fails or causes unrecoverable errors, a first troubleshooting step is to comment back the included line in /etc/my.cnf on master and slave and restart mysqld service to revert to previous good configuration.

Create replication grants on $IRPMASTER #

Replication requires a MySQL user with corresponding access rights to replicated databases. The user must be assigned a password and is designated as connecting correspondingly from $IRPMASTER or $IRPSLAVE. Unfortunately hostnames cannot be used for this and the exact IP addresses of the corresponding nodes must be specified.

The user is created only once in our procedure since after being created the database on $IRPMASTER is manually transferred to $IRPSLAVE and the user will be copied as part of this process.

Grant replication access to replication user:

Listing 2.68: Replication user and grants

mysql> CREATE USER 'irprepl'@'<mysql_slave1_ip_address>' IDENTIFIED BY '<replication_user_password>';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'irprepl'@'<mysql_masterslave1_ip_address>' REQUIRE CIPHER 'DHE-RSA-AES256-SHA';
mysql> CREATE USER 'irprepl'@'<mysql_master2_ip_address>' IDENTIFIED BY '<replication_user_password>';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'irprepl'@'<mysql_slave2_ip_address>' REQUIRE CIPHER 'DHE-RSA-AES256-SHA';

Copy IRP database configuration and database to $IRPSLAVE #

IRP failover copies the irp database and corresponding configuration file from $IRPMASTER to $IRPSLAVE before activating second server. This avoids the need to manually verify and synchronize identifiers.

Copy root’s .my.cnf config file if exists:

Listing 2.69: Copy database root user configuration file

root@IRPMASTER ~# scp /root/.my.cnf $IRPSLAVE:/root/

Copy config files:

Listing 2.70: Copy database configuration files

root@IRPMASTER ~# scp /etc/noction/db.global.conf $IRPSLAVE:/etc/noction/
root@IRPMASTER ~# scp /etc/noction/clickhouse/users.xml $IRPSLAVE:/etc/noction/clickhouse/

Preliminary copy database files:

Listing 2.71: Copy database data files

root@IRPMASTER ~# rsync -av --progress --delete --delete-after --exclude="master.info" --exclude="relay-log.info" --exclude="*-bin.*" --exclude="*-relay.*" /var/lib/mysql/ $IRPSLAVE:/var/lib/mysql/

Preliminary copy ensures that large files that take a long time to copy are synced to $IRPSLAVE without stopping MySQL daemon on $IRPMASTER and only a reduced number of differences will need to by synced while MySQL is stopped. This operation can be rerun one more time to reduce the duration of the downtime on $IRPMASTER even more.

Finish copy of database files (OS with Systemd):

Listing 2.72: Copy differences of database files (OS with Systemd)

root@IRPMASTER ~# systemctl stop mariadb clickhouse-server # RedHat Enterprise Linux
root@IRPMASTER ~# systemctl stop mysql clickhouse-server # Ubuntu
root@IRPMASTER ~# systemctl start irp-stop-nobgpd.target

systemctl start irp-shutdown-except-bgpd.target
systemctl start irp-shutdown.target

root@IRPMASTER ~# cd /var/lib/mysql && rm -vf ./master.info ./relay-log.info ./*-bin.* ./*-relay.*
root@IRPMASTER ~# rsync -av --progress --delete --delete-after /var/lib/mysql/ $IRPSLAVE:/var/lib/mysql/

The procedure above tries to reduce the downtime of MySQL daemon on $IRPMASTER. During this time Bgpd preserves IRP Improvements. Make sure this action takes less than bgpd.db.timeout.withdraw.

Start MySQL on $IRPSLAVE and check if there are no errors at /var/log/mysqld.log on RedHat Enterprise Linux or /var/log/mysql/error.log on Ubuntu.

First $IRPSLAVE must be checked.

Start MySQL on IRP master and check if there are no errors in MySQL logs as above.

Start replication (Slaves) on both $IRPMASTER and $IRPSLAVE #

MySQL Multi-Master replication uses a replication scheme where a replicating master acts as both replication master and replication slave. The steps above configured IRP nodes to be capable of acting as replication masters. Here we ensure that both nodes are capable of acting as replication slaves.

IRP provides a template command that needs to be adjusted for each $IRPMASTER and $IRPSLAVE and will instruct MySQL daemon to take the replication slave role. The template is /usr/share/doc/irp/changemasterto.template.

The template generates a different command for each $IRPMASTER and $IRPSLAVE nodes and requires multiple values to be reused from configuration settings described above. The command that is run on one node points to the other node as its master.

Make $IRPMASTER a replication slave for $IRPSLAVE:

Listing 2.73: Set $IRPMASTER as replication slave

$IRPMASTER-mysql> 
CHANGE MASTER TO 
MASTER_HOST='$IRPSLAVE-ip-address',
MASTER_USER='irprepl',
MASTER_PASSWORD='$IRPSLAVE-password>',
MASTER_PORT=3306,
MASTER_LOG_FILE= '$IRPSLAVE--bin.000001',
MASTER_LOG_POS=	<$IRPSLAVE-bin-log-position>,
MASTER_CONNECT_RETRY=10,
MASTER_SSL=1,
MASTER_SSL_CAPATH='/etc/pki/tls/certs/mysql/client/',
MASTER_SSL_CA='/etc/pki/tls/certs/mysql/client/$IRPSLAVE-ca-cert.pem',
MASTER_SSL_CERT='/etc/pki/tls/certs/mysql/client/$IRPSLAVE-client-cert.pem',
MASTER_SSL_KEY='/etc/pki/tls/private/mysql/client/$IRPSLAVE-client-key.pem',
MASTER_SSL_CIPHER='DHE-RSA-AES256-SHA';

You must manually check what values to assign to

$IRPSLAVE-bin.000001 and <$IRPSLAVE-bin-log-position>

by running the following MySQL command on $IRPSLAVE

mysql> show master status

For the initial configuration the values for $IRPSLAVE-bin.000001 and <$IRPSLAVE-bin-log-position> must be as follows:

Binlog file: $IRPSLAVE–bin.000001

Binlog position: 106

Run the commands below to run the replication and check the slave status:

Listing 2.74: Starting replication slave on $IRPMASTER

mysql> START SLAVE \G
mysql> show slave status \G

Check the Slave_IO_State, Last_IO_Errno, Last_IO_Error, Last_SQL_Errno, Last_SQL_Error values for errors. Make sure there are no errors.

Make $IRPSLAVE a replication slave for $IRPMASTER:

Listing 2.75: Set $IRPSLAVE as replication slave

$IRPSLAVE-mysql> 
CHANGE MASTER TO 
MASTER_HOST='$IRPMASTER-ip-address',
MASTER_USER='irprepl',
MASTER_PASSWORD='$IRPMASTER-password>',
MASTER_PORT=3306,
MASTER_LOG_FILE= '$IRPMASTER-bin.000001',
MASTER_LOG_POS=	<$IRPMASTER-bin-log-position>,
MASTER_CONNECT_RETRY=10,
MASTER_SSL=1,
MASTER_SSL_CAPATH='/etc/pki/tls/certs/mysql/client/',
MASTER_SSL_CA='/etc/pki/tls/certs/mysql/client/$IRPMASTER-ca-cert.pem',
MASTER_SSL_CERT='/etc/pki/tls/certs/mysql/client/$IRPMASTER-client-cert.pem',
MASTER_SSL_KEY='/etc/pki/tls/private/mysql/client/$IRPMASTER-client-key.pem',
MASTER_SSL_CIPHER='DHE-RSA-AES256-SHA';

You must manually check what values to assign to

$IRPMASTER-bin.000001and <$IRPMASTER-bin-log-position>

by running the following MySQL command on $IRPMASTER

mysql> show master status

For the initial configuration the values for $IRPMASTER-bin.000001 and <$IRPMASTER-bin-log-position> must be as follows:

Binlog file: $IRPMASTER–bin.000001

Binlog position: 106

Run the commands below to run the replication and check the slave status:

Listing 2.76: Starting replication slave

mysql> START SLAVE \G
mysql> show slave status \G

Run IRP services on $IRPMASTER and $IRPSLAVE:

Listing 2.77: Starting IRP services and Frontend (OS with Systemd)

# systemctl start irp.target

Start services on $IRPMASTER first if the actions above took very long in order to shorten MySQL downtime.

Configure Failover using Wizard on $IRPMASTER #

These steps should only be performed once SSH communication channel and MySQL Multi-Master irp database replication have been setup and are verified to work. Refer subsections of Failover Configuration above.

Run failover wizard: #

A valid failover license should be acquired before failover configuration settings become available.

Configure IRP failover: #

Follow Setup Failover wizard steps and provide the required details. Refer Setup Failover wizard for details. Once your configuration changes are submitted IRP will validate configuration and if it is valid the configuration will be synced to $IRPSLAVE.

Only after this synchronization step takes place will $IRPSLAVE node know what is its role in this setup.

Apply configuration changes to edge routers: #

Ensure designated probing IPs, BGP session(s) are also setup on edge routers. Refer to corresponding sections of this document for details.

Enable failover: #

Use IRP’s Frontend Configuration -> Global -> Failover to configure IRP failover on both nodes. Monitor both IRP nodes and ensure everything is within expectations.

It is recommended that after finishing the preparatory steps above both IRP master and slave nodes run with disabled failover for a short period of time (less than 1 hour) in order to verify that all IRP components and MySQL Multi-Master replication work as expected. Keep this time interval short to avoid a split-brain situation when the two nodes make contradictory decisions.

Synchronize RRD statistics to $IRPSLAVE #

RRD statistics is collected by both IRP failover nodes and needs not be synchronized during normal operation. Still, when IRP failover nodes are setup at a large time interval between them it is recommended that RRD files are synchronized too. This will ensure that past statistics is available on both IRP nodes. During short downtimes of each of the nodes synchronization is less usefull but can be performed in order to cover the short downtime gaps in RRD based graphs.

Sample command to synchronize IRP’s RRD statistics:

Listing 2.78: Synchronize RRD

root@IRPMASTER ~ # rsync -av /var/spool/irp/ $IRPSLAVE:/var/spool/irp

Updated on Oct 29, 2024

IRP Resources

White Papers

Discover IRP features, review use cases and make informed decisions

Case Studies

Case studies and success stories to help in the decision-making process

Videos

Watch Noction IRP videos, screencasts and client testimonials

Manuals

Technical Noction IRP documentation, deployment instructions and datasheets

Tier 1 Reports

Get a first-hand network performance view of the major Tier 1 Carriers

IRP FAQ

See answers to the questions we get asked the most about Noction IRP

NFA Resources

NFA Documentation

Product overview, user guide and the deployment instructions documents

Tips and Tricks

Practical and useful info on NFA and the overall NetFlow analysis

NFA FAQ

A series of the most common NFA questions and answers

Guides & eBooks

View All Guides

Prerequisites #

Configure communication channel from $IRPMASTER to $IRPSLAVE #

Install certificate and keys for MySQL Multi-Master replication between $IRPMASTER and $IRPSLAVE #

Configure MySQL replication on $IRPSLAVE #

Configure MySQL replication on $IRPMASTER #

Create replication grants on $IRPMASTER #

Copy IRP database configuration and database to $IRPSLAVE #

Start replication (Slaves) on both $IRPMASTER and $IRPSLAVE #

Configure Failover using Wizard on $IRPMASTER #

Run failover wizard: #

Configure IRP failover: #

Apply configuration changes to edge routers: #

Enable failover: #

Synchronize RRD statistics to $IRPSLAVE #

Company News

Blog

About Us

Press Releases

Careers

Support

Noction Releases Free IRP Lite v4.2.9 with GMI, Threat Mitigation, and Expanded OS Support

Noction IRP Quote

NFA Pricing

IRP Resources

White Papers

Case Studies

Videos

Manuals

Tier 1 Reports

IRP FAQ

NFA Resources

NFA Documentation

Tips and Tricks

NFA FAQ

Guides & eBooks

NOC as a Service

IP Transit Providers Assessment

Tier 1s Performance Dashboard

Tier 1 Carriers Report - September 2025

Noction IRP

Noction IRP Lite

Noction Flow Analyzer