Noction Flow Analyzer Documentation

Noction Flow Analyzer (NFA) is a web-based network traffic analysis, monitoring and alerting tool. The product enables engineers to optimize their networks and applications performance, control bandwidth utilization, do the proper network capacity planning, perform detailed BGP peering analysis, improve security, and minimize network incidents response time.

1.1 NFA Components

Noction Flow Analyzer contains a few fundamental components, which working together implement the main function of NFA – offer timely traffic flows information that is easy to interpret and analyze.

Databases NFA uses two databases: MySQL (configuration) and ClickHouse (Data Mart), that act relating to the central repository which stores processing results.

NFAAPId represents a set of secure web services that collect data from Databases. A valid NFA user-id is required to access most of the API services. Access NFA’s frontend to manage users or configure external User Directories. NFA API uses an authentication mechanism based on authentication tokens. The token is passed as a query parameter for all API requests that require authentication.

NFABGPd stores and keeps all routes and adds AS Path to traffic flow.

NFApushd is used to send notifications and alerts to the end-users.

NFAaggd periodically aggregates flow data that is stored in the database and flushes data according to the configuration parameters.

NFAalertd is used to detect and generate alerts based on the alert settings set by the end-user.

Msg. Broker is used for communication between the NFA components.

Frontend represents a complex browser application that interacts with NFAAPId. It offers a comprehensive set of reports, graphs and flows information that can reflect the current and historical state of a network.

1.2 Collector

The collector is one of NFA’s most important components. It receives, analyzes, and processes all traffic transiting the network and transfers data in a compatible mode to NFA Databases – MySQL and ClickHouse. It processes the most common types of Flow: NetFlow, sFlow, J-Flow, IPFIX, NetStream.

sFlow (6343 port) is a protocol designed for monitoring network, wireless and host devices. Developed by the sFlow. org Consortium, this protocol is supported by a wide range of network devices, as well as routing software and network solutions. sFlow, short for “sampled flow”, is an industry-standard for packet export at Layer 2 of the OSI model. It provides the means for exporting truncated packets, together with interface counters. It’s a packet sampling for an N number of packets with all required statistical information and expedited to the destination collector. The information details taken from the packet are the headers from Layer 3 and 4 and some information about the upper layers’ data only. For example, if the HTTP protocol is present, sFlow will guarantee data confidentiality since it will not extract the information from the packet and will not collect all network sessions.

NetFlow (2055 port) is an IP network statistics protocol developed by Cisco Systems, Inc. that offers the ability to collect IP session network traffic as it enters or exits an interface. By analyzing the data that is provided by NetFlow a network administrator can determine things such as the source and destination of traffic, class of service, and the cause of congestion. Juniper routers offer a similar feature called J-Flow which in its essence is the same Cisco NetFlow protocol.

Flow statistics are captured and stored in DB which NFA’s graphical interface subsequently offers to users as dashboards, charts, and reports with filtering, grouping, and aggregation functions.

Network devices should be first configured to forward Flow statistics to NFA in order for it to get the initial data to operate on. NFA listens to Flow stats on the default protocol ports. Flow ports can be changed from the Configuration Settings section of NFA’s Front End.

1.3 Databases

NFA processes huge volumes of data and uses two databases to store all the related information: MySQL and ClickHouse. The accumulated information is used by other NFA components to provide a graphical view of flow parameters.

MySQL is the most popular Open Source SQL database management system, developed, distributed, and supported by Oracle Corporation. It plays the role of NFA’s system data depository which possesses configuration, dashboard, device, and user information.

ClickHouse is a column-oriented database management system (DBMS) for the online analytical processing of queries (OLAP).

ClickHouse benefits:
• Extremely Fast scans that can be used for real-time queries.
• Real-time data ingestion
• Parallel processing for a single query
• Hardware efficient
• Scales well both vertically and horizontally

The most important ClickHouse tables are:
• BGP
• Database_version
• Flows1
• Flows2
• Raw
• Template_flows

Flows Tables comprise the meaningful flow data.

2. NFA Functionality

2.1 Frontend

NFA main page is designed to display a dashboard of choice and offer facilities to access all application features via its main menu, navigation buttons, and links.

2.1.1 Dashboards

NFA dashboards are the specific sets of flexible and interactive visualizations, designed for quick analysis of the network traffic data and informational awareness. Dashboards consist of widgets – containers with graphical representations of specific data, which can be added, edited, positioned, deleted or modified as you like.

• Name: The name of the dashboard
• Description: Dashboard user-defined description
• Favorite: a state marked by a star icon
• Created by: The user who created the dashboard
• List of widgets: widget names used in the dashboard
• Default status: the default dashboard the user lands on when logging into NFA

Creating a new dashboard

You can easily create a new dashboard in NFA from the All Dashboards directory.
Click the “ADD DASHBOARD” button. A pop-up will appear. Choose if you’d like to create a dashboard from scratch or build one based on an existing dashboard.

Managing Dashboards

Access any of the dashboards you’ve created or had admin rights to. Click the padlock icon in the top menu to add, edit and delete widgets or customize the dashboard’s layout.

Deleting a Dashboard

Click the “Delete” icon on the dashboard you’d like to get rid of in the All Dashboards directory.
You can only delete a dashboard if you created it, or if you‘ve been granted the corresponding admin rights.

2.1.2 Widgets

All network traffic information in NFA is graphically represented by widgets, which are the main dashboard elements. Widgets encompass a particular query focusing on the desired network feature. A library of widgets is maintained by NFA and allows users to reuse them across all dashboards.

Alternatively, you can create a new widget by duplicating an existing one. Click on the existing widget name to open it in Data Explorer. Make desired modifications and save it as a new widget.

Feel free to change the widgets graph settings, appearance as well as the legend position by clicking the appropriate buttons in the top right corner of the Data Explorer graph or the actual widget.

2.2 Data Explorer

Data Explorer provides detailed network traffic stats in both chart (when possible) and report forms. “Group & Order“, “Filters” and “Narrow by” functions are available to focus or broaden attention to the desired aspects of network traffic.

Data Explorer can be accessed either from the Main Menu or by clicking on any widget’s header on dashboards. Any grouping and filtering criteria previously setup in widgets will auto-populate in Data Explorer.

Data Explorer takes the ensuing statistics from the DB table Flows which includes but is not limited by the following:

Data Explorer functions:

• Group by – specifies how to group data.
• Filters – specify only the data of interest to include in results
• Narrow By – specify from what locations, network devices, and/or interfaces to consider the stats
• Time horizon – sets the time interval to explore
• Packets depict whether Packets, Octets, bits/s metrics are aggregated and plotted on charts
• Save | My Filters allows saving a specific set of selected filters with their corresponding values to “My Filters” library for future use
• Run query – runs the query and retrieves data
• Save as widget – prompts for a widget to be added to the library with this exact combo of filters and group by criteria
• Display as – chart type icon allows switching between different ways to plot result data

2.2.1 Group & Order

Grouping is one of the essential criteria for analyzing data. 

Grouping by source or destination indicates whether the traffic is inbound or outbound. Grouping by the port highlights what amount of traffic the network has for different applications and so on. 

Using the default unidirectional grouping, we can specify one or more Flow attributes to be analyzed. Note that the top results shown in the graph/table can be dependent on a particular traffic direction based on the selected grouping criteria.

2.2.2 Filters

Filters are used to constrain the analyzed data to a particular subset that matches filter criteria. Filters can be applied while working with Dashboards or within Data Explorer. It is a very important feature as it saves time and significantly reduces the workload.

2.2.3 Filter by Locations, Devices and Interfaces

NFA users can filter flow data in Data Explorer or dashboards by devices that have been initially added to the system as well as interfaces. There is an option to select one/multiple devices/interfaces or groups of devices assigned to specific locations (sites).

2.2.4 Time Intervals

Time intervals govern how fast and how detailed the resulting data can be. When a query extends over a long time interval or checks data far in the past the results will be less granular compared to shorter and current time intervals.

2.2.5 Percentile Reporting

95th percentile is a popular network calculation used for reporting and billing burstable network usage. It typically serves as a baseline for traffic utilization metering on a network. Starting with NFA v 21.06, percentile value calculation, be it for 95th or any other, is available for packets, octets, and flows.

Go to Data Explorer, click the % icon, specify the percentile you want NFA to calculate, add any filtering conditions, and hit the “Run Query” button. Note, NFA calculates the Xth percentile, where X is an integer between 1 and 100. Feel free to save your view as a widget and add it to any of your dashboards.

2.3 BGP Data

BGP Data is delivered as an optional add-on to Noction Flow Analyzer.

NFA overcomes the limitation of BGP support in traditional NetFlow. It employs a collection of full BGP data from BGP tables of edge routers, extracting the required BGP attributes. NFA extracts BGP attributes such as AS_PATH and matches the obtained data with a corresponding flow record from Flow DB tables. This enables NFA to see and filter on the full BGP path, not just the next hop, first three or last three AS numbers.

Enabling BGP data will require you to establish a BGP session between at least one of your routers and NFA.

2.3.1 BGP Report

Use the BGP Report section to see BGP data obtained from your devices. The BGP Route advertisements and withdrawals can now be visualized both in a graph and table mode.

Filter, group, and search data according to your needs. Create and add custom widgets to any of your dashboards.

2.3.2 BGP Sankey Diagram

NFA offers a great way to visualize the Internet traffic routing criteria along with traffic volume using a Sankey type diagram. Its extensive filtering capabilities can provide you with a clear picture of the paths your traffic is taking, the countries regions or cities you traffic originates and terminates in, traffic volume distribution by different paths, best potential new peering candidates, and a lot more.

The list of available filters is listed below:

2.4 Alerts

NFA lets you set up a robust and customizable alert system that can proactively notify you when important conditions are detected in your network traffic data. You can configure alerts based on different characteristics and parameters of your network traffic: volume changes, frequency, specific traffic type existence, duration, baseline or a complex combination of such characteristics.

2.4.1 Creating Alerts

1. Go to Alerts > My Alerts and click the “Create New Alert” button.

2. Enter a meaningful Name and Description for the Alert. Select an appropriate Priority Level: Low, High or Critical.

7. Review your Alert details, Notification Channels and Save Alert.

2.4.2 My Alerts

My Alerts section contains a list of Alerts that have been created by your NFA users. Depending on the user access level you can edit, duplicate, delete alerts or turn them on/off.

2.4.3 Active Alerts

Active Alerts section allows you to view the triggered alert details, triggered alert date/time and allows you to reset (acknowledge) alerts.

All triggered alerts in NFA show up with UTC timestamps. This is specifically useful for teams using NFA from multiple geographical time zones.

2.4.4 History of Alerts

All triggered alerts are saved in the History of Alerts section. Use the available options to search and sort the alert incidents.

3. Management

3.1 Inventory

The Inventory section accumulates information about all types of network devices being used in NFA and assigns them meaningful names. Devices must be assigned to locations/sites to further enhance NFA’s grouping and filtering capabilities.

3.1.1 Adding Devices

To add a new device, go to Management > Inventory. Click the “ADD DEVICE” button in the top right corner. A dialog box will appear and ask to provide the following information: Device Name, Device Description, Device Type, and Exporter IP.

Specify if you’d like to override the flow sampling rate for the device you are adding. Define and set up the data origin (Flow, BGP, or the Internal Database) priority for a distinct set of attributes. Click “NEXT STEP” to proceed.

Enabling BGP data export from the device you are about to add is optional.

First, configure an iBGP or eBGP session between NFA and your router(s).

Next, fill out both OUR AS and PEER AS fields under the BGP settings tab for eBGP. With eBGP, the route-reflector does not need to be configured, and the router side just needs to be set up as though it’s a transit customer. If OUR AS field is left blank, an iBGP session will be established.

The BGP SESSION control is set to ON by default.

Both SNMPv2c and SNMPv3 can be defined. Since SNMPv3 supports authentication and encryption, we recommend using this version when possible. Once set up, interface names and descriptions will appear as the “narrow by” options in NFA’s Data Explorer section.

3.1.2 Managing Devices

All devices added to NFA are listed under Management > Inventory. Devices are grouped by location. Small icons next to Flow, BGP and/or SNMP indicate their state. Click a small arrow next to the device name to see the additional information such as the Exporter IP, Sample rate, Flow type and Flow rate or turn ON/OFF the BGP session or SNMP.

3.1.3 Deleting Devices

To delete a device you’ll need to deactivate it first. Click the OFF switch for a particular device. Next, click the delete icon. A dialog box will appear asking you to confirm the deletion.

3.2 Configuration Settings

NFA has a large set of configuration settings available in the front end to fine-tune the system’s behavior.

API:

• SSL – Enables/disables SSL support
• SSL Redirect – Enables/disables SSL redirect from a non-SSL port
• SSL listening IPv4/IPv6 address – SSL (HTTPS) listening IPs
• SSL listening port – SSL (HTTPS) listening port
• Non-SSL IPv4/IPv6 listening address – Non-SSL (HTTP) listening IPs
• Non-SSL listening port – Non-SSL (HTTP) listening port
• Front-end access restriction – Enables/disables NFA front-end access restriction
• Allowed IP addresses – Lists the IPs or Subnets with access to the NFA frontend

Flow Collector:

• NetFlow/J-Flow/NetStream/IPFIX listening port – Port on which the collector listens for NetFlow/JFlow/Netstream/IPFIX packets.
• sFlow listening port – Port on which collector listens for sFlow packets
• Buffer size (packets) – The incoming packets buffer size
• Flow stream stop event threshold (seconds) – If flows stream was stopped, it defines the period of time after which FlowStreamStop event would be sent.
• Extended exporter address determination logic – Enables/Disables extended exporter address determination logic using IPFIX/NetFlow informations elements: 403:originalExporterIPv4Address, 404:originalExporterIPv6Address, 130:exporterIPv4Address, 131:exporterIPv6Address
• BGP attributes Caching – Enables/Disables BGP attributes caching
• BGP Attributes cache size (in number of entries) – Specifies the maximum number of cache entries in BGP attributes requester cache
• BGP attributes cache TTU (Time to Use) – Specifies the maximum number of seconds for TTU in BGP attributes requester cache of an entry
• BGP attributes related requests timeout (milliseconds)
• Accept flow records from enabled devices only – when enabled, NFA accepts flow records sent by exporter IP of the configured devices only

BGP:

• KeepAlive message interval (seconds) – The interval between two consecutive BGP keepalive messages
• Hold time interval (seconds) – Specifies how long NFA will wait for incoming BGP messages before it assumes the neighbor is dead.

LOGS:

• APId log level – Specifies the log level for APId. The drop-down menu lists log levels in order, from most severe to least severe ones.
• Flowd log level – Specifies the log level for Flowd. The drop-down menu lists log levels in order, from most severe to least severe ones.
• Aggd log level – Specifies the log level for Aggd. The drop-down menu lists log levels in order, from most severe to least severe ones.
• BGPd log level – Specifies the log level for BGPd. The drop-down menu lists log levels in order, from most severe to least severe ones.
• Alertd log level – Specifies the log level for Alertd. The drop-down menu lists log levels in order, from most severe to least severe ones.
• Pushd log level – Specifies the log level for Pushd. The drop-down menu lists log levels in order, from most severe to least severe ones.
• SNMPd log level – Specifies the log level for SNMPId. The drop-down menu lists log levels in order, from most severe to least severe ones
• NAPId log level – Specifies the log level for NAPId. The drop-down menu lists log levels in order, from most severe to least severe ones

STATS:

• Collector data1 granularity (minutes) – Specifies the aggregation granularity for data stored in flows1 table
• Collector data2 granularity (minutes) – Specifies the aggregation granularity for data stored in flows2 table
• Collector data3 granularity (minutes) – Specifies the aggregation granularity for data stored in flows3 table
• Collector data1 keep time (minutes) – Specifies the time for which to keep data in flows1 table
• Collector data2 keep time (minutes) – Specifies the time for which to keep data in flows2 table
• Collector data3 keep time (minutes) – Specifies the time for which to keep data in flows3 table
• BGP data keep time (days) – Specifies the time to keep data in the BGP table (used for BGP Report).

RESET DEFAULTS

• Changes all settings back to the default values.

REPORT CACHE:

• Query caching – Enables/disables caching of the query results
• Cached query lifetime (minutes) – Specifies the lifetime of the cached query
• Query cache max size (megabytes) – Specifies the maximum cache size

SNMP:

• Time between successive interface info requests (seconds) – The time interval between requests for the available interface details sent out by NFA.
• SNMP requests timeout (seconds) – The time interval that NFA waits for a response message from an agent. Increase the SNMP timeout value if there is higher latency in your network.
• Number of retries before failing a request – If a response from an SNMP Agent is not received before the timeout, then NFA retries the request the indicated number of times before reporting a failure.
• Maximum number of repetitions for GetBulk request – value for max repetitions field in the GETBULK PDU.

3.3 Custom Groups

Custom Groups allow you to streamline and logically organize your network analysis experience by creating unique sets of data parameters to personalize your view in many areas of the Noction Flow Analyzer. For instance, you can create custom IP groups for your company’s specific departments, various geographic locations, or any other administrative/business requirement.

3.4 System Notifications

3.4.1 System Notifications Overview

System notifications are used to communicate to users the state of their NFA instance and/or any of its components. They are triggered by a range of preconfigured system-level events.

The list of events that can generate notifications is provided below.

Once an NFA component is started, stopped or reconfigured it raises the following events:

• NFA BGP session Disconnected
• NFA BGP session Established

• Flow Stream Start
• Flow Stream Stop

3.4.2 System Notification Channels Configuration

In order for Notifications to be delivered correctly, the corresponding email, Telegram or Slack channel configuration shall be provided. Go to Management > System Notifications > Notification Channels.

3.4.3 System Notification Subscriptions

System Notifications are sent only if a valid subscription to events has been created.

Find the list of your active subscriptions under Management > System Notifications > Notification Subscriptions. Search through existing subscriptions, sort, view, edit, or delete them.

3.4.4 Notification Text Details

When a subscribed event is fired NFA will send notifications. The notification email will consist of the following:

1. Subscription topic as specified in the subscription
2. From email address – as configured in the email channel
3. Time – date-time of the last event that caused the notification. In case of rate limitation, this might be older than the time of the email.
4. Textual description of the event and any related details.
5. Older events which have been retained due to rate limitations. Keep in mind that all past events are aggregated into a single email.

3.5 User Management and User Directories

3.5.1 User Management

NFA includes a User Management function accessible under Management main menu section,
that allows the following:

• review and filter the list of users
• edit and delete existing user records
• add new users

3.5.2 LDAP user directories

LDAP user directories can be added, updated and removed from NFA by accessing Management > User Management. Each user directory takes a series of parameters specific for the protocol.

The example below offers a generic set of parameters required to configure NFA to use Active Directory for access management.

The general tab covers:

• User directory name – the name assigned to the directory within NFA
• User directory type
• State – a toggle to enable or disable a user directory,
• Order specifies when this user directory will be examined by NFA compared to other user directories

• User directory hostname in the form of either IP address or domain name (LDAP/LDAPS)
• User directory port
• SSL, TLS, or no encryption selector
• Certificate verification toggle and TLS CA Certificate file options in case the TLS encryption is selected
• The binding user name that NFA uses to authenticate itself
• Bind password assigned to NFA

• The Generic, LDAP POSIX, or Active Directory type selector
• Base DN specifies the root distinguished name and user subtree
• Object Class – an attribute that defines the characteristics of an object in the directory
• Object Filter – a search criterion used to find objects in the directory that match a specific set of attributes
• Username, Email, and Full Name fields map the User Directory attributes to NFA user attributes

3.6 License Status

NFA is a licensed product and requires users to obtain and register their license in the application.
The license status is accessed by navigating to Management > License.

router(config)# flow exporter-map EM
router(config-fem)# destination 10.1.1.5
router(config-fem)# source gi0/0
router(config-fem)# transport udp 2055
router(config-fem)# version v9
router(config-fem)# template data timeout 60
router(config-fem)# options interface-table timeout 60
router(config-fem)# exit

router(config)# sampler-map SM
router(config-sm)# random 1 out-of 1000
router(config)# exit

router(config)# flow monitor-map FMM
router(config-fmm)# record ipv4
router(config-fmm)# exporter EM
router(config-fmm)# cache timeout active 60
router(config-fmm)# cache timeout inactive 60
router(config-fmm)# exit

router(config)# interface Gi0/0
router(config-if)# flow ipv4 monitor FMM sampler SM egress
router(config-if)# exit

flow exporter EXPORTER-1
 destination 172.16.10.2
 export-protocol netflow-v9
 transport udp 2055
 exit
!
flow record v4_r1
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect counter bytes long
collect counter packets long
!
flow monitor FLOW-MONITOR-1
 record v4_r1
 exporter EXPORTER-1
!
interface GigabitEthernet 0/0/0
 ip address 172.16.6.2 255.255.255.0
 ip flow monitor FLOW-MONITOR-1 input

ip flow-export version 9
ip flow-export destination $NFA_IP 2055
interface $Interface_to_ISP1
 ip flow ingress
 ip flow egress

chassis {
    fpc 0 {
        sampling-instance nfa-instance;
    }
}                                       
interfaces {
    xe-0/0/0 {
        unit 0 {
            family inet {
                sampling {
                    input;
                    output;
                }
            }
        }
    }
}
forwarding-options {
    sampling {
        instance {
            inst1 {
                input {
                    rate 1024;
                }
                family inet {
                    output {
                        flow-server X.X.X.X {
                            port 2055;  
                            version-ipfix {
                                template {
                                    ipfix-templatev4;
                                }
                            }
                        }
                        inline-jflow {
                            source-address Y.Y.Y.Y;
                        }
                    }
                }
            }
        }
    }
}
services {
    flow-monitoring {
        version-ipfix {
            template ipfix-templatev4 {
                flow-active-timeout 60;
                flow-inactive-timeout 60;
                template-refresh-rate {
                    seconds 60;         
                }
                ipv4-template;
            }
        }
    }
}

chassis {
    fpc 0 {
        sampling-instance nfa-instance;
    }
}                                       
interfaces {
    xe-0/0/0 {
        unit 0 {
            family inet {
                sampling {
                    input;
                    output;
                }
            }
        }
    }
}
forwarding-options {
    sampling {
        instance {
            nfa-instance {
                input {
                    rate 1024;
                }
                family inet {
                    output {
                        flow-server X.X.X.X {
                            port 2055;  
                            version9 {
                                template {
                                    v9-templatev4;
                                }
                            }
                        }
                        inline-jflow {
                            source-address Y.Y.Y.Y;
                        }
                    }
                }
            }
        }
    }
}
services {
    flow-monitoring {
        version9 {
            template v9-templatev4 {
                flow-active-timeout 60;
                flow-inactive-timeout 60;
                template-refresh-rate {
                    seconds 60;         
                }
                ipv4-template;
            }
        }
    }
}

!
sflow run
sflow source $SOURCE
sflow destination $DESTINATION $PORT
sflow polling-interval 10
sflow sample $SAMPLING-RATE
!

ip traffic-flow set interfaces=$ISP cache-entries=1M enabled=yes active-flowtimeout=5 inactive-flow-timeout=60
ip traffic-flow target set dst-address=$NFA_IP port=2055 src-address=$ROUTER_IP
version=9 v9-template-refresh=100 v9-template-timeout=300

[Router] interface <$upstream_interface>
[Router-$upstream_interface] ip netstream sampler fix-packets 1200 inbound
[Router-$upstream_interface] ip netstream sampler fix-packets 1200 outbound
[Router-$upstream_interface] quit

[Router] ip netstream timeout active 20
[Router] ip netstream timeout inactive 100
[Router] ip netstream tcp-flag enable

[Router] ip netstream export source $router_source_IP
[Router] ip netstream export host $NFA_IP 2055

[Router] ip netstream export version 9

[Router] interface <$upstream_interface>
[Router-$upstream_interface] ip netstream inbound
[Router-$upstream_interface] ip netstream outbound
[Router-$upstream_interface] quit