Tier 1 carrier performance report: January, 2018

The report covers broadly used Tier 1 carriers performance results in the US and the general spread for the month of January 2018. For the US results, rare data-points from distant locations e.g. Alaska have been omitted.

The presented analysis is based on more than 948 million successful probes that span the entire month. All data is aggregated per carrier on a daily basis and accounts for many thousands of successful probes. A control group (labeled C) is used as a base for comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers..

Fig. 1. Average Loss and Latency in January 2018
The charts include a control group C (gray) to allow cross comparison.

The values for December 2017 are included for cross comparison.

Fig. 2. Average Loss and Latency in December 2017
The charts include a control group C (gray) to allow cross comparison.

Average packet loss analysis:

• NTT kept the leading position in January 2018, as in December 2017, and is closely followed by Zayo and Level 3;
• Zayo and Cogent results for the month of December 2017 are close to the control group level;
• A higher level of average loss has been kept by GTT, Telia and Hurricane Electric;
• There aren’t any significant changes in the distribution of average packet loss for the analyzed Tier 1 carriers when comparing January 2018 and December 2017 data.

Average latency analysis:

• In January 2018, Level 3, and NTT have registered lower average latency than the control group level value. XO, Zayo, and Telia have registered the results close to the control group level;
• The higher level of average latency has been registered by GTT, followed by Hurricane Electric, Cogent and CenturyLink;
• A significant change in average latency for the month of January, compared to December 2017, has not been observed.

The charts below illustrate the performance of each carrier in comparison to the control group level.

Fig. 3. Better or worse Loss and Latency in January 2018
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in December 2017
The numbers are differences from average control group.

In comparison with the control group for the month of January, 2018:

• NTT, XO, and Level 3 showed better results in terms of Loss. The leader of this group, NTT, registered a lower packet loss average than the control group level (2,7%) by ~ 0.5%, followed by XO and Level 3 (~ 0.3%). Zayo has shown approximately the same result as the control group. GTT showed substantially worse result in terms of Loss, exceeding the packet loss average by 0,9%. Telia surpassed the packet loss average by ~ 0.5%, followed by Hurricane Electric with ~ 0.4% ;
• Level 3 and NTT presented better results in terms of Latency. Level 3 latency was lower than the average latency of the control group (125,8 ms) by ~ 10,3 ms, and NTT by ~ 9,6 ms. GTT’s average latency surpassed the control group value by ~ 22,3 ms. Hurricane Electric was highlighted with a latency surplus of ~ 16,5 ms compared to the control group average.

In comparison with the data for the month of December, 2017:

• NTT and Level 3 remained the leaders of the group. Telia showed the same results as in December 2017 in terms of Loss. XO has considerably improved its loss value, migrating into the leaders group. GTT increased the packet loss average in comparison with the previous month. Cogent, Telia, and Hurricane Electric kept the same results as in December 2017 and remained in the group of Tier 1’s with the higher packet loss than the control group average;
• Level 3, and NTT kept their leader positions in terms of Latency. XO increased its average latency and migrate into the losers group. Other Tier 1s kept approximately the same latency results as in December 2017.

Loss

For the Loss analysis we use a scatter plot, where average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All data-points below the diagonal represent the better performing carriers and vice versa.

Abnormally large losses are still registered for a large number of datapoints. We consider excessive an average above 4.5% packet loss. Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions are deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption. More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier January 2018
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier December 2017
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

In January 2018, based on ALL data-points, the best position in terms of Loss was held by Centurylink, followed by Level 3, XO and Zayo. Other Tier 1s registered poorer results. Centurylink, after the cut off at the 4.5% control group level has been applied, kept the same position as in the ALL data-points representation while XO showed a significant improvement. These results suggest that Centurylink and XO have registered preponderantly lower value of loss for packets which transit it, below the 4.5%.

As in December 2017, Centurylink and XO are present in the winner group for both graphics. GTT has considerably worsened the results for ALL data-points. XO has improved its statistics in both graphics. The positive tendency was registered by GTT, Hurricane Electric, Cogent, Telia, NTT, and Zayo however they still maintained the negative results as in December 2017.

Latency

For the Latency analysis we use a similar scatter plot to the one we used for Loss. It displays control group values on the diagonal while highlighting individual carrier measurements on the horizontal and on the vertical axis. Datapoints placed significantly and consistently below the average highlight better performing carriers while data-points above the average highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

The results above linger around the diagonal with the following observations:

• NTT was mostly present within the latency diapazon of 85 – 265 ms;
• Level 3 and Cogent have registered results which depict traffic from local to very long distances (60 – 220 ms);
• GTT has been mostly present within the latency diapazon of 120 – 200 ms;
• Telia and XO are spotted within the latency diapazon of 60 – 175 ms;
• CenturyLink has shown the latency between 85 -195 ms;
• Zayo is spotted within the latency diapazon of 80 – 175 ms;
• Hurricane Electric has been mostly present within the 60 – 215 ms latency range.

Fig. 10. Average latency gains/losses by carrier Values averaged for the difference between carrier performance and the average group in that network.

The differences in latency from the control group shown above are averaged with the expectation that better or worse performance will cancel out if the differences are caused by the measurement noise.

The results show that during the month of January 2018 in comparison to December 2017:

• XO maintained the leading position, reducing its average RTT by ~13 ms. GTT, and CenturyLink improved their average latency for each packet by ~ 3 ms. Telia, Zayo, and Level 3 improved their results with ~1 ms for each packet;
• Hurricane Electric and NTT kept the previous tendency, adding to their RTT about 3 ms, and 1 ms respectively for each packet. Cogent improved its results adding ~ 2 ms for each packet.