IAB COVID-19 Workshop: Interconnection Changes in the United States

Introduction During the early weeks and months of the COVID-19 pandemic , significant changes to Internet usage occurred as a result of a sudden global shift to people working, studying and quarantining at home. One aspect that this affected was interconnection between networks, which this paper studies. In 2016, the Interconnection Measurement Project was launched. The IMP platform initially collected interconnection-related data from seven U.S.-based cable-based Internet Service Providers (ISPs) and an shared some of the details and findings. The initial focus of the project was to explore utilization characteristics of interconnect links during a period of time when video traffic was steadily increasing. The project concluded that there was ample aggregate capacity on interconnect links between ISPs and peers (including content providers), and that ISPs continually added capacity to their interconnects to keep pace with the growth in traffic. This IAB paper looks specifically at the long-term interconnection data from one of those ISPs, Comcast. We examine the long-term pre-COVID-19 trend as well as what occurred as COVID-19 impacted the Internet from February 2020 through August 2020. We also include observations from Comcast concerning interconnection changes during this timeframe. We hope that this information will be useful to the IAB workshop and the Internet community more broadly. It may serve as an interesting and useful historical reference in the future.

Long-Term Interconnection Data The IMP platform collected interconnection data starting in 2016, roughly four years prior to the COVID-19-driven shift in Internet usage. This provides an interesting capability for a before and after view of interconnection. A full explanation of the data can be found in Section 3 of the . At a high level, the maindata collected encompasses:

Timestamp (representing a five-minute interval)
Region (representing an aggregated link group)
Anonymized partner network
Access ISP
Total ingress bytes
Total egress bytes
Capacity

Utilization is captured based on sampled IPFIX records, with a packet sampling rate of 1/1,000. SNMP polling data yields information about the capacity on each link. The IMP platform does not have direct access to partner network identities---this dataset only includes an anonymous identifier corresponding to that particular partner. However, in collaboration with Comcast, IMP has worked to identify specific partner networks in the dataset for specific peers in the interest of detail COVID-19-related study. Focusing on Comcast within this dataset reveals several trends, in both aggregate capacity and utilization, as well as how utilization and capacity changed during the period of the COVID-19 pandemic. The data also reveals how traffic volumes changed for specific peers during this time period. The IMP data that tracks capacity shows Comcast steadily adding capacity from mid-2018 to present, with a significant increase in the rate of additional capacity from the beginning of 2020, increasing further in the second and third quarters of 2020. Specifically, we analyzed the rate at which capacity was added during these periods, on a month-by-month basis. We found that Comcast was adding aggregate capacity on its interconnects at *nearly twice* the rate as it was being added during 2019. Over a long timeframe, Comcast's daily peak hour interconnection port utilization has remained consistently around 90%. During the period at the end of March and the beginning of April 2020, daily peak utilization briefly increased to about 97% but steadily returned to normal levels in a matter of weeks by the end of April 2020, as Comcast quickly increased the rate at which it added capacity to keep pace with growing traffic demands. We also explored traffic volumes associated with each Comcast peer, comparing the ranks and volumes of each individual peer as measured on September 1, 2020 as compared to January 1, 2020, in both the upstream and downstream directions. Doing so allowed us to understand both the magnitude of changes in traffic volumes and ratios, as well as how specific individual peers deviated from normal baseline behavior, in terms of both upstream and downstream traffic rates and ratios. The data suggests that both the upstream and downstream directions saw some changes, although the deviations in upstream traffic patterns were greater: the Pearson coefficient for downstream traffic was 0.977, whereas for upstream traffic the Pearson correlation coefficient was only 0.935, indicating a more significant shift in upstream traffic ratios during this timeframe.

Comcast's COVID-19-Related Experiences Comcast observed a wide range of significant changes in Internet usage as residential users remained at home and shifted to working and studying from home. Changes in usage patterns observed in the access network in sum led to changes in the traffic flowing to interconnected networks. As the pandemic developed, there was a wide variety of changes in traffic volumes. At some locations in the network little change was detected while other locations saw a huge growth in the volume of traffic. At the peak of the surge, in March and April 2020, the average amount of growth observed across locations and types was roughly 33%. Voice & video conferencing (conferencing hereafter) jumped as much as 285% and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers . As this continued into May 2020 we observed conferencing remained up roughly 210-285%, VPN traffic up 30-40% and gaming downloads up 20-80%, and web-based streaming video consumption up 20-40% . In this several week period, traffic essentially grew at or more than it had in the prior year, which was significant growth in a short period of time. In the months following the onset of the pandemic Comcast observed:

Overall average downstream peak growth is up 13% (up as much as 20% at times).
Overall average upstream peak growth is up 36%.
In the access network, an average of 771 network augments per week were performed, peaking at over 1,800 in a single week and over with over 7 weeks with more than 1,000 per week. For comparison, the average earlier in the year was roughly 350 per week.
In the core network, over 500 augments were made in order to add 146 Tbps in capacity.
On a daily basis roughly 700,000 automated speed tests from customer homes were conducted in order to gauge the customer experience during this time. Average speeds to customers (both downstream and upstream) have generally remained at or above 105% of advertised speeds since March 1, 2020 in all regions. National average speeds have remained between 110% - 115% of advertised speeds over the same period. and
The share of streaming video as a percentage of total traffic is declined slightly from 67% to 63%. Despite strong growth, conferencing occupies a small share of total traffic and grew from 1% share to 4%. But gaming software released have driven significant download spikes since late April 2020.
For interconnection, peering coordinators across operators worked cooperatively and quickly to cut through any red tape and add new capacity as quickly as possible.
In 2019, settlement free interconnection capacity - a subset of overall interconnect types - grew by 15%. Between January and August 2020, driven by COVID-19 changes, there was an overall 37% increase in capacity from that prior 2019 level. And between March and October 2020 one Settlement Free Peer alone increased 115%.
Between March and October 2020 from Comcast observed other notable per-peer traffic increases of 245% and 3,900%.

Detailed Statistical Observations As briefly mentioned in previous sections, downstream traffic rates from many partners remained stable---the Pearson correlation coefficient for peak download rates between January 1, 2020 and September 2, 2020 is 0.977, indicating that the peak download rates to most peers was similar between these two time periods. On the other hand, certain peers experienced either a significant increase or decrease in peak download rates---often by two or three orders of magnitude. Similarly, other peers experienced a decrease in peak downstream rates by several orders of magnitude. On the other hand, upstream traffic rates were far less stable: In contrast, the Pearson coefficient for upload rates between January 1, 2020 and September 2, 2020 was only 0.935, suggesting more a more significant deviation in peak upstream rates. As with peak download rates, some peers experienced significant decreases, as well: in one outlier case, peak rate decreased by almost five orders of magnitude. A small handful of peers saw similar decreases. Yet, a far greater number of peers saw increases in peak upload rates by two to three orders of magnitude.

IANA Considerations This document includes no request to IANA.

Security Considerations This document includes no security considerations.