Report from the IAB COVID-19 Network Impacts Workshop 2020

Workshop Topics and Discussion

Measurement-based Observations on Network Traffic Dynamics The workshop started with a focus on measurements. A large portion of the submitted papers presented and discussed measurement data and these submissions provided a good basis get a better understanding of the situation, covering different angles and aspects of network traffic and kind of networks. Changes in Internet traffic due to the COVID-19 pandemic affected different networks in various ways. Yet all networks observed some form of change, be it a reduction in traffic, an increase in traffic, a change in working days and weekend days patterns, or a change in traffic classes. Traffic volume, directionality ratios, and its source and destination are radically different than from before COVID-19. At a high level, while traffic from home networks increased significantly, the traffic in mobile networks decreased as a result of reduced population mobility. The observed behavior in mobile networks is antagonistic, yet complementary, to the one observed in residential ISPs. In residential networks there was a strong increase in video conferencing and remote learning application traffic due to the shift for working and learning at home. With that shift, the typical diurnal usage patterns in network traffic also changed, with peak times occuring earlier in the day and lasting longer over the day - reflecting the start of the work or school day from home. This behavior is antagonistic, yet complementary, to the one observed in residential ISPs. While diurnal congestion at interconnect point as well in certain last mile network was reported, mainly in March, no persitent congestion was observed. Further, a downward trends in download throughput to certain cloud regions was measured, which can probably explained with the increase use of cloud services. This gives another indication that the scalng of shared resources in the Internet is working reasonably well enough to handle even larger changes in traffic as experience during the first nearly global lockdown of the COVID-19 pandemic.

Overall Traffic Growth The global pandemic has significantly accelerated the growth of data traffic worldwide. Based on the measurement data of one ISP, three IXPs, a metropolitan educational network, and a mobile operator, it was observed at the beginning of the workshop that overall the network was able to handle the situation well, despite a significant and sudden increase in traffic growth rate in March and April. That is, after the lockdown was implemented in March, a traffic increase of 15-20% at the ISP as well as the three IXPs was observed. That represents the traffic growth expected in a typical year which now took place in the matter of a few weeks only---a substantial increase. At DE-CIX Frankfurt, the world's largest Internet Exchange Point in terms of data throughput, the year 2020 has seen the largest increase in peak traffic within a single year since the IXP was founded in 1995. Additionally, mobile traffic has slightly receded. In access networks, the growth rate of upstream traffic also exceeded the growth in downstream traffic, reflecting increased adoption and use of video conferencing and other remote work and school applications. Most traffic increases happened during non-traditional peak hours: Before the first COVID-19 lockdowns, the main time of use was in the evening hours during the week, whereas since March it has been spread more equally across the day. That is, the increase in usage has mainly occurred outside the previous peak usage times (e.g. during the day while working from home). This means that, for the first time, network utilization on weekdays resembled that on weekends. The effects of the increased traffic volume could easily be absorbed: either by using existing reserve capacity, or by quickly switching additional bandwidth. This is one reason why the Internet was able to cope well with the pandemic during the first lockdown period. Some of the lockdowns were lifted or relaxed around May 2020. As people were allowed to perform some of their daily habits outside of their home again, as expected, there was a decrease of the traffic observed at the IXPs and the ISP; instead mobile traffic began to grow again.

Changes in Application Use The composition of data traffic has changed since the beginning of the pandemic: the use of videoconferencing services and virtual private networks (VPNs) for access to company resources from the home environment has risen sharply. In ISP and IXP network it was observed that traffic associated with web conferencing, video, and gaming increased largely in March 2020 as a result of the increasing user demand for solutions like Zoom or Microsoft Teams. For example, the relative traffic share of many "essential" applications like VPN and conferencing tools increased by more than 200%. Also, as people spent more hours at home, they tended to watch videos or play games, thus increasing entertainment traffic demands. At the same time, the traffic share for other traffic classes decreased substantially, e.g., traffic related to education, social media, and---for some periods---CDNs. In April and June, web conferencing traffic was still high compared to the pre-pandemic scenario, while a slight decrease in CDN and social media traffic was observed. During these months many people were still working from home, but restrictions had been lifted or relaxed, which likely led to an increase in in-person social activities and a decrease in online ones.

Example Campus Networks Changes in traffic have been observed at University campus networks as well, especially due to the necessary adoption of remote teaching. The Politecnico di Torino University (Italy) deployed its in-house solution for remote teaching, which caused the outgoing traffic to grow by 2.5 times, driven by more than 600 daily online classes. Incoming traffic, instead, decreased by a factor of 10 due to the cessation of any in-person activity. Based on their measurements, this change in traffic and network usage did however not lead to noticeable performance impairments, nor have significantly poor performance been observed for students in remote regions of Italy. Outgoing traffic also increased due to other remote working solutions, such as collaboration platforms, VPNs, and remote desktops. Similar changes were observed by measuring REDIMadrid , a European educational and research network, which connects 16 independent universities and research centers in the metropolitan region of Madrid. A drop of up to 55% in traffic volume on working days during the pandemic was observed. Similar to findings for ISP/IXP networks, it was observed that working days and weekend days are becoming more similar in terms of total traffic. The hourly traffic patterns reveal a traffic increase between 9 pm and 7 am. This could be due to users working more frequently at unusual times, but also potentially caused by overseas students (mainly from Latin America and East Asia as suggested by the AS numbers from which these connections came from) who accessed university network resources from their home countries. Given the fact that the users of the academic network (e.g., students and research staff) had to leave the campus as a response to lockdown measures, also the traffic in and out (i.e., ingress and egress) ratio changed drastically. Prior to the lockdown, the incoming traffic was much larger then the outgoing traffic. This changed to a more balanced ratio. This change of traffic asymmetry can be explained by the nature of remote work. On the one end, users connected to the network services mainly to access resources, hence the increase in outgoing traffic. On the other end, all external (i.e., Internet-based) resources requested during work were no longer accessed from the educational network but from the users' homes.

Mobile Networks and Mobility Mobile network data usage appeared to decline following the imposition of localized lockdown measures, as these reduced typical levels of mobility and roaming. measured the cellular network of O2 UK to evaluate how the changes in people's mobility impacted traffic patterns. By analyzing cellular network signalling information regarding users' device mobility activity, they observed a decrease of 50% in mobility (according to different mobility metrics) in the UK during the lockdown period. As they found no correlation between this reduction in mobility and the number of confirmed COVID-19 cases, only the enforced government order was effective in significantly reducing mobility and this reduction was more significant in densely populated urban areas than in rural areas. For London, specifically, it could be observed from the mobile network data that approximately 10% of the residents temporarily relocated during the lockdown. These mobility changes had immediate implications in traffic patterns of the cellular network. The downlink data traffic volume aggregated for all bearers (including conversational voice) decreased for all UK by up to 25% during the lockdown period. This correlates with the reduction in mobility that was observed country-wide, which likely resulted in people relying more on broadband residential Internet access to run download intensive applications such as video streaming. The observed decrease in the radio cell load, with a reduction of approximately 15% across the UK after the stay-at-home order, further corroborates the drop in cellular connectivity usage. The total uplink data traffic volume, on the other hand, experienced little changes (between -7% and +1,5%) during lockdown. This was mainly due to the increase of 4G voice traffic (i.e., VoLTE) across the UK that peaked at 150% after the lockdown compared to the national medial value before the pandemic, thus compensating for the decrease in data traffic in the uplink. Finally, it was also observed that mobility changes have a different impact on network usage in geodemographic area clusters. In densely populated urban areas, a significantly higher decrease of mobile network usage (i.e., downlink and uplink traffic volumes, radio load and active users) was observed than in rural areas. In the case of London, this was likely due to geodemographics of the central districts, which include many seasonal residents (e.g., tourists), business and commercial areas.

A Deeper Look at Interconnections Traffic at points of network interconnection noticeably increased, but most operators reacted quickly by rapidly adding additional capacity . The amount of increases varied, with some networks that hosted popular applications such as video conferencing experiencing traffic growth of several hundred to several thousand percent. At the IXP-level, it was observed that port utilization increased. This phenomenon is mostly explained by a higher traffic demand from residential users. Measurements of interconnection links at major US ISPs by CAIDA and MIT found some evidence of diurnal congestion around the March 2020 timeframe , but most of this congestion disappeared in a few weeks, which suggests that operators indeed took steps to add capacity or otherwise mitigate the congestion.

Cloud Platforms Cloud infrastructure played a key role in supporting bandwidth-intensive video conferencing and remote learning tools to practise social distancing during the COVID-19 pandemic. Network congestion between cloud platforms and access networks could impact the quality of experience of these cloud-based applications. CAIDA leveraged web-based speed test servers to perform download and upload throughput measurements from virtual machines in public cloud platforms to various access ISPs in the United States . The key findings included:

Persistent congestion events were not widely observed between cloud platforms and these networks, particular for large-scale ISPs, but we could observe large diurnal download throughput variations in peak hours from some locations to the cloud.
There was evidence of persistent congestion in the egress direction to regional ISPs serving suburban areas in the U.S. Their users could have suffered from poor video streaming or file download performance from the cloud.
The macroscopic analysis over 3 months (June-August, 2020) revealed downward trends in download throughput from ISPs and educational networks to certain cloud regions. We believe that increased use of the cloud in the pandemic could be one of the factors that contributed to the decreased performance.

Last-Mile Congestion The last mile is the centerpiece of broadband connectivity, where poor last-mile performance generally translates to poor quality of experience. In a recent IMC'20 research paper Fontugne et al. investigated last-mile latency using traceroute data from RIPE Atlas probes located in 646 ASes and looked for recurrent performance degradation . They found that in normal times Atlas probes in only 10% ASes experience persistent last-mile congestion, but they recorded 55% more congested ASes during the COVID-19 outbreak. This deterioration caused by stay-at-home measures is particularly marked in networks with a very large number of users and certain parts of the world. They found Japan to be the most impacted country in their study looking specifically at NTT OCN, but noting similar observations for several Japanese networks, including IIJ (AS2497). From mid-2020 onwards, they however observed better performance than before the pandemic. In Japan, this was partly due to the deployments originally planned for accommodating the Tokyo Olympics, and more generally, it reflects the efforts of network operators to cope with these exceptional circumstances. The pandemic has demonstrated that its adaptive design and proficient community can keep the Internet operational during such unprecedented events. Also, from the numerous research and operational reports recently published, the pandemic is apparently shaping a more resilient Internet, as Nietzsche wrote, "What does not kill me makes me stronger".

User Behaviour The type of traffic needed by the users also changed in 2020. Upstream traffic increased due the use of video conferences, remote schooling, and similar applications. The NCTA and Comcast reported that while downstream traffic grew 20%, upstream traffic grew as much as 30% to 37% . Vodafone reported that upstream traffic grew 100% in some markets . Ericsson's Consumer Lab surveyed users for their usage and experiences during the crisis. Some of the key findings in were:

9 in 10 users increased Internet activities, and time spent connected increased. In addition, 1 in 5 started new online activities, many in the older generation felt that they were helped by video calling, parents felt that their children's education was helped, and so on.
Network performance was, in general, found satisfactory. 6 in 10 were very satisfied with fixed broadband, and 3 in 4 felt that mobile broadband was the same or better compared to before the crisis. Consumers valued resilience and quality of service as the most important task for network operators.
Smartphone application usage changed, with fastest growth in apps related to COVID-19 tracking and information, remote working, e-learning, wellness, education, remote health consultation, and social shared experience applications. Biggest decreases were in travel and booking, ride hailing, location, and parking applications.

Some of the behaviours are likely permanent changes . The adoption of video calls and other new services by many consumers, such as the older generation, is likely going to have a long-lasting effect. Surveys in various organizations point to a likely long-term increase in the number of people interested in remote work .

Operational Practices and Architectural Considerations The second and third day of the workshop were held based on more open discussions focussed on operational issues and the architectural issues arising or other conclusions that could be reached.

Digital Divide Measurements from Fastly confirmed that Internet traffic volume, in multiple countries, rose rapidly at the same time as COVID cases increased and lockdown policies came into effect. Download speeds also decreased, but in a much less dramatic fashion than overall bandwidth usage increased. School closures led to a dramatic increase in traffic volume in many regions, and other public policy announcements triggered large traffic shifts. This suggests that governments might usefully coordinate with operators to allow time for pre-emptive operational changes, in some cases. Measurements from the US showed that download rates correlate with income levels. However, download rates in the lowest income zip codes increased as the pandemic progressed, closing the divide with higher income areas. One possible reason for this in the data is decisions by some ISPs, such as Comcast and Cox, that increased speeds for users on lower-cost certain plans and in certain areas. This suggests that network capacity was available, and that the correlation between income and download rates was not necessarily due to differences in the deployed infrastructure in different regions; although it was noted that certain access link technologies provide more flexibility than others in this regard.

Applications The web conferencing systems (e.g., Microsoft Teams, Zoom, Webex) saw incredible growth, with overnight traffic increases of 15-20% in response to public policy changes, such as lockdowns. This required significant and rapid changes in infrastructure provisioning. Major video providers (YouTube, etc.) reduced bandwidth by 25% in some regions. It was suggested that this had a huge impact on quality of videoconferencing systems until networks could scale to handle full bit-rate, but other operators of some other services saw limited impact. Updates to popular games has a significant impact on network load. Some discussions were reported between ISPs, CDNs, and the gaming industry on possibly coordinating various high-bandwidth update events, similar to what was done for entertainment/video download speeds. There was an apparently difficult interplay between bulk download and interactive real-time applications, potentially due to buffer bloat and queuing delays. It was noted that operators have experience of rapid growth of Internet traffic. New applications with exponential growth are not that unusual in the network, and the traffic spike due to the lockdown was not that unprecedented for many. Many operators have tools and mechanisms to deal with this. Ensuring that knowledge if shared is a challenge. Following these observations traffic prioritisation was discussed, starting from DSCP marking, basically wondering if a minimal priority marking scheme would have helped during the pandemic, e.g. by allowing marking of less-than-best-effort traffic. That discussion quickly devolved into a more general QoS and observability discussion, and as such also touching on the effects of increased encryption. The group was not, unsurprisingly, able to resolve the different perspectives and interests involved in that, but the discussion demonstrated that progress is made (and being less heated).

Observability It is clear that there is a contrast in experience. Many operators reported few problems, in terms of metrics such as measured download bandwidth, while video conferencing applications experienced significant usability problems running on those networks. The interaction between application providers and network providers worked very smoothly to resolve these issues, supported by strong personal contacts and relationships. But it seems clear that the metrics used by many operators to understand their network performance don't fully capture the impact on certain applications, and there is an observability gap. Do we need more tools to figure out the various impacts on user experience? These types of applications use surprising amounts of Forward Error Correction (FEC). Applications hide lots of loss to ensure a good user experience. This makes it harder to observe problems. The network can be behaving poorly, but experience can be good enough. Resiliency measures can improve the user experience but hide severe problems. There may be a missing feedback loop between application developers and operators. It's clear that it's difficult for application providers and operators to isolate problems. Is a problem due to the local WiFi, the access network, cloud network, etc.? Metrics from access points would help, but in general lack of observability into the network as a whole is a real concern when it comes to debugging performance issues. Further, it's clear that it can be difficult to route problem reports to the person who can fix them, across multiple networks in the Internet. COVID-enhanced cooperation made it easier to debug problems; lines of communication are important.

Security The increased threats and network security impacts arising from COVID-19 fall into two areas: (1) the agility of malicious actors to spin up new campaigns using COVID-19 as a lure, and (2) the increased threat surface from a rapid shift towards home working. During 2020, there was a shift to home working generally, and in the way in which people use the network, with IT departments rolling out new equipment quickly and using technologies like VPNs for the first time, while others put existing solutions under much greater load. As VPN technology became more widespread and more used, it arguably became a more valuable target; one Advanced (APT29) was successful in using recently published exploits in a range of VPN software to gain initial footholds. Of all scams detected by the UK NCSC (United Kingdom National Cyber Security Centre) that purported to originate from UK Government, more related to COVID-19 than any other subject. There are other reports of a strong rise in phishing, fraud, and scams related to COVID . Although, from the data seen to date, the overall levels of cyber crime have not increased, there was certainly a shift in activity - as both the NCSC and CISA (DHS Cybersecurity and Infrastructure Security Agency) saw a growing use of COVID-19 related themes by malicious cyber actors as a lure. Attackers used COVID-19 related scams and phishing emails to target: individuals, small and medium businesses, large organisations, and organisations involved in both national and international COVID-19 responses (healthcare bodies, pharmaceutical companies, academia and medical research organisations). New targets, for example organisations involved in COVID-19 vaccine development were attacked using VPN exploits, highlighting the potential consequences of vulnerable infrastructure. It's unclear how to effectively detect and counter these attacks at scale. Approaches such as using Indicators of Compromise and crowd-sourced flagging of suspicious emails were found to be effective in the response to COVID-19-related scams, and observing DNS to detect malicious use is widespread and effective. The use of DNS over HTTPS offers privacy benefits but current deployment models can bypass these existing protective DNS measures. It was also noted that when everyone moves to performing their job online, lack of understanding of security becomes a bigger issue. Is it reasonable to expect every user of the Internet to take password training? Or is there a fundamental problem with a technical solution? Modern advice advocates a layered approach to security defences, with user education forming just one of those layers. Communication platforms such as Zoom are not new: many people have used them for years, but as COVID-19 saw an increasing number of organisations and individuals turning to these technologies, they became an attractive target, due to increased usage. In turn, there was an increase in malicious cyber actor activity, either hijacking online meetings that were not secured with passwords or leveraging unpatched software as an attack vector. How can new or existing measures protect users from the attacks levied against the next vulnerable service? Overall, it may be that there were fewer security challenges than expected arising from many people suddenly working from home. However, the agility of attackers, the importance of robust and scalable defence mechanisms, and some existing security problems and challenges may have become even more obvious and acute with an increased use of Internet-based services, particularly in a pandemic situation and times of uncertainty, where users can be more vulnerable to social engineering techniques and attacks.

Discussion There is a concern that we're missing observability for the network as a whole. Each application provider and operator has their own little lens. No-one has the big-picture view of the network. How much of a safety margin do we need? Some of the resiliency comes from us not running the network too close to its limit. This allows traffic to shift, and gives headroom for the network to cope. The best effort nature of the network may help here. Techniques to run the network closer to its limits improve performance in the usual case, but highly optimised networks may be less robust. Finally, it was observed that we get what we measure. There may be an argument for operators to shift their measurement focus perhaps away from pure capacity, to rather measure QoE or resilience. The Internet is a critical infrastructure, and people are realising that now. We should use this as a wake-up-call to improve resilience, both in protocol design and operational practice, not necessarily to optimise for absolute performance or quality of experience.

Conclusions There is a wealth of data about the performance of the Internet during the crisis. The main conclusion from the various measurements is that fairly large shifts occurred. And those shifts were not merely about changing one application for another, they actually impacted traffic flows and directions, and caused in many cases a significant traffic increase. Early reports also seem to indicate that the shifts have gone relatively smoothly from the point of view of overall consumer experience. An important but not so visible factor that led to this was that many people and organizations where highly motivated to ensure good experience. A lot of collaboration happened in the background, problems were corrected, many providers significantly increased their capacity, and so on. On the security front, the COVID-19 crisis showcased the agility with which malicious actors can move in response to a shift in user Internet usage, and the vast potential of the disruption and damage that they can inflict. Equally, it showed the agility of defenders, when they have access to the tools and information they need to protect users and networks, and showcased the power of Indicators of Compromise when defenders around the world are working together against the same problem. In general, the Internet also seems well suited for adapting to new situations, at least within some bounds. The Internet is designed for flexibility and extensibility, rather than optimized for today's particular traffic. This makes it possible to use it for many applications, in many deployment situations, and make changes as needed. The generality is present in many parts of the overall system, from basic Internet technology to browsers, from name servers to content delivery networks and cloud platforms. When usage changes, what is needed is often merely different services, perhaps some re-allocation of resources, as well as consequent application and continuation of existing security defences, but not fundamental technology or hardware changes. On the other hand, this is not to say that no improvements are needed:

Improve observability to better understand the health of the Internet: Going forward, the critical nature that the Internet plays in our lives means that the health of the Internet needs to receive significant attention. Understanding how well networks work is not just a technical matter, it is also of crucial importance to the people and economy of the societies using it. Projects and research that monitor Internet and services performance in a broad scale and across different networks are therefore important.
Maintaining defensive mechanisms to be used in times of crisis: Malicious cyber actors are continually adjusting their tactics to take advantage of new situations, and the COVID-19 pandemic is no exception. Malicious actors used the high appetite for COVID-19 related information as an opportunity to deliver malware and ransomware, and to steal user credentials. Against the landscape of a shift to working from home and an increase in users vulnerable to attack, and as IT departments were often overwhelmed by rolling out new infrastructure and devices, IoC sharing was a vital part of the response to COVID-19 related scams and attacks.
The pandemic has shown how the effects of the digital divide can be amplified during a crisis. More attention is needed to ensure that broadband is available to all, and that Internet services equally serve different groups.
We need to continue to work on all the other improvements that are seen as necessary anyway, such as further improvements in security, ability for networks and applications to collaborate better, etc.
Informal collaboration between different parties needs to continue and be strengthened.