HUGE Congratulations to MOMENT Lab members Udit Paul and Jiamo Liu on receiving the Distinguished Paper Award (Best Long Paper) at ACM IMC'22! Wow!!! This work is so important and timely given the recent focus on crowdsourced speed test measurements for policy-related decision-making. Our paper argues that using these measurements without understanding their context is problematic. Though the community had internalized that speed-test results can be misleading, no one knew the extent of skews in such crowdsourced datasets. One significant roadblock in contextualizing these measurements was to infer their subscription tier (10 Mbps vs. 100 Mbps plan). A speed test result of 10 Mbps is just fine if the user subscribed to a 10 Mbps plan, but it is quite problematic if it subscribed to a 100 Mbps plan. We developed a novel BST methodology that infers the subscription tier for crowdsourced measurements. Applying this methodology, we quantify the skews in public (FCC’s MBA, M-Lab) and private (Ookla) crowdsourced datasets. Our analysis shows how lack of context contributes to misleading conclusions. For instance, the connection type (Ethernet vs. WiFi), the WiFi band and signal strength, the end-user device resources, and even the specific speed test vendor (Ookla or M-Lab) all significantly impact the performance that is achieved. We conclude with a set of recommendations for speed test vendors and the FCC to contextualize speed test data and correctly interpret measured performance. Check out the paper!
Thu, 10/27/2022