Sampling Bias & Caveats

PUTTING OUR WORK INTO CONTEXT

Sampling Bias

• Bias towards large cities. One of the drawbacks of using open call-for-service data is that the places most likely to provide open data access are larger cities. Presumably this is attributed to the cost and/or overhead of preparing data for public consumption. The large number of datasets that are housed on paid platforms seems to support this idea.

• Possible bias towards more wealthy cities. Though we did not attempt to confirm this bias, it seems plausible that the cities represented in this review skew towards more wealthy due to the fact that additional effort and technology is required to share call-for-service data publicly.

• Geographic distribution. As shown below the distribution of cities is not even across the country. Only one city was included with open data from outside the 48 contiguous states. Thirteen states did not have any representation at all and no datasets from US territories were included. This was not intensional, but we also took no steps to correct our sampling since this sample set met our initial needs.

The 127 datasets we used represented 37 states.

Caveats

• This is an informal exploration. Although we spent a lot of time, care, and energy in our work our intension was not to perform primary research or to draw conclusions about the 911 system.

• Our objective was to prototype a data aggregation and standardization system. The strengths that Code for America brings to this work are volunteer mobilization and the ability to create purpose-built software. While our underlying hypothesis remains that a multi-jurisdictional 911 dataset will be valuable to researchers and stakeholders in the 911 system, our primary goal for 2021-2022 is to develop a viable method for aggregating and standardizing data. Our objective for this exploratory phase is not to produce or support conclusions about the workings of the 911 system.

• We have acquired valuable knowledge about open call-for-service data. Although our primary objective is to show the viability of using crowdsourcing to generate actionable multi-city datasets, we have also learned a lot about open 911 data in the United States. We have published this knowledge base specifically to share our learnings with other entities interested in the viability of using open call-for-service data.