Identifying Statistical Bias in Dataset Replication
Logan Engstrom,u00a0Andrew Ilyas,u00a0Shibani Santurkar,u00a0Dimitris Tsipras,u00a0Jacob Steinhardt,u00a0Aleksander Madry
Dataset replication is a useful tool for assessing whether improvements in test accuracy on a specific benchmark correspond to improvements in modelsu2019 ability to generalize reliably. In this work, we present unintuitive yet significant ways in which standard approaches to dataset replication introduce statistical bias, skewing the resulting observations. We study ImageNet-v2, a replication of the ImageNet dataset on which models exhibit a significant (11-14%) drop in accuracy, even after controlling for selection frequency, a human-in-the-loop measure of data quality. We show that after remeasuring selection frequencies and correcting for statistical bias, only an estimated 3.6% of the original 11.7% accuracy drop remains unaccounted for. We conclude with concrete recommendations for recognizing and avoiding bias in dataset replication. Code for our study is publicly available: https://git.io/data-rep-analysis.
