Open AccessPosted Content
Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
John J. Miller,Rohan Taori,Aditi Raghunathan,Shiori Sagawa,Pang Wei Koh,Vaishaal Shankar,Percy Liang,Yair Carmon,Ludwig Schmidt +8 more
TLDR
In this article, the authors empirically show that out-of-distribution performance is strongly correlated with the performance of a wide range of models and distribution shifts and provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.Abstract:
For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.read more
Citations
More filters
Posted Content
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Andreas Fürst,Elisabeth Rumetshofer,Viet Hung Tran,Hubert Ramsauer,Fei Tang,Johannes M. Lehner,David P. Kreil,Michael K Kopp,Günter Klambauer,Angela Bitto-Nemling,Sepp Hochreiter +10 more
TL;DR: This article proposed contrastive leave-one-out boost (CLOOB) which replaces the original embedding by retrieved embeddings in the InfoLOOB objective, which stabilizes the Info-Lob objective.
Posted Content
On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias.
TL;DR: The authors theoretically and empirically show that MLM pretraining makes models robust to lexicon-level spurious features, and they also explore the efficacy of pretrained masked language models in causal settings.
Proceedings ArticleDOI
On the Robustness of Reading Comprehension Models to Entity Renaming
TL;DR: Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, Xiang Ren as mentioned in this paper , 2019 Conference of the Association for Computational Linguistics: Human Language Technologies.
Posted Content
On the Robustness of Reading Comprehension Models to Entity Renaming.
TL;DR: The authors proposed a general and scalable method to replace person names with names from a variety of sources, ranging from common English names to names from other languages to arbitrary strings, and found that this can further improve the robustness of MRC models.
References
More filters
Book
Monocular Model-Based 3D Tracking of Rigid Objects: A Survey
Vincent Lepetit,Pascal Fua +1 more
TL;DR: This survey reviews the different techniques and approaches that have been developed by industry and research on 3D tracking and includes a comprehensive study of the massive literature on the subject.
Posted Content
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Dan Hendrycks,Steven Basart,Norman Mu,Saurav Kadavath,Frank Wang,Evan Dorundo,Rahul Desai,Tyler Zhu,Samyak Parajuli,Mike Guo,Dawn Song,Jacob Steinhardt,Justin Gilmer +12 more
TL;DR: It is found that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work.
Proceedings Article
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
TL;DR: This paper standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications, and proposes a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations.
Proceedings Article
Certified Adversarial Robustness via Randomized Smoothing
TL;DR: In this paper, randomized smoothing is used to obtain an ImageNet classifier with a certified top-1 accuracy of 49% under adversarial perturbations with less than 0.5.
Proceedings ArticleDOI
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
Battista Biggio,Fabio Roli +1 more
TL;DR: A thorough overview of the evolution of this research area over the last ten years and beyond is provided, starting from pioneering, earlier work on the security of non-deep learning algorithms up to more recent work aimed to understand the security properties of deep learning algorithms, in the context of computer vision and cybersecurity tasks.