Open AccessPosted Content
Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
John J. Miller,Rohan Taori,Aditi Raghunathan,Shiori Sagawa,Pang Wei Koh,Vaishaal Shankar,Percy Liang,Yair Carmon,Ludwig Schmidt +8 more
TLDR
In this article, the authors empirically show that out-of-distribution performance is strongly correlated with the performance of a wide range of models and distribution shifts and provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.Abstract:
For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.read more
Citations
More filters
Posted Content
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Andreas Fürst,Elisabeth Rumetshofer,Viet Hung Tran,Hubert Ramsauer,Fei Tang,Johannes M. Lehner,David P. Kreil,Michael K Kopp,Günter Klambauer,Angela Bitto-Nemling,Sepp Hochreiter +10 more
TL;DR: This article proposed contrastive leave-one-out boost (CLOOB) which replaces the original embedding by retrieved embeddings in the InfoLOOB objective, which stabilizes the Info-Lob objective.
Posted Content
On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias.
TL;DR: The authors theoretically and empirically show that MLM pretraining makes models robust to lexicon-level spurious features, and they also explore the efficacy of pretrained masked language models in causal settings.
Proceedings ArticleDOI
On the Robustness of Reading Comprehension Models to Entity Renaming
TL;DR: Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, Xiang Ren as mentioned in this paper , 2019 Conference of the Association for Computational Linguistics: Human Language Technologies.
Posted Content
On the Robustness of Reading Comprehension Models to Entity Renaming.
TL;DR: The authors proposed a general and scalable method to replace person names with names from a variety of sources, ranging from common English names to names from other languages to arbitrary strings, and found that this can further improve the robustness of MRC models.
References
More filters
Journal ArticleDOI
A Survey on Transfer Learning
Sinno Jialin Pan,Qiang Yang +1 more
TL;DR: The relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift are discussed.
Proceedings ArticleDOI
Rethinking the Inception Architecture for Computer Vision
TL;DR: In this article, the authors explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization.
Dissertation
Learning Multiple Layers of Features from Tiny Images
TL;DR: In this paper, the authors describe how to train a multi-layer generative model of natural images, using a dataset of millions of tiny colour images, described in the next section.
Journal ArticleDOI
Squeeze-and-Excitation Networks
TL;DR: This work proposes a novel architectural unit, which is term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels and finds that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at minimal additional computational cost.
Proceedings ArticleDOI
Xception: Deep Learning with Depthwise Separable Convolutions
TL;DR: This work proposes a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions, and shows that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset, and significantly outperforms it on a larger image classification dataset.