David Krueger

TL;DR: The analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

TL;DR: A deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE) is proposed, based on the idea that a good representation is one in which the data has a distribution that is easy to model.

TL;DR: Non-linear Independent Component Estimation (NICE) as discussed by the authors is a deep learning framework for modeling complex high-dimensional densities based on the idea that a good representation is one in which the data has a distribution that is easy to model.

TL;DR: This work introduces the principle of Risk Extrapolation (REx), and shows conceptually how this principle enables extrapolation, and demonstrates the effectiveness and scalability of instantiations of REx on various OoD generalization tasks.

TL;DR: The authors examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness, showing that deep networks tend to prioritize learning simple patterns first.