Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: this http URL.

Self-Normalizing Neural Networks

Overview.- An Overview of Empirical Processes.- Overview of Semiparametric Inference.- Case Studies I.- Empirical Processes.- to Empirical Processes.- Preliminaries for Empirical Processes.- Stochastic Convergence.- Empirical Process Methods.- Entropy Calculations.- Bootstrapping Empirical Processes.- Additional Empirical Process Results.- The Functional Delta Method.- Z-Estimators.- M-Estimators.- Case Studies II.- Semiparametric Inference.- to Semiparametric Inference.- Preliminaries for Semiparametric Inference.- Semiparametric Models and Efficiency.- Efficient Inference for Finite-Dimensional Parameters.- Efficient Inference for Infinite-Dimensional Parameters.- Semiparametric M-Estimation.- Case Studies III.

Introduction to empirical processes and semiparametric inference

For simple random walk on aN-vertex graph, the mean time to cover all vertices is at leastcN log(N), wherec>0 is an absolute constant. This is deduced from a more general result about stationary finite-state reversible Markov chains. Under weak conditions, the covering time for such processes is at leastc times the covering time for the corresponding i.i.d. process.

Lower bounds for covering times for reversible Markov chains and random walks on graphs

This is an update of, and a supplement to, a 1986 survey paper by the author on basic
properties of strong mixing conditions.

/pdf/basic-properties-of-strong-mixing-conditions-a-survey-and-3evi1rp4rq.pdf

Basic properties of strong mixing conditions. A survey and some open questions

We discuss briefly the very interesting concept of Brownian distance covariance developed by Szekely and Rizzo (2009) and describe two possible extensions. The first extension is for high dimensional data that can be coerced into a Hilbert space, including certain high throughput screening and functional data settings. The second extension involves very simple modifications that may yield increased power in some settings. We commend Szekely and Rizzo for their very interesting work and recognize that this general idea has potential to have a large impact on the way in which statisticians evaluate dependency in data.

/pdf/brownian-distance-covariance-1xjy2lylae.pdf

Brownian distance covariance

Introduction to strong mixing conditions

This is a survey of the basic properties of strong mixing conditions for sequences of random variables. The focus will be on the “structural” properties of these conditions, and not at all on limit theory. For a discussion of central limit theorems and related results under these conditions, the reader is referred to Peligrad [60] or Iosifescu [50]. This survey will be divided into eight sections, as follows: 

1.

Measures of dependence




2.

Five strong mixing conditions




3.

Mixing conditions for two or more sequences




4.

Mixing conditions for Markov chains




5.

Mixing conditions for Gaussian sequences




6.

Some other special examples




7.

The behavior of the dependence coefficients




8.

Approximation of mixing sequences by other random sequences.

Basic Properties of Strong Mixing Conditions

For strictly stationary random fields indexed by $\mathbb{R}^d$ or $\mathbb{Z}^d$, certain versions of the "strong mixing" condition are equivalent to corresponding versions of the "$\rho$-mixing" condition.

/pdf/equivalent-mixing-conditions-for-random-fields-2pq7i9pfud.pdf

Richard C. Bradley

Papers

Basic properties of strong mixing conditions. A survey and some open questions

Introduction to strong mixing conditions

Basic Properties of Strong Mixing Conditions

Equivalent Mixing Conditions for Random Fields

Multilinear forms and measures of dependence between random variables