Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

/pdf/deep-residual-learning-for-image-recognition-b75jg61qrq.pdf

Deep Residual Learning for Image Recognition

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. 
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

The transfer of knowledge allows to rapidly set up a functioning BCI system for a novel user without user specific calibration. For spelling paradigms based on event-related potentials (ERP) of the EEG, a recent unsupervised classification method is able to train the classifier online by constantly adapting to the user on the fly. Hence, an explicit calibration phase can be avoided. However, due to the random initialization of the classifier, this method requires a warm-up time before it performs on the same level as a supervised trained classifier. This warm-up effect can be reduced by transfer learning. We present a thorough leave-one-user-out offline analysis (n=9 users) and additional preliminary online results from a spatial auditory ERP spelling study (AMUSE paradigm) on inter-subject transfer of an unsupervised adaptive classifier. For the online study, a classifier trained on data of n=8 previous users was transferred to two unseen users and further adapted online. The performance was evaluated in one online copy spelling session per user. Both, the offline simulations and the online results indicate that the transfer approach reduces the warm-up time by approx. 50 %.

https://biblio.ugent.be/publication/5709355/file/5709379.pdf

Transferring Unsupervised Adaptive Classifiers Between Users of a Spatial Auditory Brain-Computer Interface

Introduction and method Previous neurophysiological studies of emotions have focused on the affective response in the emotional valence of a situation in which the reaction is rooted in perception or memories [1]. Furthermore emotions have been investigated with regard to the trait of a subject, e.g. anger-out vs. anger control [2] and regarding motivational direction, e.g. approach vs. withdrawal [3]. Aiming at an enhancement of human-computer interaction by incorporating the emotional state of the user, a novel type of investigation is required. Neuronal correlates of emotional reactions related to interaction (e.g. annoyance due to one's own failure or an error of the machine; joy of success) have to be analyzed and methods for their detection in real-time need to be developed. In the present study we have acquired multi-channel EEG in four subjects while they were interacting with computer applications that have been specifically designed in order to provoke – in alternating phases – neural, positive or negative (stress, annoyance) emotions. In particular, a two-player variant of a two-alternative forced-choice task had to be performed while in alternating periods either one or the other player was given "unfair" preferential treatment by providing the task stimulus slightly in advance. This bias could not be noticed by the players.

/pdf/neuronal-correlates-of-emotions-in-human-machine-interaction-16mogmdf3n.pdf

Neuronal correlates of emotions in human-machine interaction

In this short abstract I will discuss recent directions where deep learning is used for analysing brain imaging data, both in the context of BCI and fMRI – summarizing steps taken by the BBCI team and co-workers. It is the nature of this short text that many pointers to research are given all of which show a high overlap to prior own contributions (this is not only unavoidable but intentional) or will touch upon ongoing unpublished respectively pre-published work.

Explainable Deep Learning for Analysing Brain Data

Real world data often exhibit inhomogeneity, e.g., the noise level, the sampling distribution or the complexity of the target function may change over the input space. In this paper, we try to isolate local function complexity in a practical, robust way. This is achieved by first estimating the locally optimal kernel bandwidth as a functional relationship. Specifically, we propose Spatially Adaptive Bandwidth Estimation in Regression (SABER), which employs the mixture of experts consisting of multinomial kernel logistic regression as a gate and Gaussian process regression models as experts. Using the locally optimal kernel bandwidths, we deduce an estimate to the local function complexity by drawing parallels to the theory of locally linear smoothing. We demonstrate the usefulness of local function complexity for model interpretation and active learning in quantum chemistry experiments and fluid dynamics simulations.

Estimating Local Function Complexity via Mixture of Gaussian Processes

Atomization energies are an important measure of chemical stability. Machine learning is used to model atomization energies of a diverse set of organic molecules, based on nuclear charges and atomic positions only [1]. Our scheme maps the problem of solving the molecular time-independent Schrodinger equation onto a non-linear statistical regression problem. Kernel ridge regression [2] models are trained on and compared to reference atomization energies computed using density functional theory (PBE0 [3] approximation to Kohn-Sham level of theory [4,5]). We use a diagonalized matrix representation of molecules based on the inter-nuclear Coulomb repulsion operator in conjunction with a Gaussian kernel. Validation on a set of over 7000 small organic molecules from the GDB database [6] yields mean absolute error of ~10 kcal/mol, while reducing computational effort by several orders of magnitude. Applicability is demonstrated for prediction of binding energy curves using augmentation samples based on physical limits.

/pdf/modeling-of-molecular-atomization-energies-using-machine-23c21gtkfz.pdf

Klaus-Robert Müller

Papers

Transferring Unsupervised Adaptive Classifiers Between Users of a Spatial Auditory Brain-Computer Interface

Neuronal correlates of emotions in human-machine interaction

Explainable Deep Learning for Analysing Brain Data

Estimating Local Function Complexity via Mixture of Gaussian Processes

Modeling of molecular atomization energies using machine learning