Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

/pdf/deep-residual-learning-for-image-recognition-b75jg61qrq.pdf

Deep Residual Learning for Image Recognition

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. 
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

[Artificial intelligence: a solution for the lack of pathologists?]

Künstliche Intelligenz als Lösung des PathologInnenmangels?

Looking at the past 30 years of research into the so-called P300 BCI reveals an almost exponential growth of publications on this topic as indexed in Pubmed. The striking increase has started around 2010 and is currently plateauing at a high rate. Certain aspects of the P300, such as stimulus presentation, feedback modality, classification procedures, or application to be controlled have been intensively investigated. The bulk of studies comprises one session approaches in the laboratory environment with healthy subjects, but patient end-users are increasingly included in studies. However, a lack of long-term studies with end-users in the field is obvious and indicates a translational gap. On the basis of more than 300 studies included, we discuss reasons for this gap and propose remedies; however, this paper presents a coarse overview only and constitutes the basis for a thorough meta-analysis. INTRODUCTION The motivation for this overview arose from the observation that a strikingly high amount of manuscripts to be reviewed by the authors do not sufficiently take into account and build upon existing results within the field of the P300 BCI. The foundations of the P300 BCI were laid with the first description of the P300 by [1]  an event-related potential that could be elicited in a so-called oddball paradigm with rare target and frequent standard stimuli. In 1988 Farwell and Donchin [2] implemented the oddball paradigm in a stimulation set-up to control a BCI. Letters and numbers were presented in a matrix on a monitor. Rows and columns of the matrix were flashed in random order and participants were required to focus attention on the letter to be selected. Focusing of attention was reinforced by asking the participants to count how often the target letter flashed. By this procedure, each letter to be selected becomes a rare target in comparison to all other letters, which turn into standard stimuli. Ever since, this paradigm has been adopted, adapted, changed, and extended in many ways, and reliable target selection in healthy subjects and patients with disease alike has been demonstrated in a plethora of studies. The primary aim of the P300 BCI has been communication and control to replace lost function in patients with severe motor impairment. Only recently have other aspects, such as for rehabilitation after post-stroke aphasia, i.e. to improve lost function (see the BNCI Horizon 2020 website for a categorization of BCI according to their application purpose), gained more attention [e.g., 3]. Despite the many studies aiming at improving different aspects of the P300 BCI such as accuracy, information transfer rate (ITR), or usability, P300 BCIs are not used in daily life by the targeted end-users with disease albeit most studies claim this to be the final goal and motivation for the experiment at hand. Thus, we face two gaps: (1) a translational gap, which prevents the many positive results arriving at the end-users home and (2) an awareness gap, i.e. researchers sub-optimally build on the results achieved by the community. Therefore, approaches to the P300-BCI and controlled applications remain idiosyncratic rather than evoking a joint effort toward bridging the translational gap. In the following we will give a brief overview of the topics covered by P300 BCI related publications (Table 1) and then discuss questions and answers which we consider relevant for the field of the P300 BCI. Those may partially transfer to BCIs with other input signals. MATERIALS AND METHODS We used Pubmed as source for our literature search on the P300 BCI. We entered the search terms Brain AND Computer AND Interface. This search yielded n=733 hits. Those were then screened for true P300 BCI content and studies in languages other than English were excluded (n=12). Based on the abstracts, studies were then categorized according to the participants included (healthy subjects vs. participants with disease), environment of data collection (laboratory vs. field), number of (daily) sessions (one vs. more than one). Other categories were: use of existing data sets, reviews, and theory and frameworks. During the process of categorization, the category sample unclear had to be created because it could not always be derived from the abstract what kind of sample was included. . Proceedings of the 7th Graz Brain-Computer Interface Conference 2017 DOI: 10.3217/978-3-85125-533-1-49

The P300 BCI: on its Way to End-Users?

Machine-learning force fields (MLFF) should be accurate, computationally and data efficient, and applicable to molecules, materials, and interfaces thereof. Currently, MLFFs often introduce tradeoffs that restrict their practical applicability to small subsets of chemical space or require exhaustive datasets for training. Here, we introduce the Bravais-Inspired Gradient-Domain Machine Learning (BIGDML) approach and demonstrate its ability to construct reliable force fields using a training set with just 10-200 geometries for materials including pristine and defect-containing 2D and 3D semiconductors and metals, as well as chemisorbed and physisorbed atomic and molecular adsorbates on surfaces. The BIGDML model employs the full relevant symmetry group for a given material, does not assume artificial atom types or localization of atomic interactions and exhibits high data efficiency and state-of-the-art energy accuracies (errors substantially below 1 meV per atom) for an extended set of materials. Extensive path-integral molecular dynamics carried out with BIGDML models demonstrate the counterintuitive localization of benzene--graphene dynamics induced by nuclear quantum effects and allow to rationalize the Arrhenius behavior of hydrogen diffusion coefficient in a Pd crystal for a wide range of temperatures.

BIGDML: Towards Exact Machine Learning Force Fields for Materials.

The rational design of molecules with desired properties is a long-standing challenge in chemistry. Generative neural networks have emerged as a powerful approach to sample novel molecules from a learned distribution. Here, we propose a conditional generative neural network for 3d molecular structures with specified structural and chemical properties. This approach is agnostic to chemical bonding and enables targeted sampling of novel molecules from conditional distributions, even in domains where reference calculations are sparse. We demonstrate the utility of our method for inverse design by generating molecules with specified composition or motifs, discovering particularly stable molecules, and jointly targeting multiple electronic properties beyond the training regime.

Klaus-Robert Müller

Papers

[Artificial intelligence: a solution for the lack of pathologists?]

Künstliche Intelligenz als Lösung des PathologInnenmangels?

The P300 BCI: on its Way to End-Users?

BIGDML: Towards Exact Machine Learning Force Fields for Materials.

Inverse design of 3d molecular structures with conditional generative neural networks.