Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

/pdf/deep-residual-learning-for-image-recognition-b75jg61qrq.pdf

Deep Residual Learning for Image Recognition

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.

Long short-term memory

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. 
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

This chapter contains sections titled: Classes of PDP Models, Specific Versions of the General Parallel Activation Model, Sigma-Pi Units, Conclusion, Acknowledgments

A general framework for parallel distributed processing

Explorations in parallel distributed processing

ion for this example. We have taken such features as has television as a microfeature. In a more realistic example, we would expect television to itself be a particular pattern over a set of units that are used to represent many different varieties of television. There might be many variations on the television pattern corresponding to variations among televisions. Moreover, since televisions in bedrooms may be systematically different (perhaps smaller) than televisions in living rooms , we would expect that these correlations would be picked up and there would be a context dependency between the particular version of television and the remaining objects in the room. In such a case the units that participate in the representation of television would play the role of a slot in a schema, and the particular pattern of activation on these units would represent the characteristics of the slot filler. Figure 6 shows several examples of the processing of this network. These runs started by " clamping " one of the descriptors on (that is , by setting the value to 1 and not letting it change) and then letting the system find a goodness-of-fit maximum. In the first example , the descriptor oven was clamped on. In such a case , we expect that the system will bring those units most tightly bound to the oven unit on and turn off those units negatively correlated to oven or other units that it turns on. On the assumption that oven is a central feature of the kitchen schema the pattern the system eventually turns on is just that which might be said to correspond to the default kitchen schema. The strengths of each of the 40 units is shown along with the " goodness-of-fit" of the state after every 20 updates. The system begins with oven and ceiling on and then adds coffee-cup (weakly), then sink and refrigerator concludes that the room is small adds toaster and coffeepot and finally ends up at a maximum with ceiling, walls, window, small, telephone, clock, coffee-cup, drapes, stove, sink, refrigerator, toaster, cupboard, coffeepot and oven. other words, it finds the default or prototype kitchen. Similarly, runs of the system starting with desk, bathtub, sofa or bed clamped lead to goodness maxima corresponding to the prototype or default office PSYCHOLOGICAL PROCESSES bathroom , living room , or bedroom , respectively. It is, as previously noted , these maxima that we believe correspond roughly to instantiations of schemata for kitchens , offices , bathrooms , living rooms, and bedrooms. The system receives input in the form of having some of the descriptors clamped from the outside. It then finds the best interpretation of the input through this process of hill climbing. As it climbs, the system " fills in " the relevant descriptors of the scene in question. In the case of the network we created for this example, there are essentially five maxima-one corresponding to each of the different room types. There are 240 possible binary states in which the system could potentially settle, but, in fact , when it is started by clamping exactly one descriptor, it will only settle into one of five states. This roughly corresponds to the view that this data base contains five schemata defined over the set of 40 descriptors. There are , as we shall see numerous subschemata which involve subpatterns within the whole pattern. 00000000000000000000 g6~puter I f~~Yhange bathtub television dresser coffee-pot cupboard toaster r~frigerator: sink stove 00 drapes . .. fire-place ash-tray . a a 00000000000000000 coffee-cup , easy-chair sofa floor-lamp picture . 00000000000000 clock .. ...... ... Idesk-chair. I books , carpet ! bookshelf . . . i typewriter .. ... .....

Schemata and sequential thought processes in PDP models

It is demonstrated how a modality-specific semantic memory system can account for category-specific impairments after brain damage. In Experiment 1, the hypothesis that visual and functional knowledge play different roles in the representation of living things and nonliving things is tested and confirmed. A parallel distributed processing model of semantic memory in which knowledge is subdivided by modality into visual and functional components is described. In Experiment 2, the model is lesioned, and it is confirmed that damage to visual semantics primarily impairs knowledge of living things, and damage to functional semantics primarily impairs knowledge of nonliving things. In Experiment 3, it is demonstrated that the model accounts naturally for a finding that had appeared problematic for a modality-specific architecture, namely, impaired retrieval of functional knowledge about living things. Finally, in Experiment 4, it is shown how the model can account for a recent observation of impaired knowledge of living things only when knowledge is probed verbally.

https://apps.dtic.mil/sti/pdfs/ADA242377.pdf

A computational model of semantic memory impairment: modality specificity and emergent category specificity.

This book presents the official, formal definition of the programming language ML including the rules for grammar and static and dynamic semantics. ML is the most well-developed and prominent of a new group of functional programming languages. On the cutting edge of theoretical computer science, ML embodies the ideas of static typing and polymorphism and has also contributed a number of novel ideas to the design of programming languages.Contents: Syntax of the Core. Syntax of Modules. Static Semantics for the Core. Static Semantics for Modules. Dynamic Semantics for Modules. Programs.Appendixes: Derived Forms. Full Grammar. The Initial Static Basis. The Initial Dynamic Basis. The Development of ML.

http://web.stanford.edu/class/psych209a/ReadingsByDate/01_11/handbook_ch1_and_2.pdf

James L. McClelland

Papers

A general framework for parallel distributed processing

Explorations in parallel distributed processing

Schemata and sequential thought processes in PDP models

A computational model of semantic memory impairment: modality specificity and emergent category specificity.

Explorations in parallel distributed processing: a handbook of models, programs, and exercises