Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

/pdf/deep-residual-learning-for-image-recognition-b75jg61qrq.pdf

Deep Residual Learning for Image Recognition

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. 
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Scene graph prediction --- classifying the set of objects and predicates in a
visual scene --- requires substantial training data. The long-tailed
distribution of relationships can be an obstacle for such approaches, however,
as they can only be trained on the small set of predicates that carry
sufficient labels. We introduce the first scene graph prediction model that
supports few-shot learning of predicates, enabling scene graph approaches to
generalize to a set of new predicates. First, we introduce a new model of
predicates as functions that operate on object features or image locations.
Next, we define a scene graph model where these functions are trained as
message passing protocols within a new graph convolution framework. We train
the framework with a frequently occurring set of predicates and show that our
approach outperforms those that use the same amount of supervision by 1.78 at
recall@50 and performs on par with other scene graph models. Next, we extract
object representations generated by the trained predicate functions to train
few-shot predicate classifiers on rare predicates with as few as 1 labeled
example. When compared to strong baselines like transfer learning from existing
state-of-the-art representations, we show improved 5-shot performance by 4.16
recall@1. Finally, we show that our predicate functions generate interpretable
visualizations, enabling the first interpretable scene graph model.

Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction

Processor enabled products, each with its own user interface, have pervaded everyday life. To facilitate interaction design exploration for novel devices, we have developed d.tools, a design tool for prototyping the bits and the atoms of physical user interfaces in concert. It enables designers without specialized engineering or programming knowledge to quickly build functional interactive prototypes. d.tools offers a visual authoring environment that allows for drag-and-drop specification of interaction models for tangible user interfaces in a matter of minutes.

https://hci.stanford.edu/publications/2005/hartmann-dtools-uist05.pdf

Visually Prototyping Physical UIs through Statecharts

Creative tools today strive to amplify our ability to create high-quality work. However, experiencing failure is also an important part of mastering creative skills. While experts have developed strategies for engaging in risky experiments and learning from mistakes, novices lack the experience and mindset needed to use failures as opportunities for growth. Current tools intimidate the unsure novice, as they are designed around showcasing success or critiquing finished work, rather than providing safe spaces for experimentation. To better support experiences of failure for novices, we instead propose flipping the value of failure in creativity tools from something to avoid to something to pursue actively. To do this, we develop a taxonomy of creative activities that people engage in when they aim to succeed. We then invert this taxonomy to derive a new set of creative activities where deliberate failure can provide a path towards creative confidence. Lastly, we envision possible creativity support tools as examples of the potential value of supporting activities where failure is encouraged and showcased.

Designing Creativity Support Tools for Failure

Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data. To scale and widen the applicability of crowdsourcing, we present a technique that produces extremely rapid judgments for binary and categorical labels. Rather than punishing all errors, which causes workers to proceed slowly and deliberately, our technique speeds up workers' judgments to the point where errors are acceptable and even expected. We demonstrate that it is possible to rectify these errors by randomizing task order and modeling response latency. We evaluate our technique on a breadth of common labeling tasks such as image verification, word similarity, sentiment analysis and topic classification. Where prior work typically achieves a 0.25x to 1x speedup over fixed majority vote, our approach often achieves an order of magnitude (10x) speedup.

Embracing Error to Enable Rapid Crowdsourcing

A diverse representation of different demographic groups in AI training data sets is important in ensuring that the models will work for a large range of users. To this end, recent efforts in AI fairness and inclusion have advocated for creating AI data sets that are well-balanced across race, gender, socioeconomic status, and disability status. In this paper, we contribute to this line of work by focusing on the representation of age by asking whether older adults are represented proportionally to the population at large in AI data sets. We examine publicly-available information about 92 face data sets to understand how they codify age as a case study to investigate how the subjects' ages are recorded and whether older generations are represented. We find that older adults are very under-represented; five data sets in the study that explicitly documented the closed age intervals of their subjects included older adults (defined as older than 65 years), while only one included oldest-old adults (defined as older than 85 years). Additionally, we find that only 24 of the data sets include any age-related information in their documentation or metadata, and that there is no consistent method followed across these data sets to collect and record the subjects' ages. We recognize the unique difficulties in creating representative data sets in terms of age, but raise it as an important dimension that researchers and engineers interested in inclusive AI should consider.

Michael S. Bernstein

Papers

Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction

Visually Prototyping Physical UIs through Statecharts

Designing Creativity Support Tools for Failure

Embracing Error to Enable Rapid Crowdsourcing

Understanding the Representation and Representativeness of Age in AI Data Sets