In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Techniques for crawling hyperlinked documents are provided. Hyperlinked documents to be crawled are grouped by host and the host to be crawled next is selected according to a stall time of the host. The stall time can indicate the earliest time that the host should be crawled and the stall times can be a predetermined amount of time, vary by host and be adjusted according to actual retrieval times from the host.

Distributed crawling of hyperlinked documents

We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods In ENAS, a controller learns to discover neural network architectures by searching for an optimal path within a larger model The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set Meanwhile the model corresponding to the selected path is trained to minimize the cross entropy loss On the Penn Treebank dataset, ENAS can discover a novel architecture thats achieves a test perplexity of 578, which is state-of-the-art among automatic model design methods on Penn Treebank On the CIFAR-10 dataset, ENAS can design novel architectures that achieve a test error of 289%, close to the 265% achieved by standard NAS (Zoph et al, 2017) Most importantly, our experiments show that ENAS is more than 10x faster and 100x less resource-demanding than NAS

/pdf/faster-discovery-of-neural-architectures-by-searching-for-3yrjceost3.pdf

Faster Discovery of Neural Architectures by Searching for Paths in a Large Model

A method of processing a request, performed by a respective server, is provided in which a request is received from a client. After receiving the request, a determination is made as to whether at least a first predefined number of other servers have a task-processing status for the request indicating that the other servers have undertaken performance of a task-processing operation for the request. When less than the first number of other servers in the set of other servers have the task-processing status for the request, a processing-status message is sent to one or more of the servers in the set of other servers indicating that the respective server is performing the task-processing operation. Upon completion of the task-processing, a result of the processing is sent to the client contingent upon a status of the other servers in the set of other servers.

Redundant data requests with cancellation

A distributed storage system is provided. The distributed storage system includes multiple front-end servers and zones for managing data for clients. Data within the distributed storage system is associated with a plurality of accounts and divided into a plurality of groups, each group including a plurality of splits, each split being associated with a respective account, and each group having multiple tablets and each tablet managed by a respective tablet server of the distributed storage system. Data associated with different accounts may be replicated within the distributed storage system using different data replication policies. There is no limit to the amount of data for an account by adding new splits to the distributed storage system. In response to a client request for a particular account's data, a front-end server communicates such request to a particular zone that has the client-requested data and returns the client-requested data to the requesting client.

Jeffrey Dean

Papers

Distributed crawling of hyperlinked documents

Faster Discovery of Neural Architectures by Searching for Paths in a Large Model

Redundant data requests with cancellation

Organizing Data in a Distributed Storage System

Classifying data objects