In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Computing numeric representations of words in a high-dimensional space

Provided is a method and system for indexing documents in a collection of linked documents. A link log, including one or more pairings of source documents and target documents is accessed. A sorted anchor map, containing one or more target document to source document pairings, is generated. The pairings in the sorted anchor map are ordered based on target document identifiers.

Anchor tag indexing in a web crawler system

We compare several algorithms for identifying mirrored hosts on the World Wide Web. The algorithms operate on the basis of URL strings and linkage data: the type of information about Web pages easily available from Web proxies and crawlers. Identification of mirrored hosts can improve Web-based information retrieval in several ways: first, by identifying mirrored hosts, search engines can avoid storing and returning duplicate documents. Second, several new information retrieval techniques for the Web make inferences based on the explicit links among hypertext documents—mirroring perturbs their graph model and degrades performance. Third, mirroring information can be used to redirect users to alternate mirror sites to compensate for various failures, and can thus improve the performance of Web browsers and proxies. We evaluated four classes of “top-down” algorithms for detecting mirrored host pairs (that is, algorithms that are based on page attributes such as URL, IP address, and hyperlinks between pages, and not on the page content) on a collection of 140 million URLs (on 230,000 hosts) and their associated connectivity information. Our best approach is one which combines five algorithms and achieved a precision of 0.57 for a recall of 0.86 considering 100,000 ranked host pairs.

A comparison of techniques to find mirrored hosts on the WWW

A method for sampling the performance of a computer system is provided. The computer system includes a plurality of functional units. The method selects transactions to be processed by a particular functional unit of the computer system. State information is stored while the selected transactions are processed by the functional unit. The state information is analyzed to guide optimization.

Apparatus and method for monitoring a computer system to guide optimization

Whole-Program Optimization of Object-Oriented Languages by Jeffrey Adgate Dean Chairperson of the Supervisory Committee: Professor Craig Chambers Department of Computer Science and Engineering This dissertation examines the use of whole-program optimization as a way of improving the performance of object-oriented programming languages. Although object-oriented programming conveys a number of software engineering benefits, heavy application of its trademark feature, dynamic dispatching, imposes severe performance penalties when programs are compiled using traditional compilation techniques. Several new techniques that rely on whole-program optimization are described, and these techniques substantially improve the performance of object-oriented programs written in Cecil, Java, C++, and Modula-3. Among the new techniques is class hierarchy analysis, which provides the compiler with knowledge of the class hierarchy of the entire program. This is an especially important optimization, because it allows programmers to write their programs using dynamic dispatching for all operations, which preserves maximal flexibility and extensibility, but permits the compiler to optimize away this flexibility when it is unused by a particular program. Exhaustive class testing allows the compiler to optimize message sends that have a limited degree of polymorphism by inserting tests to partition the potential classes of objects that can appear at a call site. A selective specialization algorithm combines static information with dynamic profile data to determine where it is profitable to compile multiple, specialized versions of a single source routine. Inlining trials provide a means of making inlining decisions that take into account the effects of post-inlining optimizations. Whole-program optimization in an interactive programming environment is made possible through the use of selective recompilation, a technique for invalidating only compiled code that is affected by a programming change. The techniques described in the thesis have been implemented in the Vortex compiler, a wholeprogram optimizing compiler developed as part of this dissertation. Whole-program not only speeds up existing features of object-oriented languages, but it also allows new features to be added to languages with little or no cost, enabling more general models of dispatching that result in more natural and uniform programming languages. The benefits of the optimization techniques described in this thesis are shown to already be important, and their importance is likely to only increase as people adopt more and more object-oriented programming styles.

https://researcher.watson.ibm.com/researcher/files/us-groved/TR-UW-CSE-96-06-02.pdf

Jeffrey Dean

Papers

Computing numeric representations of words in a high-dimensional space

Anchor tag indexing in a web crawler system

A comparison of techniques to find mirrored hosts on the WWW

Apparatus and method for monitoring a computer system to guide optimization

Whole-program optimization of object-oriented languages