In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

다수의 하드웨어 디바이스에 대한 기계 학습 모델 연산용 배치를 결정하는 방법이 기술된다. 이 방법은 다수의 하드웨어 디바이스상에서 분산 처리를 위해 배치될 기계 학습 모델을 특정하는 데이터를 수신하는 단계와; 데이터로부터, 연산 임베딩의 시퀀스를 생성하는 단계로서, 그 시퀀스내의 각 연산 임베딩은 기계 학습 모델의 처리를 수행하는데 필요한 하나 이상의 개별 연산을 특징화하고; 배치 순환 신경망의 복수의 네트워크 파라미터의 복수의 네트워크 파라미터의 제1 값에 따라 배치 순환 신경망을 사용하여 연산 임베딩 시퀀스를 처리하여, 복수의 디바이스에 대한 시퀀스내의 연산 임베딩에 의해 특징화된 연산들의 배치를 정의하는 네트워크 출력을 생성하는 단계와; 네트워크 출력에 의해 정의된 배치에 따라 다수의 디바이스상에 연산들을 배치함으로써 다수의 하드웨어 디바이스에 의해 처리하기 위한 기계 학습 모델을 스케줄링하는 단계를 포함한다.

강화 학습을 통한 디바이스 배치 최적화

PROBLEM TO BE SOLVED: To provide a method for estimating statistics of characteristics of a memory subsystem. SOLUTION: By a method for analyzing a memory transaction processed through a memory 120 of the computer system 100, one group of addresses of a memory is selected. While the selected transaction is processed through the memory 120, state information from specific successive transactions is recorded to the selected addresses. The selection and recording stages are repeated until end conditions are coincident. Then the recorded state information is statistically analyzed to estimate statistics of characteristics of memory interaction between contexts of the computer system.

Method for estimating statistic of characteristic of memory system interaction between contexts in computer system

PROBLEM TO BE SOLVED: To provide a method for estimating a statistical value for the mutual operation of a memory sub-system. SOLUTION: In a method for estimating the statistical value of the characteristic of a transaction processed by the memory sub-system of a computer system, the memory transaction processed by the memory sub-system is selected at random. While the selected transaction is processed by the memory sub-system, the state of the system is recorded as a sample. The recorded state from the sub-set of the selected transaction is statistically analyzed and the statistical value of the memory transaction is estimated.

Method for estimating statistical value of characteristic of memory system transaction

Optimizing compilers for object-oriented languages apply static class analysis and other techniques to try to deduce precise information about the possible classes of the receivers of messages; if successful, dynamically-dispatched messages can be replaced with direct procedure calls and potentially further optimized through inline-expansion. By examining the complete inheritance graph of a program, which we call class hierarchy analysis, the compiler can improve the quality of static class information and thereby improve run-time performance. In this paper we present class hierarchy analysis and describe techniques for implementing this analysis effectively in both statically- and dynamically-typed languages and also in the presence of multi-methods. We also discuss how class hierarchy analysis can be supported in an interactive programming environment and, to some extent, in the presence of separate compilation. Finally, we assess the bottom-line performance improvement due to class hierarchy analysis alone and in combination with two other "competing" optimizations, profile-guided receiver class prediction and method specialization.

Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis.

컴퓨터 저장 매체상에 인코딩된 컴퓨터 프로그램들을 포함하는 방법들, 시스템들 및 장치들은, 계산 그래프를 프로세싱하기 위해 클라이언트로부터 요청을 수신하고; 상기 계산 그래프를 나타내는 데이터를 획득하고 -상기 계산 그래프는 복수의 노드들 및 방향 에지들을 포함하고, 각 노드는 각각의 오퍼레이션을 나타내고, 각각의 방향 에지는 각각의 제1 노드를 상기 각각의 제1 노드에 의해 표현되는 오퍼레이션의 출력을 입력으로서 수신하는 오퍼레이션을 나타내는 각각의 제2 노드에 연결함-; 요청된 오퍼레이션을 수행하기 위해 복수의 이용 가능한 디바이스들을 식별하고; 상기 계산 그래프를 복수의 서브그래프들로 분할하고 -서브그래프 각각은 상기 계산 그래프 내의 하나 이상의 노드들을 포함함-; 그리고 각각의 서브그래프에 대해, 상기 서브그래프 내의 상기 하나 이상의 노드들에 의해 표현된 오퍼레이션들을 오퍼레이션을 위한 복수의 이용 가능한 디바이스들 내의 각각의 이용 가능한 디바이스에 할당하는 것을 포함한다.

Jeffrey Dean

Papers

강화 학습을 통한 디바이스 배치 최적화

Method for estimating statistic of characteristic of memory system interaction between contexts in computer system

Method for estimating statistical value of characteristic of memory system transaction

Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis.

계산 그래프들 프로세싱