We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

/pdf/imagenet-classification-with-deep-convolutional-neural-1190x9a6b9.pdf

ImageNet Classification with Deep Convolutional Neural Networks

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.

/pdf/dropout-a-simple-way-to-prevent-neural-networks-from-4gtr92okhv.pdf

Dropout: a simple way to prevent neural networks from overfitting

This paper presents a unified probabilistic framework for denoising and dereverberation of speech signals. The framework transforms the denoising and dereverberation problems into Bayes-optimal signal estimation. The key idea is to use a strong speech model that is pre-trained on a large data set of clean speech. Computational efficiency is achieved by using variational EM, working in the frequency domain, and employing conjugate priors. The framework covers both single and multiple microphones. We apply this approach to noisy reverberant speech signals and get results substantially better than standard methods.

/pdf/speech-denoising-and-dereverberation-using-probabilistic-37045bqguz.pdf

Speech Denoising and Dereverberation Using Probabilistic Models

Red-eye detection based on red region detection with eye confirmation initially identifies pixels that correspond to the color of red-eye within an image. A determination is then made as to whether these identified pixels and surrounding areas are part of an eye or not part of an eye. Those identified pixels that are determined to be part of an eye are the detected red-eye regions.

Red-eye detection based on red region detection with eye confirmation

A “Music Mapper” automatically constructs a set coordinate vectors for use in inferring similarity between various pieces of music In particular, given a music similarity graph expressed as links between various artists, albums, songs, etc, the Music Mapper applies a recursive embedding process to embed each of the graphs music entries into a multi-dimensional space This recursive embedding process also embeds new music items added to the music similarity graph without reembedding existing entries so long a convergent embedding solution is achieved Given this embedding, coordinate vectors are then computed for each of the embedded musical items The similarity between any two musical items is then determined as either a function of the distance between the two corresponding vectors In various embodiments, this similarity is then used in constructing music playlists given one or more random or user selected seed songs or in a statistical music clustering process

Client-based generation of music playlists via clustering of music similarity vectors

Architecture that monitors interaction data (e.g., search queries, query results and click-through rates), and provides users with links to other users that fall into similar categories with respect to the foregoing monitored activities (e.g., providing links to individuals and groups that share common interests and/or profiles). A search engine can be interactively coupled with one or more social networks, and that maps individuals and/or groups within respective social networks to subsets of categories associated with searches. A database stores mapped information which can be continuously updated and reorganized as links within the system mapping become stronger or weaker. The architecture can comprise a social network system that includes a database for mapping search-related information to an entity of a social network, and a search component for processing a search query for search results and returning a link to an entity of a social network based on the search query.

Search engine that identifies and uses social networks in communications, retrieval, and electronic commerce

Methods and apparatus for displaying images, e.g., text characters, on display devices which have multiple separately controllable luminance elements per pixel are described. Multiple alpha values (1004) are generated per pixel, e.g., one per pixel sub-component. Filtering (1006) is then performed to generate a filtered alpha value (1008) for each pixel sub-component. The filtering operation (1006) uses as its input multiple alpha values, e.g., 5 per pixel. The alpha values (1004) used for the filtering operation (1006) may come from neighbouring pixels (e.g. 932, 934). Accordingly, in the case where character glyphs are combined to form larger images, e.g., texts strings, the filtering operation performed on glyph edge pixels depends on the content of a neighbouring character glyph. In one embodiment unfiltered alpha values (1004) used to represent character glyphs are stored in a glyph cache. After caching, the unfiltered alpha values (1004) from multiple glyphs are combined and then the filtering operation (1006) is performed. To avoid repeated filtering of character glyphs, in another embodiment, character glyphs are padded along their vertical edges, filtered and then stored in the glyph cache. Glyphs are output from the cache as required and then combined with filtered alpha values (1008) corresponding to overlapping image portions being summed. As a result of the summing processes, correct filtered alpha values are generated along glyph edges for the combined image corresponding to multiple glyphs.

John Platt

Papers

Speech Denoising and Dereverberation Using Probabilistic Models

Red-eye detection based on red region detection with eye confirmation

Client-based generation of music playlists via clustering of music similarity vectors

Search engine that identifies and uses social networks in communications, retrieval, and electronic commerce

Methods and apparatus for filtering and caching data representing images