We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

/pdf/imagenet-classification-with-deep-convolutional-neural-1190x9a6b9.pdf

ImageNet Classification with Deep Convolutional Neural Networks

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.

/pdf/dropout-a-simple-way-to-prevent-neural-networks-from-4gtr92okhv.pdf

Dropout: a simple way to prevent neural networks from overfitting

Providing for browsing a summary of content formed of keywords that can scale to a user-defined level of detail is disclosed herein. Components of a system can include a summarization component that extracts keywords related to the content and associates the keywords with portions thereof, and a zooming component that displays a number of keywords based on a keyword/keyphrase relevance rank and a zoom factor. Additionally, a speech to text component can translate speech associated with the content into text, wherein the keywords are extracted from the translated text. Consequently, the claimed subject matter can present a variable hierarchy of keywords to form a scalable summary of such recorded content.

Scalable summaries of audio or visual content

We develop physically-based graphics models of non-rigid objects capable of heat conduction, thermoelasticity, melting and fluid-like behaviour in the molten state. These deformable models feature non-rigid dynamics governed by Lagrangian equations of motion and conductive heat transfer governed by the heat equation for non-homogeneous, non-isotropic media. In its solid state, the discretized model is an assembly of hexahedral finite elements in which thermoelastic units interconnect particles situated in a lattice. The stiffness of a thermoelastic unit decreases as its temperature increases, and the unit fuses when its temperature exceeds the melting point. The molten state of the model involves a molecular dynamics simulation in which ‘fluid’ particles that have broken free from the lattice interact through long-range attraction forces and short-range repulsion forces. We present a physically-based animation of a thermoelastic model in a simulated physical world populated by hot constraint surfaces.

Heating and melting deformable models

Methods and systems for increasing the contrast with which thin-stemmed characters can be displayed in sub-pixel rendering processes are disclosed. Some fonts, such as Courier New, have relatively thin stems and other character (902) features that would otherwise be rendered with low contrast when displayed using an oversampling process combined with a rendering process that treats individual pixel sub-components as separate luminous intensity sources. In order to avoid such low contrast features and to avoid the need for revising the font itself, the alpha values for the pixel sub-components that correspond to the character (903) features are selectively increased to simulate larger dimensions of the character features and to increase the contrast with which the character dimensions are displayed. The methods involve oversampling image data representing a character by obtaining multiple samples for each of a plurality of pixel sub-components of a pixel. The alpha values associated with pixel sub-components at or near the position of the character features are increased to give the appearance of larger dimensions of the character features and greater contrast in the displayed image.

Adjusting character dimensions to compensate for low contrast character features

We have created new networks to unmix signals which have been mixed either with time delays or via filtering. We first show that a subset of the Herault-Jutten learning rules fulfills a principle of minimum output power. We then apply this principle to extensions of the Herault-Jutten network which have delays in the feedback path. Our networks perform well on real speech and music signals that have been mixed using time delays or filtering.

/pdf/networks-for-the-separation-of-sources-that-are-superimposed-2js4gurln8.pdf

Networks for the Separation of Sources that are Superimposed and Delayed

Provided is mechanism for guiding computer searches (e.g., web, Internet, intranet...). Machine learning and reasoning mechanisms are employed together with a search engine to facilitate intelligent guidance of queries and results based on a query and responses to computer generated inquires. Also, disclosed is a mechanism that enables users to validate computer-based information. Users can receive a validity metric associated with computer generated or provided results indicative of the veracity of such results. Validation systems and methods are disclosed to facilitate determining the veracity of results including those that employ humans (e.g., referrals, voting...) and/or automated means (e.g., source analysis, data mining...).

John Platt

Papers

Scalable summaries of audio or visual content

Heating and melting deformable models

Adjusting character dimensions to compensate for low contrast character features

Networks for the Separation of Sources that are Superimposed and Delayed

Intelligently guiding search based on user dialog