We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

/pdf/imagenet-classification-with-deep-convolutional-neural-1190x9a6b9.pdf

ImageNet Classification with Deep Convolutional Neural Networks

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.

/pdf/dropout-a-simple-way-to-prevent-neural-networks-from-4gtr92okhv.pdf

Dropout: a simple way to prevent neural networks from overfitting

A continuous weight-update device for a synaptic element including at least one MOS transistor comprises a floating node having a capacitance associated therewith, the floating gate comprising at least a part of the floating node, first and second input lines, first and second error lines, an electron tunneling structure coupled to the floating node for tunneling electrons from the floating node, and an electron injecting structure coupled to the floating node for injecting electrons onto the floating node. Control circuitry is responsive to signals on the first input and error lines, for activating the electron tunneling structure, and control circuitry is responsive to signals on the second input and error lines, for activating the electron injecting structure. Circuitry is provided for driving signals onto the first and second input and error lines. Both a single synapse and an array of synapses incorporating the continuous weight-update device are also taught.

Continuous synaptic weight update mechanism

A linear voltage-to-current converter (LVCC) circuit includes two transistors, one P-channel and one N-channel. The input voltage is applied to the gates of both transistors. The drains of the two transistors are connected. The source of the p-type transistor is connected to a first voltage rail, and the source of the N-channel is connected to a second voltage rail of lower voltage. The output is the difference between the current through the P-channel transistor and the N-channel transistor. A linear current-to-voltage converter (LCVC) circuit is similar to the LVCC circuit, except that the gates of the transistors are tied to the drains of the transistors. The input current is supplied to the drains, and the output voltage is the voltage of the drains.

Linear, continuous-time, two quadrant multiplier

There is a rapidly increasing interest in crowdsourcing for data labeling. By crowdsourcing, a large number of labels can be often quickly gathered at low cost. However, the labels provided by the crowdsourcing workers are usually not of high quality. In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels. Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty. We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle. We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels.

Regularized Minimax Conditional Entropy for Crowdsourcing

A two-layer network according to the present invention is comprised of a first-layer array of electrically-adaptable synaptic elements, inter-layer connection circuitry comprised of electrically adaptable elements, and a second-layer array of electrically-adaptable synaptic elements. Electrons may be placed onto and removed from a floating node associated with at least one MOS transistor in each electrically adaptable element, usually comprising the gate of the transistor, in an analog manner, by application of first and second electrical control signals. A first electrical control signal controls the injection of electrons onto the floating node from an electron injection structure and the second electrical control signal controls the removal of electrons from the floating node by an electron removal structure. Each synaptic element in the synaptic array comprises an adaptable CMOS inverter or other amplifier circuit. The inputs to all first-layer synaptic elements in a row are connected to a common row input line. Adapt inputs to all synaptic elements in a column are connected together to a common column adapt line. The outputs of all first layer synaptic elements in a column are connected to a common sense amplifier on a sense line. The outputs of the sense amplifiers are connected to the inputs of the synaptic elements of the second layer of the array. The outputs of all synaptic elements in a given row in the second layer of the array are connected to a common row output line. In order to adapt the synaptic elements in the array, the voltages to which the synaptic elements in a given column of the first layer of the array is to be adapted are placed onto the input voltage lines, and the synaptic elements in column n are then simultaneously adapted by assertion of an adapt signal on the adapt line for the column. The voltages to which the synaptic elements of the second layer of the array are to be adapted are placed on the row outputs lines.

Two layer neural network comprised of neurons with improved input range and input offset

A linear voltage-to-current converter (LVCC) circuit includes two transistors, one P-channel and one N-channel The input voltage is applied to the gates of both transistors The drains of the two transistors are connected The source of the p-type transistor is connected to a first voltage rail, and the source of the N-channel is connected to a second voltage rail of lower voltage The output is the difference between the current through the P-channel transistor and the N-channel transistor A linear current-to-voltage converter (LCVC) circuit is similar to the LVCC circuit, except that the gates of the transistors are tied to the drains of the transistors The input current is supplied to the drains, and the output voltage is the voltage of the drains

John Platt

Papers

Continuous synaptic weight update mechanism

Linear, continuous-time, two quadrant multiplier

Regularized Minimax Conditional Entropy for Crowdsourcing

Two layer neural network comprised of neurons with improved input range and input offset

Circuits for linear conversion between voltages and currents