Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

Cascade-Correlation is a new architecture and supervised learning algorithm for artificial neural networks. Instead of just adjusting the weights in a network of fixed topology. Cascade-Correlation begins with a minimal network, then automatically trains and adds new hidden units one by one, creating a multi-layer structure. Once a new hidden unit has been added to the network, its input-side weights are frozen. This unit then becomes a permanent feature-detector in the network, available for producing outputs or for creating other, more complex feature detectors. The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.

/pdf/the-cascade-correlation-learning-architecture-i72409omez.pdf

The Cascade-Correlation Learning Architecture

Fundamental developments in feedforward artificial neural networks from the past thirty years are reviewed. The history, origination, operating characteristics, and basic theory of several supervised neural-network training algorithms (including the perceptron rule, the least-mean-square algorithm, three Madaline rules, and the backpropagation technique) are described. The concept underlying these iterative adaptation algorithms is the minimal disturbance principle, which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. The two principal kinds of online rules that have developed for altering the weights of a network are examined for both single-threshold elements and multielement networks. They are error-correction rules, which alter the weights of a network to correct error in the output response to the present input pattern, and gradient rules, which alter the weights of a network during each pattern presentation by gradient descent with the objective of reducing mean-square error (averaged over all training patterns). >

/pdf/30-years-of-adaptive-neural-networks-perceptron-madaline-and-3p7ko3fnas.pdf

30 years of adaptive neural networks: perceptron, Madaline, and backpropagation

Artificial neural networks (ANNs) constitute a class of flexible nonlinear models designed to mimic biological neural systems. In this entry, we introduce ANN using familiar econometric terminology and provide an overview of ANN modeling approach and its implementation methods. † Correspondence: Chung-Ming Kuan, Institute of Economics, Academia Sinica, 128 Academia Road, Sec. 2, Taipei 115, Taiwan; ckuan@econ.sinica.edu.tw. †† I would like to express my sincere gratitude to the editor, Professor Steven Durlauf, for his patience and constructive comments on early drafts of this entry. I also thank Shih-Hsun Hsu and Yu-Lieh Huang for very helpful suggestions. The remaining errors are all mine.

Artificial neural networks

From the Publisher:
Connectionist Speech Recognition: A Hybrid Approach describes the theory and implementation of a method to incorporate neural network approaches into state-of-the-art continuous speech recognition systems based on Hidden Markov Models (HMMs) to improve their performance. In this framework, neural networks (and in particular, multilayer perceptrons or MLPs) have been restricted to well-defined subtasks of the whole system, i.e., HMM emission probability estimation and feature extraction. The book describes a successful five year international collaboration between the authors. The lessons learned form a case study that demonstrates how hybrid systems can be developed to combine neural networks with more traditional statistical approaches. The book illustrates both the advantages and limitations of neural networks in the framework of a statistical system. Using standard databases and comparing with some conventional approaches, it is shown that MLP probability estimation can improve recognition performance. Other approaches are discussed, though there is no such unequivocal experimental result for these methods. Connectionist Speech Recognition: A Hybrid Approach is of use to anyone intending to use neural networks for speech recognition or within the framework provided by an existing successful statistical approach. This includes research and development groups working in the field of speech recognition, both with standard and neural network approaches, as well as other pattern recognition and/or neural network researchers. This book is also suitable as a text for advanced courses on neural networks or speech processing.

Connectionist Speech Recognition: A Hybrid Approach

Parallelizable optimization techniques are applied to the problem of learning in feedforward neural networks. In addition to having superior convergence properties, optimization techniques such as the Polak-Ribiere method are also significantly more efficient than the Backpropagation algorithm. These results are based on experiments performed on small boolean learning problems and the noisy real-valued learning problem of hand-written character recognition.

/pdf/efficient-parallel-learning-algorithms-for-neural-networks-73pap4c7fj.pdf

Efficient Parallel Learning Algorithms for Neural Networks

Disclosed is a fault tolerant CMOS image sensor that includes circuitry for identifying defective pixels and masking them during image generation. Masking may involve, in one example, replacing the output of a given pixel with an average of the output of surrounding non-faulty pixels. Thus, while image sensors may be fabricated with some number of faulty pixels, the images produced by such sensors will not have undesirable bright or dark spots. The disclosed sensor includes (a) one or more pixels (active or passive) capable of providing outputs indicative of a quantity of radiation to which each of the one or more pixels has been exposed; and (b) one or more circuit elements electrically coupled to the one or more pixels and configured to identify and correct faulty pixels in the CMOS imager. The one more pixels each include a photodiode diffusion formed in a well and a tap to power or ground also formed in the well. The disclosed sensor also identifies pixels that were initially acceptable but later became defective. The newly defective pixels so identified may then be masked to thereby increase the CMOS detector lifetime.

Pixel correction system and method for CMOS imagers

An integrated circuit (IC) package includes at least one light source disposed together with an IC structure within an encapsulation structure. The material forming the encapsulation structure is generally opaque. Accordingly, the light source and at least a portion of the IC are not visible to the unaided human eye. The thickness and geometry of the encapsulation are such that when the light source is caused to emit light, the encapsulation structure permits at least a portion of that light to be visible to a user. The IC may be a portion of a fingerprint sensor, exposed for receiving a fingertip of a user. The light source, or a plurality of such light sources, may be functional (e.g., providing visual indication of the of the condition or state of an item of hardware or software, assist the user in operation of a device), be primarily aesthetic, or a combination thereof.

Pseudo-Translucent Integrated Circuit Package

An improved fingerprint sensing device is provided with multiple sensing apparatus, two or more of which operating on different sensing principles. For example, a capacitive sensor may be integrally formed with an optical sensor on a single substrate. Ideally, the multiple sensing apparatus are disposed such that they may sense nearly identical portions of a fingerprint simultaneously. A primary sensor may be employed to establish the identity of a user based on a fingerprint, while a secondary sensor may be employed to establish that the fingerprint is part of a live human (anti-spoofing). Integrated light sources may be provided to drive an optical sensor. The light sources may also provide visual cues for usage, and enhance the aesthetics of the device.

Hybrid multi-sensor biometric identification device

An integrated leadframe and bezel structure includes a planar carrier frame, a plurality of bonding leads, a die pad region, and a bezel structure. The bezel structure includes a bending portion shaped and disposed to facilitate a portion of said bezel structure being bent out of the plane of said carrier frame. A sensor IC may be secured to the die pad region, and wire bonds made to permit external connection to the sensor IC. The bezel structure includes portions which are bent such that their upper extent is in or above a sensing surface. The assembly is encapsulated, exposing on the top surface part of the bezel portions and the upper surface of the sensor IC, and on the bottom surface the contact pads. Two or more bezel portions may be provided, one or more on each side of the sensor IC.

Alan Kramer

Papers

Efficient Parallel Learning Algorithms for Neural Networks

Pixel correction system and method for CMOS imagers

Pseudo-Translucent Integrated Circuit Package

Hybrid multi-sensor biometric identification device

Integrated leadframe and bezel structure and device formed from same