Showing papers on "Convolutional neural network published in 1998"

Gradient-based learning applied to document recognition

Abstract: Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Evaluation of convolutional neural networks for visual recognition

Abstract: Convolutional neural networks provide an efficient method to constrain the complexity of feedforward neural networks by weight sharing and restriction to local connections. This network topology has been applied in particular to image classification when sophisticated preprocessing is to be avoided and raw images are to be classified directly. In this paper two variations of convolutional networks-neocognitron and a modification of neocognitron-are compared with classifiers based on fully connected feedforward layers with respect to their visual recognition performance. For a quantitative experimental comparison with standard classifiers two very different recognition tasks have been-chosen: handwritten digit recognition and face recognition. In the first example, the generalization of convolutional networks is compared to fully connected networks; in the second example human face recognition is investigated under constrained and variable conditions, and the limitations of convolutional networks are discussed.

Boxlets: A Fast Convolution Algorithm for Signal Processing and Neural Networks

Abstract: Signal processing and pattern recognition algorithms make extensive use of convolution. In many cases, computational accuracy is not as important as computational speed. In feature extraction, for instance, the features of interest in a signal are usually quite distorted. This form of noise justifies some level of quantization in order to achieve faster feature extraction. Our approach consists of approximating regions of the signal with low degree polynomials, and then differentiating the resulting signals in order to obtain impulse functions (or derivatives of impulse functions). With this representation, convolution becomes extremely simple and can be implemented quite effectively. The true convolution can be recovered by integrating the result of the convolution. This method yields substantial speed up in feature extraction and is applicable to convolutional neural networks.

Face Recognition: A Hybrid Neural Network Approach

Abstract: Faces represent complex, multidimensional, meaningful visual stimuli and developing a computa- tional model for face recognition is difficult (Turk and Pentland, 1991). We present a hybrid neural network solution which compares favorably with other methods. The system combines local image sam- pling, a self-organizing map neural network, and a convolutional neural network. The self-organizing map provides a quantization of the image samples into a topological space where inputs that are nearby in the original space are also nearby in the output space, thereby providing dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides for partial invariance to translation, rotation, scale, and deformation. The convolutional network extracts successively larger features in a hierarchical set of layers. We present results using the Karhunen-Loe transform in place of the self-organizing map, and a multilayer perceptron in place of the convolu- tional network. The Karhunen-Lo` eve transform performs almost as well (5.3% error versus 3.8%). The multilayer perceptron performs very poorly (40% error versus 3.8%). The method is capable of rapid classification, requires only fast, approximate normalization and preprocessing, and consistently exhibits better classification performance than the eigenfaces approach (Turk and Pentland, 1991) on the database considered as the number of images per person in the training database is varied from 1 to 5. With 5 images per person the proposed method and eigenfaces result in 3.8% and 10.5% error respectively. The recognizer provides a measure of confidence in its output and classification error approaches zero when rejecting as few as 10% of the examples. We use a database of 400 images of 40 individuals which con- tains quite a high degree of variability in expression, pose, and facial details. We analyze computational complexity and discuss how new classes could be added to the trained recognizer.

Application of Artificial Neural Networks to Medical Image Pattern Recognition: Detection of Clustered Microcalcifications on Mammograms and Lung Cancer on Chest Radiographs

Abstract: Three neural network models were employed to evaluate their performances in the recognition of medical image patterns associated with lung cancer and breast cancer in radiography. The first method was a pattern match neural network. The second was a conventional backpropagation neural network. The third method was a backpropagation trained neocognitron in which the signal propagation is operated with the convolution calculation from one layer to the next. In the convolution neural network (CNN) experiment, several output association methods and trainer imposed driving functions in conjunction with the convolution neural network are proposed for general medical image pattern recognition. An unconventional method of applying rotation and shift invariance is also used to enhance the performance of the neural nets. We have tested these methods for the detection of microcalcifications on mammograms and lung nodules on chest radiographs. Pre-scan methods were previously described in our early publications. The artificial neural networks act as final detection classifiers to determine if a disease pattern is presented on the suspected image area. We found that the convolution neural network, which internally performs feature extraction and classification, achieves the best performance among the three neural network models. These results show that some processing associated with disease feature extraction is a necessary step before a classifier can make an accurate determination.

Structured neural networks for pattern recognition

Abstract: This paper proposes a novel approach for the design of structures of neural networks for pattern recognition. The basic idea lies in subdividing the whole classification problem in smaller and simpler problems at different levels, each managed by appropriate components of a complex neural architecture. Three neural structures are presented and applied in a surveillance system aimed at monitoring a railway waiting room classifying potential dangerous situations. Each architecture is composed by nodes, which are actual multilayer perceptrons trained to discriminate between subsets of classes until a complete separation among the classes is achieved. This approach showed better performances with respect to a classical statistical classification procedures and to a single neural network.

Effect of Batch Learning in Multilayer Neural Networks.

Abstract: This paper discusses batch gradient descent learning in mul-tilayer networks with a large number of statistical training data. We emphasize on the diierence between regular cases, where the prepared model has the same size as the true function , and overrealizable cases, where the model has surplus hidden units to realize the true function. First, experimental study on multilayer perceptrons and linear neural networks (LNN) shows that batch learning induces strong overtrain-ing on both models in overrealizable cases, which means the degrade of generalization error by surplus units can be alleviated. We theoretically analyze the dynamics in LNN, and show that this overtraining is caused by shrinkage of the parameters corresponding to surplus units.

Design of neural nets for character recognition

Abstract: In this paper, the possibility of using Artificial Neural Networks (ANNs) in the field of character recognition is discussed. Our study is undertaken on theoretical and practical investigations of two feedforward models (the Prototype Multilayer Perceptron (MLP) and the Fully Connected model) by using the backpropagation training algorithm. We introduce a fully connected network of three layers in order to make a classification between two characters T and C without being affected by shift in position, rotation, or scaling. A complete analog implementation is presented by using D-MOS transistors acting as synaptic weights and bipolar transistors to represent the nonlinear sigmoid function. Simulation results for fully connected networks are compared with those of traditional techniques (prototype MLP model) in order to recognize more characters.

A new method to construct FNN with linear output neurons

Abstract: In this paper, a new network-growing method for a multilayer feedforward neural network (FNN) is proposed. It has the following distinctive features: 1) The network starts training with a small network and gradually grows its hidden neurons. 2) The activation function of its output neurons is a linear function. Moreover, its application in pattern recognition is also discussed. Simulation results show that the new algorithm achieves a higher recognition rate and converges faster than the conventional backpropagation algorithm and it can avoid the trap of local minima through increasing the hidden neurons.