Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

/pdf/statistical-pattern-recognition-a-review-1kz1b0tte4.pdf

Statistical pattern recognition: a review

We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically.

On combining classifiers

From the Publisher:
Pattern recognition has long been studied in relation to many different (and mainly unrelated) applications, such as remote sensing, computer vision, space research, and medical imaging. In this book Professor Ripley brings together two crucial ideas in pattern recognition; statistical methods and machine learning via neural networks. Unifying principles are brought to the fore, and the author gives an overview of the state of the subject. Many examples are included to illustrate real problems in pattern recognition and how to overcome them.This is a self-contained account, ideal both as an introduction for non-specialists readers, and also as a handbook for the more expert reader.

Pattern recognition and neural networks

This machine uses the one-to-one correspondence between spelled-speech utterances and the printed alphabet symbols. Letter recognition was done by first extracting essential features from the information picked up by a photocell scanner and then matching these features with a stored set of data. In order to reduce the amount of memory required to generate letter sounds, redundant phonemes were eliminated and a selected set of basic phonemes was extracted by a segmentation process. Letter sounds were then synthesized by concatenation of these basic phonemes. Also vowels and vowel-like sounds have quasi-periodic waveforms. These sounds were reproduced satisfactorily by repeating over and over again a pitch period extracted from the original waveforms. These algorithms of letter and phoneme synthesis offer considerable saving of memory space and cost in constructing a digital spelled-speech reading machine for the blind. Further reduction in memory storage was accomplished by an encoding scheme. For the experiments, a computer was used to synthesize the letter sounds and to simulate this machine. Experimental results indicated that blind subjects could read spelled sentences between 60 and 70 words/m with 90-percent correct intelligibility after only 1 h of contact with the letter sounds.

Development of a Digital Spelled-Speech Reading Machine for the Blind

This paper presents linear and bilinear shape transformations including basic transformations, analyzes their geometric properties, and provides computer algorithms. The shape transformations can be used to simplify the recognition of Roman letters, Chinese characters and other pictorial patterns by normalizing their shapes to the standard forms. Important theoretical analyses have been performed to illustrate that the linear and bilinear transformations are applicable to computer recognition of digitized patterns. A number of pictorial examples have been computed to confirm the analyses and conclusions made.

Shape transformation models and their applications in pattern recognition

A fast iteration algorithm is proposed to approximate the reduced set vectors shared by each binary SVM solution for multi-class classification simultaneously. The iteration algorithm can be applied to the general kernel types such as k(/spl par/ x - x' /spl par//sup 2/) and k(x/sup T/x'). In addition, we present a fast block algorithm in the test phase to speed up the classification further. Experimental results have shown that the classification speeds on MNIST and Hanwang handwritten digit databases on P4 1.7 Ghz were about 16,000 and 10,895 patterns per second without sacrificing the classification accuracy of the original SVM system. The speed-up factor of 110 on MNIST database has been achieved.

Algorithms of fast SVM evaluation based on subspace projection

Many quantitative metrics have already been proposed to measure software complexity for different applications. A similar but different application of these measurements could also be developed for formal evaluation of expert systems (ES) which have some characteristics in common with conventional software. But they also have many distinct features of their own, for example, the rules employed in an experts system do not directly determine the execution order, instead they represent the expertise for solving problems in a general way, so the meanings implied by rules are different from those implied by conventional statements. In this paper, some problems concerning the complexity measurement of rule-based ES are discussed, they include complexity description, complexity model, criteria for measurement, and the measurement. A new approach to measure rulebase complexity is presented. It takes three factors into account: 1) content of the rulebase; 2) connectivity among the rules; and 3) size of the rulebase. In order to assess the effectiveness of this new approach, the complexity of 21 sample rulebase has been measured. The results show that this new measurement is more accurate than that of the “number of rules” of “Buchanan's solution space complexity”; suggesting that it is a better indicator of the rulebase complexity. Furthermore, the computation model of this measurement exhibits characteristics of being meaningful, reasonable, reliable and cost-effective. Another advantage is that the measurement results could be used as an estimator for maintenance or developed efforts, especially in applications where experimental data are lacking.

Measuring the complexity of rule-based expert systems

An intelligent document processing system which can process unconstrained format-free documents is presented. A new document format definition language called Document Architecture Language (DAL) which can handle both rectangular and irregular blocks is presented. Combined with pattern recognition techniques, DAL can be widely used in document analysis and understanding with very wide scopes including general newspaper articles, editorials, financial forms, advertisements, front page of magazine, etc. Bank cheques are employed as an example. >

Ching Y. Suen

Papers

Development of a Digital Spelled-Speech Reading Machine for the Blind

Shape transformation models and their applications in pattern recognition

Algorithms of fast SVM evaluation based on subspace projection

Measuring the complexity of rule-based expert systems

Document architecture language (DAL) approach to document processing