Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

/pdf/statistical-pattern-recognition-a-review-1kz1b0tte4.pdf

Statistical pattern recognition: a review

We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically.

On combining classifiers

From the Publisher:
Pattern recognition has long been studied in relation to many different (and mainly unrelated) applications, such as remote sensing, computer vision, space research, and medical imaging. In this book Professor Ripley brings together two crucial ideas in pattern recognition; statistical methods and machine learning via neural networks. Unifying principles are brought to the fore, and the author gives an overview of the state of the subject. Many examples are included to illustrate real problems in pattern recognition and how to overcome them.This is a self-contained account, ideal both as an introduction for non-specialists readers, and also as a handbook for the more expert reader.

Pattern recognition and neural networks

Sophisticated hybrid schemes of the homogeneous and heterogeneous classifiers for cursive word recognition are presented. Two homogeneous MLPs (multi­layer perceptrons) are combined into a new single powerful classifier at the architectural level, and HMM (hidden Markov model) is added to the new classifier as a heterogeneous one at the output level. This is based on the idea that classifiers with more different methodologies and different features can better complement each other. The presented scheme achieves a recognition rate of 92.7% for English legal words of a CENPARMI database, a performance which is better than several previous hybrid schemes reported in the literature.

Hybrid schemes of homogeneous and heterogeneous classifiers for cursive word recognition

We propose a new approach to detect fake coins using their images in this paper. A coin image is represented in the dissimilarity space, which is a vector space constructed by comparing the image with a set of prototypes. Each dimension measures the dissimilarity between the image under consideration and a prototype. In order to obtain the dissimilarity between two coin images, the local keypoints on each image are detected and described. Based on the characteristics of the coin, the matched keypoints between the two images can be identified in an efficient manner. A post-processing procedure is further proposed to remove mismatched keypoints. Due to the limited number of fake coins in real life, one-class learning is conducted for fake coin detection, so only genuine coins are needed to train the classifier. Extensive experiments have been carried out to evaluate the proposed approach on different data sets. The impressive results have demonstrated its validity and effectiveness.

An Image-Based Approach to Detection of Fake Coins

There are many difficulties facing a handwritten Arabic recognition system such as unlimited variation in character shapes. This paper describes a new method for handwritten Arabic character recognition. We propose a novel efficient approach for the recognition of off-line Arabic handwritten characters. The approach is based on novel preprocessing operations, structural statistical and topological features from the main body of the character and also from the secondary components. Evaluation of the importance and accuracy of the selected features was made. Our method based on the selected features and the system was built, trained and tested by CENPRMI dataset. We used SVM (RBF) and KNN for classification to find the recognition accuracy. The proposed algorithm obtained promising results in terms of accuracy; with recognition rates of 89.2% for SVM. Compared with other related works and also our recently published work we find that our result is the highest among them.

/pdf/investigating-of-preprocessing-techniques-and-novel-features-37xevqidfe.pdf

Investigating of Preprocessing Techniques and Novel Features in Recognition of Handwritten Arabic Characters

Our concern is nonlinear clustering on large-scale dataset. While existing popular kernels (RBF, Polynomials, Spatial Pyramid, etc.) are popularly used for implicitly mapping data into a high-dimensional or infinite dimensional space in order to generalise linear clustering methods, using these kernels cannot make kernel clustering approaches directly applicable for large scale dataset, since large scale kernel matrix or similarity matrix consumes a lot of memory (e.g., 7,450 GB memory over 1 million samples of data). To solve this problem, we introduce an Euler clustering approach. Euler clustering employs Euler kernels in order to intrinsically map the input data onto a complex space of the same dimension as the input or twice, so that Euler clustering can get rid of kernel trick and does not need to rely on any approximation or random sampling on kernel function/matrix, whilst performing a more robust nonlinear clustering against noise and outliers. Moreover, since the original Euler kernel cannot generate a non-negative similarity matrix and thus is inapplicable to spectral clustering, we introduce a positive Euler kernel, and more importantly we have proved when it can generate a non-negative similarity matrix. We apply Euler kernel and the proposed positive Euler kernel to kernel $k$ -means and spectral clustering so as to develop Euler $k$ -means and Euler spectral clustering, respectively. An efficient Stiefel-manifold-based gradient method and an equivalent weighted positive Euler $k$ -means are derived for fast computation of Euler spectral clustering and further alleviating the impact of discretization of the cluster membership indicators in Euler spectral clustering. The results show that the proposed Euler clustering approach achieves overall better clustering performance compared to using popular Mercer kernels and approximation models, whilst keeping the computational complexity of the same magnitude as the most popular linear clustering method $k$ -means.

Euler Clustering on Large-Scale Dataset

Robust segmentation of an iris image plays an important role in iris recognition. Most state-of-the-art iris segmentation algorithms focus on the processing of the ideal iris images that are captured in a controlled environment. In this paper, we process the unideal iris images that are acquired in an unconstrained situation and are affected severely by gaze deviation, eyelids and eyelashes occlusion, non uniform intensity, motion blur, reflections, etc. The novelty of this research effort is that we apply the modified Chan-Vese curve evolution scheme, which extracts the intensity information in local regions at a controllable scale, to find the pupil and iris boundaries accurately. A data fitting energy is defined in terms of a contour and two fitting functions that locally approximate the image intensities on the two sides of the contour. This energy is then incorporated into a variational level set formulation with a regularization term. Due to the kernel function used in energy functional, the extracted intensity information of the local regions is deployed to guide the motion of the contour, which thereby assists the curve evolution scheme to cope with the intensity inhomogeneity that occurs in the same region. The contours represented by the proposed variational level set method may break and merge naturally during evolution, and thus, the topological changes are handled automatically. The verification performance of the proposed scheme is validated using the UBIRIS Version 2, the ICE 2005, and the WVU unideal datasets.

Ching Y. Suen

Papers

Hybrid schemes of homogeneous and heterogeneous classifiers for cursive word recognition

An Image-Based Approach to Detection of Fake Coins

Investigating of Preprocessing Techniques and Novel Features in Recognition of Handwritten Arabic Characters

Euler Clustering on Large-Scale Dataset

Unideal iris segmentation using region-based active contour model