Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

This tutorial provides an overview of the basic theory of hidden Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and gives practical details on methods of implementation of the theory along with a description of selected applications of the theory to distinct problems in speech recognition. Results from a number of original sources are combined to provide a single source of acquiring the background required to pursue further this area of research. The author first reviews the theory of discrete Markov chains and shows how the concept of hidden states, where the observation is a probabilistic function of the state, can be used effectively. The theory is illustrated with two simple examples, namely coin-tossing, and the classic balls-in-urns system. Three fundamental problems of HMMs are noted and several practical techniques for solving these problems are given. The various types of HMMs that have been studied, including ergodic as well as left-right models, are described. >

/pdf/a-tutorial-on-hidden-markov-models-and-selected-applications-1vlzbxbk68.pdf

A tutorial on hidden Markov models and selected applications in speech recognition

From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Statistical approaches to processing natural language text have become dominant in recent years This foundational text is the first comprehensive introduction to statistical natural language processing (NLP) to appear The book contains all the theory and algorithms needed for building NLP tools It provides broad but rigorous coverage of mathematical and linguistic foundations, as well as detailed discussion of statistical methods, allowing students and researchers to construct their own implementations The book covers collocation finding, word sense disambiguation, probabilistic parsing, information retrieval, and other applications

/pdf/foundations-of-statistical-natural-language-processing-1ou5e6c2q6.pdf

Foundations of Statistical Natural Language Processing

The probability of error in decoding an optimal convolutional code transmitted over a memoryless channel is bounded from above and below as a function of the constraint length of the code. For all but pathological channels the bounds are asymptotically (exponentially) tight for rates above R_{0} , the computational cutoff rate of sequential decoding. As a function of constraint length the performance of optimal convolutional codes is shown to be superior to that of block codes of the same length, the relative improvement increasing with rate. The upper bound is obtained for a specific probabilistic nonsequential decoding algorithm which is shown to be asymptotically optimum for rates above R_{0} and whose performance bears certain similarities to that of sequential decoding algorithms.

Error bounds for convolutional codes and an asymptotically optimum decoding algorithm

It is shown that, particularly for terrestrial cellular telephony, the interference-suppression feature of CDMA (code division multiple access) can result in a many-fold increase in capacity over analog and even over competing digital techniques. A single-cell system, such as a hubbed satellite network, is addressed, and the basic expression for capacity is developed. The corresponding expressions for a multiple-cell system are derived. and the distribution on the number of users supportable per cell is determined. It is concluded that properly augmented and power-controlled multiple-cell CDMA promises a quantum increase in current cellular capacity. >

On the capacity of a cellular CDMA system

1. Introduction. Definition and Purpose. Basic Limitations of the Conventional Approach. Spread Spectrum Principles. Organization of the Book. 2. Random and Pseudorandom Signal Generation. Purpose. Pseudorandom Sequences. Maximal Length Linear Shift Register Sequences. Randomness Properties of MLSR Sequences. Conclusion. Generating Pseudorandom Signals (Pseudonoise) from Pseudorandom Sequences. First- and Second-Order Statistics of Demodulator Output in Multiple Access Interference. Statistics for QPSK Modulation by Pseudorandom Sequences. Examples. Bound for Bandlimited Spectrum. Error Probability for BPSK or QPSK with Constant Signals in Additive Gaussian Noise and Interference. Appendix 2A: Optimum Receiver Filter for Bandlimited Spectrum. 3. Synchronization of Pseudorandom Signals. Purpose. Acquisition of Pseudorandom Signal Timing. Hypothesis Testing for BPSK Spreading. Hypothesis Testing for QPSK Spreading. Effect of Frequency Error. Additional Degradation When N is Much Less Than One Period. Detection and False Alarm Probabilities. Fixed Signals in Gaussian Noise (L=1). Fixed Signals in Gaussian Noise with Postdetection Integration (L>1). Rayleigh Fading Signals (L>/=1). The Search Procedure and Acquisition Time. Single-Pass Serial Search (Simplified). Single-Pass Serial Search (Complete). Multiple Dwell Serial Search. Time Tracking of Pseudorandom Signals. Early-Late Gate Measurement Statistics. Time Tracking Loop. Carrier Synchronization. Appendix 3A: Likelihood Functions and Probability Expressions. Bayes and Neyman-Pearson Hypothesis Testing. Coherent Reception in Additive White Gaussian Noise. Noncoherent Reception in AWGN for Unfaded Signals. Noncoherent Reception of Multiple Independent Observations of Unfaded Signals in AWGN. Noncoherent Reception of Rayleigh-Faded Signals in AWGN. 4. Modulation and Demodulation of Spread Spectrum Signals in Multipath and Multiple Access Interference. Purpose. Chernoff and Battacharyya Bounds. Bounds for Gaussian Noise Channel. Chernoff Bound for Time-Synchronous Multiple Access Interference with BPSK Spreading. Chernoff Bound for Time-Synchronous Multiple Access Interference with QPSK Spreading. Improving the Chernoff Bound by a Factor of 2. Multipath Propagation: Signal Structure and Exploitation. Pilot-Aided Coherent Multipath Demodulation. Chernoff Bounds on Error Probability for Coherent Demodulation with Known Path Parameters. Rayleigh and Rician Fading Multipath Components. Noncoherent Reception. Quasi-optimum Noncoherent Multipath Reception for M-ary Orthogonal Modulation. Performance Bounds. Search Performance for Noncoherent Orthogonal M-ary Demodulators. Power Measurement and Control for Noncoherent Orthogonal M-ary Demodulators. Power Control Loop Performance. Power Control Implications. Appendix 4A: Chernoff Bound with Imperfect Parameter Estimates. 5. Coding and Interleaving. Purpose. Interleaving to Achieve Diversity. Forward Error Control Coding - Another Means to Exploit Redundancy. Convolutional Code Structure. Maximum Likelihood Decoder - Viterbi Algorithm. Generalization of the Preceding Example. Convolutional Code Performance Evaluation. Error Probability for Tailed-off Block. Bit Error Probability. Generalizations of Error Probability Computation. Catastrophic Codes. Generalization to Arbitrary Memoryless Channels - Coherent and Noncoherent. Error Bounds for Binary-Input, Output-Symmetric Channels with Integer Metrics. A Near-Optimal Class of Codes for Coherent Spread Spectrum Multiple Access. Implementation. Decoder Implementation. Generating Function and Performance. Performance Comparison and Applicability. Orthogonal Convolutional Codes for Noncoherent Demodulation of Rayleigh Fading Signals. Implementation. Performance for L-Path Rayleigh Fading. Conclusions and Caveats. Appendix 5A: Improved Bounds for Symmetric Memoryless Channels and the AWGN Channel. Appendix 5B: Upper Bound on Free Distance of Rate 1/n Convolutional Codes. 6. Capacity, Coverage, and Control of Spread Spectrum Multiple Access Networks. General. Reverse Link Power Control. Multiple Cell Pilot Tracking and Soft Handoff. Other-Cell Interference. Propagation Model. Single-Cell Reception - Hard Handoff. Soft Handoff Reception by the Better of the Two Nearest Cells. Soft Handoff Reception by the Best of Multiple Cells. Cell Coverage Issues with Hard and Soft Handoff. Hard Handoff. Soft Handoff. Erlang Capacity of Reverse Links. Erlang Capacity for Conventional Assigned-Slot Multiple Access. Spread Spectrum Multiple Access Outage - Single Cell and Perfect Power Control. Outage with Multiple-Cell Interference. Outage with Imperfect Power Control. An Approximate Explicit Formula for Capacity with Imperfect Power Control. Designing for Minimum Transmitted Power. Capacity Requirements for Initial Accesses. Erlang Capacity of Forward Links. Forward Link Power Allocation. Soft Handoff Impact on Forward Link. Orthogonal Signals for Same-Cell Users. Interference Reduction with Multisectored and Distributed Antennas. Interference Cancellation. Epilogue. References and Bibliography. Index.

Cdma: Principles of Spread Spectrum Communication

A system and method for communicating information signals using spread spectrum communication techniques. PN sequences are constructed that provide orthogonality between the users so that mutual interference will be reduced, allowing higher capacity and better link performance. With orthogonal PN codes, the cross-correlation is zero over a predetermined time interval, resulting in no interference between the orthogonal codes, provided only that the code time frames are time aligned with each other. In an exemplary embodiment, signals are communicated between a cell-site and mobile units using direct sequence spread spectrum communication signals. In the cell-to-mobile link, pilot, sync, paging and voice channels are defined. Information communicated on the cell-to-mobile link channels are, in general, encoded, interleaved, bi-phase shift key (BPSK) modulated with orthogonal covering of each BPSK symbol along with quadrature phase shift key (QPSK) spreading of the covered symbols. In the mobile-to-cell link, access and voice channels are defined. Information communicated on the mobile-to-cell link channels are, in general, encoded, interleaved, orthogonal signaling along with QPSK spreading.

Andrew J. Viterbi

Papers

Error bounds for convolutional codes and an asymptotically optimum decoding algorithm

On the capacity of a cellular CDMA system

On the capacity of a cellular CDMA system

Cdma: Principles of Spread Spectrum Communication

System and method for generating signal waveforms in a CDMA cellular telephone system