Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

We present a unified framework to study graph kernels, special cases of which include the random walk (Gartner et al., 2003; Borgwardt et al., 2005) and marginalized (Kashima et al., 2003, 2004; Mahet al., 2004) graph kernels. Through reduction to a Sylvester equation we improve the time complexity of kernel computation between unlabeled graphs with n vertices from O(n6) to O(n3). We find a spectral decomposition approach even more efficient when computing entire kernel matrices. For labeled graphs we develop conjugate gradient and fixed-point methods that take O(dn3) time per iteration, where d is the size of the label set. By extending the necessary linear algebra to Reproducing Kernel Hilbert Spaces (RKHS) we obtain the same result for d-dimensional edge kernels, and O(n4) in the infinite-dimensional case; on sparse graphs these algorithms only take O(n2) time per iteration in all cases. Experiments on graphs from bioinformatics and other application domains show that these techniques can speed up computation of the kernel by an order of magnitude or more. We also show that certain rational kernels (Cortes et al., 2002, 2003, 2004) when specialized to graphs reduce to our random walk graph kernel. Finally, we relate our framework to R-convolution kernels (Haussler, 1999) and provide a kernel that is close to the optimal assignment kernel of kernel of Frohlich et al. (2006) yet provably positive semi-definite.

Graph Kernels

Finite-machines have been used in various domains of natural language processing. We consider here the use of a type of transducer that supports very efficient programs: sequential transducers. We recall classical theorems and give new ones characterizing sequential string-to-string transducers. Transducers that outpur weights also play an important role in language and speech processing. We give a specific study of string-to-weight transducers, including algorithms for determinizing and minizizing these transducers very efficiently, and characterizations of the transducers admitting determinization and the corresponding algorithms. Some applications of these algorithms in speech recognition are described and illustrated.

/pdf/finite-state-transducers-in-language-and-speech-processing-19eowcpmyo.pdf

Finite-state transducers in language and speech processing

Compte tenu du progres effectue recemment dans le domaine de la desambiguisation du sens des mots, les As. font ici le point sur l'etat de la recherche dans ce domaine depuis ces 50 dernieres annees et considerent les prochaines etapes a franchir. Dans un 1 e r temps, ils passent en revue les principales approches de la desambiguisation du sens des mots : des premieres tentatives effectuees dans le cadre de la traduction automatique aux methodes actuelles basees sur les corpus, en passant par les methodes basees sur l'intelligence artificielle et les methodes utilisant des bases de connaissance. Dans un 2 n d temps, ils examinent les problemes laisses en suspens (le role du contexte, la division des sens, l'evaluation des resultats) et proposent quelques orientations pour la recherche future

Introduction to the special issue on word sense disambiguation: the state of the art

This is a fully-searchable, complete text of the MIT Encyclopedia of the Cognitive Sciences (MITECS) on a dual-platform CD-ROM. Since the 1970s the cognitive sciences have offered multidisciplinary ways of understanding the mind and cognition. The MIT Encyclopedia of the Cognitive Sciences (MITECS) is a landmark, comprehensive reference work that represents the methodological and theoretical diversity of this changing field. At the core of the encyclopedia are 471 concise entries, from Acquisition and Adaptationism to Wundt and X-bar Theory. Each article, written by a leading researcher in the field, provides an accessible introduction to an important concept in the cognitive sciences, as well as references or further readings. Six extended essays, which collectively serve as a roadmap to the articles, provide overviews of each of six major areas of cognitive science: Philosophy; Psychology; Neurosciences; Computational Intelligence; Linguistics and Language; and Culture, Cognition, and Evolution. For both students and researchers, MITECS will be an indispensable guide to the current state of the cognitive sciences. System requirements: Compatible with Windows 95, Windows NT (16MB of RAM available to Acrobat Reader; 10MB hard-disk space); Windows 3.1 and 3.11 for Workgroups (12MB hard-disk space); Macintosh and Power Macintosh (8MB of RAM available to Acrobat Reader, Apple System Software version 7.1.2 or later, and 12.5MB hard-disk space).

The MIT encyclopedia of the cognitive sciences

Finite-state automata are a very effective tool in natural language processing. However, in a variety of applications and especially in speech precessing, it is necessary to consider more general machines in which arcs are assigned weights or costs. We briefly describe some of the main theoretical and algorithmic aspects of these machines. In particular, we describe an efficient composition algorithm for weighted transducers, and give examples illustrating the value of determinization and minimization algorithms for weighted automata.

Weighted Automata in Text and Speech Processing

We present the concepts of weighted language, transduction and automaton from algebraic automata theory as a general framework for describing and implementing decoding cascades in speech and language processing. This generality allows us to represent uniformly such information sources as pronunciation dictionaries, language models and lattices, and to use uniform algorithms for building decoding stages and for optimizing and combining them. In particular, a single automata join algorithm can be used either to combine information sources such as a pronunciation dictionary and a context-dependency model during the construction of a decoder, or dynamically during the operation of the decoder. Applications to speech recognition and to Chinese text segmentation will be discussed.

/pdf/weighted-rational-transductions-and-their-application-to-pu28yfdqqp.pdf

Weighted rational transductions and their application to human language processing

We describe a lattice generation method that pr oduces highquality lattices with less than 10% increased computation over standard Viterbi decoding. Using the North American Business News (NAB) task, we show our method is within 0.2% in lattice word-error rate of ‘full lattices’, which are those that contain all the recognitionhypotheses within the search beam. Our method is closely related to previous lattice generation met hods, but applies to more general network topologies. We also give real-time results on the NAB task, in which we generate lattices in a first pass and then rescore them with stronger acoustic and language models in a second pass. We are able to achieve at 3x real-time a word error rate of 11.2% on the Eval ’95 test set, which is only 1.7% worse than AT&T’s official benchmark result that year using what was then a 1000x real-time system.

Efficient general lattice generation and rescoring.

Certain restrictions on possible scopings of quantified noun phrases in natural language are usually expressed in terms of formal constraints on binding at a level of logical form. Such reliance on the form rather than the content of semantic interpretations goes against the spirit of compositionality. I will show that those scoping restrictions follow from simple and fundamental facts about functional application and abstraction, and can be expressed as constraints on the derivation of possible meanings for sentences rather than constraints of the alleged forms of those meanings.

Categorial semantics and scoping

We present a system for the incremental interpretation of natural-language utterances in context. The main goal of the work is to account for the innuences of context on interpretation , while preserving compositionality to the extent possible. To achieve this goal, we introduce a representational device, conditional interpretations, and a rule system for constructing them. Conditional interpretations represent the potential contributions of phrases to the interpretation of an utterance. The rules specify how phrase interpretations are combined and how they are elaborated with respect to context. The control structure deened by the rules determines the points in the interpretation process at which suucient information becomes available to carry out speciic inferential interpretation steps, such as determining the plausibility of particular referential connections or modiier attachments. We have implemented these ideas in Candide, a system for interactive acquisition of procedural knowledge.

Fernando Pereira

Papers

Weighted Automata in Text and Speech Processing

Weighted rational transductions and their application to human language processing

Efficient general lattice generation and rescoring.

Categorial semantics and scoping

Incremental interpretation