Document recognition and retrieval technologies complement one another, providing improved access to increasingly large document collections. While recognition and retrieval of textual information is fairly mature, with wide-spread availability of optical character recognition and text-based search engines, recognition and retrieval of graphics such as images, figures, tables, diagrams, and mathematical expressions are in comparatively early stages of research. This paper surveys the state of the art in recognition and retrieval of mathematical expressions, organized around four key problems in math retrieval (query construction, normalization, indexing, and relevance feedback), and four key problems in math recognition (detecting expressions, detecting and classifying symbols, analyzing symbol layout, and constructing a representation of meaning). Of special interest is the machine learning problem of jointly optimizing the component algorithms in a math recognition system, and developing effective indexing, retrieval and relevance feedback algorithms for math retrieval. Another important open problem is developing user interfaces that seamlessly integrate recognition and retrieval. Activity in these important research areas is increasing, in part because math notation provides an excellent domain for studying problems common to many document and graphics recognition and retrieval applications, and also because mature applications will likely provide substantial benefits for education, research, and mathematical literacy.

/pdf/recognition-and-retrieval-of-mathematical-expressions-2x3ms5stvn.pdf

Recognition and retrieval of mathematical expressions

Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition

Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models

In this paper, we introduce Track, Attend, and Parse (TAP), an end-to-end approach based on neural networks for online handwritten mathematical expression recognition (OHMER). The architecture of TAP consists of a tracker and a parser. The tracker employs a stack of bidirectional recurrent neural networks with gated recurrent units (GRU) to model the input handwritten traces, which can fully utilize the dynamic trajectory information in OHMER. Followed by the tracker, the parser adopts a GRU equipped with guided hybrid attention (GHA) to generate notations. The proposed GHA is composed of a coverage-based spatial attention, a temporal attention, and an attention guider. Moreover, we demonstrate the strong complementarity between offline information with static-image input and online information with ink-trajectory input by blending a fully convolutional networks-based watcher into TAP. Inherently, unlike traditional methods, this end-to-end framework does not require the explicit symbol segmentation and a predefined expression grammar for parsing. Validated on a benchmark published by the CROHME competition, the proposed approach outperforms the state-of-the-art methods and achieves the best reported results with an expression recognition accuracy of 61.16% on CROHME 2014 and 57.02% on CROHME 2016, using only official training dataset.

Track, Attend, and Parse (TAP): An End-to-End Framework for Online Handwritten Mathematical Expression Recognition

A global learning approach for an online handwritten mathematical expression recognition system

In this paper, we propose a new framework for online handwritten mathematical expression recognition In this approach, we consider handwritten mathematical expressions as the output of stroke generation processes based on a stochastic context-free grammar which generates handwritten expressions stochastically We estimate the most likely expression candidate derived from the grammar, rather than solving one by one the three major problems in mathematical expression recognition: symbol segmentation/recognition, 2D structure recognition, and expression syntax analysis With this method, we can simultaneously recognize the symbols and structure of an expression within the grammatical constraint Experiments revealed that this simultaneous estimation decreases errors in symbol segmentation and recognition, and that these errors are reduced as grammatical restriction is strengthened

https://hal.inria.fr/inria-00104743/document

R. Yamamoto

Papers

On-Line Recognition of Handwritten Mathematical Expression Based on Stroke-Based Stochastic Context-Free Grammar