IEEE Transactions on Pattern Analysis and Machine Intelligence

Document recognition and retrieval technologies complement one another, providing improved access to increasingly large document collections. While recognition and retrieval of textual information is fairly mature, with wide-spread availability of optical character recognition and text-based search engines, recognition and retrieval of graphics such as images, figures, tables, diagrams, and mathematical expressions are in comparatively early stages of research. This paper surveys the state of the art in recognition and retrieval of mathematical expressions, organized around four key problems in math retrieval (query construction, normalization, indexing, and relevance feedback), and four key problems in math recognition (detecting expressions, detecting and classifying symbols, analyzing symbol layout, and constructing a representation of meaning). Of special interest is the machine learning problem of jointly optimizing the component algorithms in a math recognition system, and developing effective indexing, retrieval and relevance feedback algorithms for math retrieval. Another important open problem is developing user interfaces that seamlessly integrate recognition and retrieval. Activity in these important research areas is increasing, in part because math notation provides an excellent domain for studying problems common to many document and graphics recognition and retrieval applications, and also because mature applications will likely provide substantial benefits for education, research, and mathematical literacy.

/pdf/recognition-and-retrieval-of-mathematical-expressions-2x3ms5stvn.pdf

Recognition and retrieval of mathematical expressions

We present QuickDraw, a prototype sketch-based drawing tool, that facilitates drawing of precise geometry diagrams that are often drawn by students and academics in several scientific disciplines. Quickdraw can recognize sketched diagrams containing components such as line segments and circles, infer geometric constraints relating recognized components, and use this information to beautify the sketched diagram. Beautification is based on a novel algorithm that iteratively computes various sub-components of the components using an extensible set of deductive rules. We conducted a user study comparing QuickDraw with four state-of-the-art diagramming tools: Microsoft PowerPoint, Cabri II Plus, Geometry Expressions and Geometer's SketchPad. Our study demonstrates a strong interest among participants for the use of sketch-based software for drawing geometric diagrams. We also found that QuickDraw enables users to draw precise diagrams faster than the majority of existing tools in some cases, while having them make fewer corrections.

/pdf/quickdraw-improving-drawing-experience-for-geometric-3mfd5o5c59.pdf

QuickDraw: improving drawing experience for geometric diagrams

This paper presents the interaction design of, and demonstration of technical feasibility for, intelligent tutoring systems that can accept handwriting input from students. Handwriting and pen input offer several affordances for students that traditional typing-based interactions do not. To illustrate these affordances, we present evidence, from tutoring mathematics, that the ability to enter problem solutions via pen input enables students to record algebraic equations more quickly, more smoothly (fewer errors), and with increased transfer to non-computer-based tasks. Furthermore our evidence shows that students tend to like pen input for these types of problems more than typing. However, a clear downside to introducing handwriting input into intelligent tutors is that the recognition of such input is not reliable. In our work, we have found that handwriting input is more likely to be useful and reliable when context is considered, for example, the context of the problem being solved. We present an intelligent tutoring system for algebra equation solving via pen-based input that is able to use context to decrease recognition errors by 18% and to reduce recognition error recovery interactions to occur on one out of every four problems. We applied user-centered design principles to reduce the negative impact of recognition errors in the following ways: (1) though students handwrite their problem-solving process, they type their final answer to reduce ambiguity for tutoring purposes, and (2) in the small number of cases in which the system must involve the student in recognition error recovery, the interaction focuses on identifying the student's problem-solving error to keep the emphasis on tutoring. Many potential recognition errors can thus be ignored and distracting interactions are avoided. This work can inform the design of future systems for students using pen and sketch input for math or other topics by motivating the use of context and pragmatics to decrease the impact of recognition errors and put user focus on the task at hand.

http://pact.cs.cmu.edu/pubs/Anthony,%20Yang%20%26%20Koedinger.pdf

A paradigm for handwriting-based intelligent tutors

This report describes recent advances in the area of the recognition of on-line handwritten mathematical notation. We describe architectures, symbol classification methods, and techniques for the structural analysis of mathematical expressions. We also survey applications specialized for mathematical notation.

/pdf/a-survey-on-recognition-of-on-line-handwritten-mathematical-64r57m548t.pdf

A survey on recognition of on-line handwritten mathematical notation

Computer-based geometry systems have been widely used for teaching and learning, but largely based on mouse-and-keyboard interaction, these systems usually require users to draw figures by following strict task structures defined by menus, buttons, and mouse and keyboard actions. Pen-based designs offer a more natural way to develop geometry theorem proofs with hand-drawn figures and scripts. This paper describes a pen-based geometry theorem proving system that can effectively recognize hand-drawn figures and hand-written proof scripts, and accurately establish the correspondence between geometric components and proof steps. Our system provides dynamic and intelligent visual assistance to help users understand the process of proving and allows users to manipulate geometric components and proof scripts based on structures rather than strokes. The results from evaluation study show that our system is well perceived and users have high satisfaction with the accuracy of sketch recognition, the effectiveness of visual hints, and the efficiency of structure-based manipulation.

/pdf/intelligent-understanding-of-handwritten-geometry-theorem-20tqbjmfyu.pdf

Intelligent understanding of handwritten geometry theorem proving

Raw digital ink is informal and unstructured. Its editing, especially its selection, is often inefficient. In this paper, we present approaches to structuralize raw digital ink as multiple hierarchies to facilitate its selection. First a link model is built to organize ink as a mesh-like structure. Based on the link model, the isolated stroke groups form patches. In each patch, textual and graphical areas are separated. Then, each textual area is segmented into text lines, and each text line is partitioned to words. We also design gestures for selecting structured ink. Experiments showed that our ink-structuralizing approaches are effective and selecting structured ink by our gestures considerably outperforms selecting raw ink.

Structuralizing digital ink for efficient selection

In this paper, we describe a multimodal error correction mechanism. It allows the user to correct errors in continuous handwriting recognition naturally by simultaneously using pen gesture and speech. A multimodal fusion algorithm is designed to enhance recognition accuracies of handwriting and speech through cross-modal influence. We have performed preliminary evaluation experiments and the results show that this multimodal mechanism can efficiently correct the errors in continuous handwriting recognition.

Multimodal error correction for continuous handwriting recognition in pen-based user interfaces

In recognition-based user interface, users' satisfaction is determined not only by recognition accuracy but also by effort to correct recognition errors. In this paper, we introduce a crossmodal error correction technique, which allows users to correct errors of Chinese handwriting recognition by speech. The focus of the paper is a multimodal fusion algorithm supporting the crossmodal error correction. By fusing handwriting and speech recognition, the algorithm can correct errors in both character extraction and recognition of handwriting. The experimental result indicates that the algorithm is effective and efficient. Moreover, the evaluation also shows the correction technique can help users to correct errors in handwriting recognition more efficiently than the other two error correction techniques.

Crossmodal error correction of continuous handwriting recognition by speech

Chinese text entry is challenging on mobile devices which rely on keypad input. Entering one character may require many key presses. This paper proposes a multimodal text entry technique for Chinese. In this method, Chinese user can enter Chinese text by simultaneously using the simplified phonemic input method named Jianpin with keypad and speech utterance. The key of the technique is a multimodal fusion algorithm, which synchronizes speech and keypad input and fuses redundant information from two modalities to get the best candidate. A preliminary evaluation shows that users appreciate this technique and it could reduce key presses and enhance the input efficiency.

Xugang Wang

Papers

Intelligent understanding of handwritten geometry theorem proving

Structuralizing digital ink for efficient selection

Multimodal error correction for continuous handwriting recognition in pen-based user interfaces

Crossmodal error correction of continuous handwriting recognition by speech

Multimodal Chinese text entry with speech and keypad on mobile devices