Current interactive user interface construction tools are often more of a hindrance than a benefit during the early stages of user interface design. These tools take too much time to use and force designers to specify more of the design details than they wish to at this early stage. Most interface designers, especially those who have a background in graphic design, prefer to sketch early interface ideas on paper or on a white-board. We are developing an interactive tool that allows designers to quickly sketch an interface using an electronic pad and stylus. Our tool preserves the important properties of paper: A rough drawing can be produced very quickly and the medium is very flexible. However, unlike a paper sketch this electronic sketch can easily be exercised and modified. In addition, our system allows designers to examine, annotate, and edit a complete history of the design. When the designer is satisfied with this early prototype, the system can transform the sketch into a complete, finished interface in a specified look-and-feel. This transformation takes place with the guidance of the designer. By supporting the early design phases of the software life-cycle, our tool should both ease the prototyping of user interfaces and improve the speed with which a final interface can be created.

/pdf/interactive-sketching-for-the-early-stages-of-user-interface-2bt1i0hesu.pdf

Interactive sketching for the early stages of user interface design

We propose interactive beautification, a technique for rapid geometric design, and introduce the technique and its algorithm with a prototype system Pegasus. The motivation is to solve the problems with current drawing systems: too many complex commands and unintuitive procedures to satisfy geometric constraints. Interactive beautification system receives the user's freestroke and beautifies it by considering geometric constraints among segments. A single stroke is beautified one after another, preventing accumulation of recognition errors or catastrophic deformation. Supported geometric constraints includes perpendicularity, congruence, symmetry, etc., which were not seen in existing freestroke recognition systems. In addition, the system generates multiple candidates as a result of beautification to solve the problem of ambiguity. Using the technique, the user can draw precise diagrams rapidly satisfying geometric relations without using any editing commands. Interactive beautification is achieved by three sequential processes; 1) inferring underlining geometric constraints based on the spatial relationships among the input stroke and the existing segments, 2) generating multiple candidates combining inferred constraints appropriately, and 3) evaluating the candidates to find the most plausible candidate and to remove the inappropriate candidates. An user study was performed using the prototype system, a commercial CAD, and an OO-based drawing system. The result showed that the users can draw required diagrams more rapidly and more precisely using the prototype system.

Interactive beautification: a technique for rapid geometric design

CALI is a fast, simple and compact online recognizer that identifies Scribbles (multi-stroke geometric shapes) drawn with a stylus on a digitizing tablet. Our method is able to identify shapes of different sizes and rotated at arbitrary angles, drawn with dashed, continuous strokes or overlapping lines. We use temporal adjacency to allow users to input the most common shapes in drawing such as triangles, lines, rectangles, circles, diamonds and ellipses, using multiple strokes. We have further extended this approach to identify useful shapes such as arrows, crossing lines and unistroke gesture commands and have developed a library of software components to make this software generally available. The recognition algorithm uses Fuzzy Logic and geometric features, combined with an extensible set of heuristics to classify scribbles. More recently we developed a trainable version of the recognizer to allow users to easily add new shape classes to the initial core set. Evaluation results show recognition rates over 97% for the non-trainable and 95% for the trainable version.

/pdf/cali-an-online-scribble-recognizer-for-calligraphic-5fne8oq4a0.pdf

CALI: An Online Scribble Recognizer for Calligraphic Interfaces

Constraint multiset grammars provide a general, high-level framework for the definition of visual languages. They are a new formalism based on multiset rewriting. We give a formal semantics for constraint multiset grammars, investigate the theoretical complexity of parsing with these grammars and give an incremental parsing algorithm. >

Constraint multiset grammars

We disclose a trainable recognizer for multi-stroke symbols. Pen speed and the curvature are examined to segment a stroke into the constituent primitives. The primitives are determined to be lines or arcs using, for example, a least squares best fit method. Symbol recognition (training) results in the construction of definitions consisting of the constituent geometric primitives (lines and arcs), the properties of individual primitives, and the geometric relationships between them. A definition is learned by examining a few examples of a symbol and identifying which properties and relationships occur frequently. Thereafter, during symbol recognition (matching), an unknown symbol is identified by determining which definition matches the unknown with the least error. We disclose two recognition methods. One assumes that the primitives of a symbol are always drawn in the same order. This method is fast, but requires some care from the drawer. The other method uses a form of best-first search, with a speculative quality metric and pruning, to recognize symbols when the drawing order is varied. During both training and recognition, multiple primitives can be drawn in the same pen stroke.

/pdf/recognizing-multi-stroke-symbols-4tvk2c57m2.pdf

Recognizing multi-stroke symbols

Diagram editing is an attractive application of gestural interfaces and pen-based computers which promise a new input paradigm where users communicate with computers in diagram languages by using gestures. A key problem in building gesture-based diagram editors is the recognition of handsketched diagrams. Existing approaches concentrate either on gesture recognition or on parsing visual languages, there has been a lack of integrated recognition concepts. This paper presents novel concepts and techniques based on an incremental paradigm of gesture recognition and a cooperative communication between modules for pattern recognition and for diagram parsing. These concepts and techniques have been used successfully to build several experimental gesture-based and syntax-directed diagram editors.

Rui Zhao

Papers

Incremental recognition in gesture-based and syntax-directed diagram editors