Cell phones and other portable devices are equipped with a variety of technologies by which existing functionality can be improved, and new functionality can be provided. Some relate to visual search capabilities, and determining appropriate actions responsive to different image inputs. Others relate to processing of image data. Still others concern metadata generation, processing, and representation. Yet others relate to coping with fixed focus limitations of cell phone cameras, e.g., in reading digital watermark data. Still others concern user interface improvements. A great number of other features and arrangements are also detailed.

Methods and Systems for Content Processing

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient "purposive" approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction.

/pdf/visual-interpretation-of-hand-gestures-for-human-computer-f24iuo372e.pdf

Visual interpretation of hand gestures for human-computer interaction: a review

Newness and distinctiveness is claimed in the features of ornamentation as shown inside the broken line circle in the accompanying representation.

Display screen or portion thereof with graphical user interface

Many surface computing prototypes have employed gestures created by system designers. Although such gestures are appropriate for early investigations, they are not necessarily reflective of user behavior. We present an approach to designing tabletop gestures that relies on eliciting gestures from non-technical users by first portraying the effect of a gesture, and then asking users to perform its cause. In all, 1080 gestures from 20 participants were logged, analyzed, and paired with think-aloud data for 27 commands performed with 1 and 2 hands. Our findings indicate that users rarely care about the number of fingers they employ, that one hand is preferred to two, that desktop idioms strongly influence users' mental models, and that some commands elicit little gestural agreement, suggesting the need for on-screen widgets. We also present a complete user-defined gesture set, quantitative agreement scores, implications for surface technology, and a taxonomy of surface gestures. Our results will help designers create better gesture sets informed by user behavior.

https://www.microsoft.com/en-us/research/wp-content/uploads/2009/04/SurfaceGestures_CHI2009.pdf

User-defined gestures for surface computing

We present a sketching interface for quickly and easily designing freeform models such as stuffed animals and other rotund objects. The user draws several 2D freeform strokes interactively on the screen and the system automatically constructs plausible 3D polygonal surfaces. Our system supports several modeling operations, including the operation to construct a 3D polygonal surface from a 2D silhouette drawn by the user: it inflates the region surrounded by the silhouette making wide areas fat, and narrow areas thin. Teddy, our prototype system, is implemented as a Java™ program, and the mesh construction is done in real-time on a standard PC. Our informal user study showed that a first-time user typically masters the operations within 10 minutes, and can construct interesting 3D models within minutes.

/pdf/teddy-a-sketching-interface-for-3d-freeform-design-qlbvzvgvwj.pdf

Teddy: a sketching interface for 3D freeform design

Toolglass widgets are new user interface tools that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor. They can be positioned with one hand while the other positions the cursor. The widgets provide a rich and concise vocabulary for operating on application objects. These widgets may incorporate visual filters, called Magic Lens filters, that modify the presentation of application objects to reveal hidden information, to enhance data of interest, or to suppress distracting information. Together, these tools form a see-through interface that offers many advantages over traditional controls. They provide a new style of interaction that better exploits the user’s everyday skills. They can reduce steps, cursor motion, and errors. Many widgets can be provided in a user interface, by designers and by users, without requiring dedicated screen space. In addition, lenses provide rich context-dependent feedback and the ability to view details and context simultaneously. Our widgets and lenses can be combined to form operation and viewing macros, and can be used over multiple applications. CR

Toolglass and magic lenses: the see-through interface

This paper presents an application that uses hand gesture input to control a computer while giving a presentation. In order to develop a prototype of this application, we have defined an interaction model, a notation for gestures, and a set of guidelines to design gestural command sets. This works aims to define interaction styles that work in computerized reality environments. In our application, gestures are used for interacting with the computer as well as for communicating with other people or operating other devices.

Charade: remote control of objects using free-hand gestures

An experimental GUI paradigm is presented which is based on the design goals of maximizing the amount of screen used for application data, reducing the amount that the UI diverts visual attentions from the application data, and increasing the quality of input. In pursuit of these goals, we integrated the non-standard UI technologies of multi-sensor tablets, toolglass, transparent UI components, and marking menus. We describe a working prototype of our new paradigm, the rationale behind it and our experiences introducing it into an existing application. Finally, we presents some ot the lessons learned: prototypes are useful to break the barriers imposed by conventional GUI design and some of their ideas can still be retrofitted seamlessly into products. Furthermore, the added functionality is not measured only in terms of user performance, but also by the quality of interaction, which allows artists to create new graphic vocabularies and graphic styles.

/pdf/the-design-of-a-gui-paradigm-based-on-tablets-two-hands-and-54x4aqx0jb.pdf

The design of a GUI paradigm based on tablets, two-hands, and transparency

A novel input device called the Rockin’Mouse is described and evaluated. The Rockin’Mouse is a four degree-of-freedom input device that has the same shape as a regular mouse except that the bottom of the Rockin’Mouse is rounded so that it can be tilted. This tilting can be used to control two extra degrees of freedom, thus making it suitable for manipulation in 3D environments. Like the regular mouse, the Rockin’Mouse can sense planar position and perform all the usual functions. However, in a 3D scene a regular mouse can only operate on 2 dimensions at a time and therefore manipulation in 3D requires a way to switch between dimensions. With the Rockin’Mouse, however, all the dimensions can be simultaneously controlled. In this paper we describe our design rationale behind the Rockin’Mouse, and present an experiment which compares the Rockin’Mouse to the standard mouse in a typical 3D interaction task. Our results indicate that the Rockin’Mouse is 30% faster and is a promising device for both 2D and 3D interaction.

/pdf/the-rockin-mouse-integral-3d-manipulation-on-a-plane-4njx3cfb4q.pdf

The Rockin'Mouse: integral 3D manipulation on a plane

In current interfaces, users select objects, apply operations, and change viewing parameters in distinct steps that require switching attention among several screen areas. Our See-Through Interface™ software reduces steps by locating tools on a transparent sheet that can be moved over applications with one hand using a trackball, while the other hand controls a mouse cursor. The user clicks through a tool onto application objects, simultaneously selecting an operation and an operand. Tools may include graphical filters that display a customized view of application objects. Compared to traditional interactors, these tools save steps, require no permanent screen space, reduce temporal modes, apply to multiple applications, and facilitate customization. This paper presents a taxonomy of see-through tools that considers variations in each of the steps they perform. As examples, we describe particular see-through tools that perform graphical editing and text editing operations. CR Categories and Subject Descriptors: I.3.6 [Computer Graphics]: Methodology and Techniques—interaction techniques; H.5.2 [Information Interfaces and Presentation): User Interfaces—interaction styles; I.3.3 [Computer Graphics]: Picture/Image Generation—viewing algorithms; 1.3.4 [Computer Graphics]: Graphics Utilities—graphics editors

/pdf/a-taxonomy-of-see-through-tools-5etqylwl9q.pdf

Thomas Baudel

Papers

Toolglass and magic lenses: the see-through interface

Charade: remote control of objects using free-hand gestures

The design of a GUI paradigm based on tablets, two-hands, and transparency

The Rockin'Mouse: integral 3D manipulation on a plane

A taxonomy of see-through tools