Complex virtual reality (VR) tasks, like 3D solid modelling, are challenging with standard input controllers. We propose exploiting the affordances and input capabilities when using a 3D-tracked multi-touch tablet in an immersive VR environment. Observations gained during semi-structured interviews with general users, and those experienced with 3D software, are used to define a set of design dimensions and guidelines. These are used to develop a vocabulary of interaction techniques to demonstrate how a tablet's precise touch input capability, physical shape, metaphorical associations, and natural compatibility with barehand mid-air input can be used in VR. For example, transforming objects with touch input, "cutting" objects by using the tablet as a physical "knife", navigating in 3D by using the tablet as a viewport, and triggering commands by interleaving bare-hand input around the tablet. Key aspects of the vocabulary are evaluated with users, with results validating the approach.

TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality

This paper presents a new approach to wearable hand gesture recognition and finger angle estimation based on the modified barometric pressure sensing. Barometric pressure sensors were encased and injected with VytaFlex rubber such that the rubber directly contacted the sensing element allowing pressure change detection when the encasing rubber was pressed. A wearable prototype consisting of an array of ten modified barometric pressure sensors around the wrist was developed and validated with experimental testing for three different hand gesture sets and finger flexion/extension trials for each of the five fingers. The overall hand gesture recognition classification accuracy was 94%. Further analysis revealed that the most important sensor location was the underside of the wrist and that when reducing the sensor number to only five optimally placed sensors, classification accuracy was still 90%. For continuous finger angle estimation, aggregate ${R}^{\textsf {2}}$ values between actual and predicted angles were thumb: 0.81 ± 0.10, index finger: 0.85±0.06, middle finger: 0.77±0.08, ring finger: 0.77 ± 0.12, and pinkie finger: 0.75 ± 0.10, and the overall average was 0.79 ± 0.05. These results demonstrate that a modified barometric pressure wristband can be used to classify hand gestures and to estimate individual finger joint angles. This approach could serve to improve the clinical treatment for upper extremity deficiencies, such as for stroke rehabilitation, by providing objective patient motor control metrics to inform and aid physicians and therapists throughout the rehabilitation process.

Hand Gesture Recognition and Finger Angle Estimation via Wrist-Worn Modified Barometric Pressure Sensing

Modeling in Augmented Reality (AR) lets users create and manipulate virtual objects in mid-air that are aligned to their real environment. We present ARPen, a bimanual input technique for AR modeling that combines a standard smartphone with a 3D-printed pen. Users sketch with the pen in mid-air, while holding their smartphone in the other hand to see the virtual pen traces in the live camera image. ARPen combines the pen's higher 3D input precision with the rich interactive capabilities of the smartphone touchscreen. We studied subjective preferences for this bimanual input technique, such as how people hold the smartphone while drawing, and analyzed the performance of different bimanual techniques for selecting and moving virtual objects. Users preferred a bimanual technique casting a ray through the pen tip for both selection and translation. We provide initial design guidelines for this new class of bimanual AR modeling systems.

ARPen: Mid-Air Object Manipulation Techniques for a Bimanual AR System with Pen & Smartphone

Gesture recognition is important for human–computer interaction and a variety of emerging research and commercial areas including virtual and augmented reality. Current approaches typically require sensors to be placed on the forearm, wrist, or directly across finger joints; however, they can be cumbersome or hinder human movement and sensation. In this paper, we introduce a novel approach to recognize hand gestures by estimating skin strain with multiple soft sensors optimally placed across the back of the hand. A pilot study was first conducted by covering the back of the hand with 40 small 2.5 mm reflective markers and using a high-precision camera system to measure skin strain patterns for individual finger movements. Optimal strain locations are then determined and used for sensor placement in a stretchable e-skin patch prototype. Experimental testing is performed to evaluate the stretchable e-skin patch performance in classifying individual finger gestures and American Sign Language 0–9 number gestures. Results showed classification accuracies of 95.3% and 94.4% for finger gestures and American Sign Language 0–9 gestures, respectively. These results demonstrate the feasibility of a stretchable e-skin patch on the back of the hand for hand gesture recognition and their potential to significantly enhance human–computer interaction.

Stretchable e-Skin Patch for Gesture Recognition on the Back of the Hand

A computer system concurrently displays, in an augmented reality environment, a representation of at least a portion of a field of view of one or more cameras that includes a respective physical object, which is updated as contents of the field of view change; and a respective virtual user interface object, at a respective location in the virtual user interface determined based on the location of the respective physical object in the field of view. While detecting an input at a location that corresponds to the displayed respective virtual user interface object, in response to detecting movement of the input relative to the respective physical object in the field of view of the one or more cameras, the system adjusts an appearance of the respective virtual user interface object in accordance with a magnitude of movement of the input relative to the respective physical object.

Systems, Methods, and Graphical User Interfaces for Interacting with Augmented and Virtual Reality Environments

Researchers at the Human Interface Technology Laboratory New Zealand (HIT Lab NZ) are investigating free-hand gestures for natural interaction with augmented-reality interfaces. They've applied the results to systems for desktop computers and mobile devices.

Tham Piumsomboon

Papers

Hands in Space: Gesture Interaction with Augmented-Reality Interfaces