A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Intuitive computing methods and systems

Augmented reality applications in design and manufacturing

This survey summarizes almost 50 years of research and development in the field of Augmented Reality AR. From early research in the1960's until widespread availability by the 2010's there has been steady progress towards the goal of being able to seamlessly combine real and virtual worlds. We provide an overview of the common definitions of AR, and show how AR fits into taxonomies of other related technologies. A history of important milestones in Augmented Reality is followed by sections on the key enabling technologies of tracking, display and input devices. We also review design guidelines and provide some examples of successful AR applications. Finally, we conclude with a summary of directions for future work and a review of some of the areas that are currently being researched.

/pdf/a-survey-of-augmented-reality-2ce6xet7y5.pdf

A Survey of Augmented Reality

In some arrangements, product packaging is digitally watermarked over most of its extent to facilitate high-throughput item identification at retail checkouts. Imagery captured by conventional or plenoptic cameras can be processed (e.g., by GPUs) to derive several different perspective-transformed views—further minimizing the need to manually reposition items for identification. Crinkles and other deformations in product packaging can be optically sensed, allowing such surfaces to be virtually flattened to aid identification. Piles of items can be 3D-modelled and virtually segmented into geometric primitives to aid identification, and to discover locations of obscured items. Other data (e.g., including data from sensors in aisles, shelves and carts, and gaze tracking for clues about visual saliency) can be used in assessing identification hypotheses about an item. A great variety of other features and arrangements are also detailed.

Methods and arrangements for identifying objects

Physical representations of data have existed for thousands of years. Yet it is now that advances in digital fabrication, actuated tangible interfaces, and shape-changing displays are spurring an emerging area of research that we call Data Physicalization. It aims to help people explore, understand, and communicate data using computer-supported physical data representations. We call these representations physicalizations, analogously to visualizations -- their purely visual counterpart. In this article, we go beyond the focused research questions addressed so far by delineating the research area, synthesizing its open challenges and laying out a research agenda.

/pdf/opportunities-and-challenges-for-data-physicalization-4mgjx9xk0v.pdf

Opportunities and Challenges for Data Physicalization

Multi-touch interaction with computationally enhanced surfaces has received considerable recent attention. Approaches to the implementation of multi-touch interaction such as Frustrated Total Internal Reflection (FTIR) and Diffused Illumination (DI) have allowed for the low cost development of such surfaces, leading to a number of technology and application innovations. Although many of these techniques have been presented in an academic setting, the practicalities of building a high quality multi-touch enabled surface, both in terms of the software and hardware, are not trivial. \\ This document aims to summarize the knowledge and experience of developers of multi-touch technology who gathered at the Bootcamp on Construction & Implementation of Optical Multi-touch Surfaces at Tabletop 2008 in Amsterdam, and seeks to provide hints and practical advice to people seeking to ``build your own'' multi-touch surface. We mostly focus on technical aspects that are important in the construction of optical multi-touch surfaces, including: infrared illumination, silicone compliant surfaces, projection screens, cameras, filters, and projectors. In addition, we outline how to integrate this hardware to allow users to create a solid multi-touch surface, and provide an overview of existing software libraries for the implementation of multi-touch applications. In addition, we discuss the problem of latency introduced by the different parts of the system. A brief description of most of the common technologies to realize (multi-) touch surfaces is provided; however, the main focus is on those that utilise optical approaches.

https://mediatum.ub.tum.de/doc/1094399/file.pdf

Multi-Touch Surfaces: A Technical Guide

We introduce the term shape resolution, which adds to the existing definitions of screen and touch resolution. We propose a framework, based on a geometric model (Non-Uniform Rational B-splines), which defines a metric for shape resolution in ten features. We illustrate it by comparing the current related work of shape changing devices. We then propose the concept of Morphees that are self-actuated flexible mobile devices adapting their shapes on their own to the context of use in order to offer better affordances. For instance, when a game is launched, the mobile device morphs into a console-like shape by curling two opposite edges to be better grasped with two hands. We then create preliminary prototypes of Morphees in order to explore six different building strategies using advanced shape changing materials (dielectric electro active polymers and shape memory alloys). By comparing the shape resolution of our prototypes, we generate insights to help designers toward creating high shape resolution Morphees.

Morphees: toward high "shape resolution" in self-actuated flexible mobile devices

Practical wearable gesture tracking requires that sensors align with existing ergonomic device forms. We show that combining EMG and pressure data sensed only at the wrist can support accurate classification of hand gestures. A pilot study with unintended EMG electrode pressure variability led to exploration of the approach in greater depth. The EMPress technique senses both finger movements and rotations around the wrist and forearm, covering a wide range of gestures, with an overall 10-fold cross validation classification accuracy of 96%. We show that EMG is especially suited to sensing finger movements, that pressure is suited to sensing wrist and forearm rotations, and their combination is significantly more accurate for a range of gestures than either technique alone. The technique is well suited to existing wearable device forms such as smart watches that are already mounted on the wrist.

/pdf/empress-practical-hand-gesture-classification-with-wrist-4izejpk0br.pdf

EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing

The advantages of paper-based maps have been utilized in the field of mobile Augmented Reality (AR) in the last few years. Traditional paper-based maps provide high-resolution, large-scale information with zero power consumption. There are numerous implementations of magic lens interfaces that combine high-resolution paper maps with dynamic handheld displays. From an HCI perspective, the main challenge of magic lens interfaces is that users have to switch their attention between the magic lens and the information in the background. In this paper, we attempt to overcome this problem by using a lightweight mobile camera projector unit to augment the paper map directly with additional information. The "Map Torchlight" is tracked over a paper map and can precisely highlight points of interest, streets, and areas to give directions or other guidance for interacting with the map.

Map torchlight: a mobile augmented reality camera projector unit

We present a flexible Machine Learning approach for learning user-specific touch input models to increase touch accuracy on mobile devices. The model is based on flexible, non-parametric Gaussian Process regression and is learned using recorded touch inputs. We demonstrate that significant touch accuracy improvements can be obtained when either raw sensor data is used as an input or when the device's reported touch location is used as an input, with the latter marginally outperforming the former. We show that learned offset functions are highly nonlinear and user-specific and that user-specific models outperform models trained on data pooled from several users. Crucially, significant performance improvements can be obtained with a small (≈200) number of training examples, easily obtained for a particular user through a calibration game or from keyboard entry data.

/pdf/a-user-specific-machine-learning-approach-for-improving-33xo50d2pc.pdf

Markus Löchtefeld

Papers

Multi-Touch Surfaces: A Technical Guide

Morphees: toward high "shape resolution" in self-actuated flexible mobile devices

EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing

Map torchlight: a mobile augmented reality camera projector unit

A user-specific machine learning approach for improving touch accuracy on mobile devices