We present insights from a gesture elicitation study conducted for TV control, during which 18 participants contributed gesture commands and rated the execution difficulty and recall likeliness of free-hand gestures for 21 television control tasks Our study complements previous work on gesture interaction design for the TV set with the first exploration of fine-grained resolution 3-D finger movements and hand gestures We report lower agreement rates than previous gesture studies (
 $${\mathcal {AR}}=158$$
 ) with 728 % recall rate and 158 % false positives, results that are explained by the complexity and variability of unconstrained finger and hand gestures However, our observations also confirm previous findings, such as people preferring related gestures for dichotomous tasks and more disagreement occurring for abstract tasks, such as “open browser” or “show the list of channels” for our specific TV scenario To reach a better understanding of our participants’ preferences for articulating finger and hand gestures, we defined five measures for Leap Motion gestures, such as gesture volume and finger-to-palm distance, which we employed to evaluate gestures performed by our participants We also contribute a set of guidelines for practitioners interested in designing free-hand gestures for interactive TV scenarios involving similar gesture acquisition technology We release our dataset consisting in 378 Leap Motion gestures described by fingertips position, direction, and velocity coordinates to foster further studies in the community This first exploration of viewers’ preferences for fine-grained resolution free-hand gestures for TV control represents one more step toward designing low-effort gesture interfaces for lean-back interaction with the TV set

On free-hand TV control: experimental results on user-elicited gestures with Leap Motion

This paper presents a method for designing tactile macrotextures with magnetic rubber sheets. In the method, named "Magnetic Plotter", a desktop digital plotting machine combined with a tiny neodymium magnet writes fine magnetic patterns on the surface of the magnetic rubber sheets. This method enables users to design magnetic fields freely with inexpensive commercially available materials as if they are drawing pictures. Moreover, when the magnetic sheets are rubbed together, unique haptic stimuli are displayed on the fingers. The haptic stimuli can be changed by the magnetic patterns designed on the rubber sheets. We developed a prototype of the Magnetic Plotter and investigated the range of the generated haptic stimuli and the texture design possibilities.

Magnetic Plotter: A Macrotexture Design Method Using Magnetic Rubber Sheets

Natural language acquisition can be divided into two distinct phases In the pre-competent language phase, early language acquisition, emphasis is on the direct experiences of the concrete surroundings In this establishing phase of language, co-occurring sensory–motor activity is crucial for language acquisition to occur Therefore, in the pre-competent language phase, the innate perceptivity to the surroundings dominates the character of the resulting neural correlate In the second phase, in post-competent language acquisition, emphasis is less on the concrete surroundings and more on interlocutors in conversation Based on competent language use, the relevance of perceptivity to online surroundings is reduced Thus, the role of the interlocutor is to elicit vicarious ‘as-if’ experiences in the learner in a process named ‘derived embodiment’ Based on contemporary neuroscience, first I analyse why direct experiences are central to the establishing phase through the notion of ‘linguification’ Second, I discuss why later conversations in the post-competent language phase substitute for direct experiences To demonstrate the challenges facing biologically inspired cognitive architectures, I propose a model pertaining to natural language acquisition in humans The model asserts that: • Language acquisition is a continuous process in which the pre-competent phase grounds the later post-competent phase • The character of the relevant online perceptions changes from the processing of concrete surroundings to the processing of words • Human language acquisition depends both on innate perceptivity working bottom-up and social exchanges working top-down in which the interlocutor directs the attention of the learner

Doubletalk – The biological and social acquisition of language

Developing computational models of spatial prepositions (such as on, in, above, etc.) is crucial for such tasks as human-machine collaboration, story understanding, and 3D model generation from descriptions. However, these prepositions are notoriously vague and ambiguous, with meanings depending on the types, shapes and sizes of entities in the argument positions, the physical and task context, and other factors. As a result truth value judgments for prepositional relations are often uncertain and variable. In this paper we treat the modeling task as calling for assignment of probabilities to such relations as a function of multiple factors, where such probabilities can be viewed as estimates of whether humans would judge the relations to hold in given circumstances. We implemented our models in a 3D blocks world and a room world in a computer graphics setting, and found that true/false judgments based on these models do not differ much more from human judgments that the latter differ from one another. However, what really matters pragmatically is not the accuracy of truth value judgments but whether, for instance, the computer models suffice for identifying objects described in terms of prepositional relations, (e.g., “the box to the left of the table”, where there are multiple boxes). For such tasks, our models achieved accuracies above 90% for most relations.

/pdf/computational-models-for-spatial-prepositions-1vpmcdemga.pdf

Computational Models for Spatial Prepositions

https://par.nsf.gov/servlets/purl/10065982

Visual mental imagery: A view from artificial intelligence.

There is ample evidence that human understanding of ordinary language relies in part on a rich capacity for imagistic mental modeling. We argue that genuine language understanding in machines will similarly require an imagistic modeling capacity enabling fast construction of instances of prototypical physical situations and events, whose participants are drawn from a wide variety of entity types, including animate agents. By allowing fast evaluation of predicates such as ‘can-see’, ‘under’, and ‘inside’, these model instances support coherent text interpretation. Imagistic modeling is thus a crucial – and not very broadly appreciated – aspect of the long-standing knowledge acquisition bottleneck in AI. We will illustrate how the need for imagistic modeling arises even in the simplest first-reader stories for children, and provide an initial feasibility study to indicate what the architecture of a system combining symbolic with imagistic understanding might look like.

/pdf/on-the-need-for-imagistic-modeling-in-story-understanding-258ey1hfdg.pdf

On the Need for Imagistic Modeling in Story Understanding

This paper describes the development of a working prototype of a novel free-space haptic human-computer interface called MIDAS Touch. MIDAS Touch works by applying physical forces to a user's finger through the production of a dynamic magnetic field. The magnetic field strength is adjusted in real-time based on the user's movement of his/her finger. A user's hand/finger motion in the real world is mapped to movement of a virtual finger in a virtual world through the use of a Leap Motion 3D tracking sensor. As a person's virtual finger collides with objects in the virtual world, the magnetic field strength is varied. In this demo, we present a case of MIDAS Touch coupled to a standard PC as a computer drawing viewer and drawing application for helping individuals with visual impairment feel what they or others have drawn.

Alex T. Wilson

Papers

On the Need for Imagistic Modeling in Story Understanding

M.I.D.A.S. touch: magnetic interactive device for alternative sight through touch