Open AccessPosted Content
Toward Natural Gesture/Speech Control of a Large Display
Reads0
Chats0
TLDR
In this article, a structured approach for studying patterns of multimodal language in the context of a 2D-display control is proposed, where gestures from observable kinematical primitives to their semantics are considered pertinent to a linguistic structure.Abstract:
In recent years because of the advances in computer vision research, free hand gestures have been explored as means of human-computer interaction (HCI). Together with improved speech processing technology it is an important step toward natural multimodal HCI. However, inclusion of non-predefined continuous gestures into a multimodal framework is a challenging problem. In this paper, we propose a structured approach for studying patterns of multimodal language in the context of a 2D-display control. We consider systematic analysis of gestures from observable kinematical primitives to their semantics as pertinent to a linguistic structure. Proposed semantic classification of co-verbal gestures distinguishes six categories based on their spatio-temporal deixis. We discuss evolution of a computational framework for gesture and speech integration which was used to develop an interactive testbed (iMAP). The testbed enabled elicitation of adequate, non-sequential, multimodal patterns in a narrative mode of HCI. Conducted user studies illustrate significance of accounting for the temporal alignment of gesture and speech parts in semantic mapping. Furthermore, co-occurrence analysis of gesture/speech production suggests syntactic organization of gestures at the lexical level.read more
Citations
More filters
Dissertation
Toward understanding human expression in human-robot interaction
TL;DR: A novel multimodal framework is presented to help reduce the total human effort and time required to communicate with intelligent devices by determining human intent using a knowledge-based architecture that combines and leverages conflicting information available across multiple natural communication modes and modalities.
Patent
Recognizing a movement of a pointing device
TL;DR: In this article, a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired.
References
More filters
Journal ArticleDOI
Visual interpretation of hand gestures for human-computer interaction: a review
TL;DR: A fraction of the recycle slurry is treated with sulphuric acid to convert at least some of the gypsum to calcium sulphate hemihydrate and the slurry comprising hemihYDrate is returned to contact the mixture of phosphate rock, phosphoric acid and recycle Gypsum slurry.
Book
Gaze and mutual gaze
Michael Argyle,Mark Cook +1 more
TL;DR: The role of gaze in human social interaction was investigated experimentally by Argyle and Cook as mentioned in this paper, who set up a research group at Oxford with Ted Crossman and Adam Kendon, to study non-verbal communication and gaze as an important aspect of this behaviour.
Proceedings ArticleDOI
Integration and synchronization of input modes during multimodal human-computer interaction
TL;DR: The present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system and revealed that the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence.
Book
Gesture and the Nature of Language
TL;DR: The nature of gesture, signed and spoken languages differently organized, and the origin of syntax: gesture as name and relation are discussed.
Journal ArticleDOI
Toward multimodal human-computer interface
TL;DR: It is clear that further research is needed for interpreting and fitting multiple sensing modalities in the context of HCI and the fundamental issues in integrating them at various levels, from early signal level to intermediate feature level to late decision level.