https://www.ee.columbia.edu/%7Eajaimes/Pubs/ajaimes_iccv05.pdf

Multimodal human-computer interaction: A survey

Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

Architecture for controlling a computer using hand gestures

Methods can include using a doorbell to wirelessly communicate with a remotely located computing device. Doorbells can include a speaker, a microphone, a camera, and a button to sound a chime. A doorbell shipping mode can detect whether the doorbell is electrically coupled to an external power source. Methods can include entering a setup mode or a standby mode in response to detecting electricity from the external power source.

Doorbell communication systems and methods

In this paper we review the major approaches to multimodal human computer interaction from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition, and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.

Multimodal human computer interaction: a survey

Computer implemented method for interactively animating an avatar in response to real world input are provided. The avatar can be transmitted between a first user and a second user using a computer program that is executed on at least one computer in a computer network. Additionally, the first user and the second user each interact using a respective computing system that is at least partially executing the computer program. The method is initiated by identifying components of the avatar representing the first user that can be modified using real-time effects. The method continues by the computing system identifying controller input from either the first user or the second user. The identification of the controller input is used to determine which of the identified components of the avatar representing the first user will be modified. In response to the identified controller input the real-time effects are applied to the identified components of the avatar that represent the first user. The avatar of the first user augmented to reflect the application of the real-time effects. In another operation, the method displays the augmented avatar of the first user on a screen connected to the computing system of one or both of the first and second users.

Methods for interactive communications with real time effects and avatar environment interaction

The present invention is a method and apparatus for providing an enhanced automatic drive-thru experience to the customers in a vehicle by allowing use of natural hand gestures to interact with digital content. The invention is named Virtual Touch Ordering System (VTOS). In the VTOS, the virtual touch interaction is defined to be a contact free interaction, in which a user is able to select graphical objects within the digital contents on a display system and is able to control the processes connected to the graphical objects, by natural hand gestures without touching any physical devices, such as a keyboard or a touch screen. Using the virtual touch interaction of the VTOS, the user is able to complete transactions or ordering, without leaving the car and without any physical contact with the display. A plurality of Computer Vision algorithms in the VTOS processes a plurality of input image sequences from the image-capturing system that is pointed at the customers in a vehicle and performs the virtual touch interaction by natural hand gestures. The invention can increase the throughput of drive-thru interaction and reduce the delay in wait time, labor cost, and maintenance cost.

Method and apparatus for providing virtual touch interaction in the drive-thru

The present invention is a system and method for detecting and analyzing motion patterns of individuals present at a multimedia computer terminal from a stream of video frames generated by a video camera and the method of providing visual feedback of the extracted information to aid the interaction process between a user and the system. The method allows multiple people to be present in front of the computer terminal and yet allow one active user to make selections on the computer display. Thus the invention can be used as method for contact-free human-computer interaction in a public place, where the computer terminal can be positioned in a variety of configurations including behind a transparent glass window or at a height or location where the user cannot touch the terminal physically.

Method and system for detecting conscious hand movement patterns and computer-generated visual feedback for facilitating human-computer interaction

The present invention is a method and apparatus for attracting the attention of people in public places and engaging them in a touch-free interaction with a multimedia display using an image-capturing system and a set of Computer Vision algorithms as a means of informing the public as well as collecting data about/from the users. The invention is named, Virtual Touch Entertainment (VTE) Platform. The VTE Platform comprises of a series of interaction states, such as the Wait State, the Attraction State, the User Engagement State, the User Interaction State, and the Interaction Termination State. The modules in these interaction states handle complicated tasks assigned to them, such as attracting the users, training the users, providing the multimedia digital content to the users, and collecting the user data and statistics, in an efficient and intelligent manner. The user is able to experience a whole new way of interaction paradigm while getting information and entertainment through the rich digital multimedia. The system operates automatically and dynamically in real-time throughout the whole interaction process.

Method and system for virtual touch entertainment

Progress in computer vision and speech recognition technologies has recently enabled multimodal interfaces that use speech and gestures. These technologies o er promising alternatives to existing interfaces because they emulate the natural way in which humans communicate. However, no systematic work has been reported that formally evaluates the new speech/gesture interfaces. This paper is concerned with formal experimental evaluation of new human-computer interactions enabled by speech and hand gestures.The paper describes an experiment conducted with 23 subjects that evaluates selection strategies for interaction with large screen displays. The multimodal interface designed for this experiment does not require the user to be in physical contact with any device. Video cameras and long range microphones are used as input for the system. Three selection strategies are evaluated and results for Different target sizes and positions are reported in terms of accuracy, selection times and user preference. Design implications for vision/speech based interfaces are inferred from these results. This study also raises new question and topics for future research.

Experimental evaluation of vision and speech based multimodal interfaces

This paper presents a multimodal crisis management system (XISM). It employs processing of natural gesture and speech commands elicited by a user to efficiently manage complex dynamic emergency scenarios on a large display. The developed prototype system demonstrates the means of incorporating unconstrained free-hand gestures and speech in a real-time interactive interface. This paper provides insights into the design aspects of the XISM system. In particular, it addresses the issues of extraction and fusion of gesture and speech modalities to allow more natural interactive behavior. Performance characteristics of the current prototype and considerations for future work are discussed. A series of studies indicated positive response with respect to ease of interacting with the current system.

/pdf/multimodal-human-computer-interaction-for-crisis-management-3sv3c3kwid.pdf

Emilio Schapira

Papers

Method and apparatus for providing virtual touch interaction in the drive-thru

Method and system for detecting conscious hand movement patterns and computer-generated visual feedback for facilitating human-computer interaction

Method and system for virtual touch entertainment

Experimental evaluation of vision and speech based multimodal interfaces

Multimodal human-computer interaction for crisis management systems