A method for man machine interaction with an electronic device associated with an electronic display comprises capturing images of at least one hand positioned over an input device, tracking position or posture of the hand from the images; switching from interaction based on interaction with an input device to pointing device emulation in response to detecting a gesture performed with the hand, and emulating a pointing device based on the tracking, with the hand no longer performing the gesture.

Vision based pointing device emulation

A system and method are provided for controlling a device based on computer vision. Embodiments of the system and method of the invention are based on receiving a sequence of images of a field of view; detecting movement of at least one object in the images; applying a shape recognition algorithm on the at least one moving object; confirming that the object is a user hand by combining information from at least two images of the object; and tracking the object to control the device.

Computer vision gesture based control of a device

An electronic device coupleable to a display screen includes a camera system that acquires optical data of a user comfortably gesturing in a user-customizable interaction zone having a z 0 plane, while controlling operation of the device. Subtle gestures include hand movements commenced in a dynamically resizable and relocatable interaction zone. Preferably (x,y,z) locations in the interaction zone are mapped to two-dimensional display screen locations. Detected user hand movements can signal the device that an interaction is occurring in gesture mode. Device response includes presenting GUI on the display screen, creating user feedback including haptic feedback. User three-dimensional interaction can manipulate displayed virtual objects, including releasing such objects. User hand gesture trajectory clues enable the device to anticipate probable user intent and to appropriately update display screen renderings.

Method and system enabling natural user interface gestures with an electronic system

As a model is only an abstraction of the real system, unmodeled dynamics, parameter variations, and disturbances can result in poor performance of a conventional controller based on this model. In such cases, a conventional controller cannot remain well tuned. This paper presents the control of a spherical rolling robot by using an adaptive neuro-fuzzy controller in combination with a sliding-mode control (SMC)-theory-based learning algorithm. The proposed control structure consists of a neuro-fuzzy network and a conventional controller which is used to guarantee the asymptotic stability of the system in a compact space. The parameter updating rules of the neuro-fuzzy system using SMC theory are derived, and the stability of the learning is proven using a Lyapunov function. The simulation results show that the control scheme with the proposed SMC-theory-based learning algorithm is able to not only eliminate the steady-state error but also improve the transient response performance of the spherical rolling robot without knowing its dynamic equations.

/pdf/adaptive-neuro-fuzzy-control-of-a-spherical-rolling-robot-4lun26e1i3.pdf

Adaptive Neuro-Fuzzy Control of a Spherical Rolling Robot Using Sliding-Mode-Control-Theory-Based Online Learning Algorithm

We present HaptoMime, a mid-air interaction system that allows users to touch a floating virtual screen with hands-free tactile feedback. Floating images formed by tailored light beams are inherently lacking in tactile feedback. Here we propose a method to superpose hands-free tactile feedback on such a floating image using ultrasound. By tracking a fingertip with an electronically steerable ultrasonic beam, the fingertip encounters a mechanical force consistent with the floating image. We demonstrate and characterize the proposed transmission scheme and discuss promising applications with an emphasis that it helps us 'pantomime' in mid-air.

HaptoMime: mid-air haptic interaction with a floating virtual screen

We propose a new imaging optics called the Transmissive Mirror Device (TMD). It consists of numerous micromirrors 
placed perpendicular to the surface of a flat, thin metal plate. The micro-mirror array is implemented 
by the inner walls of minute square holes, which are densely pitted on the device. The basic mode of operation 
is based on two reflections by a pair of adjacent mutually perpendicular mirrors, i.e., a dihedral corner reflector. 
Although the principal of operation is based on reflection by mirrors, the device is also transmissive and deflects 
light. Since this imaging system forms a real image at a plane symmetric point, the depth of the 3D image is 
inverted. Its optical defects are low optical transmittance and stray light caused by non-reflected light and that 
reflected once. We manufactured the device experimentally with nano-precision machining technology and also 
evaluated it.

Transmissive optical imaging device with micromirror array

As optical elements for bending a light beam when it passes through an element face constituting one plane, a plurality of unit optical elements for reflecting light on one or more mirror surfaces arranged perpendicularly or at an angle nearly perpendicular to the element face are disposed to constitute an optical imaging element. Light emitted from projection object disposed on one side of the element face is reflected on the mirror surface when it passes through the element face so that a real image is formed in a space on the other side of the element face where a physical entity does not exist, thus constituting a display equipped with the optical imaging element. An aerial picture projection system not existing heretofore is also provided.

Imageing element and display

This paper deals with the problem of controlling the position and attitude of a spherical rolling robot. The driving torque for the rolling robot is generated by a new type of mechanism equipped with a gyro. We derive two types of models, a kinematic model and a dynamic model, from the equations of motion for the robot. A feedback controller for the kinematic model is designed by using our previous method that is based on Lyapunov control. And then, from the controller, we obtain a feedback controller for the dynamic model by backstepping technique. The effectiveness of both controllers is demonstrated by numerical simulations

Position and attitude control of a spherical rolling robot equipped with a gyro

A real mirror image forming optical system is constructed from a half-mirror, and from a retroreflector placed in a position where it recursively reflects either the light passing through or reflected by that half-mirror, so that light emitted from an object to be projected placed behind the half-mirror can pass through either a first light path consisting of passing through the half-mirror, recursive reflection by the retroreflector, and reflection by the half-mirror; or a second light path consisting of reflection by the half-mirror, recursive reflection by the retroreflector, and passing through the half-mirror; thus by one or both of these light paths a real mirror image of the object to be projected is formed at a planar symmetric position with respect to the half-mirror surface.

Floating image interaction device and its program

A volumetric display system using an optical imaging device consisting of numerous dihedral corner reflectors placed perpendicular to the surface of a metal plate is proposed. Image formation by the dihedral corner reflector array (DCRA) is free from distortion and focal length. In the proposed volumetric display system, a two-dimensional real image is moved by a mirror scanner to scan a three-dimensional (3D) space. Cross-sectional images of a 3D object are displayed in accordance with the position of the image plane. A volumetric image is observed as a stack of the cross-sectional images. The use of the DCRA brings compact system configuration and volumetric real image generation with very low distortion. An experimental volumetric display system including a DCRA, a galvanometer mirror, and a digital micro-mirror device was constructed to verify the proposed method. A volumetric image consisting of 1024×768×400 voxels was formed by the experimental system.

Satoshi Maekawa

Papers

Transmissive optical imaging device with micromirror array

Imageing element and display

Position and attitude control of a spherical rolling robot equipped with a gyro

Floating image interaction device and its program

Floating volumetric image formation using a dihedral corner reflector array device