Paul Milgram received the BASc degree from the University of Toronto in 1970, the MSEE degree from the Technion (Israel) in 1973 and the PhD degree from the University of Toronto in 1980 From 1980 to 1982 he was a ZWO Visiting Scientist and a NATO Postdoctoral in the Netherlands, researching automobile driving behaviour From 1982 to 1984 he was a Senior Research Engineer in Human Engineering at the National Aerospace Laboratory (NLR) in Amsterdam, where his work involved the modelling of aircraft flight crew activity, advanced display concepts and control loops with human operators in space teleoperation Since 1986 he has worked at the Industrial Engineering Department of the University of Toronto, where he is currently an Associate Professor and Coordinator of the Human Factors Engineering group He is also cross appointed to the Department of Psychology In 1993-94 he was an invited researcher at the ATR Communication Systems Research Laboratories, in Kyoto, Japan His research interests include display and control issues in telerobotics and virtual environments, stereoscopic video and computer graphics, cognitive engineering, and human factors issues in medicine He is also President of Translucent Technologies, a company which produces "Plato" liquid crystal visual occlusion spectacles (of which he is the inventor), for visual and psychomotor research

/pdf/a-taxonomy-of-mixed-reality-visual-displays-54v3x9pjl3.pdf

A Taxonomy of Mixed Reality Visual Displays

Methods and systems for processing touch inputs are disclosed. The invention in one respect includes reading data from a multipoint sensing device such as a multipoint touch screen where the data pertains to touch input with respect to the multipoint sensing device, and identifying at least one multipoint gesture based on the data from the multipoint sensing device.

Gestures for touch sensitive input devices

Disclosed herein is a multi-functional hand-held device capable of configuring user inputs based on how the device is to be used. Preferably, the multifunctional hand-held device has at most only a few physical buttons, keys, or switches so that its display size can be substantially increased. The multifunctional hand-held device also incorporates a variety of input mechanisms, including touch sensitive screens, touch sensitive housings, display actuators, audio input, etc. The device also incorporates a user-configurable GUI for each of the multiple functions of the devices.

Multi-functional hand-held device

In this paper we discuss augmented reality (AR) displays in a general sense, within the context of a reality-virtuality (RV) continuum, encompassing a large class of `mixed reality' (MR) displays, which also includes augmented virtuality (AV). MR displays are defined by means of seven examples of existing display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish different MR display systems from each other are presented, first by means of a table in which the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and then by means of a three dimensional taxonomic framework comprising: extent of world knowledge, reproduction fidelity, and extent of presence metaphor. A principal objective of the taxonomy is to clarify terminology issues and to provide a framework for classifying research across different disciplines.

/pdf/augmented-reality-a-class-of-displays-on-the-reality-3h6y5kudax.pdf

Augmented Reality : A class of displays on the reality-virtuality continuum

A system for performing minimally invasive cardiac procedures. The system includes a pair of surgical instruments that are coupled to a pair of robotic arms. The instruments have end effectors that can be manipulated to hold and suture tissue. The robotic arms are coupled to a pair of master handles by a controller. The handles can be moved by the surgeon to produce a corresponding movement of the end effectors. The movement of the handles is scaled so that the end effectors have a corresponding movement that is different, typically smaller, than the movement performed by the hands of the surgeon. The scale factor is adjustable so that the surgeon can control the resolution of the end effector movement. The movement of the end effector can be controlled by an input button, so that the end effector only moves when the button is depressed by the surgeon. The input button allows the surgeon to adjust the position of the handles without moving the end effector, so that the handles can be moved to a more comfortable position. The system may also have a robotically controlled endoscope which allows the surgeon to remotely view the surgical site. A cardiac procedure can be performed by making small incisions in the patient's skin and inserting the instruments and endoscope into the patient. The surgeon manipulates the handles and moves the end effectors to perform a cardiac procedure such as a coronary artery bypass graft.

Method and apparatus for performing minimally invasive cardiac procedures

A haptic feedback planar touch control used to provide input to a computer. A touch input device includes a planar touch surface that inputs a position signal to a processor of the computer based on a location of user contact on the touch surface. The computer can position a cursor in a displayed graphical environment based at least in part on the position signal, or perform a different function. At least one actuator is also coupled to the touch input device and outputs a force to provide a haptic sensation to the user contacting the touch surface. The touch input device can be a touchpad separate from the computer's display screen, or can be a touch screen. Output haptic sensations on the touch input device can include pulses, vibrations, and spatial textures. The touch input device can include multiple different regions to control different computer functions.

Haptic feedback for touchpads and other touch control

In order to enhance operator performance and understanding within remote environments, most research and development of telepresence systems has been directed towards improving the fidelity of the link between operator and environment. Although higher fidelity interfaces are important to the advancement of a telepresence system, the beneficial effects of corrupting the link between operator and remote environment by introducing abstract perceptual information into the interface called virtual fixtures are described. >

Virtual fixtures: Perceptual tools for telerobotic manipulation

A haptic feedback touch control used to provide input to a computer. A touch input device includes a planar touch surface that provides position information to a computer based on a location of user contact. The computer can position a cursor in a displayed graphical environment based at least in part on the position information, or perform a different function. At least one actuator is also coupled to the touch input device and outputs a force to provide a haptic sensation to the user. The actuator can move the touchpad laterally, or a separate surface member can be actuated. A flat E-core actuator, piezoelectric actuator, or other types of actuators can be used to provide forces. The touch input device can include multiple different regions to control different computer functions.

Haptic interface for laptop computers and other portable devices

An apparatus (14) for controlling and providing force feedback using an interface device manipulated by a user and connected to a host computer system (12). A microprocessor (26) is provided local to the interface device and reads sensor data from sensors that describes the positioning of a user object moved by the user, such as a joystick. The microprocessor controls actuators (30) to provide forces on the user object. The host computer sends high level host commands to the local microprocessor and the microprocessor independently implements a local reflex process based on the high level command to provide force values to actuators using sensor data and other parameters. A host command protocol includes a variety of different types of host commands and associated command parameters. By providing a relatively small set of high level host commands, the protocol further shifts the computational burden from the host computer to the local microprocessor.

Method and apparatus for controlling force feedback interface systems utilizing a host computer

A method and apparatus for interfacing the motion of a user-manipulable object with an electrical or computer system includes a user object physically contacted by a user. A gimbal mechanism is coupled to the user object, such as a joystick or a medical tool, and provides at least two degrees of freedom to the user object. The gimbal mechanism preferably includes multiple members, at least two of which are formed as a unitary member which provides flex between the selected members. An actuator applies a force along a degree of freedom to the user object in response to electrical signals produced by the computer system. A sensor detects a position of the user object along the degree of freedom and outputs sensor signals to the computer system. Another embodiment includes a host computer system and a local microprocessor, separate from the host computer, for communicating with the host computer and controlling the forces output by the actuators according to a processor subroutine selected in accordance with a host command, sensor signals, and timing information. Another embodiment of the interface apparatus uses voice coil actuators that produce forces in either linear or rotary degrees of freedom using currents applied in a magnetic fields. A friction drive mechanism of the present invention can be coupled between an actuator and a gimbal mechanism. Force from the actuator is transmitted to the gimbal mechanism through frictional contact of members of the friction drive mechanism.

Louis B. Rosenberg

Papers

Haptic feedback for touchpads and other touch control

Virtual fixtures: Perceptual tools for telerobotic manipulation

Haptic interface for laptop computers and other portable devices

Method and apparatus for controlling force feedback interface systems utilizing a host computer

Computer interface apparatus including linkage having flex