This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.

/pdf/a-survey-of-augmented-reality-17irtaxt7z.pdf

A survey of augmented reality

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.

/pdf/a-tutorial-on-visual-servo-control-5cy6rje63g.pdf

A tutorial on visual servo control

Preface. 1. Introduction. 2. Rigid Motions and Homogeneous Transformations. 3. Forward and Inverse Kinematics. 4. Velocity Kinematics-The Jacobian. 5. Path and Trajectory Planning. 6. Independent Joint Control. 7. Dynamics. 8. Multivariable Control. 9. Force Control. 10. Geometric Nonlinear Control. 11. Computer Vision. 12. Vision-Based Control. Appendix A: Trigonometry. Appendix B: Linear Algebra. Appendix C: Dynamical Systems. Appendix D: Lyapunov Stability. Index.

/pdf/robot-modeling-and-control-2n3memrp6g.pdf

Robot Modeling and Control

https://www.cs.cmu.edu/~illah/PAPERS/socialroboticssurvey.pdf

A survey of socially interactive robots

Computer vision systems for monitoring people and collecting valuable demographic information in a social environment is an important research problem. It is expected that such a system will play an increasingly important role in enhancing user's experience and can significantly improve the intelligibility of a human computer interaction (HCI) system. For example, a robust gender classification system can provide a basis for passive surveillance and access to a smart building using demographic information or can provide valuable consumer statistics in a public place. The option of an audio cue in addition to the visual cue promises a robust solution with high accuracy and ease-of-use in human computer interaction systems. This paper investigates gender classification using Support Vector Machines (SVMs). The visual (thumbnail frontal face) and the audio (features from speech data) cues were considered for designing the classifier. Three different representations of the data, namely, raw data, principle component analysis (PCA) and non-negative matrix factorization (NMF) were used for the experimentation with visual signal. For speech, mel-cepstral coefficient and pitch were used for the experimentation. It was found that the best overall classification rates obtained using the SVM for the visual and speech data were 95.31% and 100%, respectively, on data set collected in laboratory environment. The performance of the SVM was compared with two simple classifiers namely, the nearest prototype neighbor and the k-nearest neighbor on all feature sets. It was found that the SVM outperformed the other two classifiers on all datasets. To further understand the robustness issues, the proposed approach has been applied on a large balanced (roughly equal distribution of gender, ethnicity and age group) data-base consisting of 8000 faces collected in real world environment. While, the results are very promising it indicates more to be done to make a statistically meaningful conclusion.

Support vector learning for gender classification using audio and visual cues

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.

Prosody based audiovisual coanalysis for coverbal gesture recognition

For many surgical procedures, computer-based training has become an increasingly attractive alternative to traditional training methods. One of the key problems in computer-based surgical training is automatic skill evaluation, which in turn requires skill modelling. To model and evaluate human skills it is necessary to provide the means by which the skills can be measured and interpreted by computers. This paper proposes a new approach for observing continuous, long sequence of hand movements in surgical operation, and then modelling and evaluating the skill demonstrated in the observation. This involves a video-based technique for tracking the hand during a surgical exercise. Because of the non-contact nature of the tracking technique, there is minimal interference with the skill execution, unlike other methods that employ instrumented gloves. To increase the robustness of hand tracking, a Kalman filter (see Appendix A for the system and measurement model) is employed together with a set of coloured markers on the surgical glove. For modelling the surgical skill, a stochastic approach is proposed that uses Hidden Markov Models (HMMs). Using this technique, person-independent models can be developed through human demonstration of particular surgical skills. To automatically evaluate a person’s skill, an objective evaluation criterion is proposed that is based on the log probability of an observation sequence for the given HMM. The probability measures the stochastic similarity between the performance of the observation sequence and the performance represented by the model. This paper also describes an implemented prototype system and experiments that establish the feasibility of the proposed approach for surgical skill modelling and evaluation.

Visual modelling and evaluation of surgical skill

A novel interface system for accessing geospatial data (GeoMIP) has been developed that realizes a user-centered multimodal speech/gesture interface for addressing some of the critical needs in crisis management. In this system we primarily developed vision sensing algorithms, speech integration, multimodality fusion, and rule-based mapping of multimodal user input to GIS database queries. A demo system of this interface has been developed for the Port Authority NJ/NY and is explained here.

Multimodal interface platform for geographical information systems (GeoMIP) in crisis management

Knowledge of the complex molecular structures of living cells is being accumulated at a tremendous rate Key technologies enabling this success have been, high performance computing and powerful molecular graphics applications, but the technology is beginning to seriously lag behind challenges posed by the size and number of new structures and by the emerging opportunities in drug design and genetic engineering A visual computing environment is being developed which permits interactive modeling of biopolymers by linking a 3D molecular graphics program with an efficient molecular dynamics simulation program executed on remote high-performance parallel computers The system will be ideally suited for distributed computing environments, by utilizing both local 3D graphics facilities and the peak capacity of high-performance computers for the purpose of interactive biomolecular modeling To create an interactive 3D environment three input methods will be explored: (1) a six degree of freedom "mouse" for controlling the space shared by the model and the user; (2) voice commands monitored through a microphone and recognized by a speech recognition interface; (3) hand gestures, detected through cameras and interpreted using computer vision techniques Controlling 3D graphics connected to real time simulations and the use of voice with suitable language semantics, as well as hand gestures, promise great benefits for many types of problem solving environments Our focus on structural biology takes advantage of existing sophisticated software, provides concrete objectives, defines a well-posed domain of tasks and offers a well-developed vocabulary for spoken communication

Rajeev Sharma

Papers

Support vector learning for gender classification using audio and visual cues

Prosody based audiovisual coanalysis for coverbal gesture recognition

Visual modelling and evaluation of surgical skill

Multimodal interface platform for geographical information systems (GeoMIP) in crisis management

A visual computing environment for very large scale biomolecular modeling