Augmented Reality for Robotics: A Review

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics.

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

When humans interact with each other, they often make use of deictic gestures such as pointing to help pick out targets of interest to their conversation. In the field of Human-Robot Interaction, research has repeatedly demonstrated the utility of enabling robots to use such gestures as well. Recent work in augmented, mixed, and virtual reality stands to enable enormous advances in robot deixis, both by allowing robots to gesture in ways that were not previously feasible, and by enabling gesture on robotic platforms and environmental contexts in which gesture was not previously feasible. In this paper, we summarize our own recent work on using augmented, mixed, and virtual-reality techniques to advance the state-of-the-art of robot-generated deixis.

Augmented, Mixed, and Virtual Reality Enabling of Robot Deixis

Touch gestures can be a very important aspect when developing mobile applications with enhanced reality. The main purpose of this research was to determine which touch gestures were most frequently used by engineering students when using a simulation of a projectile motion in a mobile AR applica‐ tion. A randomized experimental design was given to students, and the results showed the most commonly used gestures to visualize are: zoom in “pinch open”, zoom out “pinch closed”, move “drag” and spin “rotate”.

Virtual, Augmented and Mixed Reality: Interaction, Navigation, Visualization, Embodiment, and Simulation : 10th International Conference, VAMR 2018, Held as Part of HCI International 2018, Las Vegas, NV, USA, July 15-20, 2018, Proceedings, Part I

This paper aims to illustrate the rapidly growing use of robots in the retail sector by considering a range of products and applications.,Following a short introduction, this paper discusses robotic products and deployments in the following applications: inventory management, the provision of customer assistance, cleaning and hazard detection and product delivery. It concludes with a brief discussion.,Robots are rapidly making inroads in the retail sector. They have been shown to reduce manpower requirements, improve efficiency, reduce costs and speed-up deliveries. As applications grow and the technologies mature, retail is expected to become a major robot user and will offer significant business opportunities.,This study provides a detailed insight into the emerging and rapidly developing retail robotics business.

Strong prospects for robots in retail

Policing of road traffic is listed in the most hazardous tasks since many traffic police personnel injured in a series of accidents at intersections A life-size traffic cop robot called “IWI” is invented as an alternative to human traffic police in the street for directing traffic on point duty, and this paper proposes a human-robot cooperation scheme with wearable augmented glasses as an HMI to obtain a more immersing teleoperation experience with the aid of wearable augmented reality glasses First, a humanoid robot is tailored for mobile surveillance with raspberry pi camera based eyes and omnidirectional Mecanum wheels Second, the real-time video stream acquired by the on-site robot is distributed to multiple terminals, such as the central sever, the wearable augmented reality glasses and the mobile control tablet Third, with a depth of focus estimation the augmented indications are displayed to aid to understand the remote scenario Finally, the human policing results, such as STOP, PULL OVER, TURN-LEFT, are compiled and programmed as simplified patterns to control the robot body/hand pantomime Experimental results show that the proposed methodology and control scheme is feasible in real-time application with high real-time performance of less than 05s latency, and open possibilities of easing the traffic jam via simultaneously scheduling multiple traffic cop robots

Real-time human-in-the-loop remote control for a life-size traffic police robot with multiple augmented reality aided display terminals

Precisely imitating human motions in real-time poses a challenge for the robots due to difference in their physical structures. This paper proposes a human–computer interaction method for remotely manipulating life-size humanoid robots with a new metrics for evaluating motion similarity. First, we establish a motion capture system to acquire the operator’s motion data and retarget it to the standard bone model. Secondly, we develop a fast mapping algorithm, by mapping the BVH (BioVision Hierarchy) data collected by the motion capture system to each joint motion angle of the robot to realize the imitated motion control of the humanoid robot. Thirdly, a DTW (Dynamic Time Warping)-based trajectory evaluation method is proposed to quantitatively evaluate the difference between robot trajectory and human motion, and meanwhile, visualization terminals render it more convenient to make comparisons between two different but simultaneous motion systems. We design a complex gesture simulation experiment to verify the feasibility and real-time performance of the control method. The proposed human-in-the-loop imitation control method addresses a prominent non-isostructural retargeting problem between human and robot, enhances robot interaction capability in a more natural way, and improves robot adaptability to uncertain and dynamic environments.

/pdf/motion-similarity-evaluation-between-human-and-a-tri-co-2lf7yzm0.pdf

Motion Similarity Evaluation between Human and a Tri-Co Robot during Real-Time Imitation with a Trajectory Dynamic Time Warping Model

This paper aims to present a human-in-the-loop natural teaching paradigm based on scene-motion cross-modal perception, which facilitates the manipulation intelligence and robot teleoperation.,The proposed natural teaching paradigm is used to telemanipulate a life-size humanoid robot in response to a complicated working scenario. First, a vision sensor is used to project mission scenes onto virtual reality glasses for human-in-the-loop reactions. Second, motion capture system is established to retarget eye-body synergic movements to a skeletal model. Third, real-time data transfer is realized through publish-subscribe messaging mechanism in robot operating system. Next, joint angles are computed through a fast mapping algorithm and sent to a slave controller through a serial port. Finally, visualization terminals render it convenient to make comparisons between two motion systems.,Experimentation in various industrial mission scenes, such as approaching flanges, shows the numerous advantages brought by natural teaching, including being real-time, high accuracy, repeatability and dexterity.,The proposed paradigm realizes the natural cross-modal combination of perception information and enhances the working capacity and flexibility of industrial robots, paving a new way for effective robot teaching and autonomous learning.

Natural teaching for humanoid robot via human-in-the-loop scene-motion cross-modal perception

Inspired by the concept of Tri-Co Robots, a real-time motion imitation based human-robot interaction method is proposed to telemanipulate a life-size humanoid robot. First, a motion capture system is established to retarget the accurate motion data to a skeletal model. Second, the real-time data transfer for a human-in-the-loop control system is realized through the publish-subscribe messaging mechanism in ROS (Robot Operating System). Third, a fast mapping algorithm is developed to convert BVH (BioVision Hierarchy) data into corresponding joint angles which are then encapsulated in a designed communication protocol and sent to a low-level slave controller through a serial port. Finally, visualization terminals render it more convenient to make comparisons between two different but simultaneous motion systems. Experimentation on complicated gesture imitation shows the feasibility of the proposed methodology, and the synchronous latency of less than 0.5 seconds validated its real-time performance. The proposed human-in-the-loop imitation control method enhances robot interaction capability in a more natural way and improves robot adaptability to uncertain and dynamic environments.

Human-robot Interaction Oriented Human-in-the-loop Real-time Motion Imitation on a Humanoid Tri-Co Robot

This paper presents a robot prototype designed to overcome the lack of current security monitoring system: low flexibility, limited monitoring range, disability of copping with sudden special circumstances. It’s designed and built for remote live-video security surveillance and object recognition and capture in highly structured small space environment in industrial field, involving remote controlling, live video taking, distance and position identifying of an object, trajectory planning and object capturing. The robot is controlled by means of human-machine interface. The operator drives the robotic device and performs the related tasks by means of a control user interface developed on android APP. The robot computer operates robot grasping and manipulation automatically, that is, performs the precise localization of a specific object, computes adequate capturing sequence and controls the motion (robot moving and object capturing) of all the mechanical components. Throughout this paper, the specific design of every module of the robotic device is presented. The device has been built. Related laboratory tests have been done to check the robot performance. Results show excellent robot controllability, targeted object finder and live field operations. Some figures and tables showing overall performance are given.

Xudong Li

Papers

Real-time human-in-the-loop remote control for a life-size traffic police robot with multiple augmented reality aided display terminals

Motion Similarity Evaluation between Human and a Tri-Co Robot during Real-Time Imitation with a Trajectory Dynamic Time Warping Model

Natural teaching for humanoid robot via human-in-the-loop scene-motion cross-modal perception

Human-robot Interaction Oriented Human-in-the-loop Real-time Motion Imitation on a Humanoid Tri-Co Robot

Remote Live-Video Security Surveillance via Mobile Robot with Raspberry Pi IP Camera