This paper presents ORB-SLAM3, the first system able to perform visual, visual-inertial and multi-map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models. The first main novelty is a feature-based tightly-integrated visual-inertial SLAM system that fully relies on Maximum-a-Posteriori (MAP) estimation, even during the IMU initialization phase. The result is a system that operates robustly in real-time, in small and large, indoor and outdoor environments, and is 2 to 5 times more accurate than previous approaches. The second main novelty is a multiple map system that relies on a new place recognition method with improved recall. Thanks to it, ORB-SLAM3 is able to survive to long periods of poor visual information: when it gets lost, it starts a new map that will be seamlessly merged with previous maps when revisiting mapped areas. Compared with visual odometry systems that only use information from the last few seconds, ORB-SLAM3 is the first system able to reuse in all the algorithm stages all previous information. This allows to include in bundle adjustment co-visible keyframes, that provide high parallax observations boosting accuracy, even if they are widely separated in time or if they come from a previous mapping session. Our experiments show that, in all sensor configurations, ORB-SLAM3 is as robust as the best systems available in the literature, and significantly more accurate. Notably, our stereo-inertial SLAM achieves an average accuracy of 3.6 cm on the EuRoC drone and 9 mm under quick hand-held motions in the room of TUM-VI dataset, a setting representative of AR/VR scenarios. For the benefit of the community we make public the source code.

ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM

Augmented Reality (AR) technologies for supporting maintenance operations have been an academic research topic for around 50 years now. In the last decade, major progresses have been made and the AR technology is getting closer to being implemented in industry. In this paper, the advantages and disadvantages of AR have been explored and quantified in terms of Key Performance Indicators (KPI) for industrial maintenance. Unfortunately, some technical issues still prevent AR from being suitable for industrial applications. This paper aims to show, through the results of a systematic literature review, the current state of the art of AR in maintenance and the most relevant technical limitations. The analysis included filtering from a large number of publications to 30 primary studies published between 1997 and 2017. The results indicate a high fragmentation among hardware, software and AR solutions which lead to a high complexity for selecting and developing AR systems. The results of the study show the areas where AR technology still lacks maturity. Future research directions are also proposed encompassing hardware, tracking and user-AR interaction in industrial maintenance is proposed.

/pdf/a-systematic-review-of-augmented-reality-applications-in-2a9ygqrdpa.pdf

A systematic review of augmented reality applications in maintenance

We present a survey of recent work on robot manipulation and sensing of deformable objects, a field with relevant applications in diverse industries such as medical (e.g. surgery assistance), food handling, manufacturing, and domestic chores (e.g. folding clothes). We classify the reviewed approaches into four categories based on the type of object they manipulate. Furthermore, within this object classification we divide the approaches based on the particular task they perform on the deformable object. Finally, we conclude this survey with a discussion of the current state of the art and propose future directions within the proposed classification.

/pdf/robotic-manipulation-and-sensing-of-deformable-objects-in-tajhrn5gez.pdf

Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey:

In the past two decades, augmented reality (AR) has received a growing amount of attention by researchers in the manufacturing technology community, because AR can be applied to address a wide range of problems throughout the assembly phase in the lifecycle of a product, e.g., planning, design, ergonomics assessment, operation guidance and training. However, to the best of authors’ knowledge, there has not been any comprehensive review of AR-based assembly systems. This paper aims to provide a concise overview of the technical features, characteristics and broad range of applications of AR-based assembly systems published between 1990 and 2015. Among these selected articles, two thirds of them were published between 2005 and 2015, and they are considered as recent pertinent works which will be discussed in detail. In addition, the current limitation factors and future trends in the development will also be discussed.

/pdf/a-comprehensive-survey-of-augmented-reality-assembly-4v9flaozri.pdf

A comprehensive survey of augmented reality assembly research

Disassembly planning aims to search the best disassembly sequences of a given obsolete/used product in terms of economic and environmental performances. A practical disassembly process may face great uncertainty owing to various unpredictable factors. To handle it, researchers have addressed the stochastic cost and time problems of product disassembly. In reality, the uncertain environment of product disassembly is associated with both randomness and fuzziness. Besides uncertain disassembly cost and time, the quality of disassembled components/parts in a process has uncertainty and thus needs to be assessed via expert opinions/subjects. To do so, this paper presents a new AND/OR-graph-based disassembly sequence planning problem by considering uncertain component quality and varying disassembly operational cost. Important disassembly planning models are built on the basis of different disassembly criteria. A novel hybrid intelligent algorithm integrating fuzzy simulation and artificial bee colony is proposed to solve them. Its effectiveness is well illustrated through several numerical cases and comparison with a prior method, i.e., fuzzy-simulation-based genetic algorithm. Note to Practitioners —This paper deals with the uncertainty management problem of product disassembly. It builds some fuzzy programming models for product disassembly and proposes a hybrid intelligent algorithm integrating fuzzy simulation and artificial bee colony to solve them. Previously, such a problem was handled through a methodology based on stochastic planning, which was ineffective without considering the fuzzy characteristic of completing a disassembly task. The goal of this paper is to analyze the disassembly uncertainty feature from the perspective of fuzzy programming. Both theoretical and simulation results demonstrate that the proposed approach is highly effective. The obtained results can help decision-makers better determine a disassembly process of a used/returned/obsolete product.

Disassembly Sequence Planning Considering Fuzzy Component Quality and Varying Operational Cost

The planning of disassembly sequences requires the identification of the extraction trajectories of the different parts or assemblies. The failure to find these trajectories can make a planner fail to generate correct sequences or not evaluate potential solutions. In this paper, we analyze the disassembly path-planning problem, its relation to the general path-planning problem and the main differences between both of them, such as the lack of a target configuration. We present a modification of the rapid-growing random tree-based algorithm (RRT) that addresses these differences. RRTs are easily parallelized so we analyze two different parallelization methods using dual-core-based CPUs as well as the impact of the target selection probability of the algorithm in execution time. The method described is applied to several real-world and synthetic examples.

Parallel RRT-based path planning for selective disassembly planning

Photometric bundle adjustment (PBA) accurately estimates geometry from video. However, current PBA systems have a temporary map that cannot manage scene reobservations. We present, direct sparse mapping, a full monocular visual simultaneous localization and mapping (SLAM) based on PBA. Its persistent map handles reobservations, yielding the most accurate results up to date on EuRoC for a direct method.

Direct Sparse Mapping

Mass-Spring Models (MSMs) are used to simulate the mechanical behavior of deformable bodies such as soft tissues in medical applications. Although they are fast to compute, they lack accuracy and their design remains still a great challenge. The major difficulties in building realistic MSMs lie on the spring stiffness estimation and the topology identification. In this work, the mechanical behavior of MSMs under tensile loads is analyzed before studying the spring stiffness estimation. In particular, the performed qualitative and quantitative analysis of the behavior of cubical MSMs shows that they have a nonlinear response similar to hyperelastic material models. According to this behavior, a new method for spring stiffness estimation valid for linear and nonlinear material models is proposed. This method adjusts the stress-strain and compressibility curves to a given reference behavior. The accuracy of the MSMs designed with this method is tested taking as reference some soft-tissue simulations based on nonlinear Finite Element Method (FEM). The obtained results show that MSMs can be designed to realistically model the behavior of hyperelastic materials such as soft tissues and can become an interesting alternative to other approaches such as nonlinear FEM.

Cubical Mass-Spring Model Design Based on a Tensile Deformation Test and Nonlinear Material Model

This paper proposes a new real-time Augmented Reality based tool to help in disassembly for maintenance operations. This tool provides workers with augmented instructions to perform maintenance tasks more efficiently. Our prototype is a complete framework characterized by its capability to automatically generate all the necessary data from input based on untextured 3D triangle meshes, without requiring additional user intervention. An automatic offline stage extracts the basic geometric features. These are used during the online stage to compute the camera pose from a monocular image. Thus, we can handle the usual textureless 3D models used in industrial applications. A self-supplied and robust markerless tracking system that combines an edge tracker, a point based tracker and a 3D particle filter has also been designed to continuously update the camera pose. Our framework incorporates an automatic path-planning module. During the offline stage, the assembly/disassembly sequence is automatically deduced from the 3D model geometry. This information is used to generate the disassembly instructions for workers.

Providing guidance for maintenance operations using automatic markerless Augmented Reality system

Nowadays, many companies see augmented reality (AR) as an important tool to provide new services related to their products. However, many challenges remain to be solved for the widely adoption of this technology, such as the development of suitable authoring tools and real-time and robust algorithms for the detection and tracking of objects where the virtual annotations will be anchored in the real world. This paper presents a complete framework, called ARgitu, to generate and present virtual and augmented information, including the tools required for the development of new contents. To solve the object detection and tracking in complex industrial environments, we also propose a new monocular method for 3D non-Lambertian object recognition in arbitrary environments. The method is based on the current state-of-the-art chamfer matching approaches with a reduced computational effort while maintaining their accuracy.

Iker Aguinaga

Papers

Parallel RRT-based path planning for selective disassembly planning

Direct Sparse Mapping

Cubical Mass-Spring Model Design Based on a Tensile Deformation Test and Nonlinear Material Model

Providing guidance for maintenance operations using automatic markerless Augmented Reality system

A framework for augmented reality guidance in industry