This paper presents ORB-SLAM, a feature-based monocular simultaneous localization and mapping (SLAM) system that operates in real time, in small and large indoor and outdoor environments. The system is robust to severe motion clutter, allows wide baseline loop closing and relocalization, and includes full automatic initialization. Building on excellent algorithms of recent years, we designed from scratch a novel system that uses the same features for all SLAM tasks: tracking, mapping, relocalization, and loop closing. A survival of the fittest strategy that selects the points and keyframes of the reconstruction leads to excellent robustness and generates a compact and trackable map that only grows if the scene content changes, allowing lifelong operation. We present an exhaustive evaluation in 27 sequences from the most popular datasets. ORB-SLAM achieves unprecedented performance with respect to other state-of-the-art monocular SLAM approaches. For the benefit of the community, we make the source code public.

/pdf/orb-slam-a-versatile-and-accurate-monocular-slam-system-t5f1c0a74e.pdf

ORB-SLAM: A Versatile and Accurate Monocular SLAM System

scikit-image is an image processing library that implements algorithms and utilities for use in research, education and industry applications. It is released under the liberal Modified BSD open source license, provides a well-documented API in the Python programming language, and is developed by an active, international team of collaborators. In this paper we highlight the advantages of open source to achieve the goals of the scikit-image library, and we showcase several real-world image processing applications that use scikit-image. More information can be found on the project homepage, http://scikit-image.org.

/pdf/scikit-image-image-processing-in-python-3tlzmiwc32.pdf

scikit-image: Image processing in Python

This paper presents ORB-SLAM, a feature-based monocular SLAM system that operates in real time, in small and large, indoor and outdoor environments. The system is robust to severe motion clutter, allows wide baseline loop closing and relocalization, and includes full automatic initialization. Building on excellent algorithms of recent years, we designed from scratch a novel system that uses the same features for all SLAM tasks: tracking, mapping, relocalization, and loop closing. A survival of the fittest strategy that selects the points and keyframes of the reconstruction leads to excellent robustness and generates a compact and trackable map that only grows if the scene content changes, allowing lifelong operation. We present an exhaustive evaluation in 27 sequences from the most popular datasets. ORB-SLAM achieves unprecedented performance with respect to other state-of-the-art monocular SLAM approaches. For the benefit of the community, we make the source code public.

/pdf/orb-slam-a-versatile-and-accurate-monocular-slam-system-rnvdngj54a.pdf

ORB-SLAM: a Versatile and Accurate Monocular SLAM System

We present ORB-SLAM2, a complete simultaneous localization and mapping (SLAM) system for monocular, stereo and RGB-D cameras, including map reuse, loop closing, and relocalization capabilities. The system works in real time on standard central processing units in a wide variety of environments from small hand-held indoors sequences, to drones flying in industrial environments and cars driving around a city. Our back-end, based on bundle adjustment with monocular and stereo observations, allows for accurate trajectory estimation with metric scale. Our system includes a lightweight localization mode that leverages visual odometry tracks for unmapped regions and matches with map points that allow for zero-drift localization. The evaluation on 29 popular public sequences shows that our method achieves state-of-the-art accuracy, being in most cases the most accurate SLAM solution. We publish the source code, not only for the benefit of the SLAM community, but with the aim of being an out-of-the-box SLAM solution for researchers in other fields.

ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the performance of our approach. For the first time, we show that learning dense spatial transformation in deep CNNs is effective for sophisticated vision tasks such as object detection and semantic segmentation. The code is released at https://github.com/msracver/Deformable-ConvNets.

Deformable Convolutional Networks

Methods and systems for determining and modeling admissible gripper forces are described. An example method may involve receiving data representative of a plurality of trajectories along which a robotic manipulator, while gripping one or more physical objects, previously moved the one or more objects without dropping the one or more objects. The method may also involve determining set of force vectors corresponding to the trajectories. The method may also involve determining a three-dimensional virtual model of the set, where boundaries of the model represent constraints on magnitudes of forces that are applied in a given direction to perform the trajectories. The method may then involve determining one or more subsequent trajectories that correspond to a subsequent set of force vectors that are within the boundaries of the model and along which the robotic manipulator can move a subsequent physical object at particular accelerations without dropping the subsequent object.

Methods and systems for determining and modeling admissible gripper forces for robotic devices

This disclosure is directed to a hardware system for inverse graphics capture. An inverse graphics capture system (IGCS) captures data regarding a physical space that can be used to generate a photorealistic graphical model of that physical space. In certain approaches, the system includes hardware and accompanying software used to create a photorealistic six degree of freedom (6DOF) graphical model of the physical space. In certain approaches, the system includes hardware and accompanying software used for projection mapping onto the physical space. In certain approaches, the model produced by the IGCS is built using data regarding the geometry, lighting, surfaces, and environment of the physical space. In certain approaches, the model produced by the IGCS is both photorealistic and fully modifiable.

Hardware system for inverse graphics capture

Methods and systems for remote perception assistance to facilitate robotic object manipulation are provided herein. From a model of objects in an environment of a robotic manipulator, a virtual boundary line separating two adjacent identified virtual objects may be identified. The robotic manipulator may be configured to perform a task involving a manipulation of at least one object in the environment represented by the identified virtual objects. Based on the identifying, a request for remote assistance with verifying the virtual boundary line may be sent to a remote assistor device. A response to the request, including a modification to the virtual boundary line, may then be received from the remote assistor device. The robotic manipulator may then be caused to perform the task based on the modification to the virtual boundary line.

Methods and systems for remote perception assistance to facilitate robotic object manipulation

Methods and systems for using projected patterns to facilitate mapping of an environment are provided herein. A computing system may cause fixedly-posed projectors to each provide, onto a respective area of the environment, a predetermined respective distinct pattern. The system may determine respective poses of the projectors, and further determine a map of the environment that identifies, for each distinct pattern, respective locations on one or more surfaces in the environment on which the distinct pattern is detectable. Based on sensor data the system may identify a portion of a particular distinct pattern in the environment. The system may use the map and the respective pose of a particular projector that is providing the particular pattern to make a determination that the portion is located at a new, different location compared to the map. The system may then transmit an output signal indicating the determination.

Methods and systems for using projected patterns to facilitate mapping of an environment

Example implementations relate to determining depth information using stereo sensor data. An example system may include at least one projector coupled to a robotic manipulator and configured to project a texture pattern onto an environment. The system may further include a displacer coupled to the at least one texture projector and configured to repeatedly change a position of the texture pattern within the environment. The system may also include at least two optical sensors configured to capture stereo sensor data for the environment. And the system may include a computing device configured to determine, using the stereo sensor data, an output including a virtual representation of the environment.

Ethan Rublee

Papers

Methods and systems for determining and modeling admissible gripper forces for robotic devices

Hardware system for inverse graphics capture

Methods and systems for remote perception assistance to facilitate robotic object manipulation

Methods and systems for using projected patterns to facilitate mapping of an environment

Projected pattern motion/vibration for depth sensing