Visual place recognition is a challenging problem due to the vast range of ways in which the appearance of real-world places can vary. In recent years, improvements in visual sensing capabilities, an ever-increasing focus on long-term mobile robot autonomy, and the ability to draw on state-of-the-art research in other disciplines—particularly recognition in computer vision and animal navigation in neuroscience—have all contributed to significant advances in visual place recognition systems. This paper presents a survey of the visual place recognition research landscape. We start by introducing the concepts behind place recognition—the role of place recognition in the animal kingdom, how a “place” is defined in a robotics context, and the major components of a place recognition system. Long-term robot operations have revealed that changing appearance can be a significant factor in visual place recognition failure; therefore, we discuss how place recognition solutions can implicitly or explicitly account for appearance change within the environment. Finally, we close with a discussion on the future of visual place recognition, in particular with respect to the rapid advances being made in the related fields of deep learning, semantic scene understanding, and video description.

/pdf/visual-place-recognition-a-survey-d2xtkdlp55.pdf

Visual Place Recognition: A Survey

Vision-based topological mapping and localization methods

Image-based localization technique embodiments are presented which provide a real-time approach for image-based video camera localization within large scenes that have been reconstructed offline using structure from motion or similar techniques. From monocular video, a precise 3D position and 3D orientation of the camera can be estimated on a frame by frame basis using only visual features.

Image-based localization

This paper presents a novel method and innovative apparatus for building three-dimensional 3D dense visual maps of large-scale unstructured environments for autonomous navigation and real-time localization. The main contribution of the paper is focused on proposing an efficient and accurate 3D world representation that allows us to extend the boundaries of state-of-the-art dense visual mapping to large scales. This is achieved via an omnidirectional key-frame representation of the environment, which is able to synthesize photorealistic views of captured environments at arbitrary locations. Locally, the representation is image-based egocentric and is composed of accurate augmented spherical panoramas combining photometric information RGB, depth information D, and saliency for all viewing directions at a particular point in space i.e., a point in the light field. The spheres are related by a graph of six degree of freedom DOF poses 3 DOF translation and 3 DOF rotation that are estimated through multiview spherical registration. It is shown that this world representation can be used to perform robust real-time localization in 6 DOF of any configuration of visual sensors within their environment, whether they be monocular, stereo, or multiview. Contrary to feature-based approaches, an efficient direct image registration technique is formulated. This approach directly exploits the advantages of the spherical representation by minimizing a photometric error between a current image and a reference sphere. Two novel multicamera acquisition systems have been developed and calibrated to acquire this information, and this paper reports for the first time the second system. Given the robustness and efficiency of this representation, field experiments demonstrating autonomous navigation and large-scale mapping will be reported in detail for challenging unstructured environments, containing vegetation, pedestrians, varying illumination conditions, trams, and dense traffic.

/pdf/dense-omnidirectional-rgb-d-mapping-of-large-scale-outdoor-80kydurn59.pdf

Dense Omnidirectional RGB-D Mapping of Large-scale Outdoor Environments for Real-time Localization and Autonomous Navigation

This paper describes an online place discovery and recognition engine that fuses information over time to create topologically distinct places. A key motivation is the recognition that a single image may be a poor exemplar of what constitutes a place. Images are not ‘places’ nor are they ‘documents’. Instead, by treating image-sequences as a multimodal distribution over topics – and by discovering topics incrementally and online – it is possible to both reduce the memory footprint of place recognition systems, and to improve precision and recall. Distinctive key-places are represented by a cluster topics found from the covisibility graph of a relative simultaneous localization and mapping engine – key-places inherently span many images. A dynamic vocabulary of visual words and density based clustering is used to continually estimate a set of visual topics, changes in which drive the placerecognition process. The system is evaluated using an indoor robot sequence, a standard outdoor robot sequence and a longterm sequence from a static camera. Experiments demonstrate qualitatively distinct themes associated with discovered places – from common place types such as ‘hallway’, or ‘desk-area’, to temporal concepts such as ‘dusk’, ‘dawn’ or ‘mid-day’. Compared to traditional image-based place-recognition, this reduces the information that must be stored without reducing place-recognition performance.

Incremental unsupervised topological place discovery

Topological navigation consists for a robot in navigating in a topological graph which nodes are topological places. Either for indoor or outdoor environments, segmentation into topological places is a challenging issue. In this paper, we propose a common approach for indoor and outdoor environment segmentation without elaborating a complete topological navigation system. The approach is novel in that environment sensing is performed using spherical images. Environment structure estimation is performed by a global structure descriptor specially adapted to the spherical representation. This descriptor is processed by a custom designed algorithm which detects change-points defining the segmentation between topological places.

/pdf/topological-segmentation-of-indoors-outdoors-sequences-of-1i1uf96ovm.pdf

Topological segmentation of indoors/outdoors sequences of spherical views

We present a novel approach to deal with visual loop closure detection independently from the point of view in large scale environments. Loop closure detection is the process of recognizing when a robot comes back to a previously visited location and is a key issue for place recognition, topological localization and visual Simultaneous Localization And Mapping. Our solution relies on an ego-centred spherical representation of the environment and the exploitation of two kinds of information: local appearance and global feature density. Local appearance is provided by local features extracted from the spherical representation and processed using the bag of visual words approach. Global density information is computed from the distribution over the sphere of these features thus characterizing the environment structure. The approach is purely appearance-based and does not involve any geometric information such as epipolar constraints. An experiment on an 1.5kms trajectory in an environment containing buildings and vegetation validates our algorithm and shows a reduction of the false alarm rate by about 50% for the same probability of detection compared to the standard bag of visual words approach.

A spherical representation for efficient visual loop closing

Navigating in large scale, complex and dynamic environments requires reliable representations able to capture metric, topological and semantic aspects of the scene for supporting path planing and real time motion control. In a previous work [11], we addressed metric and topological representations thanks to a multi-cameras system which allows building of dense visual maps of large scale 3D environments. The map is a set of locally accurate spherical panoramas related by 6d of poses graph. The work presented here is a further step toward a semantic representation. We aim at detecting the changes in the structural properties of the scene during navigation. Structural properties are estimated online using a global descriptor relying on spherical harmonics which are particularly well-fitted to capture properties in spherical views. A change-point detection algorithm based on a statistical Neyman-Pearson test allows us to find optimal transitions between topological places. Results are presented and discussed both for indoors and outdoors experiments.

/pdf/appearance-based-segmentation-of-indoors-outdoors-sequences-1bi6b2bdc4.pdf

Alexandre Chapoulie

Papers

Topological segmentation of indoors/outdoors sequences of spherical views

A spherical representation for efficient visual loop closing

Appearance-based segmentation of indoors/outdoors sequences of spherical views