Visual place recognition is a challenging problem in robotics and autonomous systems because the scene undergoes appearance and viewpoint changes in a changing world. Existing state-of-the-art methods heavily rely on CNN-based architectures. However, CNN cannot effectively model image spatial structure information due to the inherent locality. To address this issue, this paper proposes a novel Transformer-based place recognition method to combine local details, spatial context, and semantic information for image feature embedding. Firstly, to overcome the inherent locality of the convolutional neural network (CNN), a hybrid CNN-Transformer feature extraction network is introduced. The network utilizes the feature pyramid based on CNN to obtain the detailed visual understanding, while using the vision Transformer to model image contextual information and aggregate task-related features dynamically. Specifically, the multi-level output tokens from the Transformer are fed into a single Transformer encoder block to fuse multi-scale spatial information. Secondly, to acquire the multi-scale semantic information, a global semantic NetVLAD aggregation strategy is constructed. This strategy employs semantic enhanced NetVLAD, imposing prior knowledge on the terms of the Vector of Locally Aggregated Descriptors (VLAD), to aggregate multi-level token maps, and further concatenates the multi-level semantic features globally. Finally, to alleviate the disadvantage that the fixed margin of triplet loss leads to the suboptimal convergence, an adaptive triplet loss with dynamic margin is proposed. Extensive experiments on public datasets show that the learned features are robust to appearance and viewpoint changes and achieve promising performance compared to state-of-the-arts.

Hybrid CNN-Transformer Features for Visual Place Recognition

Sparse local feature matching is pivotal for many computer vision and robotics tasks. To improve their invariance to challenging appearance conditions and viewing angles, and hence their usefulness, existing learning-based methods have primarily focused on data augmentation-based training. In this work, we propose an alternative, complementary approach that centers on inducing bias in the model architecture itself to generate ‘rotation-specific’ features using Steerable E2-CNNs, that are then group-pooled to achieve rotation-invariant local features. We demonstrate that this high performance, rotationspecific coverage from the steerable CNNs can be expanded to all rotation angles by combining it with augmentationtrained standard CNNs which have broader coverage but are often inaccurate, thus creating a state-of-the-art rotation-robust local feature matcher. We benchmark our proposed methods against existing techniques on HPatches and a newly proposed UrbanScenes3D-Air dataset for visual place recognition. Furthermore, we present a detailed analysis of the performance effects of ensembling, robust estimation, network architecture variations, and the use of rotation priors. *Denotes authors with equal contribution 1Robotics Research Center, IIIT Hyderabad. 2QUT Centre for Robotics, Queensland University of Technology (QUT), Australia. This research is supported by MathWorks. Code link: https://github.com/abhishek-peri/ReF-official-code

ReF - Rotation Equivariant Features for Local Feature Matching

Localization is a fundamental and crucial module for autonomous vehicles. Most of the existing localization methodologies, such as signal-dependent methods (RTK-GPS and Bluetooth), simultaneous localization and mapping (SLAM), and map-based methods, have been utilized in outdoor autonomous driving vehicles and indoor robot positioning. However, they suffer from severe limitations, such as signal-blocked scenes of GPS, computing resource occupation explosion in large-scale scenarios, intolerable time delay, and registration divergence of SLAM/map-based methods. In this article, a self-localization framework, without relying on GPS or any other wireless signals, is proposed. We demonstrate that the proposed homogeneous normal distribution transform algorithm and two-way information interaction mechanism could achieve centimeter-level localization accuracy, which reaches the requirement of autonomous vehicle localization for instantaneity and robustness. In addition, benefitting from hardware and software co-design, the proposed localization approach is extremely light-weighted enough to be operated on an embedded computing system, which is different from other LiDAR localization methods relying on high-performance CPU/GPU. Experiments on a public dataset (Baidu Apollo SouthBay dataset) and real-world verified the effectiveness and advantages of our approach compared with other similar algorithms.

Onboard Sensors-Based Self-Localization for Autonomous Vehicle With Hierarchical Map

Place recognition plays an important role in robot localization and SLAM. Being able to retrieve the current position in a given map allows, for instance, localizing without relying on GPS reception. In this letter, we address the problem of point cloud-based place recognition, we especially focus on reducing the often significant training time needed by learning-based approaches. We propose a novel neural network architecture that first extracts local features using a pre-trained encoder network plus a stem architecture. The local features are aggregated to a global descriptor, which allows us to compute the similarity between locations. In line with with several existing approaches, we target the generation of descriptors, which are similar for spatially near locations and dissimilar to other places. By exploiting the recent success of feature banks, we are able to bypass the computation of the negative examples, which enables faster training, bigger batch sizes, or the use of more sophisticated networks. As a key result, able to speed up the training process by a factor of 17 against the most common training procedure while increasing also the performance.

KPPR: Exploiting Momentum Contrast for Point Cloud-Based Place Recognition

Localization is a fundamental and crucial module for autonomous vehicles. Most of the existing localization methodologies, such as signal-dependent methods (RTK-GPS and Bluetooth), simultaneous localization and mapping (SLAM), and map-based methods, have been utilized in outdoor autonomous driving vehicles and indoor robot positioning. However, they suffer from severe limitations, such as signal-blocked scenes of GPS, computing resource occupation explosion in large-scale scenarios, intolerable time delay, and registration divergence of SLAM/map-based methods. In this article, a self-localization framework, without relying on GPS or any other wireless signals, is proposed. We demonstrate that the proposed homogeneous normal distribution transform algorithm and two-way information interaction mechanism could achieve centimeter-level localization accuracy, which reaches the requirement of autonomous vehicle localization for instantaneity and robustness. In addition, benefitting from hardware and software co-design, the proposed localization approach is extremely light-weighted enough to be operated on an embedded computing system, which is different from other LiDAR localization methods relying on high-performance CPU/GPU. Experiments on a public dataset (Baidu Apollo SouthBay dataset) and real-world verified the effectiveness and advantages of our approach compared with other similar algorithms. 

Visual place recognition is a challenging task for applications such as autonomous driving navigation and mobile robot localization. Distracting elements presenting in complex scenes often lead to deviations in the perception of visual place. To address this problem, it is crucial to integrate information from only task-relevant regions into image representations. In this paper, we introduce a novel holistic place recognition model, TransVPR, based on vision Transformers. It benefits from the desirable property of the self-attention operation in Transformers which can naturally aggregate task-relevant features. Attentions from multiple levels of the Transformer, which focus on different regions of interest, are further combined to generate a global image representation. In addition, the output tokens from Transformer layers filtered by the fused attention mask are considered as key-patch descriptors, which are used to perform spatial matching to re-rank the candidates retrieved by the global image features. The whole model allows end-to-end training with a single objective and image-level supervision. TransVPR achieves state-of-the-art performance on several real-world benchmarks while maintaining low computational time and storage requirements.

TransVPR: Transformer-Based Place Recognition with Multi-Level Attention Aggregation

LiDAR-based positioning systems are widely used in unmanned systems. However, affected by the high computational complexity of high-precision positioning algorithms, the current positioning system is supported by hardware with low power efficiency and thus hard to integrate into many platforms. In this paper, we analyze features of the positioning system in autonomous driving application, design, and apply Heterogeneous LiDAR-based Positioning System, HeLPS, with software-hardware co-design methodology to achieve better efficiency. Our contributions can be concluded in three aspects. Firstly, we design the CPU-FPGA heterogeneous positioning system accelerating Iterative Closest Point (ICP) algorithm and achieves improvements on both speed and power-efficiency. Secondly, we exploit the spatial locality in the point cloud and design a new compressed data structure for fast neighbor accessing. The experiment reports a significant speedup comparing with other data structures. Lastly, we explore the data access pattern in positioning application and develop a specific cache system and out-of-order execution system reducing memory burden. Our system is deployed on a small and cheap Xilinx Zynq7000 ARM+FPGA platform, which achieves 983.3x speedup compared with Cortex-A9 CPU, and 31.8x speedup compared with i7-7820 CPU, with only 2.37W power consumption.

HeLPS: Heterogeneous LiDAR-based Positioning System for Autonomous Vehicle

In the field of autonomous driving, multi-beam light detection and ranging (3D LIDAR) system and global navigation satellite system/integrated inertial navigation system (GNSS/INS) are widely used in high-definition map construction, localization and obstacle detection. As 3D LIDAR system and INS have their own coordinate systems, the calibration of the two mentioned systems is required. In this paper, a novel algorithm for calibrating the coordinate system of 3D LIDAR and INS is proposed, which consists of three parts. The first procedure is to project two point clouds to the world coordinate system based on the initial transform matrix between 3D LIDAR and INS with the real-time data from INS. Then optimal point-to-point correspondences can be found between two frames of point cloud data through registration method. Finally, the loss function is constructed with the sum of the Euclidean distances of the corresponding points and optimized by using perturbation model of Lie algebras, so as to obtain the optimal transform matrix. With different given initial calibration parameters, test results of both simulation and real experiments validate the proposed algorithm and quantify its accuracy and robustness.

Robust Extrinsic Parameter Calibration of 3D LIDAR Using Lie Algebras

Visual place recognition (VPR) is a key issue for robotics and autonomous systems. For the trade-off between time and performance, most of methods use the coarse-to-fine hierarchical architecture, which consists of retrieving top-N candidates using global features, and re-ranking top-N with local features. However, since the two types of features are usually processed independently, re-ranking may harm global retrieval, termed re-ranking confusion. Moreover, re-ranking is limited by global retrieval. In this paper, we propose a tightly coupled learning (TCL) strategy to train triplet models. Different from original triplet learning (OTL) strategy, it combines global and local descriptors for joint optimization. In addition, a bidirectional search dynamic time warping (BS-DTW) algorithm is also proposed to mine locally spatial information tailored to VPR in re-ranking. The experimental results on public benchmarks show that the models using TCL outperform the models using OTL, and TCL can be used as a general strategy to improve performance for weakly supervised ranking tasks. Further, our lightweight unified model is better than several state-of-the-art methods and has over an order of magnitude of computational efficiency to meet the real-time requirements of robots.

Yanqing Shen

Papers

TransVPR: Transformer-Based Place Recognition with Multi-Level Attention Aggregation

HeLPS: Heterogeneous LiDAR-based Positioning System for Autonomous Vehicle

Robust Extrinsic Parameter Calibration of 3D LIDAR Using Lie Algebras

Onboard Sensors-Based Self-Localization for Autonomous Vehicle With Hierarchical Map

TCL: Tightly Coupled Learning Strategy for Weakly Supervised Hierarchical Place Recognition