The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point pair feature</i> (PPF) is widely used in manufacturing for estimating 6-D poses. The key to the success of PPF matching is to establish correct 3-D correspondences between the object and the scene, i.e., finding as many valid similar point pairs as possible. However, efficient sampling of point pairs has been overlooked in existing frameworks. In this article, we propose a revised PPF matching pipeline to improve the efficiency of 6-D pose estimation. Our basic idea is that the valid scene reference points are lying on the object’s surface and the previously sampled reference points can provide prior information for locating new reference points. The novelty of our approach is a new sampling algorithm for selecting scene reference points based on the multisubpopulation particle swarm optimization guided by a probability map. We also introduce an effective pose clustering and hypotheses verification method to obtain the optimal pose. Moreover, we optimize the progressive sampling for multiframe point clouds to improve processing efficiency. The experimental results show that our method outperforms previous methods by 6.6%, 3.9% in terms of accuracy on the public DTU and LineMOD datasets, respectively. We further validate our approach by applying it in a real robot grasping task. 

/pdf/efficient-mspso-sampling-for-object-detection-and-6-d-pose-2rpbvuzj.pdf

Efficient MSPSO Sampling for Object Detection and 6-D Pose Estimation in 3-D Scenes

Background: The planetary rover is an essential platform for planetary exploration. Visual semantic segmentation is significant in the localization, perception, and path planning of the rover autonomy. Recent advances in computer vision and artificial intelligence brought about new opportunities. A systematic literature review (SLR) can help analyze existing solutions, discover available data, and identify potential gaps. Methods: A rigorous SLR has been conducted, and papers are selected from three databases (IEEE Xplore, Web of Science, and Scopus) from the start of records to May 2022. The 320 candidate studies were found by searching with keywords and bool operators, and they address the semantic terrain segmentation in the navigation vision of planetary rovers. Finally, after four rounds of screening, 30 papers were included with robust inclusion and exclusion criteria as well as quality assessment. Results: 30 studies were included for the review, and sub-research areas include navigation (16 studies), geological analysis (7 studies), exploration efficiency (10 studies), and others (3 studies) (overlaps exist). Five distributions are extendedly depicted (time, study type, geographical location, publisher, and experimental setting), which analyzes the included study from the view of community interests, development status, and reimplementation ability. One key research question and six sub-research questions are discussed to evaluate the current achievements and future gaps. Conclusions: Many promising achievements in accuracy, available data, and real-time performance have been promoted by computer vision and artificial intelligence. However, a solution that satisfies pixel-level segmentation, real-time inference time, and onboard hardware does not exist, and an open, pixel-level annotated, and the real-world data-based dataset is not found. As planetary exploration projects progress worldwide, more promising studies will be proposed, and deep learning will bring more opportunities and contributions to future studies. Contributions: This SLR identifies future gaps and challenges by proposing a methodical, replicable, and transparent survey, which is the first review (also the first SLR) for semantic terrain segmentation in the navigation vision of planetary rovers.

/pdf/semantic-terrain-segmentation-in-the-navigation-vision-of-2yj8br58.pdf

Semantic Terrain Segmentation in the Navigation Vision of Planetary Rovers—A Systematic Literature Review

The goal of the Mars Sample Return campaign is to collect soil samples from the surface of Mars and return them to Earth for further study. The samples will be acquired and stored in metal tubes by the Perseverance rover and deposited on the Martian surface. As part of this campaign, it is expected that the Sample Fetch Rover will be in charge of localizing and gathering up to 35 sample tubes over 150 Martian sols. Autonomous capabilities are critical for the success of the overall campaign and for the Sample Fetch Rover in particular. This work proposes a novel system architecture for the autonomous detection and pose estimation of the sample tubes. For the detection stage, a Deep Neural Network and transfer learning from a synthetic dataset are proposed. The dataset is created from photorealistic 3D simulations of Martian scenarios. Additionally, the sample tubes poses are estimated using Computer Vision techniques such as contour detection and line fitting on the detected area. Finally, laboratory tests of the Sample Localization procedure are performed using the ExoMars Testing Rover on a Mars-like testbed. These tests validate the proposed approach in different hardware architectures, providing promising results related to the sample detection and pose estimation.

Hardware-Accelerated Mars Sample Localization Via Deep Transfer Learning From Photorealistic Simulations

The goal of the Mars Sample Return campaign is to collect soil samples from the surface of Mars and return them to Earth for further study. The samples will be acquired and stored in metal tubes by the Perseverance rover and deposited on the Martian surface. As part of this campaign, it is expected that the Sample Fetch Rover will be in charge of localizing and gathering up to 35 sample tubes over 150 Martian sols. Autonomous capabilities are critical for the success of the overall campaign and for the Sample Fetch Rover in particular. This work proposes a novel system architecture for the autonomous detection and pose estimation of the sample tubes. For the detection stage, a Deep Neural Network and transfer learning from a synthetic dataset are proposed. The dataset is created from photorealistic 3D simulations of Martian scenarios. Additionally, the sample tubes poses are estimated using Computer Vision techniques such as contour detection and line fitting on the detected area. Finally, laboratory tests of the Sample Localization procedure are performed using the ExoMars Testing Rover on a Mars-like testbed. These tests validate the proposed approach in different hardware architectures, providing promising results related to the sample detection and pose estimation. 

/pdf/hardware-accelerated-mars-sample-localization-via-deep-2iwdmnen.pdf

This study first attempts to observe the action of the CNN and then compares it to test Projective Template Matching and Object Detection as new approaches. In the final model selection, the accuracy of the prediction model and the computational processing time was mainly compared. At last, the combination of the Object Detection model and CNN was selected as a final location classification model with a prediction accuracy of 61%. This final model shows the optimal prediction result by first attempting to detect the common feature regions of the location image and then analyzing the overall feature characteristic. The fact is that CNN is good for training image data with common overall features for classification. This being so, we expect that training several fundamental ROIs can more efficiently train the CNN model than training the pure location images.

/pdf/substantiation-of-location-image-classification-model-using-1qojetix.pdf

Substantiation of location image classification model using projective template matching and convolutional neural network

A potential Mars Sample Return (MSR) architecture is being jointly studied by NASA and ESA. As currently envisioned, the MSR campaign consists of a series of 3 missions: sample cache, fetch and return to Earth. In this paper, we focus on the fetch part of the MSR, and more specifically the problem of autonomously detecting and localizing sample tubes deposited on the Martian surface. Towards this end, we study two machine-vision based approaches: First, a geometry-driven approach based on template matching that uses hard-coded filters and a 3D shape model of the tube; and second, a data-driven approach based on convolutional neural networks (CNNs) and learned features. Furthermore, we present a large benchmark dataset of sample-tube images, collected in representative outdoor environments and annotated with ground truth segmentation masks and locations. The dataset was acquired systematically across different terrain, illumination conditions and dust-coverage; and benchmarking was performed to study the feasibility of each approach, their relative strengths and weaknesses, and robustness in the presence of adverse environmental conditions.

Peter Ilhardt

Papers

Machine Vision based Sample-Tube Localization for Mars Sample Return

Machine Vision based Sample-Tube Localization for Mars Sample Return