As a fundamental and critical task in various visual applications, image matching can identify then correspond the same or similar structure/content from two or more images. Over the past decades, growing amount and diversity of methods have been proposed for image matching, particularly with the development of deep learning techniques over the recent years. However, it may leave several open questions about which method would be a suitable choice for specific applications with respect to different scenarios and task requirements and how to design better image matching methods with superior performance in accuracy, robustness and efficiency. This encourages us to conduct a comprehensive and systematic review and analysis for those classical and latest techniques. Following the feature-based image matching pipeline, we first introduce feature detection, description, and matching techniques from handcrafted methods to trainable ones and provide an analysis of the development of these methods in theory and practice. Secondly, we briefly introduce several typical image matching-based applications for a comprehensive understanding of the significance of image matching. In addition, we also provide a comprehensive and objective comparison of these classical and latest techniques through extensive experiments on representative datasets. Finally, we conclude with the current status of image matching technologies and deliver insightful discussions and prospects for future works. This survey can serve as a reference for (but not limited to) researchers and engineers in image matching and related fields.

/pdf/image-matching-from-handcrafted-to-deep-features-a-survey-27kn6w026w.pdf

Image Matching from Handcrafted to Deep Features: A Survey

We present Deep Global Registration, a differentiable framework for pairwise registration of real-world 3D scans. Deep global registration is based on three modules: a 6-dimensional convolutional network for correspondence confidence prediction, a differentiable Weighted Procrustes algorithm for closed-form pose estimation, and a robust gradient-based SE(3) optimizer for pose refinement. Experiments demonstrate that our approach outperforms state-of-the-art methods, both learning-based and classical, on real-world data.

/pdf/deep-global-registration-3nht5po8s3.pdf

Deep Global Registration

Deep Global Registration.

A review of multimodal image matching: Methods and applications

We introduce a comprehensive benchmark for local features and robust estimation algorithms, focusing on the downstream task -- the accuracy of the reconstructed camera pose -- as our primary metric. Our pipeline's modular structure allows easy integration, configuration, and combination of different methods and heuristics. This is demonstrated by embedding dozens of popular algorithms and evaluating them, from seminal works to the cutting edge of machine learning research. We show that with proper settings, classical solutions may still outperform the perceived state of the art. 
Besides establishing the actual state of the art, the conducted experiments reveal unexpected properties of Structure from Motion (SfM) pipelines that can help improve their performance, for both algorithmic and learned methods. Data and code are online this https URL, providing an easy-to-use and flexible framework for the benchmarking of local features and robust estimation methods, both alongside and against top-performing methods. This work provides a basis for the Image Matching Challenge this https URL.

Image Matching across Wide Baselines: From Paper to Practice

Feature correspondence selection is pivotal to many feature-matching based tasks in computer vision. Searching spatially k-nearest neighbors is a common strategy for extracting local information in many previous works. However, there is no guarantee that the spatially k-nearest neighbors of correspondences are consistent because the spatial distribution of false correspondences is often irregular. To address this issue, we present a compatibility-specific mining method to search for consistent neighbors. Moreover, in order to extract and aggregate more reliable features from neighbors, we propose a hierarchical network named NM-Net with a series of graph convolutions that is insensitive to the order of correspondences. Our experimental results have shown the proposed method achieves the state-of-the-art performance on four datasets with various inlier ratios and varying numbers of feature consistencies.

/pdf/nm-net-mining-reliable-neighbors-for-robust-feature-2fn96cw3br.pdf

NM-Net: Mining Reliable Neighbors for Robust Feature Correspondences

Feature correspondence selection is pivotal to many feature-matching based tasks in computer vision. Searching for spatially k-nearest neighbors is a common strategy for extracting local information in many previous works. However, there is no guarantee that the spatially k-nearest neighbors of correspondences are consistent because the spatial distribution of false correspondences is often irregular. To address this issue, we present a compatibility-specific mining method to search for consistent neighbors. Moreover, in order to extract and aggregate more reliable features from neighbors, we propose a hierarchical network named NM-Net with a series of convolution layers taking the generated graph as input, which is insensitive to the order of correspondences. Our experimental results have shown the proposed method achieves the state-of-the-art performance on four datasets with various inlier ratios and varying numbers of feature consistencies.

The invention relates to a method for converting a 2D video into a 3D video, belongs to the field of mode identification and computer vision, and aims to eliminate an unpredictable error brought in by scene depth estimation and viewpoint synthesis in the prior art and meanwhile greatly improve a calculation speed. The method disclosed by the invention comprises a training stage and a using stage; the training stage sequentially comprises the steps of data input, feature extraction, feature fusion, viewpoint synthesis and parameter updating; and the using stage sequentially comprises the steps of data input, feature extraction, feature fusion and viewpoint synthesis. In the training stage, a 10 -level side-by-side 3D stereoscopic video movie film is trained, scene depth estimation and viewpoint synthesis are simultaneously subjected to optimization solution and parameters are determined, so that pixel-level accuracy prediction of an output right-sided view is ensured, and an error brought in by dividing conversion from the 2D video to the 3D video into two task operations; and after training is completed, conversion from the 2D video to the 3D video can be directly carried out, so that conversion efficiency can be greatly improved, and accuracy of a finally output 3D stereoscopic video is ensured.

Chi Li

Papers

NM-Net: Mining Reliable Neighbors for Robust Feature Correspondences

NM-Net: Mining Reliable Neighbors for Robust Feature Correspondences

Method for converting 2D video into 3D video