Learning Deconvolution Network for Semantic Segmentation

Learning and then recognizing a route, whether travelled during the day or at night, in clear or inclement weather, and in summer or winter is a challenging task for state of the art algorithms in computer vision and robotics. In this paper, we present a new approach to visual navigation under changing conditions dubbed SeqSLAM. Instead of calculating the single location most likely given a current image, our approach calculates the best candidate matching location within every local navigation sequence. Localization is then achieved by recognizing coherent sequences of these “local best matches”. This approach removes the need for global matching performance by the vision front-end - instead it must only pick the best match within any short sequence of images. The approach is applicable over environment changes that render traditional feature-based techniques ineffective. Using two car-mounted camera datasets we demonstrate the effectiveness of the algorithm and compare it to one of the most successful feature-based SLAM algorithms, FAB-MAP. The perceptual change in the datasets is extreme; repeated traverses through environments during the day and then in the middle of the night, at times separated by months or years and in opposite seasons, and in clear weather and extremely heavy rain. While the feature-based method fails, the sequence-based algorithm is able to match trajectory segments at 100% precision with recall rates of up to 60%.

SeqSLAM : visual route-based navigation for sunny summer days and stormy winter nights

With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography and medical imaging. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present an architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In a nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNet.

Learning Enriched Features for Real Image Restoration and Enhancement

This paper introduces a novel large dataset for video deblurring, video super-resolution and studies the state-of-the-art as emerged from the NTIRE 2019 video restoration challenges. The video deblurring and video super-resolution challenges are each the first challenge of its kind, with 4 competitions, hundreds of participants and tens of proposed solutions. Our newly collected REalistic and Diverse Scenes dataset (REDS) was employed by the challenges. In our study, we compare the solutions from the challenges to a set of representative methods from the literature and evaluate them on our proposed REDS dataset. We find that the NTIRE 2019 challenges push the state-of-the-art in video deblurring and super-resolution, reaching compelling performance on our newly proposed REDS dataset.

/pdf/ntire-2019-challenge-on-video-deblurring-and-super-18qmvhz59m.pdf

NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study

Most of the existing learning-based single image super-resolution (SISR) methods are trained and evaluated on simulated datasets, where the low-resolution (LR) images are generated by applying a simple and uniform degradation (i.e., bicubic downsampling) to their high-resolution (HR) counterparts. However, the degradations in real-world LR images are far more complicated. As a consequence, the SISR models trained on simulated data become less effective when applied to practical scenarios. In this paper, we build a real-world super-resolution (RealSR) dataset where paired LR-HR images on the same scene are captured by adjusting the focal length of a digital camera. An image registration algorithm is developed to progressively align the image pairs at different resolutions. Considering that the degradation kernels are naturally non-uniform in our dataset, we present a Laplacian pyramid based kernel prediction network (LP-KPN), which efficiently learns per-pixel kernels to recover the HR image. Our extensive experiments demonstrate that SISR models trained on our RealSR dataset deliver better visual quality with sharper edges and finer textures on real-world scenes than those trained on simulated datasets. Though our RealSR dataset is built by using only two cameras (Canon 5D3 and Nikon D810), the trained model generalizes well to other camera devices such as Sony a7II and mobile phones.

/pdf/toward-real-world-single-image-super-resolution-a-new-39vy7sc9j4.pdf

Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model

This paper presents a novel Fast Monocular Visual Place Recognition (FMPR) with a shallow path-oriented offline learning stage and an online place recognition and tracking stage. FMPR uses a tube of frames with a humanlike key frame recognition to solve place recognition for situations with varying speeds and changing lighting conditions, which are two most commonly encountered situations in real life. We propose an offline learning to analyze the correlation of all video frames in a reference path and to extract effective feature patches of key frames with an offline feature-shifts approach to achieve real-time place recognition. Our recognition results are on the basis of both the instant feature matching of frames and the historical recognition results which impose temporal logic constraints on the movement of a vehicle. Experimental results demonstrate that our proposed method can achieve comparable or even better performance compared with the state-of-the-art methods on different challenging datasets, especially for the case which requires a trade-off between the performance and the processing time. We believe that our FMPR offers a useful alternative to computationally expensive deep learning-based methods especially for applications with battery-powered or resource-limited devices.

Fast Monocular Visual Place Recognition for Non-Uniform Vehicle Speed and Varying Lighting Environment

Vision-based localization is a temporal informative task in which we can obtain information about the ego-motion of a vehicle from the historical information via examining consecutive frames. Sufficient temporal information helps to reduce the search space of the next location. Hence, both efficiency and accuracy of the localization system can be enhanced. This paper presents a semi-supervised deep vision-based localization algorithm, using a novel tubing strategy to find the starting location of a vehicle. We group different number of consecutive frames as sets of tubes based on their temporal correlation to achieve pair searching with variable tube sizes. We also enhance an off-the-shelf network model with our modified training data generation method to improve the discrimination power of the features given by the model. Experimental results show that our proposed temporal correlation based initialization module can confidently localize the starting location of a vehicle (for a certain journey), and achieve 40% precision improvement over that of the conventional CNN approaches.

Semi-Supervised Deep Vision-Based Localization Using Temporal Correlation Between Consecutive Frames

This paper presents a railway localization algorithm, using a novel tube of frames concept with key frame based rectification. We extract effective feature patches from key frames with an offline feature-shifts approach to real-time train localization. We get the localization results from both actual calculation of frame matching and estimation based on the previous matches, the temporal information about the train motion. We focus on practical situations in which the train travels at inconstant speeds in different journeys and stops at different locations. The experimental results illustrate that our algorithm can achieve 88.8% precision with 100% recall under an acceptable range of deviation from the ground truth, which outperforms SeqSLAM, a benchmark of localization and mapping algorithm. Moreover, our algorithm is robust to illumination and less sensitive to the length of sequences than the benchmark. We also compare with the modern CNN features based approach. We show that blurring and heavy time cost are two limitations of the CNN. Our algorithm only requires 13.4 ms to process a frame on average using a regular desktop, which is 10 times faster than using the CNN approach and also faster than the benchmark, with the best result of 2.1 times faster on the sample dataset.

Fast Monocular Vision-based Railway Localization for Situations with Varying Speeds

The most challenging part of vision-based place recognition is the wide variety in appearance of places. However temporal information between consecutive frames can be used to infer the next locations of a vehicle and obtain information about its ego-motion. Effective use of temporal information is useful to narrow the search ranges of the next locations, hence an efficient place recognition system can be accomplished. This paper presents a robust vision-based place recognition method, using the recent discriminative ConvNet features and proposes a flexible tubing strategy which groups consecutive frames based on their similarities. With the tubing strategy, effective pair searching can be achieved. We also suggest to add additional variations in the appearance of places to further enhance the variety of the training data and fine-tune an off-the-shelf, CALC, network model to obtain better generalization about its extracted features. Experimental results show that our proposed temporal correlation based recognition strategy with the fine-tuned model achieves the best (0.572) F1 score improvement over the original CALC model. The proposed place recognition method is also faster than the linear full search method by a factor of 2.15.

Vision-based Place Recognition Using ConvNet Features and Temporal Correlation Between Consecutive Frames

This paper presents a key frame recognition algorithm, using novel offline feature-shifts approach and search window weights. We extract effective feature patches from key frames with an offline feature-shifts approach for real-time key frame recognition. We focus on practical situations in which blurring and shifts in viewpoints occur in our dataset. We compare our method with some conventional keypoint-based matching methods and the newest CNN features for scene recognition. The experimental results illustrate that our method can reasonably preserve the performance in key frame recognition when comparing with methods using online feature-shifts approach. Our proposed method provides larger tolerance of unmatched pairs which is useful for decision making in real-time systems. Moreover, our method is robust to illumination and blurring. We achieve 90% accuracy in a nighttime sequence while CNN approach only attains 60% accuracy. Our method only requires 33.8 ms to match a frame on average using a regular desktop, which is 4 times faster than CNN approach with only CPU mode.

Chu-Tak Li

Papers

Fast Monocular Visual Place Recognition for Non-Uniform Vehicle Speed and Varying Lighting Environment

Semi-Supervised Deep Vision-Based Localization Using Temporal Correlation Between Consecutive Frames

Fast Monocular Vision-based Railway Localization for Situations with Varying Speeds

Vision-based Place Recognition Using ConvNet Features and Temporal Correlation Between Consecutive Frames

Boosting the Performance of Scene Recognition via Offline Feature-Shifts and Search Window Weights