The video-based separation of foreground (FG) and background (BG) has been widely studied due to its vital role in many applications, including intelligent transportation and video surveillance. Most of the existing algorithms are based on traditional computer vision techniques that perform pixel-level processing assuming that FG and BG possess distinct visual characteristics. Recently, state-of-the-art solutions exploit deep learning models targeted originally for image classification. Major drawbacks of such a strategy are the lacking delineation of FG regions due to missing temporal information as they segment the FG based on a single frame object detection strategy. To grapple with this issue, we excogitate a 3D convolutional neural network (3D CNN) with long short-term memory (LSTM) pipelines that harness seminal ideas, viz., fully convolutional networking, 3D transpose convolution, and residual feature flows. Thence, an FG-BG segmenter is implemented in an encoder-decoder fashion and trained on representative FG-BG segments. The model devises a strategy called double encoding and slow decoding, which fuses the learned spatio-temporal cues with appropriate feature maps both in the down-sampling and up-sampling paths for achieving well generalized FG object representation. Finally, from the Sigmoid confidence map generated by the 3D CNN-LSTM model, the FG is identified automatically by using Nobuyuki Otsu’s method and an empirical global threshold. The analysis of experimental results via standard quantitative metrics on 16 benchmark datasets including both indoor and outdoor scenes validates that the proposed 3D CNN-LSTM achieves competitive performance in terms of figure of merit evaluated against prior and state-of-the-art methods. Besides, a failure analysis is conducted on 20 video sequences from the DAVIS 2016 dataset.

A 3D CNN-LSTM-Based Image-to-Image Foreground Segmentation

Moving object detection (MOD) in videos is a challenging task. Estimation of accurate background is the key to extracting the foreground from video frames. In this paper, we have proposed a novel compact end-to-end convolutional neural network architecture, motion saliency foreground network (MSFgNet), to estimate the background and to extract the foreground from video frames. Initially, the long streaming video is divided into a number of small video streams (SVS). The proposed network takes the SVS as an input and estimates the background frame for each SVS. Second, the saliency map is extracted using the current video frame and estimated background. Furthermore, a compact encoder–decoder network is proposed to extract the foreground from the estimated saliency maps. The performance of the proposed MSFgNet is tested on three benchmark datasets (CDnet-2014, LASIESTA, and PTIS) for MOD. The computational complexity (handling of number of parameters and execution time) and the performance of the proposed MSFgNet are compared with the existing state-of-the-art methods for MOD in terms of precision, recall, and F-measure. Performance analysis shows that the proposed network is very compact and outperforms the existing state-of-the-art methods for MOD in videos.

MSFgNet: A Novel Compact End-to-End Deep Network for Moving Object Detection

Imbalanced training sets are known to produce suboptimal maps for supervised classification. Therefore, one challenge in mapping land cover is acquiring training data that will allow classification with high overall accuracy (OA) in which each class is also mapped onto similar user’s accuracy. To solve this problem, we integrated local adaptive region and box-and-whisker plot (BP) techniques into an iterative algorithm to expand the size of the training sample for selected classes in this article. The major steps of the proposed algorithm are as follows. First, a very small initial training sample (ITS) for each class set is labeled manually. Second, potential new training samples are found within an adaptive region by conducting local spectral variation analysis. Lastly, three new training samples are acquired to capture information regarding intraclass variation; these samples lie in the lower, median, and upper quartiles of BP. After adding these new training samples to the ITS, classification is retrained and the process is continued iteratively until termination. The proposed approach was applied to three very high-resolution (VHR) remote-sensing images and compared with a set of cognate methods. The comparison demonstrated that the proposed approach produced the best result in terms of OA and exhibited superiority in balancing user’s accuracy. For example, the proposed approach was typically 2%-10% more accurate than the compared methods in terms of OA and it generally yielded the most balanced classification.

/pdf/iterative-training-sample-expansion-to-increase-and-balance-w862h7ehtc.pdf

Iterative Training Sample Expansion to Increase and Balance the Accuracy of Land Classification From VHR Imagery

Robust, discriminative and comprehensive dictionary learning for face recognition

This paper provides a benchmark of the performance of 23 classical and state-of-the-art background subtraction (BS) algorithms on visible range and near infrared range videos in the Singapore Maritime dataset. Importantly, our study indicates the limitations of the conventional performance evaluation criteria for maritime vision and proposes new performance evaluation criteria that is better suited to this problem. This paper provides insight into the specific challenges of BS in maritime vision. We identify four open challenges that plague BS methods in maritime scenario. These include spurious dynamics of water, wakes, ghost effect, and multiple detections. Poor recall and extremely poor precision of all the 23 methods, which have been otherwise successful for other challenging BS situations, allude to the need for new BS methods custom designed for maritime vision.

Object Detection in a Maritime Environment: Performance Evaluation of Background Subtraction Methods

http://www.yongxu.org/paper/Multiple%20representations%20and%20sparse%20representation%20for%20image%20classification.pdf

Multiple representations and sparse representation for image classification

Background modeling has played an important role in detecting the foreground for video analysis. In this paper, we presented a novel background modeling method for foreground segmentation. The innovations of the proposed method lie in the joint usage of the pixel-based adaptive segmentation method and the background updating strategy, which is performed in both pixel and object levels. Current pixel-based adaptive segmentation method only updates the background at the pixel level and does not take into account the physical changes of the object, which may result in a series of problems in foreground detection, e.g., a static or low-speed object is updated too fast or merely a partial foreground region is properly detected. To avoid these deficiencies, we used a counter to place the foreground pixels into two categories (illumination and object). The proposed method extracted a correct foreground object by controlling the updating time of the pixels belonging to an object or an illumination region respectively. Extensive experiments showed that our method is more competitive than the state-of-the-art foreground detection methods, particularly in the intermittent object motion scenario. Moreover, we also analyzed the efficiency of our method in different situations to show that the proposed method is available for real-time applications.

An Adaptive Background Modeling Method for Foreground Segmentation

Designing a general motion detection method that has self-adaptive parameters remains a challenging issue in video surveillance. To address this problem, in this paper, a dual-target nonparametric background modeling (DTNBM) method is proposed. This model integrates the gray value and gradient to represent each pixel, which enhances the discriminative ability of the background model. We design a simple but effective classification rule for determining whether a pixel belongs to a motionless object or dynamic background. Moreover, DTNBM provides suitable updating strategies for the two categories of pixels. Most importantly, DTNBM utilizes a dual-target updating strategy to preserve the completeness of static objects and prevent false detections that are caused by background initialization or frequent background variations. To improve the updating effectiveness and efficiency, we combine similar and random schemes for background updating. The key features of DTNBM include nonparametric modeling and a controlling threshold adaptation process, which render our method easy to use on various scenarios. Comprehensive experiments have been conducted, and the results demonstrate that DTNBM outperforms the state-of-the-art methods in foreground detection.

Zuofeng Zhong

Papers

Multiple representations and sparse representation for image classification

An Adaptive Background Modeling Method for Foreground Segmentation

A general moving detection method using dual-target nonparametric background model