VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method

In the current Web scenario a video browsing tool that produces on-the-fly storyboards is more and more a need. Video summary techniques can be helpful but, due to their long processing time, they are usually unsuitable for on-the-fly usage. Therefore, it is common to produce storyboards in advance, penalizing users customization. The lack of customization is more and more critical, as users have different demands and might access the Web with several different networking and device technologies. In this paper we propose STIMO, a summarization technique designed to produce on-the-fly video storyboards. STIMO produces still and moving storyboards and allows advanced users customization (e.g., users can select the storyboard length and the maximum time they are willing to wait to get the storyboard). STIMO is based on a fast clustering algorithm that selects the most representative video contents using HSV frame color distribution. Experimental results show that STIMO produces storyboards with good quality and in a time that makes on-the-fly usage possible.

/pdf/stimo-still-and-moving-video-storyboard-for-the-web-scenario-298l1dohr4.pdf

STIMO: STIll and MOving video storyboard for the web scenario

We address the problem of image collection summarization by learning mixtures of submodular functions. Submodularity is useful for this problem since it naturally represents characteristics such as fidelity and diversity, desirable for any summary. Several previously proposed image summarization scoring methodologies, in fact, instinctively arrived at submodularity. We provide classes of submodular component functions (including some which are instantiated via a deep neural network) over which mixtures may be learnt. We formulate the learning of such mixtures as a supervised problem via large-margin structured prediction. As a loss function, and for automatic summary scoring, we introduce a novel summary evaluation method called V-ROUGE, and test both submodular and non-submodular optimization (using the submodular-supermodular procedure) to learn a mixture of submodular functions. Interestingly, using non-submodular optimization to learn submodular functions provides the best results. We also provide a new data set consisting of 14 real-world image collections along with many human-generated ground truth summaries collected using Amazon Mechanical Turk. We compare our method with previous work on this problem and show that our learning approach outperforms all competitors on this new data set. This paper provides, to our knowledge, the first systematic approach for quantifying the problem of image collection summarization, along with a new data set of image collections and human summaries.

/pdf/learning-mixtures-of-submodular-functions-for-image-2c4qpoa7hc.pdf

Learning Mixtures of Submodular Functions for Image Collection Summarization

Although video summarization has achieved great success in recent years, few approaches have realized the influence of video structure on the summarization results. As we know, the video data follow a hierarchical structure, i.e., a video is composed of shots, and a shot is composed of several frames. Generally, shots provide the activity-level information for people to understand the video content. While few existing summarization approaches pay attention to the shot segmentation procedure. They generate shots by some trivial strategies, such as fixed length segmentation, which may destroy the underlying hierarchical structure of video data and further reduce the quality of generated summaries. To address this problem, we propose a structure-adaptive video summarization approach that integrates shot segmentation and video summarization into a Hierarchical Structure-Adaptive RNN, denoted as HSA-RNN. We evaluate the proposed approach on four popular datasets, i.e., SumMe, TVsum, CoSum and VTW. The experimental results have demonstrated the effectiveness of HSA-RNN in the video summarization task.

/pdf/hsa-rnn-hierarchical-structure-adaptive-rnn-for-video-r5c303qarp.pdf

HSA-RNN: Hierarchical Structure-Adaptive RNN for Video Summarization

Exploiting the temporal dependency among video frames or subshots is very important for the task of video summarization. Practically, RNN is good at temporal dependency modeling, and has achieved overwhelming performance in many video-based tasks, such as video captioning and classification. However, RNN is not capable enough to handle the video summarization task, since traditional RNNs, including LSTM, can only deal with short videos, while the videos in the summarization task are usually in longer duration. To address this problem, we propose a hierarchical recurrent neural network for video summarization, called H-RNN in this paper. Specifically, it has two layers, where the first layer is utilized to encode short video subshots cut from the original video, and the final hidden state of each subshot is input to the second layer for calculating its confidence to be a key subshot. Compared to traditional RNNs, H-RNN is more suitable to video summarization, since it can exploit long temporal dependency among frames, meanwhile, the computation operations are significantly lessened. The results on two popular datasets, including the Combined dataset and VTW dataset, have demonstrated that the proposed H-RNN outperforms the state-of-the-arts.

Hierarchical Recurrent Neural Network for Video Summarization

In this paper, we propose a video summarization algorithm by multiple extractions of key frames in each shot. This algorithm is based on the k-medoid clustering algorithms to find the best representative frame for each video shot. This algorithm, which is applicable to all types of descriptors, consists of extracting key frames by similarity clustering according to the given index. In our proposal, the distance between frames is calculated using a fast full search block matching algorithm based on the frequency domain. The proposed approach is computationally tractable and robust with respect to sudden changes in mean intensity within a shot. Additionally, this approach produces different key frames even in the presence of large motion. The experiments results show that our algorithm extracts multiple representatives frames in each video shot without visual redundancy, and thus it is an effective tool for video indexing and retrieval.

Video summarization by k-medoid clustering

Fast Fourier transforms (FFTs) which are O(N logN) algorithms to compute a discrete Fourier transform (DFT) of size N have been called one of the ten most important algorithms of the twentieth century. However, even though many algorithms have been developed to speed up the computation the sum of absolute difference (SAD) matching, they are exclusively designed in the spatial domain. In this paper, we propose a fast frequency algorithm to speed up the process of (SAD) matching. We use a new approach to approximate the SAD metric by cosine series which can be expressed in correlation terms. These latter can be computed using FFT algorithms. Experimental results demonstrate the effectiveness of our method when using only the first correlation terms for block and template matching in terms of accuracy and speed. The proposed algorithm is suitable for software implementations and has a deterministic execution time unlike the existing fast algorithms for SAD matching.

Adjustable SAD matching algorithm using frequency domain

We propose a new algorithm to calculate the sum square difference (SSD) metric for estimating translational motion. Using some substitutions and complex arithmetics, computation of the SSD is derived to be a correlation function of two substituting functions. The former can be computed using the fast Fourier transform (FFT) approach, which is less computationally expensive than the direct computation of the SSD. Zero padding of the block is useful both to avoid the circular character of the FFT and to computes the SSD function at once. We prove the existence of a simple and optimal correlation method for noncircular SSD matching.

Simple noncircular correlation method for exhaustive sum square difference matching

We present a new method for object tracking; we use an efficient local search scheme based on the Kalman filter and the probability product kernel KFPPK to find the image region with a histogram most similar to the histogram of the tracked target. Experimental results verify the effectiveness of this proposed system.

/pdf/an-adaptive-object-tracking-using-kalman-filter-and-2j8owpn1k4.pdf

Fedwa Essannouni

Papers

Video summarization by k-medoid clustering

Adjustable SAD matching algorithm using frequency domain

Simple noncircular correlation method for exhaustive sum square difference matching

An Adaptive Object Tracking Using Kalman Filter and Probability Product Kernel

Fast and robust object tracking via AcceptReject color histogram-based method