Descriptive video service (DVS) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the Movie Description dataset contains a parallel corpus of over 54,000 sentences and video snippets from 72 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing DVS to scripts, we find that DVS is far more visual and describes precisely what is shown rather than what should happen according to the scripts created prior to movie production.

A Dataset for Movie Description

Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the MPII Movie Description dataset (MPII-MD) contains a parallel corpus of over 68K sentences and video snippets from 94 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are far more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production.

/pdf/a-dataset-for-movie-description-3pi7axvygy.pdf

A dataset for Movie Description

Audio description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. We introduce the Large Scale Movie Description Challenge (LSMDC) which contains a parallel corpus of 128,118 sentences aligned to video clips from 200 movies (around 150 h of video in total). The goal of the challenge is to automatically generate descriptions for the movie clips. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in the challenges organized in the context of two workshops at ICCV 2015 and ECCV 2016.

Movie Description

A video copy detection system that is based on content fingerprinting and can be used for video indexing and copyright applications is proposed. The system relies on a fingerprint extraction algorithm followed by a fast approximate search algorithm. The fingerprint extraction algorithm extracts compact content-based signatures from special images constructed from the video. Each such image represents a short segment of the video and contains temporal as well as spatial information about the video segment. These images are denoted by temporally informative representative images. To find whether a query video (or a part of it) is copied from a video in a video database, the fingerprints of all the videos in the database are extracted and stored in advance. The search algorithm searches the stored fingerprints to find close enough matches for the fingerprints of the query video. The proposed fast approximate search algorithm facilitates the online application of the system to a large video database of tens of millions of fingerprints, so that a match (if it exists) is found in a few seconds. The proposed system is tested on a database of 200 videos in the presence of different types of distortions such as noise, changes in brightness/contrast, frame loss, shift, rotation, and time shift. It yields a high average true positive rate of 97.6% and a low average false positive rate of 1.0%. These results emphasize the robustness and discrimination properties of the proposed copy detection system. As security of a fingerprinting system is important for certain applications such as copyright protections, a secure version of the system is also presented.

/pdf/a-robust-and-fast-video-copy-detection-system-using-content-4ntmtnvy90.pdf

A Robust and Fast Video Copy Detection System Using Content-Based Fingerprinting

Near-duplicate (ND) detection appears as a timely issue recently, being regarded as a powerful tool for various emerging applications. In the Web 2.0 environment particularly, the identification of near-duplicates enables the tasks such as copyright enforcement, news topic tracking, image and video search. In this paper, we describe an algorithm, namely Scale-Rotation invariant Pattern Entropy (SR-PE), for the detection of near-duplicates in large-scale video corpus. SR-PE is a novel pattern evaluation technique capable of measuring the spatial regularity of matching patterns formed by local keypoints. More importantly, the coherency of patterns and the perception of visual similarity, under the scenario that there could be multiple ND regions undergone arbitrary transformations, respectively, are carefully addressed through entropy measure. To demonstrate our work in large-scale dataset, a practical framework composed of three components: bag-of-words representation, local keypoint matching and SR-PE evaluation, is also proposed for the rapid detection of near-duplicates.

Scale-Rotation Invariant Pattern Entropy for Keypoint-Based Near-Duplicate Detection

This paper presents the status of a R&D project targeting the development of computer-vision tools to assist humans in generating and rendering video description for people with vision loss. Three principal issues are discussed: (1) production practices, (2) needs of people with vision loss, and (3) current system design, core technologies and implementation. The paper provides the main conclusions of consultations with producers of video description regarding their practices and with end-users regarding their needs, as well as an analysis of described productions that lead to propose a video description typology. The current status of a prototype software is also presented (audio-vision manager) that uses many computer-vision technologies (shot transition detection, key-frame identification, key-face recognition, key-text spotting, visual motion, gait/gesture characterization, key-place identification, key-object spotting and image categorization) to automatically extract visual content, associate textual descriptions and add them to the audio track with a synthetic voice. A proof of concept is also briefly described for a first adaptive video description player which allows end users to select various levels of video description.

Towards computer-vision software tools to increase production and accessibility of video description for people with vision loss

Approach we have tested in our submitted runs: For visualbased copy detection, we find links between video shot key-frames using a probabilistic latent space model over local matches between the keyframe images. This facilitates the extraction of significant groups of local matching descriptors that may represent common semantic elements of near duplicate key-frames. For 2009, we have worked on an optimal representation of the test database. We first select the discriminant local descriptors. Then, we quantize the selected local descriptors into a hierarchical structure. For audio based copy detection, we give results with two different feature parameters: 15-bit energy difference parameters similar to [1] and a feature-based mapping of test frames to query frames. Differences we found among the runs: We submitted 1 run for the video only copy detection task (same run for Balanced and for nofa). Four runs were submitted for the ”audio only” copy detection task : • CRIM.a.NOFA.EnNN2pass: energy-diff parameter search rescored with nearest-neighbor mapping. • CRIM.a.NOFA.NN22para: search using nearest-neighbor mapping. • CRIM.a.BALANCED.EnNN2pass: lower threshold than for NOFA case. • CRIM.a.BALANCED.EnNN22wt15: fuse Energy-diff parameters search (wt 15) with nearest-neighbor mapping search. We fused the video submission from CRIM with each of the four audio only submissions to get four different submissions for audio+video copy detection task. The threshold was adjusted based on the results of 2008 a+v queries. Relative contribution of each component of our approach: For visual-based copy detection, the probabilistic latent space model over local matches between the key-frame images produces a robust and accurate filtering process in relation to all possible local matches. It works well even if there are only a few local matches between the key-frames of the copied video in question. We have introduced a new method for SIFT quantizing. It improves the time computation performance while keeping a good precision for SIFT representation. For audio only copy detection, the fingerprints obtained by mapping each test frame to the nearest query frame (NN-based fingerprints) reduced minimal NDCR by half over that obtained with energy-difference based fingerprints. This work was supported in part by the Natural Science and Engineering Research Council of Canada (NSERC) What we learned about runs/approaches and the research question(s) that motivated them : Approaches based on local descriptor matching are efficient for video copy detection but very time consuming. Our method is more adapted when there is very little common visual information to establish a link between two key-frames. Video copy detection may not need such a good precision. For audio copy detection, mapping each test frame to the nearest query frame (NN-mapping) results in robust audio copy detection. The minimal normalized detection cost rate (NDCR) for even the worst case transformations is less than 0.03 for 2008 queries, and less than 0.075 for 2009 queries. The algorithm provides easy parallel processing on a graphics processing unit, leading to a very fast search.

/pdf/crim-s-content-based-copy-detection-system-for-trecvid-3c4812efs3.pdf

CRIM's content-based copy detection system for TRECVID

We present an application of video indexing/summarization to produce Videodescription (VD) for the blinds. Audio and computer vision technologies can automatically detect and recognize many elements that are pertinent to VD which can speed-up the VD production process. We have developed and integrated many of them into a first computer-assisted VD production software. The paper presents the main outcomes of this R&D activity started 5 years ago in our laboratory. Up to now, usability performance on various video and TV series types have shown a reduction of up to 50% in the VD time production process.

A computer-vision-assisted system for Videodescription scripting

We describe a new method to find and cluster recurrent key-places in a movie. It consists of an unsupervised classification of shots that are taking place in the same physical location (key-place). Our approach is based on finding links between key-frames belonging to a same key-place. We use a probabilistic latent space model over the possible match points between the image sets. This allows extracting significant groups of local descriptor matches that may represent characteristic elements of a key-place. A preliminary test on a full-length movie gives a recognition rate of 78.0% on the key-places clustering.

Key-Places Detection and Clustering in Movies Using Latent Aspects

An improved unsupervised classification method to extract and link places features and cluster recurrent physical locations (key-places) within a movie is presented. Our approach finds links between key frames of a common key-place based on the use of a probabilistic latent space model over the possible local matches between the key frames image set. This allows the extraction of significant groups of local matching descriptors that may represent characteristic elements of a key-place. An exhaustive evaluation of our approach was conducted on in-house and public image datasets, as well as on full-length movies. Results revealed that our method is very efficient for near-duplicate object/background detection with weak overlap. Performance measurements on full-length movies indicate a recognition rate of about 75% on the key-places clustering with a false alarm rate (FAR) of approximately 2%.

Maguelonne Héritier

Papers

Towards computer-vision software tools to increase production and accessibility of video description for people with vision loss

CRIM's content-based copy detection system for TRECVID

A computer-vision-assisted system for Videodescription scripting

Key-Places Detection and Clustering in Movies Using Latent Aspects

Places Clustering of Full-Length Film Key-Frames Using Latent Aspect Modeling Over SIFT Matches