We present a new approach for modeling multi-modal data sets, focusing on the specific case of segmented images with associated text. Learning the joint distribution of image regions and words has many applications. We consider in detail predicting words associated with whole images (auto-annotation) and corresponding to particular image regions (region naming). Auto-annotation might help organize and access large collections of images. Region naming is a model of object recognition as a process of translating image regions to words, much as one might translate from one language to another. Learning the relationships between image regions and semantic correlates (words) is an interesting example of multi-modal data mining, particularly because it is typically hard to apply data mining techniques to collections of images. We develop a number of models for the joint distribution of image regions and words, including several which explicitly learn the correspondence between regions and words. We study multi-modal and correspondence extensions to Hofmann's hierarchical clustering/aspect model, a translation model adapted from statistical machine translation (Brown et al.), and a multi-modal extension to mixture of latent Dirichlet allocation (MoM-LDA). All models are assessed using a large collection of annotated images of real scenes. We study in depth the difficult problem of measuring performance. For the annotation task, we look at prediction performance on held out data. We present three alternative measures, oriented toward different types of task. Measuring the performance of correspondence methods is harder, because one must determine whether a word has been placed on the right region of an image. We can use annotation performance as a proxy measure, but accurate measurement requires hand labeled data, and thus must occur on a smaller scale. We show results using both an annotation proxy, and manually labeled data.

/pdf/matching-words-and-pictures-hbi7wazp2t.pdf

Matching words and pictures

https://www.comp.nus.edu.sg/~cs4243/projects/TextDetectionSurvey.pdf

Text information extraction in images and video: a survey

We present a statistical model for organizing image collections which integrates semantic information provided by associate text and visual information provided by image features. The model is very promising for information retrieval tasks such as database browsing and searching for images based on text and/or image features. Furthermore, since the model learns relationships between text and image features, it can be used for novel applications such as associating words with pictures, and unsupervised learning for object recognition.

/pdf/learning-the-semantics-of-words-and-pictures-3x4zlhzbqr.pdf

Learning the semantics of words and pictures

Various methods of automatic shot boundary detection have been proposed and claimed to perform reliably. Although the detection of edits is fundamental to any kind of video analysis since it segments a video into its basic components, the shots, only few comparative investigations on early shot boundary detection algorithms have been published. These investigations mainly concentrate on measuring the edit detection performance, however, do not consider the algorithms? ability to classify the types and to locate the boundaries of the edits correctly. This paper extends these comparative investigations. More recent algorithms designed explicitly to detect specific complex editing operations such as fades and dissolves are taken into account, and their ability to classify the types and locate the boundaries of such edits are examined. The algorithms? performance is measured in terms of hit rate, number of false hits, and miss rate for hard cuts, fades, and dissolves over a large and diverse set of video sequences. The experiments show that while hard cuts and fades can be detected reliably, dissolves are still an open research issue. The false hit rate for dis-solves is usually unacceptably high, ranging from 50% up to over 400%. Moreover, all algorithms seem to fail under roughly the same conditions.

/pdf/comparison-of-automatic-shot-boundary-detection-algorithms-1tzbuyakz2.pdf

Comparison of automatic shot boundary detection algorithms

The increasing availability of high-performance, low-priced, portable digital imaging devices has created a tremendous opportunity for supplementing traditional scanning for document image acquisition. Digital cameras attached to cellular phones, PDAs, or wearable computers, and standalone image or video devices are highly mobile and easy to use; they can capture images of thick books, historical manuscripts too fragile to touch, and text in scenes, making them much more versatile than desktop scanners. Should robust solutions to the analysis of documents captured with such devices become available, there will clearly be a demand in many domains. Traditional scanner-based document analysis techniques provide us with a good reference and starting point, but they cannot be used directly on camera-captured images. Camera-captured images can suffer from low resolution, blur, and perspective distortion, as well as complex layout and interaction of the content and background. In this paper we present a survey of application domains, technical challenges, and solutions for the analysis of documents captured by digital cameras. We begin by describing typical imaging devices and the imaging process. We discuss document analysis from a single camera-captured image as well as multiple frames and highlight some sample applications under development and feasible ideas for future development.

Camera-based analysis of text and documents: a survey

A number of automated shot-change detection methods for indexing a video sequence to facilitate browsing and retrieval have been proposed. Many of these methods use color histograms or features computed from block motion or compression parameters to compute frame differences. It is important to evaluate and characterize their performance so as to deliver a single set of algorithms that may be used by other researchers for indexing video databases. We present the results of a performance evaluation and characterization of a number of shot-change detection methods that use color histograms, block motion matching, or MPEG compressed data.

Performance characterization of video-shot-change detection methods

Temporal segmentation of video is a necessary first step to indexing digital video for browsing and retrieval. A number of different video temporal segmentation algorithms have been published in the literature. There has been little effort to evaluate and characterize their performance so as to deliver a single (or set of) algorithms that may be used by other researchers for indexing video databases. The present results of evaluating a number of these algorithms and characterizing their performance, specifically with respect to robustness to encoder and bitrate changes. The lessons learnt have relevance to algorithm development and evaluation in general.

Performance characterization and comparison of video indexing algorithms

Like shot changes, the presence of text in digital video is an important event that can be used to index digital video and provide extremely useful semantic information about the scene content. The special characteristics of digital video compared to document images both require and allow new robust approaches to recognition of text in video. We discuss the characteristics and special challenges of text in video and present a strategy of detecting, localizing, and segmenting text from video data for the text indexing problem. Preliminary results from our approach are presented.

Indexing text events in digital video databases

As a first step to creating a video database, a video sequence has to be segmented into several subsequences based on significant changes in the scene. This enables the media user to identify the whole or a part of the sequence and to retrieve scenes of interest from a large video database. Researchers in the past have used a histogram based inter-frame difference approach to identify significant scene changes. To determine which is the best color coordinate system for video indexing, we have evaluated the histogram based indexing method using different color coordinate systems -- RGB, HSV, YIQ, L*a*b*, L*u*v* & Munsell -- and compared the results for accuracy of indexing with reference to subjective indexing. Since it is difficult to determine the exact threshold value to obtain reasonably good results, we also propose a vide segmenting method called hierarchical histogram based indexing that segments a video sequence into several levels of subsequences using different levels of threshold.

Evaluation of video sequence indexing and hierarchical video indexing

Video indexing is an important problem that has occupied recent research efforts. The text appearing in video can provide semantic information about the scene content. Detecting and recognizing text events can provide indices into the video for content based querying. We describe a system for detecting, tracking, and extracting artificial and scene text in MPEG-1 video. Preliminary results are presented.

/pdf/a-system-for-automatic-text-detection-in-video-1putl1lv14.pdf

Ullas Gargi

Papers

Performance characterization of video-shot-change detection methods

Performance characterization and comparison of video indexing algorithms

Indexing text events in digital video databases

Evaluation of video sequence indexing and hierarchical video indexing

A system for automatic text detection in video