Automatic estimation of salient object regions across images, without any prior assumption or knowledge of the contents of the corresponding scenes, enhances many computer vision and computer graphics applications. We introduce a regional contrast based salient object detection algorithm, which simultaneously evaluates global contrast differences and spatial weighted coherence scores. The proposed algorithm is simple, efficient, naturally multi-scale, and produces full-resolution, high-quality saliency maps. These saliency maps are further used to initialize a novel iterative version of GrabCut, namely SaliencyCut, for high quality unsupervised salient object segmentation. We extensively evaluated our algorithm using traditional salient object detection datasets, as well as a more challenging Internet image dataset. Our experimental results demonstrate that our algorithm consistently outperforms 15 existing salient object detection and segmentation methods, yielding higher precision and better recall rates. We also show that our algorithm can be used to efficiently extract salient object masks from Internet images, enabling effective sketch-based image retrieval (SBIR) via simple shape comparisons. Despite such noisy internet images, where the saliency regions are ambiguous, our saliency guided image retrieval achieves a superior retrieval rate compared with state-of-the-art SBIR methods, and additionally provides important target object region information.

https://mmcheng.net/mftp/SalObj/PosterSaliency.pdf

Global contrast based salient region detection

We have witnessed great interest and a wealth of promise in content-based image retrieval as an emerging technology. While the last decade laid foundation to such promise, it also paved the way for a large number of new techniques and systems, got many new people involved, and triggered stronger association of weakly related fields. In this article, we survey almost 300 key theoretical and empirical contributions in the current decade related to image retrieval and automatic image annotation, and in the process discuss the spawning of related subfields. We also discuss significant challenges involved in the adaptation of existing image retrieval techniques to build systems that can be useful in the real world. In retrospect of what has been achieved so far, we also conjecture what the future may hold for image retrieval research.

/pdf/image-retrieval-ideas-influences-and-trends-of-the-new-age-51whw4w7su.pdf

Image retrieval: Ideas, influences, and trends of the new age

In this paper, we study the salient object detection problem for images. We formulate this problem as a binary labeling task where we separate the salient object from the background. We propose a set of novel features, including multiscale contrast, center-surround histogram, and color spatial distribution, to describe a salient object locally, regionally, and globally. A conditional random field is learned to effectively combine these features for salient object detection. Further, we extend the proposed approach to detect a salient object from sequential images by introducing the dynamic salient features. We collected a large image database containing tens of thousands of carefully labeled images by multiple users and a video segment database, and conducted a set of experiments over them to demonstrate the effectiveness of the proposed approach.

/pdf/learning-to-detect-a-salient-object-19dnoso19v.pdf

Learning to Detect a Salient Object

Most existing bottom-up methods measure the foreground saliency of a pixel or region based on its contrast within a local context or the entire image, whereas a few methods focus on segmenting out background regions and thereby salient objects Instead of considering the contrast between the salient objects and their surrounding regions, we consider both foreground and background cues in a different way We rank the similarity of the image elements (pixels or regions) with foreground cues or background cues via graph-based manifold ranking The saliency of the image elements is defined based on their relevances to the given seeds or queries We represent the image as a close-loop graph with super pixels as nodes These nodes are ranked based on the similarity to background and foreground queries, based on affinity matrices Saliency detection is carried out in a two-stage scheme to extract background regions and foreground salient objects efficiently Experimental results on two large benchmark databases demonstrate the proposed method performs well when against the state-of-the-art methods in terms of accuracy and speed We also create a more difficult benchmark database containing 5,172 images to test the proposed saliency model and make this database publicly available with this paper for further studies in the saliency field

https://faculty.ucmerced.edu/mhyang/papers/cvpr13_saliency.pdf

Saliency Detection via Graph-Based Manifold Ranking

Modeling visual attention-particularly stimulus-driven, saliency-based attention-has been a very active research area over the past 25 years. Many different models of attention are now available which, aside from lending theoretical contributions to other fields, have demonstrated successful applications in computer vision, mobile robotics, and cognitive systems. Here we review, from a computational perspective, the basic concepts of attention implemented in these models. We present a taxonomy of nearly 65 models, which provides a critical comparison of approaches, their capabilities, and shortcomings. In particular, 13 criteria derived from behavioral and computational studies are formulated for qualitative comparison of attention models. Furthermore, we address several challenging issues with models, including biological plausibility of the computations, correlation with eye movement datasets, bottom-up and top-down dissociation, and constructing meaningful performance measures. Finally, we highlight current research trends in attention modeling and provide insights for future.

State-of-the-Art in Visual Attention Modeling

Videos are composed of many shots that are caused by different camera operations, e.g., on/off operations and switch- ing between cameras. One important goal in video analysis is to group the shots into temporal scenes, such that all the shots in a single scene are related to the same subject, which could be a particular physical setting, an on-going action or a theme. In this paper, we present a general framework for temporal scene segmentation in various video domains. The proposed method is formulated in a statistical fashion and uses the Markov chain Monte Carlo (MCMC) technique to determine the boundaries between video scenes. In this approach, a set of arbitrary scene boundaries are initialized at random locations and are automat- ically updated using two types of updates: diffusion and jumps. Diffusion is the process of updating the boundaries between adjacent scenes. Jumps consist of two reversible operations: the merging of two scenes and the splitting of an existing scene. The posterior probability of the target distribution of the number of scenes and their corresponding boundary locations is computed based on the model priors and the data likelihood. The updates of the model parameters are controlled by the hypothesis ratio test in the MCMC process, and the samples are collected to generate the final scene boundaries. The major advantage of the proposed framework is two-fold: (1) it is able to find the weak boundaries as well as the strong boundaries, i.e., it does not rely on the fixed threshold; (2) it can be applied to different video domains. We have tested the proposed method on two video domains: Home Videos and Feature Films, and accurate results have been obtained. provides the chapters that correspond to the different sub- themes of the movies. In television videos, segmentation can be used to separate the commercials from the regular programs. In news broadcast programs, segmentation can be used to identify different news stories. In home videos, scene segmentation may help the consumers to logically organize the videos related to the different events (e.g., birthday, graduation, weddings, or vacation (e.g. city tours, sightseeing)).

Video Scene Segmentation Using Markov Chain

In this paper, we present the PEGASUS system. PEGASUS is an integrated news video search system with three major components: (1) user interface, where users can formulate search queries and browse the returned results; (2) server, which takes the queries from user interface, performs the searches and ranks the search results before returns them to the users; (3) data storage, which is composed of feature indexing system and video database. The PEGASUS system has the capability to allow users to perform fast multi-modality video search using video features, including features from both audio and visual portions of the videos. To search a target topic, the user first submits an initial query using the prior knowledge on the topic. Then, through a series of relevance feedback processes, a set of relevant video shots are returned by the system to the user. The user is able to further view the results using the video-on-demand (VoD) functionality of the system. The system has been constructed using over 45,000 news video shots, and it is available online of public access.

https://crcv.ucf.edu/papers/spie2006_jim.pdf

PEGASUS: an information mining system for TV news videos

In this fast paced digital age, a vast amount of videos are produced every day, such as movies, TV programs, personal home videos, surveillance video, etc. This places a high demand for effective video data analysis and management techniques. In this dissertation, we have developed new techniques for segmentation, linking and understanding of video scenes. Firstly, we have developed a video scene segmentation framework that segments the video content into story units. Then, a linking method is designed to find the semantic correlation between video scenes/stories. Finally, to better understand the video content, we have developed a spatiotemporal attention detection model for videos. Our general framework for temporal scene segmentation, which is applicable to several video domains, is formulated in a statistical fashion and uses the Markov chain Monte Carlo (MCMC) technique to determine the boundaries between video scenes. In this approach, a set of arbitrary scene boundaries are initialized at random locations and are further automatically updated using two types of updates: diffusion and jumps. The posterior probability of the target distribution of the number of scenes and their corresponding boundary locations are computed based on the model priors and the data likelihood. Model parameter updates are controlled by the MCMC hypothesis ratio test, and samples are collected to generate the final scene boundaries. The major contribution of the proposed framework is two-fold: (1) it is able to find weak boundaries as well as strong boundaries, i.e., it does not rely on the fixed threshold; (2) it can be applied to different video domains. We have tested the proposed method on two video domains: home videos and feature films. On both of these domains we have obtained very accurate results, achieving on the average of 86% precision and 92% recall for home video segmentation, and 83% precision and 83% recall for feature films. The video scene segmentation process divides videos into meaningful units. These segments (or stories) can be further organized into clusters based on their content similarities. In the second part of this dissertation, we have developed a novel concept tracking method, which links news stories that focus on the same topic across multiple sources. The semantic linkage between the news stories is reflected in the combination of both their visual content and speech content. Visually, each news story is represented by a set of key frames, which may or may not contain human faces. The facial key frames are linked based on the analysis of the extended facial regions, and the non-facial key frames are correlated using the global matching. The textual similarity of the stories is expressed in terms of the normalized textual similarity between the keywords in the speech content of the stories. The developed framework has also been applied to the task of story ranking, which computes the interestingness of the stories. The proposed semantic linking framework and the story ranking method have both been…

/pdf/video-content-extraction-scene-segmentation-linking-and-541xc0swg2.pdf

Video Content Extraction: Scene Segmentation, Linking And Attention Detection

We address the problem of detecting the conversation scenes from feature films and propose and efficient and robust method for the stated problem. This method utilizes the structural information of the movie scenes with the combination of the low-level and mid-level features. We propose and demonstrate that a finite state machine (FSM) is suitable for detecting movie scenes with conversational settings. Two major characteristics of motion pictures, motion and audio, are used in our approach. The transitions of the FSM are determined by two mid-level features of each shot in the scene: the activity intensity and the face identity. Our FSM has been experimented on over 50 clips with both positive and negative examples and produces convincing results.

Conversation detection in feature films using finite state machines

In this paper, we address the problem of the scene classifications in feature films and propose a robust framework for the stated problem. The framework utilizes the structural information of the movie scenes rather than analyzing the global low level feature values. We propose and demonstrate that the Finite State Machines (FSM) are suitable for classifying the movie scenes into three categories: conversation, explosion/gunfire and suspense. Three major characteristics of motion pictures,motion, color and audio, are used in our approach. The transitions of the FSMs are determined by the mid-level features of each shot in the scene. Our FSMs have been experimented on a large set of data with both positive and negative examples and produces impressive results.

https://crcv.ucf.edu/papers/zhai_icpr_2004.pdf

Yun Zhai

Papers

Video Scene Segmentation Using Markov Chain

PEGASUS: an information mining system for TV news videos

Video Content Extraction: Scene Segmentation, Linking And Attention Detection

Conversation detection in feature films using finite state machines

Finite State Machines in Movie Scene Classification