The invention provides a system and method for automatically indexing and retrieving multimedia content. The method may include separating a multimedia data stream into audio, visual and text components, segmenting the audio, visual and text components based on semantic differences, identifying at least one target speaker using the audio and visual components, identifying a topic of the multimedia event using the segmented text and topic category models, generating a summary of the multimedia event based on the audio, visual and text components, the identified topic and the identified target speaker, and generating a multimedia description of the multimedia event based on the identified target speaker, the identified topic, and the generated summary.

System and method for automated multimedia content indexing and retrieval

Handbook of Software Engineering and Knowledge Engineering

Video production involves the process of capturing, editing and composing video segments for delivery to a consumer. A composition must yield a coherent presentation of an event or narrative. This process can be automated if appropriate domain-specific metadata are associated with video segments and composition techniques are established. Automation leads to the support of dynamic composition and customization for applications such as news on demand. In this paper, we present techniques to achieve dynamic, real-time and cohesive video composition and customization. We also identify metrics for evaluating our techniques with respect to existing manually produced video-based news. The results of such an evaluation show that the quality of automatic composition is comparable to-and in some cases, better than-broadcast news video composition. The results also validate the assertions on which the automatic composition techniques are based.

/pdf/automatic-composition-techniques-for-video-production-472gim2it2.pdf

Automatic composition techniques for video production

Considers the problem of querying multimedia presentations based on content information. Multimedia presentations are modeled as presentation graphs, which are directed acyclic graphs that visually specify the presentations. We present a graph data model for the specification of multimedia presentations and discuss query languages as effective tools to query and manipulate multimedia presentation graphs with respect to content information. To query the information flow throughout a multimedia presentation, as well as in each individual multimedia stream, we use revised versions of temporal operators Next, Connected and Until, together with path formulas. These constructs allow us to specify and query paths along a presentation graph. We present an icon-based graphical query language, GVISUAL, that provides iconic representations for these constructs and a user-friendly graphical interface for query specification. We also present an OQL-like language, GOQL (Graph OQL), with similar constructs, that allows textual and more traditional specifications of graph queries. Finally, we introduce GCalculus (Graph Calculus), a calculus-based language that establishes the formal grounds for the use of temporal operators in path formulas and for querying presentation graphs with respect to content information. We also discuss GCalculus/S (GCalculus with Sets) which avoids highly complex query expressions by eliminating the universal path quantifier, the negation operator and the universal quantifier. GCalculus/S represents the formal basis for GVISUAL, i.e. GVISUAL uses the constructs of GCalculus/S directly.

Querying multimedia presentations based on content

Multimedia data require specialized management techniques because the representations of color, time, semantic concepts, and other underlying information can be drastically different from one another. The user's subjective judgment can also have significant impact on what data or features are relevant in a given context. These factors affect both the performance of the retrieval algorithms and their effectiveness. This textbook on multimedia data management techniques offers a unified perspective on retrieval efficiency and effectiveness. It provides a comprehensive treatment, from basic to advanced concepts, that will be useful to readers of different levels, from advanced undergraduate and graduate students to researchers and to professionals. After introducing models for multimedia data (images, video, audio, text, and web) and for their features, such as color, texture, shape, and time, the book presents data structures and algorithms that help store, index, cluster, classify, and access common data representations. The authors also introduce techniques, such as relevance feedback and collaborative filtering, for bridging the "semantic gap" and present the applications of these to emerging topics, including web and social networking.

/pdf/data-management-for-multimedia-retrieval-sud0bxos1b.pdf

Data Management for Multimedia Retrieval

A multimedia presentation refers to the presentation of multimedia data using output devices such as monitors for text and video, and speakers for audio. Each presentation consists of multimedia segments which are obtained from a multimedia data model. In this paper, we propose to express the semantic coherency of a multimedia presentation in terms of presentation inclusion and exclusion constraints that are incorporated into the multimedia data model. Thus, when a user specifies a set of segments for a presentation, the DBMS adds segments into and/or deletes segments from the set in order to satisfy the inclusion and exclusion constraints. To automate the assembly of a presentation with concurrent presentation streams, we also propose presentation organization constraints that are incorporated into the multimedia data model, independent of any presentation. We give two algorithms for automated presentation assembly and discuss their complexity. We discuss the satisfiability of inclusion and exclusion constraints when negation is allowed, and we briefly describe a prototype system that is being developed for automated presentation assembly.

Automating the assembly of presentations from multimedia databases

In this paper, we present a methodology for automated construction of multimedia presentations. Semantic coherency of a multimedia presentation is expressed in terms of presentation inclusion and exclusion constraints. When a user specifies a set of segments for a presentation, the multimedia database system adds segments into and/or deletes segments from the set in order to satisfy the inclusion and exclusion constraints. We discuss the consistency and the satisfiability of inclusion and exclusion constraints when exclusion is allowed. Users express a presentation query by (a) pointing and clicking to an initial set of desired multimedia segments to be included into the presentation, and (b) specifying an upper bound on the time length of the presentation. The multimedia database system then finds the set of segments satisfying the inclusion-exclusion constraints and the time bound. Using priorities for segments and inclusion constraints, we give two algorithms for automated presentation assembly and discuss their complexity. To automate the assembly of a presentation with concurrent presentation streams, we introduce presentation organization constraints that are incorporated into the multimedia data model, independent of any presentation. We define four types of presentation organization constraints that, together with an underlying database ordering, allow us to obtain a unique presentation graph for a given set of multimedia segments. We briefly summarize a prototype system that fully incorporates the algorithms for the segment selection problem.

Constraint-based automation of multimedia presentation assembly

We introduce a constraint-driven methodology for the automated assembly, organization and playout of presentations from multimedia databases. We use inclusion and exclusion constraints for extracting a semantically coherent set of multimedia segments. Presentation organization constraints are utilized for organizing the multimedia segments into a presentation, which in turn helps decide the playout order of the extracted multimedia segments. The playout order of the segments is represented in a presentation graph. If the specified set of organization constraints are not sufficient to construct a unique presentation graph, we propose two techniques so that a unique graph is constructible. We also propose two playout algorithms, one for the generation, start and termination of playout agents, the other for dynamic control of playout management on organized presentations. The characteristics of these algorithms are expressed in terms of presentation playout parameters.

A Constraint-Driven Approach to Automate the Organization andPlayout of Presentations in Multimedia Databases

A constraint-driven methodology for designing a multimedia presentation system from multimedia databases

We introduce a constraint-driven methodology for the automated assembly, organization and playout of presentations from multimedia databases. We use inclusion and exclusion constraints for extracting a semantically coherent set of multimedia segments. Presentation organization constraints are utilized for organizing the multimedia segments into a presentation, which in turn helps decide the playout order of the extracted multimedia segments. The playout order of the segments is represented in a presentation graph. If the specified set of organization constraints are not sufficient to construct a unique presentation graph, we propose two techniques so that a unique graph is constructible. We also propose playout algorithms, one for the generation, start and termination of playout agents, the other to provide dynamic controls for playout management of organized presentations. The characteristics of these algorithms are expressed in terms of presentation playout parameters.

Veli Hakkoymaz

Papers

Automating the assembly of presentations from multimedia databases

Constraint-based automation of multimedia presentation assembly

A Constraint-Driven Approach to Automate the Organization andPlayout of Presentations in Multimedia Databases

A constraint-driven methodology for designing a multimedia presentation system from multimedia databases

Automating the organization of presentations for playout management in multimedia databases