An intelligent electronic appliance preferably includes a user interface, data input and/or output port, and an intelligent processor. A preferred embodiment comprises a set top box for interacting with broadband media streams, with an adaptive user interface, content-based media processing and/or media metadata processing, and telecommunications integration. An adaptive user interface models the user, by observation, feedback, and/or explicit input, and presents a user interface and/or executes functions based on the user model. A content-based media processing system analyzes media content, for example audio and video, to understand the content, for example to generate content-descriptive metadata. A media metadata processing system operates on locally or remotely generated metadata to process the media in accordance with the metadata, which may be, for example, an electronic program guide, MPEG 7 data, and/or automatically generated format. A set top box preferably includes digital trick play effects, and incorporated digital rights management features.

Intelligent electronic appliance system and method

An adaptive interface for a programmable system, for predicting a desired user function, based on user history, as well as machine internal status and context. The apparatus receives an input from the user and other data. A predicted input is presented for confirmation by the user, and the predictive mechanism is updated based on this feedback. Also provided is a pattern recognition system for a multimedia device, wherein a user input is matched to a video stream on a conceptual basis, allowing inexact programming of a multimedia device. The system analyzes a data stream for correspondence with a data pattern for processing and storage. The data stream is subjected to adaptive pattern recognition to extract features of interest to provide a highly compressed representation which may be efficiently processed to determine correspondence. Applications of the interface and system include a VCR, medical device, vehicle control system, audio device, environmental control system, securities trading terminal, and smart house. The system optionally includes an actuator for effecting the environment of operation, allowing closed-loop feedback operation and automated learning.

Ergonomic man-machine interface incorporating adaptive pattern recognition based control system

Partitioning a video source into meaningful segments is an important step for video indexing. We present a comprehensive study of a partitioning system that detects segment boundaries. The system is based on a set of difference metrics and it measures the content changes between video frames. A twin-comparison approach has been developed to solve the problem of detecting transitions implemented by special effects. To eliminate the false interpretation of camera movements as transitions, a motion analysis algorithm is applied to determine whether an actual transition has occurred. A technique for determining the threshold for a difference metric and a multi-pass approach to improve the computation speed and accuracy have also been developed.

Automatic partitioning of full-motion video

An adaptive interface for a programmable system, for predicting a desired user function, based on user history, as well as machine internal status and context. The apparatus receives an input from the user and other data. A predicted input is presented for confirmation by the user, and the predictive mechanism is updated based on this feedback. Also provided is a pattern recognition system for a multimedia device, wherein a user input is matched to a video stream on a conceptual basis, allowing inexact programming of a multimedia device. The system analyzes a data stream for correspondence with a data pattern for processing and storage. The data stream is subjected to adaptive pattern recognition to extract features of interest to provide a highly compressed representation that may be efficiently processed to determine correspondence. Applications of the interface and system include a video cassette recorder (VCR), medical device, vehicle control system, audio device, environmental control system, securities trading terminal, and smart house. The system optionally includes an actuator for effecting the environment of operation, allowing closed-loop feedback operation and automated learning.

Adaptive pattern recognition based control system and method

Color histograms are used to compare images in many applications. Their advantages are efficiency, and insensitivity to small changes in camera viewpoint. However, color histograms lack spatial information, so images with very different appearances can have similar histograms. For example, a picture of fall foliage might contain a large number of scattered red pixels; this could have a similar color histogram to a picture with a single large red object. We describe a histogram-based method for comparing images that incorporates spatial information. We classify each pixel in a given color bucket as either coherent or incoherent, based on whether or not it is part of a large similarly-colored region. A color coherence vector (CCV) stores the number of coherent versus incoherent pixels with each color. By separating coherent pixels from incoherent pixels, CCV’s provide finer distinctions than color histograms. CCV’s can be computed at over 5 images per second on a standard workstation. A database with 15,000 images can be queried for the images with the most similar CCV’s in under 2 seconds. We show that CCV’s can give superior results to color his∗To whom correspondence should be addressed tograms for image retrieval.

/pdf/comparing-images-using-color-coherence-vectors-4vt1adbw0d.pdf

Comparing images using color coherence vectors

This paper presents a novel approach to processing encoded video sequences prior to decoding. Scene changes may be easily detected using DCT coefficients in JPEG and MPEG encoded video sequences. In addition, by analyzing the DCT coefficients, regions of interest may be isolated prior to decompression, increasing efficiency of any subsequent image processing steps, such as edge detection. The results are currently used in a video browser, and are part of an ongoing research project in creating large video databases. The procedure is presented in detail and several examples are exhibited.

Image processing on compressed data for large video databases

Large video databases, as well as many other applications involving video, such as multimedia, training, and `movie-on-demand' systems, require efficient steps to manipulate the enormous amount of data associated with full motion video. In this paper the incoming video is systematically and efficiently reduced via a frame selection procedure which takes advantage of the fact that the incoming video is encoded using one of several existing DCT- based standards. The procedure is performed in the frequency domain prior to video decoding. Further refinement in the frame selection step is achieved using a robust metric based upon the color histogram of the selected subset of decoded frames. The procedure is presented in detail and several examples are exhibited.

Feature management for large video databases

A computer implemented method provides a representative frame (Rframe) for a group of frames in a video sequence. The method comprises selecting a reference frame from the group of frames and storing the reference frame in a computer memory. The method defines a peripheral motion tracking region along an edge of the reference frame and successively tracks movement of boundary pixels in the tracking region, symbolizing any of the length of the shot and the presence of any caption.

Method for representing contents of a single video shot using frames

A computer implemented method for providing a representative frame (Rframe) for a group of frames in a video sequence comprises selecting a reference frame from the group of frames; storing the reference frame in a computer memory, defining a peripheral motion tracking region along an edge of the reference frame, and successively tracking movement of boundary pixels in the tracking region, symbolizing any of the length of the shot and the presence of any caption

Arding Hsu

Papers

Image processing on compressed data for large video databases

Feature management for large video databases

Method for representing contents of a single video shot using frames

Representing contents of single video shot using Rframes