We investigate the properties of a metric between two distributions, the Earth Mover's Distance (EMD), for content-based image retrieval. The EMD is based on the minimal cost that must be paid to transform one distribution into the other, in a precise sense, and was first proposed for certain vision problems by Peleg, Werman, and Rom. For image retrieval, we combine this idea with a representation scheme for distributions that is based on vector quantization. This combination leads to an image comparison framework that often accounts for perceptual similarity better than other previously proposed methods. The EMD is based on a solution to the transportation problem from linear optimization, for which efficient algorithms are available, and also allows naturally for partial matching. It is more robust than histogram matching techniques, in that it can operate on variable-length representations of the distributions that avoid quantization and other binning problems typical of histograms. When used to compare distributions with the same overall mass, the EMD is a true metric. In this paper we focus on applications to color and texture, and we compare the retrieval performance of the EMD with that of other distances.

/pdf/the-earth-mover-s-distance-as-a-metric-for-image-retrieval-h8u7zfx61r.pdf

The Earth Mover's Distance as a Metric for Image Retrieval

Image content based retrieval is emerging as an important research area with application to digital libraries and multimedia databases. The focus of this paper is on the image processing aspects and in particular using texture information for browsing and retrieval of large image data. We propose the use of Gabor wavelet features for texture analysis and provide a comprehensive experimental evaluation. Comparisons with other multiresolution texture features using the Brodatz texture database indicate that the Gabor features provide the best pattern retrieval accuracy. An application to browsing large air photos is illustrated.

Texture features for browsing and retrieval of image data

Research on ways to extend and improve query methods for image databases is widespread. We have developed the QBIC (Query by Image Content) system to explore content-based retrieval methods. QBIC allows queries on large image and video databases based on example images, user-constructed sketches and drawings, selected color and texture patterns, camera and object motion, and other graphical information. Two key properties of QBIC are (1) its use of image and video content-computable properties of color, texture, shape and motion of images, videos and their objects-in the queries, and (2) its graphical query language, in which queries are posed by drawing, selecting and other graphical means. This article describes the QBIC system and demonstrates its query capabilities. QBIC technology is part of several IBM products. >

Query by image and video content: the QBIC system

An intelligent electronic appliance preferably includes a user interface, data input and/or output port, and an intelligent processor. A preferred embodiment comprises a set top box for interacting with broadband media streams, with an adaptive user interface, content-based media processing and/or media metadata processing, and telecommunications integration. An adaptive user interface models the user, by observation, feedback, and/or explicit input, and presents a user interface and/or executes functions based on the user model. A content-based media processing system analyzes media content, for example audio and video, to understand the content, for example to generate content-descriptive metadata. A media metadata processing system operates on locally or remotely generated metadata to process the media in accordance with the metadata, which may be, for example, an electronic program guide, MPEG 7 data, and/or automatically generated format. A set top box preferably includes digital trick play effects, and incorporated digital rights management features.

Intelligent electronic appliance system and method

The goal of this work is to accurately detect and localize boundaries in natural scenes using local image measurements. We formulate features that respond to characteristic changes in brightness, color, and texture associated with natural boundaries. In order to combine the information from these features in an optimal way, we train a classifier using human labeled images as ground truth. The output of this classifier provides the posterior probability of a boundary at each image location and orientation. We present precision-recall curves showing that the resulting detector significantly outperforms existing approaches. Our two main results are 1) that cue combination can be performed adequately with a simple linear model and 2) that a proper, explicit treatment of texture is required to detect boundaries in natural images.

/pdf/learning-to-detect-natural-image-boundaries-using-local-1fw2xru2j4.pdf

Learning to detect natural image boundaries using local brightness, color, and texture cues

In the query by image content (QBIC) project we are studying methods to query large on-line image databases using the images' content as the basis of the queries. Examples of the content we use include color, texture, and shape of image objects and regions. Potential applications include medical (`Give me other images that contain a tumor with a texture like this one'), photo-journalism (`Give me images that have blue at the top and red at the bottom'), and many others in art, fashion, cataloging, retailing, and industry. Key issues include derivation and computation of attributes of images and objects that provide useful query functionality, retrieval methods based on similarity as opposed to exact match, query by image example or user drawn image, the user interfaces, query refinement and navigation, high dimensional database indexing, and automatic and semi-automatic database population. We currently have a prototype system written in X/Motif and C running on an RS/6000 that allows a variety of queries, and a test database of over 1000 images and 1000 objects populated from commercially available photo clip art images. In this paper we present the main algorithms for color texture, shape and sketch query that we use, show example query results, and discuss future directions.© (1993) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

/pdf/qbic-project-querying-images-by-content-using-color-texture-3jgprbildi.pdf

QBIC project: querying images by content, using color, texture, and shape

Images in an image database are searched in response to queries which include the visual characteristics of the images such as colors, textures, shapes, and sizes, as well as by textual tags appended to the images. Queries are constructed in an image query construction area in response to values of representations of the visual characteristics and to locations of the representations in the image query construction area.

Image query system and method

This invention is directed to relational extenders for a computer-based relational database. Each relational extender includes at least one column, in a first, business, table containing a user defined application database, dedicated to object handles for defining the complex data type of an object; a second, attribute, table containing at least one column defining a unique characteristic associated with the one object and one column dedicated to containing the object handle; and a third, metadata, table containing at least one column defining a common characteristic associated with all objects defined within the business table and one column dedicated to containing the object handle and at least one column dedicated to containing a reference to object data associated with the object. The relational extender further includes a fourth table containing a reference to each object handle column defined in the first table, and a fifth table containing the names of the second and third tables for each object defined in the first table.

Relational database extenders for handling complex data types

A system enables a user to query for key words and phrases a text document, such as a presentation slide file, and an associated audio stream, such as can be derived from an audio-video recording that is made of a presenter contemporaneously with the showing of the slides to an audience. A graphical user interface is presented in which query results for both the text document and the audio stream are displayed in a time-aligned format, to enable a user to easily and conveniently browse the text document and accompanying time-aligned audio stream based on the key words/phrases.

System and method for linking an audio stream with accompanying text material

A method and apparatus for indexing and searching content in a hardcopy document utilizes a searching assistant computing device (402) with an index table (420) stored in memory (412). The index table (420) is created in memory by scanning a 2-D barcode from a hardcopy document or alternatively by downloading indexing information from a web page via the Internet (430). A search engine (410) in the searching assistant (402) searches the index table (420) to locate a data element found in the content of the hardcopy document. The indexing information corresponding to the data element is displayed to a user as part of the search results to indicate the location of the data element in the hardcopy document.

Carlton Wayne Niblack

Papers

QBIC project: querying images by content, using color, texture, and shape

Image query system and method

Relational database extenders for handling complex data types

System and method for linking an audio stream with accompanying text material

Method and apparatus for indexing and searching content in hardcopy documents