1
Content-based Representation and Retrieval of Visual
Media: A State-of-the-Art Review
1
Philippe Aigrain
Institut de Recherche en Informatique de Toulouse, Universite Paul Sabatier
118, route de Narbonne, F-31062 Toulouse Cedex, France
HongJiang Zhang
Broadband Information Systems Lab, Hewlett-Packard Labs
1501 Page Mill Road. Palo Alto, CA94304, USA
Dragutin Petkovic
IBM Almaden Research Center
San Jose, CA 95120-6099, USA
Abstract
This paper reviews a number of recently available techniques in contentanalysis
of visual media and their application to the indexing, retrieval,abstracting, rele-
vance assessment, interactive perception, annotation and re-use of visualdocu-
ments.
1. Background
A few years ago, the problems of representation and retrieval of visualmedia were confined to
specialized image databases (geographical, medical, pilot experimentsin computerized slide
libraries), in the professional applications of the audiovisualindustries (production, broadcasting
and archives), and in computerized training or education. The presentdevelopment of multimedia
technology and information highways has put content processing of visualmedia at the core of
key application domains: digital and interactive video, large distributed digital libraries, multime-
dia publishing. Though the most important investments have been targeted at the information
infrastructure (networks, servers, coding and compression, deliverymodels, multimedia systems
architecture), a growing number of researchers have realized thatcontent processing will be a key
asset in putting together successful applications. The need for contentprocessing techniques has
been made evident from a variety of angles, ranging from achievingbetter quality in compression,
allowing user choice of programs in video-on-demand, achieving betterproductivity in video pro-
duction, providing access to large still image databases or integrating still images and video in
multimedia publishing and cooperative work.
Content-based retrieval of visual media and representation of visualdocuments in human-com-
puter interfaces are based on the availability of content representationdata (time-structure for
1. To apprear in Multimedia Tools and Applications special issue on Representation and Retrieval of Visual
Media.
2
time-based media, image signatures, object and motion data). When itis possible, the human pro-
duction of this descriptive data is so time consuming - and thus costly -that it is almost impossible
to generate it for large document spaces. There is some hope that for video documents, some of
this data will be created at production time and coded in the documentitself. Nonetheless it will
never be available for many existing documents, and when considering thehistory of media and
carriers one is lead to a very cautious estimate of how often this typeof information will be really
available even in future documents. Thus, there is a clear need forautomatic analysis tools which
are able to extract representation data from the documents.
The researchers involved in content processing efforts come from various backgrounds, for
instance:
• the publishing, entertainment, retail or document industry whereresearchers try to extend
their activity to visual documents, or to integrate them inhypertext-based new document
types,
• the AV hardware and software industry, primarily interested by digital editing tools and
other programma production tools,
• academic laboratories where research had been conducted for some time oncomputer
analysis and access to existing visual media, such as the MIT MediaLaboratory , the Insti-
tute of System Sciences in Singapore, or IRIT in France,
• large telecommunication company laboratories, where researchers areprimarily interest-
ing in cooperative work and remote access to visual media,
• the robotics vision, signal processing, image sequence processing forsecurity , or data
compression research communities who try to find new applications fortheir models of
images or human perception.
• computer hardware manufacturers developing digital library or visualmedia research pro-
grams.
These researchers originally used very different models and techniques and often conflicting
vocabulary. After a few years of lively confusion and exciting achievements, it isnow possible to
draw a clearer panorama of the state of this emerging field, and to outline some of its possible
directions of development.
In this paper, we review the methods available for different types of visual content analysis, repre-
sentation and their application and survey some open research problems.Section 2 covers various
visual features for representing and comparing image content. Sections 3reviews video content
parsing and representation algorithms and schemes, including temporalsegmentation, video
abstraction, shot comparison and soundtrack analysis. Section 4 presentapplications of visual
representation schemes in content-based image and video retrieval andbrowsing. Finally, Section
5 summaries our survey and current research directions.
2 The many facets of image similarity
Retrieval of still images by similarity, i.e. retrieving images which are similar to an already
retrieved image (retrieval by example) or to a model or schema is arelatively old idea. Some
3
might date it to the mnemotechnical ideas of the antiquity, but more seriously it appeared in spe-
cialized geographical information systems databases around 1980, inparticular in the Query by
Pictorial Example system of IMAID[CF80]. From the start, it was clearthat retrieval by similarity
called for specific definitions of what it means to be similar. In the mapping system, a satellite
image was matched to existing map images from the point of view ofsimilarity of road and river
networks, easily extracted from images by edge detection. Apart frompaper models [Aig87], it
was only in the beginning of the 90s that researchers started to look atretrieval by similarity in
large set of heterogeneous images with no specific model of their semanticcontents. The proto-
type systems of Kato[Kat92], followed by the availability of the QBIC commercial system using
several types of similarities [FSN+95] contributed to making this idea moreand more popular .
A system for retrieval by similarity rest on 3 components:
• extraction of features or image signatures from the images, and an efficient representation
and storage strategy for this precomputed data,
• a set of similarity measures, each of which captures some perceptivelymeaningful defini-
tion of similarity, and which should be efficiently computable when matching an example
with the whole database,
• a user interface for the choice of which definition(s) ofsimilarity should be applied for
retrieval, and for the ordered and visually efficient presentation of retrieved images and for
supporting relevance feedback.
Recent work has made evident that:
• A large number of meaningful types of similarity can and must be defined.Only part of
these definitions are associated with efficient feature extraction mechanisms and (dis)sim-
ilarity measures.
• Since there are many definitions of similarity and the discriminatingpower of each of the
measures is likely to degrade significantly for large image databases, the user interface and
the feature storage strategy components of the systems will play a moreand more essential
role. We will come back to this point in Section 4.1.
• Visual content based retrieval is best utilized when combined with thetraditional search,
both at user interface and the system level. The basic reason for this isthat we do not see
the possibility of content based retrieval replacing the ability ofparametric (SQL) search,
text and keywords to represent the rich semantic content of the visualmaterial (names,
places, action, prices etc.). The key is to apply content basedretrieval where appropriate,
and this is where the use of text and keywords is suboptimal. Examples ofsuch applica-
tions are where visual appearance (e.g. color, texture, shape, motion) are important search
arguments like in stock photo/video, art, retail, on-line shopping etc. Notonly content
based retrieval reduces the high variability among human indexers, but italso enables
more “fuzzy” browsing and search which in many application is anessential part of the
process. It is obvious then that content based retrieval involves stronguser interaction,
thus necessitating the development of special fast browser and UI techniques.
In this section we briefly survey the various types of similaritydefinitions and associated feature
extraction and measures for systems which do not assume any specificimage domain or a-priori
4
semantic knowledge on the images.
Gudivada has listed possible types of similarity for retrievalin[Gudivada95]: color similarity , tex-
ture similarity, shape similarity, spatial similarity, etc. Some of these types can be considered in
all or only part of one image, can be considered independently of scaleor angle or not, depending
on whether one is interested in the scene represented by the image or in theimage per se.
2.1 Color similarity
Color distribution similarity has been one of the first choices [HK92,FSN+95] because if one
chooses a proper representation and measure it can be partially reliableeven in presence of
changes in lighting, view angle, and scale. For the capture of propertiesof the global color distri-
bution in images, the need for a perceptively meaningful color modelleads to the choice of HLS
(Hue-Luminosity-Saturation) models, and of measures based on the 3first moments of color dis-
tributions[SO94] preferably to histogram distances. It has been proposedin [AJL95] to use hue
and saturation distributions only when one wants to capturelighting-independent color distribu-
tion properties which are good signatures of a scene when the scale doesnot change too much. In
this case one can identify the hue-saturation perceptive space with thecomplex unit discus and
define measures using statistical moments in this space. This isuseful to avoid biases of measures
which do not take in account the circular nature of hue, and could befurther refined to distinguish
between true spectral hues and the purples. Stricker and Orengo have argued in [SO94] for the
importance of including the 3rd moment (distribution skewness) inthe definition of the similarity
measure.
One important difficulty with color similarity is that when using it for retrieval, anuser will often
be looking for an image “with a red object such as this one”. Thisproblem of restricting color
similarity to a spatial component, and more generally of combiningspatial similarity and color
similarity is also present for texture similarity. It explains why prototype and commercial systems
have included complex ad-hoc mechanisms in their user interfaces tocombine various similarity
functions.
2.2 Texture similarity
For texture as for color, it is essential to define a well-funded perceptive space. Picard and
Liu[PL94] have shown that it is possible to do so using the Wold decomposition of the texture
considered as a luminance field. One gets three components(periodic, evanescent and random)
corresponding to the bi-dimensional periodicity, mono-dimensional orientation, and complexity
of the analyzed texture. Experiments have shown that these independentcomponents agree well
with the perceptive evaluation of texture similarity[TMY79]. Therelated similarity measures
has lead to remarkably efficient results including for the retrieval of large-scale textures such as
images of buildings and cars [PM95] In QIBC system, Tomura texture features, contrast, com-
pactness and direction, ared used [FSN+95]. But of course one is againconfronted to the problem
of combining texture information with the spatial organization of several textures (see below).
2.3 Shape similarity
A proper definition of shape similarity calls for the distinctionsbetween shape similarity in
5
images (similarity between actual geometrical shapes appearing in theimages) and shape similar-
ity between the objects depicted by the images, i.e. similarity modulo anumber of geometrical
transformations corresponding to changes in view angle, opticalparameters and scale. In some
cases, one wants to include even deformation of non-rigid bodies. Thefirst type of similarity has
attracted research work only for calibrated image databases of specialtypes of objects, such as
ceramic plates. Even, in this case, the researchers have tried todefine shape representations which
are scale independent, resting on curvature, angle statistics andcontour complexity . Systems such
as QBIC[FSN+95] use circularity, eccentricity, major axis orientation (not angle-independent)
and algebraic moment. It should be noted that in some cases the user of aretrieval system will
want a definition of shape similarity which is dependent on view angle(for instance will want to
retrieve trapezoids with an horizontal basis and not the othertrapezoids).
In the general case, a promising approach has been proposed by Sclaroff and Pent-
land[PS91,SP95] in which shapes are represented as canonicaldeformations of prototype objects.
In this approach, a “physical” model of the 2D-shape is builtusing a new form of Galerkin’ s inter-
polation method (finite-element discretization). The possibledeformation modes are analyzed
using Karhunen-Loeve transform. This yields an ordered list ofdeformation modes corresponding
to rigid body modes (translation, rotation), low-frequency non-rigidmodes associated to global
deformations and higher-frequency modes associated to localized deformations.
As for color and texture, the present schemes for shape similaritymodelling are faced with seri-
ous difficulties when images include several objects or background. Apreliminary segmentation
as well as modelling of spatial relationships between shapes is thennecessary (are we interested
in finding images where one region represent a shape similar to a givenprototype or to some spa-
tial organization of several shapes?).
2.4 Spatial similarity
Gudivada and Raghavan[GR95] have treated spatial similarity in thesituation in which it is
assumed that images have been (automatically or manually) segmentedinto meaningful objects,
each object being associated with is centroid and a symbolic name. Sucha representation is called
a symbolic image, and it is relatively easy to define similarity functions for suchimage modulo
transformations such as rotation, scaling and translation. Efforts have also been made to address
spatial similarity directly (without segmentation and objectindexing). This was the case, for
instance, in the original work of Kato[Kat92], in the limited case ofdirect spatial similarity (with-
out geometrical transformation), using a number of ad-hoc statisticalfeatures computed on very
low resolution images.
2.5 Object presence analysis
Finding in a set of images in which a particular object or type of objectappears - all images with
cars, all shots in a video in which a given character is present - is aparticular case of similarity
computation. Once again, the range of applicable methods is defined bythe invariants of the
object to be recognized. For color images, and for images whose colordoes not change, local
color distribution is efficient, and can be reliable even when changes in scale or angle occur
[NT92]. In the general case, the best results so far have been obtainedwith texture-based mod-
els[PPDH94]. A pyramidal analysis of texture (with the whole imageconsidered as the texture