Incremental Natural Language Description of
Dynamic Imagery
Gerd Herzog,* Chen-Ko Sung,+ Elisabeth Andr´e, **
Wilfried Enkelmann,+ Hans-Hellmut Nagel,+ ++ Thomas Rist, **
Wolfgang Wahlster,* ** Georg Zimmermann+
SFB 314, Project VITRA, Universit¨at des Saarlandes
D-66041 Saarbr¨ucken, Germany
* German Research Center for Artificial Intelligence (DFKI)
D-66123 Saarbr¨ucken, Germany
+ Fraunhofer–Institutf¨ur Informations- und Datenverarbeitung (IITB)
D-76131 Karlsruhe, Germany
++ Fakult¨at f¨ur Informatik, Universit¨at Karlsruhe (TH)
D-76128 Karlsruhe, Germany
Abstract
Although image understanding and natural language processing constitute
two major areas of AI, they have mostlybeenstudiedindependentlyof each other.
Only a few attempts have been concerned with the integration of computer vision
and the generation of natural language expressions for the description of image
sequences.
The aim of our joint efforts at combining a vision system and a natural lan-
guage access system is the automatic simultaneous description of dynamic im-
agery, i.e., we are interested in image interpretation and language processing on
an incremental basis. In this contribution
1
we sketch an approach towards the in-
tegration of the Karlsruhe vision system called Actions and the natural language
component Vitra developed in Saarbr¨ucken. The steps toward realization, based
1
The work described here was partly supported by the Sonderforschungsbereich 314 der Deutschen
Forschungsgemeinschaft, “K¨unstliche Intelligenz und wissensbasierte Systeme”, projects V1 (IITB,
Karlsruhe) and N2: VITRA (Universit¨at des Saarlandes).
1
In: Ch. Freksa und W. Brauer (Hrsg.), Wissensbasierte Systeme , pp. 153-162, Berlin, Heidelberg: Springer.
on available components, are outlined and the capabilities of the current system
are demonstrated.
Zusammenfassung
Obwohl das Bildverstehen und die Verarbeitung nat¨urlicher Sprache zwei der
Kerngebiete im Bereich der KI darstellen, wurden sie bisher nahezu unabh¨angig
voneinander untersucht. Nur sehr wenige Ans¨atze haben sich mit der Intergration
von maschinellem Sehen und der Generierung nat¨urlichsprachlicher
¨
Außerungen
zur Beschreibung von Bildfolgen besch¨aftigt.
Das Ziel unserer Zusammenarbeit bei der Kopplung eines bildverstehenden
Systems und eines nat¨urlichsprachlichen Zugangssystems ist die automatische
simultane Beschreibung zeitver¨anderlicher Szenen, d.h. wir sind interessiert an
Bildfolgeninterpretation und Sprachverarbeitung auf inkrementeller Basis. In
diesem Beitrag beschreiben wir einen Ansatz zur Integration des Karlsruher Bild-
folgenanalysesystems Actions und der nat¨urlichsprachlichen Komponente Vitra,
die in Saarbr¨ucken entwickelt wird. Die Schritte hin zur Realisierung, basierend
auf bereits verf¨ugbaren Komponenten, werden dargestellt und die F¨ahigkeiten
des derzeit vorhandenen Systems demonstriert.
This paper appeared in: In: C. Freksa and W. Brauer (eds.), Wissensbasierte
Systeme. 3. Int. GI-Kongreß, pp. 153–162. Berlin, Heidelberg: Springer, 1989.
2
1 Introduction
Image understanding and natural language processing are two major areas of research
within AI that have generally been studied independently of one another. Advances
in both technical fields during the last 10 years form a promising basis for the de-
sign and construction of integrated knowledge-based systems capable of translating
visual information into natural language descriptions. From the point of view of cog-
nitive science, anchoring meaning in a referential semantics is of theoretical as well as
practical interest. From the engineering perspective, the systems envisaged here could
serve such practical purposes as handling the vast amount of visual data accumulating,
for example, in medical technology, remote sensing, and traffic control.
The goal of our joint efforts at combining a vision system and a natural language
access system is the automatic simultaneous description of dynamic imagery, i.e., we
are interested in image interpretation and language processing on an incremental basis.
The conversational setting is this: the system provides a running report of the scene it
is watching for a listener who cannot see the scene her/himself, but who is assumed to
have prior knowledge about its static properties. In this paper we describe the integra-
tion of the Karlsruhe vision system Actions and the natural language component Vitra
developed in Saarbr¨ucken.
2
The steps toward realization, based on available compo-
nents, are outlined, and results already obtained in the investigation of traffic scenes
and short sequences from soccer matches will be discussed.
2 Relations to Previous Research
Following Kanade (see Kanade [1980]), it is advantageous for a discussion of machine
vision to distinguish between the 2-D picture domain and the 3-D scene domain. So
far, most machine vision approaches have been concerned (i) with the detection and
localization of significant grey value variations (corners, edges, regions) in the picture
domain, and in the scene domain (ii) with the estimation of 3-D shape descriptions,
as well as—more recently—(iii) with the evaluation of image sequences for object
tracking and automatic navigation. Among the latter approaches, the estimation of
relative motion between camera(s) and scene components as well as the estimation
of spatial structures, i.e., surfaces and objects, are focal points of activity (see Ay-
ache and Faugeras [1987], Faugeras [1988], Nagel [1988b]). Few research results
have been published about attempts to associate picture domain cues extracted from
image sequences with conceptual descriptions that could be linked directly to efforts
at algorithmic processing of natural language expressions and sentences. In this con-
text, computer-based generic descriptions for complex movements become important.
Those accessible in the image understanding literature have been surveyed in Nagel
[1988a]. Two even more recent investigations in this direction have been published
2
The acronyms stand for `Automatic Cueing and Trajectory estimation in Imagery of Objects in
Natural Scenes' and `VIsual TRAnslator'.
3
in Witkin et al. [1988] (in particular Section D) and Goddard [1988]. A few selected
approaches from the literature are outlined in the remainder of this section to provide
a background for the ideas presented here.
In Badler [1975], Badler studied the interpretation of simulated image sequences
with object motions in terms of natural language oriented concepts. His approach has
been improved by Tsotsos, who proposed a largely domain-independent hierarchy of
conceptual motion frames which is specialized further within the system Alven to ana-
lyze X-ray image sequences showing left ventricular wall motion (see Tsotsos [1985]).
Later, a similar system for the analysis of scintigraphic image sequences of the human
heart was developed by Niemann et al. (see Niemann et al. [1985]). Based on a study
of Japanese verbs, Okada developed a set of 20 semantic features to be used within the
system Supp to match those verb patterns, that are applicable to simple line drawings
(see Okada [1979]). Traffic scenes constitute one of the diverse domains of the dialog
system Ham-Ans (see Wahlster et al. [1983]). Based on a procedural referential se-
mantics for certain verbs of locomotion, the system answers questions concerning the
motions of vehicles and pedestrians. The system Naos (see Neumann [1984], Novak
[1986]) also allows for a retrospective natural language description. In Naos, event
recognition is based on a hierarchy of event models, i.e., declarative descriptions of
classes of events organized around verbs of locomotion. The more recent Epex system
(see Walter et al. [1988]) studies the handling of conceptual units of higher semantic
complexity, but still in an a posteriori way.
The natural language interfaces mentioned so far have not been connected to real
vision components, they use only simulated data. Apart from our previous results
(see Andr´e et al. [1986], Schirra et al. [1987]) the LandScan system (see Bajcsy et al.
[1985]) constitutes the only approach in which processing spans the entire distance be-
tween raw images and natural language utterances but it deals only with static scenes.
3 Simultaneous Evaluation and Natural Language De-
scription of Image Sequences
The main goal of our cooperation is the design and implementation of an integrated
system that performs a kind of simultaneous reporting, that is, evaluating an image
sequence and immediately generating a natural language description of the salient ac-
tivities corresponding to the most recent image subsequence. It is not (yet) real-time
evaluation, but our approach emphasizes concurrency of image sequence evaluation
and natural language generation.
In order to gain a realistic insight into the problems associated with such an en-
deavor, we decided toevaluate real-worldimagesequences with multiplemobile agents
or objects, based on system components which are already partially available due to
previous research efforts in the laboratories involved. Since the analysis of complex
articulated movements still exceeds our capabilities given the computational resources
4
available today, we concentrate initially on the picture domain in order to detect and
track projected object candidates, which are considered to be essentially rigid. The
crucial links between the picture domain results and the natural language process-
ing steps are provided by complex events, i.e., higher conceptual units capturing the
spatio-temporalaspects of object motions. A complex event should be understood as an
`event' in its broadest sense, comprising also notions like `episode' and `history' (see
Nagel [1988a]). The recognition of intentions and plans (see Retz-Schmidt [1988]) is,
however, outside the scope of this paper. In what follows, the term `event' will be used
to refer to complex events.
3.1 Overall Structure of the Approach
The task of generating natural language descriptions based on visual data can roughly
be subdivided into three parts: (1) constructing an abstract propositional description of
the scene, the so-called Geometrical Scene Description (GSD, see Neumann [1984]),
(2) further interpretation of this intermediate geometrical representation by recogniz-
ing complex events, and (3) selection and verbalization of appropriate propositions
derived in step 2 to describe the scene under discussion. Because of the simultaneity
of the description in our case, the three steps have to be carried out incrementally.
Figure 1: The architecture of the integrated system
5
