Interval-Based Conceptual Models
for Time-Dependent Multimedia Data
1
T.D.C. Little
†
and A. Ghafoor
‡
†
Multimedia Communications Laboratory
Department of Electrical, Computer and Systems Engineering
Boston University, Boston, Massachusetts 02215, USA
tdcl@bu.edu
‡
School of Electrical Engineering
Purdue University, West Lafayette, Indiana 47907, USA
MCL Technical Repor t 05-07-1993
Abstract–Multimedia data often have time dependencies that must be satisfied at presen-
tation time. To support a general-purpo se multimedia information system, these timing
relationships must be managed to provide utility to both the data presentation system and
the multimedia author.
In this paper we propose new conceptual models for capturing these timing relationships
and managing them as part of a database. Specifically, we introduce and define n-ary
and reverse temporal relations along with their temporal constraints. These new relations
are a generalization of our earlier temporal models and establish the basis f or conceptual
database structures and temporal access control algorithms to facilitate forward, reverse, and
partial-interval evaluation during multimedia object playout. The proposed relations are
defined to ensure a property of monotonically increasing playout deadlines to facilitate both
real-time deadline-driven playout scheduling or optimistic interval-based process playout.
Furthermore, we show a translation of t he conceptual models to a structure suitable for a
relational database.
Keywords: Temporal Modeling, Multimedia Databases, Synchronization, Scheduling.
1
In IEEE Trans. on Knowledge and Data Engineering (Special Issue: Multimedia Information Systems),
Vol. 5, No. 4, August 1993, pp. 551- 563. Portions of this work were presented at the 199 2 Workshop on
Multimedia Informations Systems held in Tempe, Arizona on February 7, 1992. The work by T.D.C. Little
is supported in part by the National Science Foundation under Grant No. IRI-9211165. The wo rk by A.
Ghafoor is supported in part by the National Science Foundation under Grant No. CDA-912177 1. The
authors also acknowledge the support of the New York State Center for Advanced Technology in Computer
Applications and Software Engineering (CASE) at Syracuse University.
1 Introduction
Multimedia refers to the integration of text, images, audio, and video in a variety of appli-
cation environments. These data can be heavily time-dependent, such as audio and video
in a motion picture, and require time-ordered playout during presentation. The task of co-
ordinating sequences of these data r equires synchronization among the interacting media as
well as within each medium. Synchronization can be applied to the playout of concurrent
or sequential streams of data and to the external events generated by a human user includ-
ing browsing, querying, and editing typical of stored-data applications. This problem of
synchronizing time-ordered sequences of data element s is fundamental to multimedia data.
Timing relationships between the media can be implied, as in the simultaneous acqui-
sition of voice and video, or can be explicitly formulated, as in the case of a multimedia
document with voice-annotated text. In either situation, the characteristics of each medium,
and the relationships among them must be established in order to provide synchronization
in the presence of vastly different presentation requirements. Consider the familiar canned
multimedia slide presentation in which a series of verbal annotations coincides with a series
of images. The presentation of the annotations and the slides occur in a sequential man-
ner. Synchronization points correspond to the change o f an image and the end of a verbal
annotation, representing a coarse-gra in synchronization between objects. A fine-g r ained ex-
ample of synchronization is the lip-sync of audio and video which usually requires 25 or 30
synchronization points per second.
A multimedia system must preserve the timing relationships among the elements of the
object presentation at these points by the process of multimedia synchro nization. A mul-
timedia database management system (MDBMS) must have the capability for managing
the aspects of time required for time-dep endent media. This problem is different from t he
provision o f historical databases, temporal query languages [25, 27], or time-critical query
evaluation [14]. Time-dependent multimedia objects require special considerations for pre-
sentation due to their real-time playout characteristics as data need to be delivered from
the storage devices based on a prespecified schedule. Furthermore, presentation of a single
object can occur over an extended duration (e.g., a motion picture). Fig. 1 illustrates such
a time dependency between elements of a compo site multimedia object. In this example, a
sequence of text and image elements are played-out in succession based on such a prespecified
schedule.
The tolerance of data to timing skew and jitter during playout varies widely depending on
2
time
1
t
2
t
3
t
4
t
5
t
6
t
7
t
Figure 1: Time-Dependent Data Presentation
the medium. Audio and video require tight b ounds on the order of hundreds of milliseconds,
whereas synchronous text and images can tolerate skew on the order of seconds. Furthermore,
audio and video can tolerate different absolute timing requirements during playout as the
human ear can discern dropouts in audio data more readily than of video. Based on the
data’s tolerance to skew a nd jitter during playout, two approaches to providing synchronous
playout of time-dependent data streams have been propo sed. These consist of a real-time
scheduling approach [19], and an optimistic interval-based process approach as proposed in
this paper.
In addition to simple linear playout of time-dependent data sequences, other modes of
data access are also possible due to the unique nature of the multimedia data objects and
should be supported by a MDBMS. These include,
• Reverse
• Fast-forward
• Fast-backward
• Midpo int suspension
• Midpo int resumption
These Temporal Access Control (TAC) operations are feasible with existing technologies;
however, when non-sequential stora ge devices are used with complex data compression al-
gorithms, and random communication delays are introduced due to data distribution, the
provision of these capabilities can be very difficult. Examples include viewing a motion
picture backwards or reversing an animation of a series o f images (sequence reversal), rapid
viewing of a long sequence o f time-dependent data by increasing the rate of presentation
or by skipping some data (fast-forward or fast-backward), and stopping and starting of a
3
motion picture at an arbitrary point (midpoint suspension, resumption and partial interval
evaluation).
In this paper, we propose temporal-interval-based models and constraints which provide
a basis for a proposed conceptual data representation and algorithmic support of the afo re-
mentioned TAC functionality. The work represents major extension and generalization o f
our earlier models presented in [17]; however, we do not consider the dynamic properties
of user interaction (e.g., Stotts and Furuta [24]). The uncertainty created by random user
interaction is an a dditional complexity in managing time in multimedia information systems.
The remainder of the paper is organized as f ollows. In Section 2, we review related work
on time-dependent data storag e. In Section 3, we describe interval-based modeling schemes,
including our proposed models for establishing a conceptual database structure. Section 4
describes a conceptual data representation based on the new temporal models, including an
example using a relational implementation. Section 5 describes algo rithms for accessing the
proposed conceptual models in the context of a database. We discuss characteristics of the
overall modeling methodology in Section 6, and in Section 7 we conclude the paper.
2 Background and Related Work
The primary requirements for the support of time-dependent data playout in an MDBMS
include the means for the identification of temporal relations between multimedia da ta ob-
jects, temporal conceptual database schema development, physical schema design, and syn-
chronous a ccess for data retrieval. In this section we briefly describe these requirements,
introduce various terminology, and describe related work.
As indicated in the introduction, time-dependent data differ from historical data which
do not specifically require timely playout. Typically, time-dependent data are stored using
mature technologies possessing mechanisms to ensure synchronous playout (e.g., VCRs or
audio tape recorders). With such mechanisms, dedicated hardware provides a constant rate
of playout for homogeneous, periodic sequences of data, and concurrency in data streams
is provided by independent physical data paths. When this type of data is migrated to
more general-purpose computer data storage systems (e.g., disks), many interesting new
capabilities are possible, including random access to the tempora l dat a sequence and time-
dependent playout of static data (animation). However, the generality of such a system
eliminates the dedicated physical data paths and the implied data structures of sequentia l
4
storage. Therefore, a general MDBMS needs to support new access para digms including a
retrieval mechanism for large amounts of multimedia data, and must provide conceptual and
physical database schemata to support these paradigms. Furthermore, a MDBMS must also
accommodate the performance limitations of the computer.
At the conceptual level, the temporal aspects of data must be modeled. Da t a can have
natural or implied time dependencies, (e.g., audio and video r ecorded simultaneously). These
data streams often are described as continuous because recorded data elements form a con-
tinuum during playout, i.e., elements are played-out contiguously in time. Static data,
which lack time dependencies, can have synthetic temporal relationships (e.g., Fig. 1). The
combination of natural and synthetic time dependencies can describe the overall tempor al
requirements of any pre-orchestrated multimedia presentation. Temporal information can be
encapsulated in the description of the data using the object-oriented paradigm (e.g., Gibbs
[10], Herrtwich [12]). Using such schemes, tempor al information such as a time reference,
playout time units, tempo ral relationships, and required time offsets can be maintained for
specific multimedia objects. If the data are periodic, this approach can define the time
dependencies for an entire sequence by defining the period or frequency of playout (e.g.,
30 frames/s for video). For mixed-type, time-dependent data, there have been several pro-
posals for their conceptual modeling and interchange format specification, most based on
temporal-interva l-based schemes [5, 8, 13, 23]. However, these works either neglect to con-
sider the implications on t he development of conceptual database structures to support TAC
operations or do not comprehensively model time-dependent data.
Once a conceptual temporal model is established for a multimedia object, the multimedia
data must be mapped to the physical system to facilitate database access and retrieval. For
time-dependent multimedia data this presents some interesting challenges. Problems arise
due to the strict timing requirements for playout of time-dependent data. The multimedia
types of audio and video require very large amounts of storage space and must be maintained
in secondary storage. In order to meet the presentation requirements for these data, vari-
ous physical storage organizations have been proposed, such as storing data in contiguous
blocks on a disk in the same order as playout. Some recent work on data placement on
physical storage for audio data retrieval is described by Yu et al. [28, 30], Gemmell and
Christo doulakis [9], Rangan and Vin [22], and Christodoulakis and Faloutsos [7]. We do not
address physical storage organizations here.
The integration of conceptual and physical data models with system supp ort for data
delivery yields the functionality necessary to construct multimedia applications. System
5
