Adapting multimedia Internet content for universal access

TL;DR: This work presents a system that adapts multimedia Web documents to optimally match the capabilities of the client device requesting it using a representation scheme called the InfoPyramid that provides a multimodal, multiresolution representation hierarchy for multimedia.

Abstract: Content delivery over the Internet needs to address both the multimedia nature of the content and the capabilities of the diverse client platforms the content is being delivered to. We present a system that adapts multimedia Web documents to optimally match the capabilities of the client device requesting it. This system has two key components. 1) A representation scheme called the InfoPyramid that provides a multimodal, multiresolution representation hierarchy for multimedia. 2) A customizer that selects the best content representation to meet the client capabilities while delivering the most value. We model the selection process as a resource allocation problem in a generalized rate distortion framework. In this framework, we address the issue of both multiple media types in a Web document and multiple resource types at the client. We extend this framework to allow prioritization on the content items in a Web document. We illustrate our content adaptation technique with a web server that adapts multimedia news stories to clients as diverse as workstations, PDA's and cellular phones.

Summary

1. INTRODUCTION

• Network appliances, or information appliances, are computing devices that are network enabled.
• The network appliances vary widely in their features such as screen size, resolution, color depth, computing power, storage and software.
• In this paper the authors present a system that adapts multimedia Web content to optimally match the resources and capabilities of diverse client devices.
• A customizer that selects the best versions of content items from the InfoPyramids to meet the client resources while delivering the most “value.”.

1.1 RELATED WORK

• Much work (for a small sampling, see [3,4,5,6]) has been done on adapting video to bandwidth variations, and sometimes screen size, by selecting a suitable compression scheme.
• Certain resources, such as effective network bandwidth, costs and patience of the users can be different for similar client devices.
• For example images are transcoded only to images and video to video.
• The content is authored in XML [23], allowing the author to provide more information to the transcoding and customization system than can be deduced from an HTML page.
• The key benefit of this serverbased system is that due to the guidance provided by the author, significantly greater level of customization can be performed than is possible in transcoding proxies.

1.2 OUTLINE

• In Section 2.1 the authors describe the multimedia content the system handles.
• In Section 2.2 the authors define the client capabilities considered.
• The InfoPyramid, described in Section 2.4, provides a multi-modal, multi-resolution representation hierarchy for multimedia.
• Section 2.6 describes how the customization module uses the client device characteristics as constraints to dynamically pick the best content representation.
• The authors demonstrate how this server adapts multimedia news stories to the varying resources on clients ranging from workstations on LANs through PDAs to cellular phones and pagers.

2. SYSTEM ARCHITECTURE

• The content source contains the multimedia content to be delivered by the Web server.
• A novel data representation, the InfoPyramid, is used to store the multiple resolutions and modalities of the transcoded content, along with any associated 9 meta-data.
• This transcoding is done off-line, during content creation time.
• When the Web server receives a request, it first determines the capabilities of the requesting client device.
• This selected content is then rendered in a suitable delivery format (for example, HTML) for delivery to the client.

2.1 CLIENT DEVICES

• The types of devices that can access the Internet are rapidly expanding beyond the workstation on LAN that most multimedia Internet content is authored for [1,2,7].
• Even traditional computers such as workstations, laptops and PC may vary widely in their display and specially in their network bandwidth.
• The browsers designed to meet the special needs of handicapped people can be modeled as client devices with specific capabilities [20].
• Thus, the authors see that to fulfill the promise of universal access to the Internet, devices with very diverse capabilities need to be catered to.
• Currently, the system considers the following client device characteristics:.

1. Screen.

• Color/monochrome and bits/pixel, also known as Color depth.
• Currently, the system is told the effective network bandwidth to the client.
• For storage constrained devices, the payload will be defined as the storage space.
• The system can determine the client device capabilities and resources by a number of mechanisms.
• Many sites require users to login, or place cookies at the user allowing client capabilities to be retrieved from stored profiles.

2.2 CONTENT

• While the system described in this paper can be applied to various types of multimedia documents, the authors will restrict their discussion to Web pages.
• The authors also provide an ingestion module that takes in HTML documents and generates an XML document.
• For content authored in HTML, the authors are also working on an extension to HTML that allows the author to introduce more information for content customization using XML.
• This extended HTML will also allow this content adaptation system to be deployed at proxies.
• For simplicity, the authors will first consider only atomic content items, and then, in Section 5.2, deal with composite items.

2.3 CONTENT ANALYSIS

• The authored content is analyzed to extract information that will be useful in transcoding and customization.
• The authors determine the following resource requirements: 1. Content size in bits.
• Text items may not have fixed width and height, but display area may be computed from assumptions about font sizes and layouts.
• The streaming bit-rate for video and audio is the minimum bit-rate required to properly support their transport.
• The authors currently analyze images to determine their type and purpose [22,23].

2.4 INFOPYRAMID

• Multimedia content description is key to various tasks, such as searching, filtering and delivery.
• 13 MPEG 7 [24] is working on standards for multimedia content description.
• Features and semantics at different resolutions can be obtained from raw data or transformed data at different resolutions, thus resulting in a feature or semantic pyramid.
• Methods generate content descriptors from the features of the data, or analyze, manipulate, provide modality translation, or process the data in various ways, also known as 14 Methods and Rules.
• Rather than forcing a strong separation between the data and the content description meta-data, the InfoPyramid offers a continuum between the data, various abstractions of the data, and content description data.

2.5 TRANSCODING

• Content transcoders populate the InfoPyramid structure with multi-resolution, multi-modal versions of the content.
• The authors have implemented a number of transcoding modules for handling video and images and imported others for text, images, video and audio.
• The systems performs the following transcodings of content along the dimensions of resolution and modality : Images: Resolution - Spatial size reduction, color depth reduction, lossy compression.
• Modality - Related text, embedded text, semantic labels.
• If the authors have client devices that have black and white screens, the images will be converted to black and white.

2.6 CUSTOMIZATION

• The customization module uses the client device characteristics as constraints to pick the best content representation.
• The best representation is the one that maximizes content value for that client device.
• The customization utilizes a value-resource framework, which is generalization of rate-distortion (Section 3).
• In Section 4, the authors model the selection problem as one of optimal allocation of the resources on the client among the different versions of the multimedia items of the Web document.
• The authors show that different models for the relationship between the value and the resource requirements lead to different optimal resource allocation strategies.

2.7 RENDERING

• When the server receives a request from a client, it (1) looks up the InfoPyramid that corresponds to the request, (2) determines client capabilities, and (3) forwards the InfoPyramid and the client profile to a customization module.
• The customization module selects the InfoPyramid content items that best suit the client.
• This XML document is then rendered in HTML or, potentially, into other formats such as Lotus Notes(TM), or PalmPilot(TM) prc files.
• The HTML page delivered to the client has appropriate textual elements incorporated and links to the right media versions.
• Thus, subsequent requests to the media elements included in the page do not entail any processing overhead at the server.

2.8 CACHE

• Every time the system sees a client with a new set of capabilities, it generates a new client id and stores the client capabilities under that client id.
• It then asks the cache to check if a customized copy for the requested document exists for that client id.
• Since, for a busy site, the number of requests for a document is typically much larger than the number of different client devices, the cache can result in significant improvement in response times.
• To effectively handle this, the cost of performing customization versus the variation in the resources will need to be considered.
• Alternatively, one can group clients with very similar capabilities under the same client id.

3. CONTENT VALUE

• Image or video compression can be viewed as adapting the content to meet bit resource constraints.
• One framework for compressing to meet bit resource constraints [26,28] has built on the rate-distortion (R-D) theory due to Shannon [27].
• One problem with the MSE based distortion measure is that it may not correspond to the perceived loss of fidelity [31].
• This also allows us to compare document items that were in different media types.
• The content value is a useful construct that helps us analyze various dynamic content adaptation policies in a theoretical rate-distortion based framework and draw parallels with compression.

4. RESOURCE ALLOCATION

• While iV and iR are discrete, the authors will first consider them to be continuos, and then deal with the discrete case.
• ClientR is the maximum resource available at the client.

4.1 ANALYTIC FUNCTIONS

• Content value, as an alternative to distortion, makes it possible for authors or users to specify value judgements about various transcoded versions of the content.
• This is not to suggest that there actually exist such a simple mechanism for assigning value (or distortion).
• Starting with the item with the largest RUF, allocate the maximum resources that each item can use till all the resources are depleted.
• In a similar vein, Equation 6 will give us the optimal solution for all other concave functions.
• To account for the discrete values, the authors use the following algorithm: 1. For each item i, let iR′ be the resource selected by the optimization process.

4.2 ARBITRARY FUNCTIONS

• For this case, the authors adapt a technique by Shoham and Gersho [33].
• The optimal version iM ′ 23 is given by sweeping a line with slope λ , from the top-left to the bottom-right, till it meets the concave hull of these points.
• Points outside the concave hull are not in the solution space.
• A text transcript of video may take more screen space but have less value, so it is out of the solution space.

5.1 PRIORITIES

• In the resource allocation strategies discussed in Section 4, no matter how the value to resource relationship is defined, the items with the least resource requirements for their original versions (i.e. with the highest RUF), get precedence in the allocation of resources.
• Web page that has one color photograph of the event covered in the news story.
• One can argue that in the original Web document, the larger images were more 25 important as more resources were given to them.
• The authors can model these intrinsic priorities as proportional to ln(image size).

5.2 COMPOSITE ITEMS

• The optimal resource iR′ thus allocated to each composite item i is in turn used as the resource constraint for its constituent items.
• The authors then allocate this resource iR′ among the children of the composite item i.
• This resource allocation is repeated till the items being considered are atomic.
• When the authors have priorities assigned to items, they similarly modify the resource allocation strategy described in Section 5.1.
• For a composite item, the number of its different versions is combinatorial in the number of its children item.

5.3 MULTIPLE RESOURCES

• A client may have a different number of capabilities and resources.
• A handheld PC (HPC) may not be capable of displaying video, and a PDA may not be capable of displaying color images.
• Let there be r 26 different resources kclientR that the authors have to consider.
• The version so selected may not be optimal.
• To find the optimal set, a search (possibly combinatorial) may be required.

5.4 MUTUAL DEPENDENCE

• For finding the optimal content adaptation schemes the authors assumed that the content items on a Web page are independent of each other.
• For a news story, if the text to the story has to be discarded due to space limitations, then delivering the pictures for the story may not be very useful.
• The authors partial solution is to use composite items (Section 5.2).
• The authors consider dependent items as composite items and allocate resources first to the 27 composite item.
• A better solution would be to extend rate-distortion techniques for handling dependent blocks, such as [34,35,26], to the valueresource framework.

5.5 SPACE AND TIME

• The authors have not addressed two difficult and important issues (1) temporal relationships or synchronization and (2) spatial relationships or layout.
• The authors have limited ourselves to ourselves to Web (HTML) documents using http over TCP/IP.
• These do not provide for specifying any synchronization among the different component items of a Web page.
• For video the authors assume that the video and audio are multiplexed on the same stream or that player/plugin handles synchronization issues.
• Currently the transcoded or adapted content may not preserve the spatial layout of the original document.

6. A MULTIMEDIA NEWS SYSTEM

• The authors have implemented a Web server that customized Web pages to the capabilities of the client requesting them, employing the content adaptation process described above as an extension to this server [19].
• Based on a template InfoPyramid for news stories, these raw content items are ingested into InfoPyramids.
• The video modality is also converted to images by extracting key-frames.
• Some of the content adaptation is based on client device capabilities, and some on resource allocation.
• If the authors increase the wait-time to 300secs, increasing the total number of bits available at the client, they see that in Figure 6.c there are more images that get downloaded.

7. SUMMARY

• In this paper the authors have presented a system for adapting multimedia Internet content.
• Web content to client devices with diverse capabilities.
• The authors used InfoPyramids to represent content transcoded into multiple resolution and modalities.
• In the value-resource framework, content adaptation is analogous to compressing multimedia documents to meet resource constraints imposed by the client device.
• The server based content adaptation approach allows precise control over the process by the publisher.

8. REFERENCES

• H. Bharadvaj, A. Joshi, and S. Auephanwiriyakul, An active transcoding proxy to support mobile Web access", Proc. 17th IEEE Symposium on Reliable Distributed Systems, October 1998. [16].
• A. Ortega and K. Ramchandran, Rate-Distortion Methods for Image and Video Compression, IEEE Signal Processing Magazine, Nov. 1998. [27].

Figures

Figure 6: Dynamic adaptation of same original content to different resources on client. Client resources are specified as screen width x screen height, bandwidth, wait-time.

Figure 5. The workstation gets the full text of the story and streaming video. The PDA gets a text summary and two key-frames in 2-bit gray-scale. The cellular phone gets a short title and audio.

Figure 4 Optimal value is obtained at the point where a line with slope λ meets the concave hull. The version selected has value closest to this optimal.

Figure 1. Internet content adaptation system architecture

Figure 2. An InfoPyramid for a video item.

Figure 3 Table showing value V for different functional relationships between value and resource used in terms of bytes.

Full text

Adapting Multimedia Internet Content for Universal Access
Rakesh Mohan, John R. Smith and Chung-Sheng Li
IBM T.J. Watson Research Center
PO Box 704, Yorktown Heights, NY 10598
{rakesh, jrsmith, csli}@watson.ibm.com
ABSTRACT
Content delivery over the Internet needs to address both the multimedia nature of the content and
the capabilities of the diverse client platforms the content is being delivered to. We present a
system that adapts multimedia Web documents to optimally match the capabilities of the client
device requesting it. This system has two key components:
(1) A representation scheme called the InfoPyramid that provides a multi-modal, multi-resolution
representation hierarchy for multimedia.
(2) A customizer that selects the best content representation to meet the client capabilities while
delivering the most value.
We model the selection process as a resource allocation problem in a generalized rate-distortion
framework. In this framework, we address the issue of both multiple media types in a Web
document and multiple resource types at the client. We extend this framework to allow
prioritization on the content items in a Web document. We illustrate our content adaptation
technique with a web server that adapts multimedia news stories to clients as diverse as
workstations, PDAs and cellular phones.
Keywords: Multimedia, Internet, content adaptation, transcoding, compression, rate-distortion,
universal access, information appliances.

2
1. INTRODUCTION
Network appliances, or information appliances, are computing devices that are network enabled.
They typically have fewer resources than personal computers and are geared towards a limited
number of applications. Some current examples of network appliances are hand-held computers
(HPCs), personal digital assistants (PDAs), set-top boxes, screen telephones, smart cellular
phones and network computers. In “ubiquitous” or “pervasive” computing, consumers will use
different network appliances to connect to the internet for different applications, from
entertainment to banking, from different settings, from living rooms to cars. Sources, such as The
Economist [1] and IDC [2], predict that the sales of network appliances will significantly outstrip
that of personal computers after the year 2002. Therefore, within a decade, network appliances
will replace personal computers as the client device of choice for viewing Web content.
Currently multimedia content is authored with the personal computer as the target client device.
Web documents, which have rapidly become the largest deployed form of multimedia, are also
authored specifically for personal computers with reasonable wired network connections.
However, network appliances are very different from the typical PC on a modem or LAN. The
network appliances vary widely in their features such as screen size, resolution, color depth,
computing power, storage and software. They also use a variety of network connections ranging
from cable to mobile, with different bandwidth, connection characteristics and costs. For
example screen sizes for PCs are in the 800x600 to 1024x780 pixels range, for HPCs in the
480x240 to 640x240 range, for PDAs in the 160x160 to 320x240 range, and for smart cellular
phones and pagers about 20x5 characters. Screen colors range from 24 bit and 8 bit color to 4 bit
and 2 bit gray-level. The network bandwidths range from 10 Mbs for ethernet LANs to 28Kbs
for phone modems, to effective rates of 1 KBs for CDPD wireless modems [7]. The diversity of
these devices will make it difficult and expensive to author multimedia content separately for
each individual type of device. Therefore, technologies that can adapt multimedia content to
diverse client devices will become critical in the coming pervasive computing era.

3
In this paper we present a system that adapts multimedia Web content to optimally match the
resources and capabilities of diverse client devices. This system employs two key technologies:
1. A progressive data representation scheme called the InfoPyramid [25]. Content items on a
Web page are transcoded into multiple resolution and modality versions so that they can be
rendered on different devices. For example, a video item is transcoded in to a set of images so
that it can be rendered on a device not capable of displaying video. The InfoPyramid
provides a multi-modal, multi-resolution representation for the content items and their
transcoded versions.
2. A customizer that selects the best versions of content items from the InfoPyramids to meet
the client resources while delivering the most “value.” The customizer allocates resources on
the client among the items in the document. This resource allocation results in the selection
of the appropriate resolution or modality of the content items. If the client has limited
resources (such as a PDA or pager), some of the content items may not get any resources
assigned and thus not be delivered to the client. We propose a novel value-resource
framework for the customizer. This value-resource framework allows us to design and
analyze a number of content adaptation strategies.
We illustrate this content adaptation with a multimedia news delivery system that adapts to
clients ranging from workstations to cellular phones.
1.1 RELATED WORK
Much work (for a small sampling, see [3,4,5,6]) has been done on adapting video to bandwidth
variations, and sometimes screen size, by selecting a suitable compression scheme. These
systems consider only a single type of media, not composite multimedia documents. Also, while
a range of bandwidth variations is accounted for, drastically different clients, such as those that
can not handle video, are not addressed.

4
Web content adaptation can be performed either at the server, at the client, at an intermediate
proxy, or some combination of the three.
Some client devices adapt content at the device. For example, Windows-CE devices change
color-depth (for example, from 24 bit color to 4bit gray-level) of images. The drawback is that
network appliances have low network bandwidth, which results in slow access to pages with rich
multimedia content. Another problem is that network appliances are often restricted in their
computational power, which makes content adaptation at the device slow, or even impossible.
Most content adaptation systems [7,8,9,10,11,12,13,14,15,16,18] are http proxy based. The proxy
intercepts client device’s requests for Web pages, fetches the requested content, adapts it and
sends the adapted version to the client. This content adaptation is often termed “transcoding”.
In the TranSend project [7] a proxy transcodes Web content on the fly. The adaptation, which
they term “distillation,” is primarily limited to image compression and reduction of image size
and color space. Postscript documents are converted to HTML. In [8], video is also converted
into different frame-rates and encodings using a video gateway [6]. A “refinement” mechanism
allows clients to request the original version of the content. Further reports of this work are
provided in [9,10]. Based on this work, a company, Proxinet [16], has been started that provides
a proxy which customizes content for a special browser on the 3Com PalmPilot (TM) [17].
Bickmore and Schilit [11] also propose a proxy based mechanism. Images are scaled by pre-
defined scaling factors. The emphasis of the work is on textual, specifically HTML, content.
They use a number of heuristics and a planner to perform outlining and elision of the content to
fit the Web page on the client’s screen.
The Spyglass Prism (TM) [13], a commercial product, is another transcoding proxy. It
compresses (JPEG) images and changes the size and color depth of images. It also performs
some HTML filtering and modification, like removal of Java and JavaScript, and conversion of
tables to lists.

5
AvantGo [18] offers a solution similar to Proxinet. They have a special browser for the PalmPilot
and a content adaptation system that is deployed on a PC. The transcoder at the PC downloads
Web content and customizes it for the PalmPilot browser. When the PalmPilot is “synchronized”
with the PC, the transcoded content is transferred to it for offline browsing.
Content adaptation upstream of the client results in a faster response time [7,8]. If the client has a
slow link to the transcoding proxy, and the proxy has a fast link to the Web servers providing the
original content, then the client device sees a faster response time even with the added time taken
for adapting the content at the proxy. Based on this observation, Intel launched the QuickWeb
(TM) [12] service that compresses images at a proxy. There is no customization for different
client devices. The sole purpose of the service is to improve response times for PCs connected
over slow links such as modems. This improvement in response time is even more significant
when the adaptation is performed at the Web servers, where the transcoded content can be pre-
cached.
These transcoding proxies typically consider a few client devices and employ static, ad-hoc,
content adaptation strategies. A common policy [7-13] is to scale all images by a fixed factor.
Thus, these transcoding proxies fail to dynamically address the variation in the resource
requirements of different Web documents. The set of client devices will also grow more diverse.
Certain resources, such as effective network bandwidth, costs and patience of the users can be
different for similar client devices. The static adaptation policies used by these systems do not
handle well this variability in Web content and client resources.
A Web document is a multimedia document composed of various components in different media.
None of the existing transcoding systems (with the possible exception of [11, 14]) consider the
requirements of the entire Web page or relationships between the components. Each component
is transcoded separately. Also, these systems only consider transcoding within the same
modality. For example images are transcoded only to images and video to video. This limits the
client devices that can be supported.

Frequently asked questions

Q1. What are the contributions in "Adapting multimedia internet content for universal access" ?

The authors present a system that adapts multimedia Web documents to optimally match the capabilities of the client device requesting it. This system has two key components: ( 1 ) A representation scheme called the InfoPyramid that provides a multi-modal, multi-resolution representation hierarchy for multimedia. ( 2 ) A customizer that selects the best content representation to meet the client capabilities while delivering the most value. In this framework, the authors address the issue of both multiple media types in a Web document and multiple resource types at the client. The authors extend this framework to allow prioritization on the content items in a Web document. 

Q2. What is the effect of a temporary change in the cache hit ratio?

Temporal variations in resources on the client, such as bandwidth, CPU resources, storage, etc., will reduce the cache hit ratio. 

Q3. What is the key benefit of the serverbased system?

The key benefit of this serverbased system is that due to the guidance provided by the author, significantly greater level of customization can be performed than is possible in transcoding proxies. 

Q4. How do the authors generalize rate distortion theory to a value-resource framework?

The authors generalize rate-distortion theory to a value-resource framework by considering different versions of a content item in an InfoPyramid as analogous to different compressions, and different client resources as analogous to the bit-rate. 

Q5. How do the authors allocate resources for a composite item?

To account for composite items, the authors allocate resources using ( )iii RfV = where f is a concave analytic function and the items i under consideration may be composite. 

Q6. What is the optimal resource iR′ for each item?

The optimal resource iR′ thus allocated to each composite item i is in turn used as the resource constraint for its constituent items. 

Q7. What is the process for determining the client capabilities?

When the server receives a request from a client, it (1) looks up the InfoPyramid that corresponds to the request, (2) determines client capabilities, and (3) forwards the InfoPyramid and the client profile to a customization module. 

Q8. What is the effect of a cache?

for a busy site, the number of requests for a document is typically much larger than the number of different client devices, the cache can result in significant improvement in response times. 

Q9. How can the concept be further generalized?

The InfoPyramid concept can be further generalized by using other axes such as fault/loss tolerance, numerical complexity, interaction modality, etc. 

Q10. How many key-frames did the story have?

Since the stories have an average of over a dozen key-frames, this allowed us to test out the effect of adaptation for both small and large differences among client resources. 

Q11. What is the common way that search engines generate documents?

Many Internet applications, such as search engines, customized news sites, etc., generate documents dynamically in response to a user request. 

Q12. How do the authors find the optimal version of iM?

The optimal version iM ′23is given by sweeping a line with slope λ , from the top-left to the bottom-right, till it meets the concave hull of these points. 

Q13. What is the advantage of transcoding proxies?

The server shares the benefit of transcoding proxies in speeding content delivery as the customized content is often much smaller than the original content.