Showing papers by "Kathleen R. McKeown published in 1992"

Generating cross-references for multimedia explanation

Abstract: When explanations include multiple media, such as text and illustrations, a reference to an object can be made through a combination of media. We call part of a presentation that references material elsewhere a cross-reference. We are concerned here with how textual expressions can refer to parts of accompanying illustrations. The illustration to which a cross-reference refers should also satisfy the specific goal of identifying an object for the user. Thus, producing an effective cross-reference not only involves text generation, but may also entail modifying or replacing an existing illustration and in some cases, generating an illustration where previously none was needed. In this paper, we describe the different types of cross-references that COMET (COordinated Multimedia Explanation Testbed) generates and show the roles that both its text and graphics generators play in this process.

Extracting constraints on word usage from large text corpora

Abstract: Our research focuses on the identification of word usage constraints from large text corpora. Such constraints are useful both for the problem of selecting vocabulary for language generation and for disambiguating lexical meaning in interpretation. We are developing systems that can automatically extract such constraints from corpora and empirical methods for analyzing text. Identified constraints will be represented in a lexicon that will be tested computationally as part of a natural language system. We are also identifying lexical constraints for machine translation using the aligned Hansard corpus as training data and are identifying many-to-many word alignments.

Extending Language Generation to Multiple Media

Abstract: When explaining how to use a technical device humans will often utilize a combination of language and graphics. It is a rare instruction manual that does not contain illustrations. Multimodal presentation systems combining natural language and graphics take advantage of both the individual strength of each communication mode and the fact that both modes can be employed in parallel. It is an important goal of this research not simply to merge the verbalization results of a natural language generator and the visualization results of a knowledge-based graphics design component, but to carefully coordinate natural language and graphics in such a way that they generate a multiplicative improvement in communication capabilities. Allowing all of the modalities to refer to and depend upon each other is a key to the richness of multimodal communication. In the WIP system that plans and coordinates multimodal presentations in which all material is generated by the system, we have integrated multiple AI components such as planning, knowledge representation, natural language generation, and graphics generation. The current prototype of WIP generates multimodal explanations and instructions for assembling, using, maintaining or repairing physical devices. As we try to substantiate our results with cross-language and cross-application evidence WIP is currently able to generate simple German or English explanations for using an espresso-machine or assembling a lawn-mower. In WIP we combined and extended only formalisms that have reached a certain level of maturity: in particular, terminological logics, RST-based planning, constraint processing techniques, and tree adjoining grammars with feature unification. One of the important insights we gained from building the WIP system is that it is actually possible to extend and adapt many of the fundamental concepts developed to date in AI and computational linguistics for the generation of natural language in such a way that they become useful for the generation of graphics and text-picture combinations as well. This means that an interesting methodological transfer from the area of natural language processing to a much broader computational model of multimodal communication seems possible. In particular, semantic and pragmatic concepts like coherence, focus, communicative act, discourse model, reference, implicature, anaphora, or scope ambiguity take an extended meaning in the context of text-picture combinations. A basic principle underlying the WIP model is that the various constituents of a multimodal presentation should be generated from a common representation of what is to be conveyed. This raises the question of how to decompose a given communicative goal into subgoals to be realized by the modespecific generators, so that they complement each other. To address this problem, we explored computational models of the cognitive decision processes coping with questions such as what should go into text, what should go into graphics, and which kinds of links between the verbal and non-verbal fragments are necessary. The task of the knowledge-based presentation system WIP is the context-sensitive generation of a variety of multimodal documents from an input including a presentation goal. The presentation goal is a formal representation of the communicative intent specified by the back-end application system. WIP is a highly adaptive interface, since all of its output is generated on the fly and customized for the intended target audience and situation. The quest for adaptation is based on the fact that is impossible to anticipate the needs and requirements of each potential user in an infinite number of presentation situations. We view the design of multimodal presentations including text and graphics design as a subarea of general communication design. We approximate the fact that communication is always situated by introducing generation parameters (see Fig. 1) in our model. The current system includes a choice between user