Showing papers by "Luke Zettlemoyer published in 2000"

Programming by example: visual generalization in programming by example

Abstract: When a user selects a graphical object on the screen, for example, most PBE systems describe the object in terms of the properties of the underlying application data. If the user selects a link on a Web page, the PBE system might represent that selection based on the link’s HTML properties. Here, we explore a different, and radical, approach—using the visual properties of the interaction elements themselves, including size, shape, color, and appearance—to describe user intentions. Only recently has the speed of image processing made feasible PBE systems’ real-time analysis of screen images. We have not yet realized the goal of a PBE system that uses “visual generalization” but feel this approach is important enough to warrant describing and promoting the idea publicly. (Visual generalization means the inference of general patterns in user behavior based on the visual properties and relationships of user interface objects.) Visual information can supplement the information available from other sources, suggesting new kinds of generalizations not possible from application data alone. In addition, these generalizations can map more closely to user intentions, especially beginning users, who rely on the same visual information when making selections. Moreover, visual generalization can sometimes remove one of the main stumbling blocks—reliance on application

Visual generalization in programming by example

Abstract: Publisher Summary In programming-by-example (PBE) systems, the system records the actions performed by a user in the interface and produces a generalized program that can be used later in analogous examples. A key issue is how to describe the actions and objects selected by the user, which determines what kind of generalizations will be possible. This chapter explores an approach using visual properties of the interaction elements themselves, such as size, shape, color, and appearance of graphical objects to describe user intentions. Visual information can supplement information available from other sources and opens up the possibility of new kinds of generalizations not possible from the application data alone. In addition, these generalizations can map more closely to the intentions of users, especially beginning users, who rely on the same visual information when making selections. Finally, visual generalization can sometimes remove one of the worst obstacles preventing the use of PBE with commercial applications—that is—reliance on application program interfaces (APIs). When necessary, PBE systems can work exclusively from the visual appearance of applications and do not need explicit cooperation from the API.

The user interface as an agent environment

Abstract: Theoretically motivated planning systems often make assumptions about their environments, in areas such as the predictability of action e ects, static behavior of the environment, and access to state information. We nd a striking correspondence between these assumptions and the properties of graphical user interfaces. We have developed a novel type of interface agent, called an ibot, to exploit these correspondences. Ibots can interact with o -the-shelf applications through the user interface rather than programmatically, gaining access to functionality not readily available to arti cial agents by other means. In this paper we describe the relationship between these agents and the theoretical and heuristic properties of user interfaces. We demonstrate the feasibility of our approach to interface agents with an implemented prototype that interacts with an unmodi ed application for graphical illustration.

User Interface Softbots

Abstract: Human-computer interaction (HCI) and artificial intelligence (AI) share a long history of research. Concepts such as problem spaces, goals and operators, rationality, and computational models of cognition have significantly influenced research directions in both fields. Recently the concept of agents has sparked a common interest among AI and HCI researchers. Our demonstration focuses on interface agents, those that assist the user in the rich, often complex environment of the graphical user interface (Maes 1994; Lieberman 1995). Our general interest lies in the interaction between agents and their environments. Conventional interface agents interact with other applications through an application programming interface (API) or access to source code. We have developed a novel class of agents we call interface softbots, or ibots, that control interactive applications through the graphical user interface, as human users do (Zettlemoyer & St. Amant 1999; Zettlemoyer, St. Amant, & Dulberg 1999). Our ibots are based on a programmable substrate that provides sensors and effectors for this purpose. Sensor modules take pixel-level input from the display, run the data through image processing algorithms, and build a structured representation of visible interface objects. Effector modules generate mouse and keyboard gestures to manipulate these objects. These sensors and effectors act as the eyes and hands of an artificial user, controlled by an external cognitive system. Together the sensors, effectors, and controller provide a general-purpose means of managing interactive applications, through the same medium as a real user.