An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

The various methods and devices described herein relate to devices which, in at least certain embodiments, may include one or more sensors for providing data relating to user activity and at least one processor for causing the device to respond based on the user activity which was determined, at least in part, through the sensors. The response by the device may include a change of state of the device, and the response may be automatically performed after the user activity is determined.

Automated Response to and Sensing of User Activity in Portable Devices

A virtual assistant uses context information to supplement natural language or gestural input from a user. Context helps to clarify the user's intent and to reduce the number of candidate interpretations of the user's input, and reduces the need for the user to provide excessive clarification input. Context can include any available information that is usable by the assistant to supplement explicit user input to constrain an information-processing problem and/or to personalize results. Context can be used to constrain solutions during various phases of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Using context information to facilitate processing of commands in a virtual assistant

A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain.

Method and apparatus for building an intelligent automated assistant

Among other things, techniques and systems are disclosed for implementing contextual voice commands. On a device, a data item in a first context is displayed. On the device, a physical input selecting the displayed data item in the first context is received. On the device, a voice input that relates the selected data item to an operation in a second context is received. The operation is performed on the selected data item in the second context.

Contextual voice commands

We describe a method for generating causal explanations, in natural language, of the simulated behavior of physical devices. The method is implemented in DME, a system that helps formulate mathematical simulation models from a library of model fragments using a Compositional Modeling approach. Because explanations are generated from models that are dynamically constructed from modular pieces, several of the limitations of conventional explanation techniques are overcome. Since the explanation system has access to the derivation of mathematical equations from the original model specification, the system can explain low-level quantitative behavior predicted by conventional simulation techniques in terms of salient behavioral abstractions such as physical processes, idealized components, and operating modes. Instead of relying on ad hoc causal models, crafted specifically for the explanation task, the program infers causal relationships among parameters in a constraint-based equation model. Rather than using canned, top-down templates, the text generator composes textual annotations associated with individual model fragments into coherent sentences. We show how these techniques can be combined to produce a variety of explanations about simulated systems.

Machine-generated Explanations of Engineering Models: A Compositional Modeling Approach.

Generating explanations of device behavior is a long-standmg goal of AI research in reasoning about physical systems. Much of the relevant work has concentrated on new methods for modeling and simulation, such as qualitative physics, or on sophisticated natural language generation, in which the device models are specially crafted for explanatory purposes. We show how two techniques from the modeling research--compositional modeling and causal ordering-- can be effectively combined to generate natural language explanations of device behavior from engineering models. The explanations offer three advances over the data displays produced by conventional simulation software: (1) causal interpretations of the data, (2) summaries at appropriate levels of abstraction (physical mechanisms and component operating modes), and (3) query-driven, natural language summaries. Furthermore, combining the compositional modeling and causal ordering techniques allows models that are more scalable and less brittle than models designed solely for explanation. However, these techniques produce models with detail that can be distracting in explanations and would be removed in hand-crafted models (e.g., intermediate variables). We present domain-independent filtering and aggregation techniques that overcome these problems.

Patrice O. Gautier

Papers

Machine-generated Explanations of Engineering Models: A Compositional Modeling Approach.

Generating explanations of device behavior using compositional modeling and causal ordering