An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

A virtual assistant uses context information to supplement natural language or gestural input from a user. Context helps to clarify the user's intent and to reduce the number of candidate interpretations of the user's input, and reduces the need for the user to provide excessive clarification input. Context can include any available information that is usable by the assistant to supplement explicit user input to constrain an information-processing problem and/or to personalize results. Context can be used to constrain solutions during various phases of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Using context information to facilitate processing of commands in a virtual assistant

A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain.

Method and apparatus for building an intelligent automated assistant

An electronic device may capture a voice command from a user. The electronic device may store contextual information about the state of the electronic device when the voice command is received. The electronic device may transmit the voice command and the contextual information to computing equipment such as a desktop computer or a remote server. The computing equipment may perform a speech recognition operation on the voice command and may process the contextual information. The computing equipment may respond to the voice command. The computing equipment may also transmit information to the electronic device that allows the electronic device to respond to the voice command.

Electronic Devices with Voice Command and Contextual Data Processing Capabilities

The method includes automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle. In some implementations, determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle (e.g., via a wired or wireless communication techniques and/or protocols). The method also includes, responsive to the determining, invoking a listening mode of a virtual assistant implemented by the electronic device. In some implementations, the method also includes limiting the ability of a user to view visual output presented by the electronic device, provide typed input to the electronic device, and the like.

Automatically adapting user interfaces for hands-free interaction

A method, system and program product 100 usable by domain developers having any experience level in creating domain models. A representation of domain model knowledge is derived from a domain specification. The domain specification includes multiple potential domain objects, e.g., tables of APIs functional arguments, and each of the potential domain objects include one or more attributes. Potential domain objects are selected one at a time 102 from the specification and offered to the developer. The developer decides 104 whether or not to include the potential domain object in the domain model. If the developer decides to include the potential domain object 106, then the system provides a default name 108, i.e., the table name or argument name, and allows the developer to rename the selected domain object 110. Then, after having selected the object, potential attributes 112, e.g., table columns 1122, are selected from the object and offered to the developer 116. If the developer decides to include a potential attribute, then a default name, i.e., the column name or name extracted from an API function, is offered 1126 for the selected attribute and the developer is allowed to rename attributes 1128. Once all the potential domain objects have been offered 118 to the developer and the developer has either decided to include the potential objects or not, the system checks the domain model for nesting structure 200. If domain objects include attributes that are shared with other domain objects 2006, then those domain objects may be reorganized such that some domain objects include instances of identically named attributes from other domain objects.

Interactive tool for semi-automatic creation of a domain model

A Higher Order Command Dialog System (HOCS) 250 for enabling voice control to a user interface is provided. The HOCS can record (302) a sequence of action steps a user performs while navigating a menu system to perform a task, prompt (304) a user to create an HOC for the task, and associate (306) the sequence of actions steps with a Higher Order Command (HOC) for performing the task. The HOC can include multi-modal inputs (120/260) and prompt a user for non-specific additional information (124) required in performing the task. The HOCS can store the HOC as a voice tag or a user-input command.

Method and system for a user interface using higher order commands

A method (900), device (200) and system (100) provide, in response to text/linguistic input, one of a set of pre-determined meanings which is the most likely intended meaning of that input. A trained meaning discriminator is generated from an annotated training corpus and a meaning discriminator trainer. The trained meaning discriminator generates a meaning vector from an input utterance. The intended meaning encoder analyzes the meaning vector to determine the most likely intended meaning and confidence measures.

Method, device and system for noise-tolerant language understanding

A method, system, and tool product for creating a grammar for a natural language dialog system from a device description is provided. The system can include a device descriptor for identifying configuration, interface, object, and attribute information of the device, a speech grammar for identifying one or more rules generated from the device descriptor that are supported by the device, a speech recognition system for invoking rules of the speech grammar, and a processor for facilitating interoperability and development of distributed applications and providing delineated coverage of the one or more rules in view of the device descriptor.

Interactive tool for semi-automatic generation of a natural language grammar from a device descriptor

A system, method and program product for creating a grammar 100 for a natural language dialog system from a domain model. A template grammar 104 with parameterized rules to be instantiated is opened. A new grammar 102, initially empty, is created. Instances of general purpose grammar rules are created 106 and tentatively added to the new grammar. The general purpose grammar rules instances are domain objects of the new grammar, each including one or more attributes. The developer is offered the option of deciding whether to include each object 112 and which attributes 118 are kept in the new grammar. An umbrella rule is created 108 for each broad category of queries among the general purpose grammar rules. Each umbrella rule includes a domain-object-independent non-terminal in a left-hand side and a set of expansions of the non-terminal in a right-hand side. Objects are selectively included 112 as domain objects in the new grammar. Then, umbrella rules are created for domain object attributes 120 and, object attributes are selectively included 124 in the new grammar. Attributes are classified as either simple or complex 1204. Complex attributes relate one object to a subsidiary domain object. Then, a grammar checker is applied to the new grammar 200, 206 to prevent inclusion of unreachable non-terminals or non-terminating expansions in the final natural language dialog system grammar. Unreachable non-terminals may be removed or new rules may be added making them reachable at the developer's option 204. Non-terminating expansions may also be removed or appropriate rules may be added, also at the developer's option 208.

Dale W. Russell

Papers

Interactive tool for semi-automatic creation of a domain model

Method and system for a user interface using higher order commands

Method, device and system for noise-tolerant language understanding

Interactive tool for semi-automatic generation of a natural language grammar from a device descriptor

Interactive tool for semi-automatic creation of a natural language grammar from a domain model