An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

The various methods and devices described herein relate to devices which, in at least certain embodiments, may include one or more sensors for providing data relating to user activity and at least one processor for causing the device to respond based on the user activity which was determined, at least in part, through the sensors. The response by the device may include a change of state of the device, and the response may be automatically performed after the user activity is determined.

Automated Response to and Sensing of User Activity in Portable Devices

A virtual assistant uses context information to supplement natural language or gestural input from a user. Context helps to clarify the user's intent and to reduce the number of candidate interpretations of the user's input, and reduces the need for the user to provide excessive clarification input. Context can include any available information that is usable by the assistant to supplement explicit user input to constrain an information-processing problem and/or to personalize results. Context can be used to constrain solutions during various phases of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Using context information to facilitate processing of commands in a virtual assistant

A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain.

Method and apparatus for building an intelligent automated assistant

Among other things, techniques and systems are disclosed for implementing contextual voice commands. On a device, a data item in a first context is displayed. On the device, a physical input selecting the displayed data item in the first context is received. On the device, a voice input that relates the selected data item to an operation in a second context is received. The operation is performed on the selected data item in the second context.

Contextual voice commands

Communications are monitored for user intentions related to user tasks such as meeting scheduling and the like 308 . When an intention (e.g. mention of a meeting) is determined from the communications, candidate task actions (e.g., scheduling meeting) are initiated and/or stored 310 . At an appropriate event trigger (e.g. completion of communications), a user can be prompted with a list of candidate task actions and can select which actions to perform 312 . Action profiles 320 can also be incorporated to allow customization of how task actions are completed. These profiles can be called at the time of task action execution to guide the action fulfillment. The profiles can be user and/or system generated and/or provided by third parties and the like. The task actions are typically performed utilizing task related applications to fully execute the task action so that additional user input is not required.

Communication-prompted user assistance

Technology for testing a target recognition, analysis, and tracking system is provided. A searchable repository of recorded and synthesized depth clips and associated ground truth tracking data is provided. Data in the repository is used by one or more processing devices each including at least one instance of a target recognition, analysis, and tracking pipeline to analyze performance of the tracking pipeline. An analysis engine provides at least a subset of the searchable set responsive to a request to test the pipeline and receives tracking data output from the pipeline on the at least subset of the searchable set. A report generator outputs an analysis of the tracking data relative to the ground truth in the at least subset to provide an output of the error relative to the ground truth.

Validation analysis of human target

Technologies are described herein for launching an online support chat session in response to an occurrence of a support event on a personal computer and transmitting detailed event information via the chat session for display to support personnel. When an error or other event occurs on a computer, an online support chat session is established using an online support chat client associated with the error or event. The pertinent information about the error or event along with additional data regarding the computer and the user are transmitted via the online support chat session for display to the support personnel.

Initiating a support chat session in response to the occurrence of a support event with transmission of detailed event information

Architecture that guides a user through a complex set of steps of a task that can span multiple applications and services, and/or multiple subprograms/services within a single application. A publication mechanism publishes categorized tasks and task steps to the end user client. Task publication allows end user/community modification or task creation for self use or for publishing to other users. A user interface (UI) associates a specific step with the larger task and helps the user to return back to the list of steps and otherwise navigate at the user discretion. A profile mechanism facilitates the storage of user choices over time and the development of a user profile from which future or similar task completion can be based. The architecture serves as a teaching tool that exposes the user to a variety of applications and services of which the user might now otherwise be aware.

Jon D. Pulsipher

Papers

Communication-prompted user assistance

Validation analysis of human target

Initiating a support chat session in response to the occurrence of a support event with transmission of detailed event information

Guidance across complex tasks