An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

A virtual assistant uses context information to supplement natural language or gestural input from a user. Context helps to clarify the user's intent and to reduce the number of candidate interpretations of the user's input, and reduces the need for the user to provide excessive clarification input. Context can include any available information that is usable by the assistant to supplement explicit user input to constrain an information-processing problem and/or to personalize results. Context can be used to constrain solutions during various phases of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Using context information to facilitate processing of commands in a virtual assistant

A method and apparatus are provided for building an intelligent automated assistant. Embodiments of the present invention rely on the concept of “active ontologies” (e.g., execution environments constructed in an ontology-like manner) to build and run applications for use by intelligent automated assistants. In one specific embodiment, a method for building an automated assistant includes interfacing a service-oriented architecture that includes a plurality of remote services to an active ontology, where the active ontology includes at least one active processing element that models a domain. At least one of the remote services is then registered for use in the domain.

Method and apparatus for building an intelligent automated assistant

An electronic device may capture a voice command from a user. The electronic device may store contextual information about the state of the electronic device when the voice command is received. The electronic device may transmit the voice command and the contextual information to computing equipment such as a desktop computer or a remote server. The computing equipment may perform a speech recognition operation on the voice command and may process the contextual information. The computing equipment may respond to the voice command. The computing equipment may also transmit information to the electronic device that allows the electronic device to respond to the voice command.

Electronic Devices with Voice Command and Contextual Data Processing Capabilities

The method includes automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle. In some implementations, determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle (e.g., via a wired or wireless communication techniques and/or protocols). The method also includes, responsive to the determining, invoking a listening mode of a virtual assistant implemented by the electronic device. In some implementations, the method also includes limiting the ability of a user to view visual output presented by the electronic device, provide typed input to the electronic device, and the like.

Automatically adapting user interfaces for hands-free interaction

One embodiment of the present invention provides a system that uses a sound sensor to adjust the audio output of a device. During operation, the system uses the sound sensor to determine an ambient sound level for the environment in the proximity of the device. The system then adjusts a volume setting for the device adaptively based on the determined ambient sound level. Adaptively adjusting the volume setting allows the device to adapt to its audio environment and ameliorates potentially-disruptive audio outputs.

Method and apparatus for using a sound sensor to adjust the audio output for a device

One or more location-based clients can be activated on a mobile device for providing location-based services. The location-based clients can be provided with information (e.g., presets, defaults) related to the current location and/or mode of the mobile device. The information can be obtained from one or more network resources. In some implementations, a number of location-based clients can run concurrently on the mobile device and share information.

Location-Aware Mobile Device

Systems and methods for providing real-time feedback to karaoke users are provided. The systems and methods for providing users with real-time feedback while they are singing karaoke generally relate to receiving the user's vocals, determining whether the user is singing on key/pitch and providing real-time feedback to the user while the karaoke song is being sung. The feedback will be positive feedback if user is on key/pitch and it will be negative feedback if user is off key/pitch. For example, the feedback signal if the user is singing too low can be an exaggerated low signal of the user's own voice. This will encourage the user to sing at a higher pitch.

Methods and systems for providing real-time feedback for karaoke

Systems and methods are provided for detecting ambient light with reduced sensitivity to infrared sources. An electronic device may include an infrared sensor, an ambient light sensor, a decoder, and a processor. The infrared sensor may detect an intensity of infrared light. The ambient light sensor may be configured to detect incident light and to generate an electronic signal indicative of an intensity of visible light. The decoder may be configured to receive the intensity of infrared light and to generate an intensity of decoded infrared light. The processor, which may be coupled to the decoder and the ambient light sensor, may be configured to substitute an alternate electronic signal for the electronic signal if the intensity of infrared light exceeds a threshold amount.

Ambient light sensor with reduced sensitivity to noise from infrared sources

The present invention can provide solutions to many common imaging problems, such as, for example, unevenly distributed illumination, shadows, white balance adjustment, colored ambient light and high dynamic range imaging Imaging systems and methods can be provided through a computer (eg, laptop or desktop) such that the system or method can take advantage of the computer's processing power to provide functionality that goes beyond typical camera Such an imaging system may include an imaging device, a camera, a light source and a user interface

Peter H. Mahowald

Papers

Method and apparatus for using a sound sensor to adjust the audio output for a device

Location-Aware Mobile Device

Methods and systems for providing real-time feedback for karaoke

Ambient light sensor with reduced sensitivity to noise from infrared sources

Illumination systems and methods for computer imagers