An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

An electronic device may capture a voice command from a user. The electronic device may store contextual information about the state of the electronic device when the voice command is received. The electronic device may transmit the voice command and the contextual information to computing equipment such as a desktop computer or a remote server. The computing equipment may perform a speech recognition operation on the voice command and may process the contextual information. The computing equipment may respond to the voice command. The computing equipment may also transmit information to the electronic device that allows the electronic device to respond to the voice command.

Electronic Devices with Voice Command and Contextual Data Processing Capabilities

A method for operating a voice trigger is provided. In some implementations, the method is performed at an electronic device including one or more processors and memory storing instructions for execution by the one or more processors. The method includes receiving a sound input. The sound input may correspond to a spoken word or phrase, or a portion thereof. The method includes determining whether at least a portion of the sound input corresponds to a predetermined type of sound, such as a human voice. The method includes, upon a determination that at least a portion of the sound input corresponds to the predetermined type, determining whether the sound input includes predetermined content, such as a predetermined trigger word or phrase. The method also includes, upon a determination that the sound input includes the predetermined content, initiating a speech-based service, such as a voice-based digital assistant.

Voice trigger for a digital assistant

A user request is received from a mobile client device, where the user request includes at least a speech input and seeks an informational answer or performance of a task. A failure to provide a satisfactory response to the user request is detected. In response to detection of the failure, information relevant to the user request is crowd-sourced by querying one or more crowd sourcing information sources. One or more answers are received from the crowd sourcing information sources, and the response to the user request is generated based on at least one of the one or more answers received from the one or more crowd sourcing information sources.

Crowd sourcing information to fulfill user requests

A device can be configured to receive speech input from a user. The speech input can include a command for accessing a restricted feature of the device. The speech input can be compared to a voiceprint (e.g., text-independent voiceprint) of the user's voice to authenticate the user to the device. Responsive to successful authentication of the user to the device, the user is allowed access to the restricted feature without the user having to perform additional authentication steps or speaking the command again. If the user is not successfully authenticated to the device, additional authentication steps can be request by the device (e.g., request a password).

Device access using voice authentication

A language model for speech recognition may be discriminatively pruned. In some embodiments, the language model is discriminatively pruned by computing a discriminative objective function value for one or more n-grams in the language model, and selecting one or more n-grams to prune based at least in part on a threshold value. In some embodiments, the language model is discriminatively pruned to a sufficiently small number of n-grams such that transcription of audio inputs may occur in real time, or such that the pruned language model may be stored on a device with relatively limited electronic storage capacity.

Discriminative language model pruning

We investigate the incorporation of larger time-scale information, such as prosody, into standard speaker ID systems. Our study is based on the Extended Data Task of the NIST 2001 Speaker ID evaluation, which provides much more test and training data than has traditionally been available to similar speaker ID investigations. In addition, we have had access to a detailed prosodic feature database of Switchboard-I conversations, including data not previously applied to speaker ID. We describe two baseline acoustic systems, an approach using Gaussian Mixture Models, and an LVCSR-based speaker ID system. These results are compared to and combined with two larger time-scale systems: a system based on an “idiolect” language model. and a system making use of the contents of the prosody database. We find that, with sufficient test and training data, suprasegmental information can significantly enhance the performance of traditional speaker ID systems.

Using prosodic and lexical information for speaker identification

Natural language controlled devices may be configured to activate command recognition in response to one or more wake words Techniques are provided to receive a candidate word for evaluation as a wake word that activates a natural language control functionality of a computing device The candidate word may include one or more words or sounds Values for multiple wake word metrics are then determined The candidate word is evaluated based on the various wake word metrics

Wake word evaluation

Neural networks may be used in certain automatic speech recognition systems. To improve performance of these neural networks, they may be updated/retrained during run time by training the neural network based on the output of a speech recognition system or based on the output of the neural networks themselves. The outputs may include weighted outputs, lattices, weighted N-best lists, or the like. The neural networks may be acoustic model neural networks or language model neural networks. The neural networks may be retrained after each pass through the network, after each utterance, or in varying time scales.

Adaptive neural network speech recognition models

Features are disclosed for automatically identifying a speaker. Artifacts of automatic speech recognition (“ASR”) and/or other automatically determined information may be processed against individual user profiles or models. Scores may be determined reflecting the likelihood that individual users made an utterance. The scores can be based on, e.g., individual components of Gaussian mixture models (“GMMs”) that score best for frames of audio data of an utterance. A user associated with the highest likelihood score for a particular utterance can be identified as the speaker of the utterance. Information regarding the identified user can be provided to components of a spoken language processing system, separate applications, etc.

Frederick V. Weber

Papers

Discriminative language model pruning

Using prosodic and lexical information for speaker identification

Wake word evaluation

Adaptive neural network speech recognition models

Automatic speaker identification using speech recognition features