Newness and distinctiveness is claimed in the features of ornamentation as shown inside the broken line circle in the accompanying representation.

Display screen or portion thereof with graphical user interface

An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

Systems and methods are provided for receiving speech and non-speech communications of natural language questions and/or commands, transcribing the speech and non-speech communications to textual messages, and executing the questions and/or commands. The invention applies context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for one or more users presenting questions or commands across multiple domains. The systems and methods creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context of the speech and non-speech communications and presenting the expected results for a particular question or command.

Systems and methods for responding to natural language speech utterance

Traditional interactive applications are limited to using only the input that users explicitly provide. As users move away from traditional desktop computing environments and move towards mobile and ubiquitous computing environments, there is a greater need for applications to leverage from implicit information, or context. These types of environments are rich in context, with users and devices moving around and computational services becoming available or disappearing over time. This information is usually not available to applications but can be useful in adapting the way in which it performs its services and in changing the available services. Applications that use context are known as context-aware applications. 
This research in context-aware computing has focused on the development of a software architecture to support the building of context-aware applications. While developers have been able to build context-aware applications, they have been limited to using a small variety of sensors that provide only simple context such as identity and location. This dissertation presents a set of requirements and component abstractions for a conceptual supporting framework. The framework along with an identified design process makes it easier to acquire and deliver context to applications, and in turn, build more complex context-aware applications. 
In addition, an implementation of the framework called the Context Toolkit is discussed, along with a number of context-aware applications that have been built with it. The applications illustrate how the toolkit is used in practice and allows an exploration of the design space of context-aware computing. This dissertation also shows how the Context Toolkit has been used as a research testbed, supporting the investigation of difficult problems in context-aware computing such as the building of high-level programming abstractions, dealing with ambiguous or inaccurate context data and controlling access to personal context.

/pdf/providing-architectural-support-for-building-context-aware-1y6c5sg04e.pdf

Providing architectural support for building context-aware applications

https://www.ee.columbia.edu/%7Eajaimes/Pubs/ajaimes_iccv05.pdf

Multimodal human-computer interaction: A survey

A system, method, and article of manufacture are provided for navigating an electronic data source by means of spoken language where a portion of the data link between a mobile information appliance of the user and the data source utilizes wireless communication. When a spoken input request is received from a user who is using the mobile information appliance, it is interpreted. The resulting interpretation of the request is thereupon used to automatically construct an operational navigation query to retrieve the desired information from one or more electronic network data sources, which is transmitted to the mobile information appliance.

Mobile navigation of network-based electronic information using spoken input

A highly mobile, ambient computing environment is disclosed for serving a knowledge worker away from the their desk The present invention allows a knowledge worker to get increased leverage from personal, networked, and interactive computing devices while in their car, airplane seat, or in a conference room with others An Open Agent Architecture is used to incorporate elements such as GPS agents, speech recognition, and opportunistic connectivity among meeting participants Communication and cooperation between agents are brokered by one or more facilitators, which are responsible for matching requests, from users and agents, with descriptions of the capabilities of other agents It is not generally required that a user or agent know the identities, locations, or number of other agents involved in satisfying a request, and relatively minimal effort is involved in incorporating new agents and “wrapping” legacy applications Extreme flexibility is achieved through an architecture organized around the declaration of capabilities by service-providing agents, the construction of arbitrarily complex goals by users and service-requesting agents, and the role of facilitators in delegating and coordinating the satisfaction of these goals, subject to advice and constraints that may accompany them

Using a community of distributed electronic agents to support a highly mobile, ambient computing environment

In this paper, we discuss how multiple input modalities may be combined to produce more natural user interfaces. To illustrate this technique, we present a prototype map-based application for a travel planning domain. The application is distinguished by a synergistic combination of handwriting, gesture and speech modalities; access to existing data sources including the World Wide Web; and a mobile handheld interface. To implement the described application, a hierarchical distributed network of heterogeneous software agents was augmented by appropriate functionality for developing synergistic multimodal applications.

/pdf/multimodal-maps-an-agent-based-approach-1e51o1j1gt.pdf

Multimodal Maps: An Agent-Based Approach

A system, method, and article of manufacture are provided for navigating an electronic data source by means of spoken language where a portion of the data link between a mobile information appliance of the user and the data source utilizes wireless communication. When a spoken input request is received from a user, it is interpreted. The resulting interpretation of the request is thereupon used to automatically construct an operational navigation query. The navigation query is routed to one or more agents, which use the navigation query to retrieve the desired information from one or more electronic network data sources.

System, method, and article of manufacture for agent-based navigation in a speech-based data navigation system

The design and development of the Open Agent Architecture (OAA) 1 system has focused on providing access to agentbased applications through an intelligent, cooperative, distributed, and multimodal agent-based user interfaces. The current multimodal interface supports a mix of spoken language, handwriting and gesture, and is adaptable to the user’ s preferences, resources and environment. Only the primary user interface agents need run on the local computer, thereby simplifying the task of using a range of applications from a variety of platforms, especially low-powered computers such as Personal Digital Assistants (PDAs). An important consideration in the design of the OAA was to facilitate mix-andmatch: to facilitate the reuse of agents in new and unanticipated applications, and to support rapid prototyping by facilitating the replacement of agents by better versions. The utility of the agents and tools developed as part of this ongoing research project has been demonstrated by their use as infrastructure in unrelated projects.

/pdf/multimodal-user-interfaces-in-the-open-agent-architecture-30sq5voh8q.pdf

Luc Julia

Papers

Mobile navigation of network-based electronic information using spoken input

Using a community of distributed electronic agents to support a highly mobile, ambient computing environment

Multimodal Maps: An Agent-Based Approach

System, method, and article of manufacture for agent-based navigation in a speech-based data navigation system

Multimodal user interfaces in the Open Agent Architecture