A system for integrating video programming with the vast information resources of the Internet. A computer-based system receives a video program with embedded uniform resource locators (URLs). The URLs, the effective addresses of locations or Web sites on the Internet, are interpreted by the system and direct the system to the Web site locations to retrieve related Web pages. Upon receipt of the Web pages by the system, the Web pages are synchronized to the video content for display. The video program signal can be displayed on a video window on a conventional personal computer screen. The actual retrieved Web pages are time stamped to also be displayed, on another portion of the display screen, when predetermined related video content is displayed in the video window. As an alternative, the computer-based system receives the URLs directly through an Internet connection, at times specified by TV broadcasters in advance. The system interprets the URLs and retrieves the appropriate Web pages. The Web pages are synchronized to the video content for display in conjunction with a television program being broadcast to the user at that time. This alternative system allows the URLs to be entered for live transmission to the user.

Enhanced video programming system and method for incorporating and displaying retrieved integrated Internet information segments

An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

A method and system that provides filtered data from a data system (16). In one embodiment that system includes an API (application programming interface) and associated software modules to enable third party applications to access an enterprise data system. Administrators are enabled to select specific user interface (UI) objects (72), such as screens, views, applets, columns and fields to voice or pass-through enable via a GUI (108) that presents a tree depicting a hierarchy of the UI objects (72) within a user interface of an application (14). An XSLT style sheet is then automatically generated to filter out data pertaining to UI objects (72) that were not voice or pass-through enabled. In response to a request for data, unfiltered data are retrieved from the data system and a specified style sheet is applied to the unfiltered data to return filtered data pertaining to only those fields and columns that are voice or pass-through enabled.

Method and system for enabling connectivity to a data system

https://www.ee.columbia.edu/%7Eajaimes/Pubs/ajaimes_iccv05.pdf

Multimodal human-computer interaction: A survey

A system and method for implementing conversational protocols for distributed conversational networking architectures and/or distributed conversational applications, as well as real-time conversational computing between network-connected pervasive computing devices and/or servers over a computer network. The implementation of distributed conversational systems/applications according to the present invention is based, in part, on a suitably defined conversational coding, transport and control protocols. The control protocols include session control protocols, protocols for exchanging of speech meta-information, and speech engine remote control protocols.

Conversational networking via transport, coding and control conversational protocols

Systems and methods for building multi-modal browsers applications and, in particular, to systems and methods for building modular multi-modal browsers using a DOM (Document Object Model) and MVC (Model-View-Controller) framework that enables a user to interact in parallel with the same information via a multiplicity of channels, devices, and/or user interfaces, while presenting a unified, synchronized view of such information across the various channels, devices and/or user interfaces supported by the multi-modal browser. The use of a DOM framework (or specifications similar to DOM) allows existing browsers to be extended without modification of the underling browser code. A multi-modal browser framework is modular and flexible to allow various fat client and thin (distributed) client approaches.

Systems and methods for implementing modular DOM (Document Object Model)-based multi-modal browsers

JMUI (Journal on Multimodal User Interfaces), Special issue “Best of affective computing and intelligent Guidelines for multimodal user interface design. support, human multi-modal information processing. characteristics to the design of a user-oriented and guidelines of multimodal interface design. Artifact lifecycle management, Consumer and user, Interfaces in Automated.Aug 2 Aug 7Los Angeles, CA, USAThursday, 6 August 2015 / HCI International 20152015.hci.international/thursday​CachedDefining and Optimizing User Interfaces Information Complexity for AI Design and Development of Multimodal Applications: A Vision on Key Issues and Traditional Heuristics and Industry Guidelines to Evaluate Multimodal Digital Artifacts

Guidelines for multimodal user interface design

A new application programming language is provided which is based on user interaction with any device which a user is employing to access any type of information. The new language is referred to herein as a “Conversational Markup Language (CML). In a preferred embodiment, CML is a high level XML based language for representing “dialogs” or “conversations” the user will have with any given computing device. For example, interaction may comprise, but is not limited to, visual based (text and graphical) user interaction and speech based user interaction. Such a language allows application authors to program applications using interaction-based elements referred to herein as “conversational gestures.” The present invention also provides for various embodiments of a multimodal browser capable of supporting the features of CML in accordance with various modality specific representations, e.g., HTML based graphical user interface (GUI) browser, VoiceXML based speech browser, etc.

Systems and methods for providing conversational computing via javaserver pages and javabeans

Systems and methods for building speech-based applications using reusable dialog components based on VoiceXML (Voice eXtensible Markup Language). VoiceXML reusable dialog components can be used for building a voice interface for use with multi-modal, multi-channel and conversational applications that offer universal access to information anytime, from any location, using any pervasive computing device regardless of its I/O modality. In one embodiment, a framework for reusable dialog components built within the VoiceXML specifications is based on the tag and ECMAScript parameter objects to pass parameters, configuration and results. This solution is interpreted at the client side (VoiceXML browser). In another embodiment, a framework for reusable dialog components is based on JSP (Java Server Pages) and beans that generate VoiceXML subdialogs. This solution can be evaluated at the server side. These frameworks can be mixed and matched depending on the application.

Reusable voiceXML dialog components, subdialogs and beans

In this paper we introduce JustSpeak, a universal voice control solution for non-visual access to the Android operating system. JustSpeak offers two contributions as compared to existing systems. First, it enables system wide voice control on Android that can accommodate any application. JustSpeak constructs the set of available voice commands based on application context; these commands are directly synthesized from on-screen labels and accessibility metadata, and require no further intervention from the application developer. Second, it provides more efficient and natural interaction with support of multiple voice commands in the same utterance. We present the system design of JustSpeak and describe its utility in various use cases. We then discuss the system level supports required by a service like JustSpeak on other platforms. By eliminating the target locating and pointing tasks, JustSpeak can significantly improve experience of graphic interface interaction for blind and motion-impaired users.

T. V. Raman

Papers

Systems and methods for implementing modular DOM (Document Object Model)-based multi-modal browsers

Guidelines for multimodal user interface design

Systems and methods for providing conversational computing via javaserver pages and javabeans

Reusable voiceXML dialog components, subdialogs and beans

JustSpeak: enabling universal voice control on Android