There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

In 2000, the Cohn-Kanade (CK) database was released for the purpose of promoting research into automatically detecting individual facial expressions. Since then, the CK database has become one of the most widely used test-beds for algorithm development and evaluation. During this period, three limitations have become apparent: 1) While AU codes are well validated, emotion labels are not, as they refer to what was requested rather than what was actually performed, 2) The lack of a common performance metric against which to evaluate new algorithms, and 3) Standard protocols for common databases have not emerged. As a consequence, the CK database has been used for both AU and emotion detection (even though labels for the latter have not been validated), comparison with benchmark algorithms is missing, and use of random subsets of the original database makes meta-analyses difficult. To address these and other concerns, we present the Extended Cohn-Kanade (CK+) database. The number of sequences is increased by 22% and the number of subjects by 27%. The target expression for each sequence is fully FACS coded and emotion labels have been revised and validated. In addition to this, non-posed sequences for several types of smiles and their associated metadata have been added. We present baseline results using Active Appearance Models (AAMs) and a linear support vector machine (SVM) classifier using a leave-one-out subject cross-validation for both AU and emotion detection for the posed data. The emotion and AU labels, along with the extended image data and tracked landmarks will be made available July 2010.

/pdf/the-extended-cohn-kanade-dataset-ck-a-complete-dataset-for-3ui317eamh.pdf

The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression

Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.

/pdf/multimodal-deep-learning-2qi0e2kef0.pdf

Multimodal Deep Learning

Matrix Factorization Techniques for Recommender Systems

In professional sport, an enormous amount of fine-grain performance data can be generated at near millisecond intervals in the form of vision-based tracking data. One of the first sports to embrace this technology has been tennis, where Hawk-Eye technology has been used to both aid umpiring decisions, and to visualize shot trajectories for broadcast purposes. Despite the high-level of accuracy of the tracking systems and the sheer volume of spatiotemporal data they generate, the use of this data for player performance analysis and prediction has been lacking. In this research, we use ball and player tracking data from “Hawk-Eye” to discover unique player styles and predict within-point events. We move beyond current analysis that only incorporates coarse match statistics (i.e., serves, winners, number of shots, and volleys) and use spatial and temporal information which better characterizes the tactics and tendencies of each player. Using a probabilistic graphical model, we are able to model player behaviors which enables us to: 1) find the factors such as location and speed of the incoming shot which are most conducive to a player hitting a winner (i.e., “sweet-spot”) or cause an error, and 2) do “live in-point” prediction - based on the shots being played during a rally we estimate the probability of the outcome (e.g., winner, continuation, or error) and the location of the next shot. As player behavior depends on the opponent, we use model adaptation to enhance our prediction. We show the utility of our approach by analyzing the play of Djokovic, Nadal, and Federer at the 2012 Australian Tennis Open.

Forecasting the Next Shot Location in Tennis Using Fine-Grained Spatiotemporal Tracking Data

Acoustically, car cabins are extremely noisy and as a consequence audio-only, in-car voice recognition systems perform poorly. As the visual modality is immune to acoustic noise, using the visual lip information from the driver is seen as a viable strategy in circumventing this problem by using audio visual automatic speech recognition (AVASR). However, implementing AVASR requires a system being able to accurately locate and track the drivers face and lip area in real-time. In this paper we present such an approach using the Viola-Jones algorithm. Using the AVICAR [1] in-car database, we show that the Viola- Jones approach is a suitable method of locating and tracking the driver's lips despite the visual variability of illumination and head pose for audio-visual speech recognition system.

/pdf/lip-detection-for-audio-visual-speech-recognition-in-car-2esd1kb1r4.pdf

Lip detection for audio-visual speech recognition in-car environment

Methods and systems for measuring group behavior are provided. Group behavior of different groups may be measured objectively and automatically in different environments including a dark environment. A uniform visible signal comprising images of members of a group may be obtained. Facial motion and body motions of each member may be detected and analyzed from the signal. Group behavior may be measured by aggregating facial motions and body motions of all members of the group. A facial motion such as a smile may be detected by using the Fourier Lucas-Kanade (FLK) algorithm to register and track faces of each member of a group. A flow-profile for each member of the group is generated. Group behavior may be further analyzed to determine a correlation of the group behavior and the content of the stimulus. A prediction of the general public's response to the stimulus based on the analysis of the group behavior is also provided.

Methods and systems for measuring group behavior

In recent work, we have concentrated on the problem of lipreading
from non-frontal views (poses). In particular, we have focused on
the use of profile views, and proposed two approaches for
lipreading on basis of visual features extracted from such views:
(a) Direct statistical modeling of the features, namely use of
view-dependent statistical models; and (b) Normalization of such
features by their projection onto the ``space'' of frontal-view
visual features, which allows employing one set of statistical
models for all available views. The latter approach has been
considered for two only poses (frontal and profile views), and
for visual features of a specific dimensionality. In this paper,
we further extend this work, by investigating its applicability
to the case where data from three views are available (frontal,
left- and right-profile). In addition, we examine the effect of
visual feature dimensionality on the pose-normalization approach.
Our experiments demonstrate that results generalize well to three
views, but also that feature dimensionality is crucial to the
effectiveness of the approach. In particular, feature
dimensionality larger than 30
is detrimental to multi-pose visual speech recognition performance.

An Extended Pose-Invariant Lipreading System

Visual activity detection of lip movements can be used to overcome the poor performance of voice activity detection based solely in the audio domain, particularly in noisy acoustic conditions. However, most of the research conducted in visual voice activity detection (VVAD) has neglected addressing variabilities in the visual domain such as viewpoint variation. In this paper we investigate the effectiveness of the visual information from the speaker's frontal and profile views (i.e left and right side views) for the task of VVAD. As far as we are aware, our work constitutes the first real attempt to study this problem. We describe our visual front end approach and the Gaussian mixture model (GMM) based VVAD framework, and report the experimental results using the freely available CUAVE database. The experimental results show that VVAD is indeed possible from profile views and we give a quantitative comparison of VVAD based on frontal and profile views The results presented are useful in the development of multi-modal Human Machine Interaction (HMI) using a single camera, where the speaker's face may not always be frontal.

/pdf/visual-voice-activity-detection-using-frontal-versus-profile-4naunli331.pdf

Patrick Lucey

Papers

Forecasting the Next Shot Location in Tennis Using Fine-Grained Spatiotemporal Tracking Data

Lip detection for audio-visual speech recognition in-car environment

Methods and systems for measuring group behavior

An Extended Pose-Invariant Lipreading System

Visual Voice Activity Detection Using Frontal versus Profile Views