There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

In 2000, the Cohn-Kanade (CK) database was released for the purpose of promoting research into automatically detecting individual facial expressions. Since then, the CK database has become one of the most widely used test-beds for algorithm development and evaluation. During this period, three limitations have become apparent: 1) While AU codes are well validated, emotion labels are not, as they refer to what was requested rather than what was actually performed, 2) The lack of a common performance metric against which to evaluate new algorithms, and 3) Standard protocols for common databases have not emerged. As a consequence, the CK database has been used for both AU and emotion detection (even though labels for the latter have not been validated), comparison with benchmark algorithms is missing, and use of random subsets of the original database makes meta-analyses difficult. To address these and other concerns, we present the Extended Cohn-Kanade (CK+) database. The number of sequences is increased by 22% and the number of subjects by 27%. The target expression for each sequence is fully FACS coded and emotion labels have been revised and validated. In addition to this, non-posed sequences for several types of smiles and their associated metadata have been added. We present baseline results using Active Appearance Models (AAMs) and a linear support vector machine (SVM) classifier using a leave-one-out subject cross-validation for both AU and emotion detection for the posed data. The emotion and AU labels, along with the extended image data and tracked landmarks will be made available July 2010.

/pdf/the-extended-cohn-kanade-dataset-ck-a-complete-dataset-for-3ui317eamh.pdf

The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression

Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.

/pdf/multimodal-deep-learning-2qi0e2kef0.pdf

Multimodal Deep Learning

Matrix Factorization Techniques for Recommender Systems

Motivated by the success of free-parts based representations in face recognition, we have attempted to address some of the problems associated with applying such a philosophy to the task of speaker-independent visual speech recognition. A major problem with canonical area-based approaches in automatic visual speech recognition is the dependence these approaches have on locating and tracking the speakers region of interest (ROI) correctly. By employing a free-parts representation,we assume that the position/structure of patches within the mouth image can be relaxed so they can "freely" move to varying extents, hence reducing the influence of the front-end effect. In this paper, we show that by using a free-parts representation we gain some robustness against the problem of ROI localisation and tracking compared to current area-based feature extraction techniques such as the discrete cosine transform (DCT). Also in this paper, we expose the importance of representation for the task of visual speech recognition highlighted by the poor results current representations yield.

https://ci2cv.net/media/papers/2005_free_parts_Lucey.pdf

Using a Free-Parts Representation for Visual Speech Recognition

Visual information fromaspeaker's mouthregion is knowntoimprove automatic speech recognition robustness. How- ever, thevastmajority ofaudio-visual automatic speech recog- nition (AVASR)studies assumefrontal imagesofthespeaker's face. Incontrast, this paperinvestigates extracting visual speech information fromthespeaker's profile view, and,toourknowl- edge, constitutes thefirst realattempt toattack thisproblem. AswithanyAVASRsystem, theoverall recognition performance depends heavily onthevisual front end.Thisisespecially the casewithprofile-view data, asthefacial features areheavily compacted compared tothefrontal scenario. Inthispaper, we particularly describe ourvisual front endapproach, andreport experiments onamulti-subject, small-vocabulary, bimodal, multi- sensory database thatcontains synchronously captured audio withfrontal andprofile facevideo. Ourexperiments showthat AVASRispossible fromprofile views withmoderate performance degradation compared tofrontal video data.

https://ieeexplore.ieee.org/iel5/4064504/4064505/04064511.pdf

LipreadingUsing ProfileVersusFrontal Views

Motivated by the success of free-parts based representatio s in face recognition [1] we have attempted to address some of the problems associated with applying such a philosophy to the task of speaker-independent automatic speech reading. Hitherto, a major problem with canonical area-based approaches in automatic speech reading is the intrinsic lac k of training observations due to the visual speech modality’ s low sample rate and large variability in appearance. We believe a free-parts representation can overcome many of these limitations due to its natural ability to generalize b y producing many observations from a single mouth image, whilst still preserving the ability to discriminate betwee n various visual-speech units. This approach additionally r equires a modification to traditional techniques employed fo r the estimation of hidden Markov Models (HMMs), whose resultant models we currently refer to as free-parts HMMs (FP-HMMs). Results will be presented on the CUAVE audiovisual speech database.

Improved speech reading through a free-parts representation.

A system and method of generating trackable frames from a broadcast video feed are provided herein. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. The computing system generates a set of frames for classification using a principal component analysis model. The set of frames are a subset of the plurality of video frames. The computing system partitions each frame of the set of frames into a plurality of clusters. The computing system classifies each frame of the plurality of frames as trackable or untrackable. Trackable frames capture a unified view of the sporting event. The computing system compares each cluster to a predetermined threshold to determine whether each cluster comprises at least a threshold number of trackable frames. The computing system classifies each cluster that includes at least the threshold number of trackable frames as trackable.

System and method for generating trackable video frames from broadcast video

The ability to predict what shot a batsman will attempt given the type of ball and match situation is both one of the most challenging and strategically important tasks in cricket 
The goal of the batsman is to score as many runs without being dismissed, whilst for bowlers their goal is to stem the flow of runs and ideally to dismiss their opponent Getting the best batsman vs bowler match-up is of paramount importance For example, for the fielding team, the choice of bowler against the opposition star batsman could be the key difference between winning or losing Therefore, the ability to have a predefined playbook (as in the NFL) which would allow a team to predict how best to set their fielders given the context of the game, the batsman they are bowling to and bowlers at their disposal would give them a significant strategic advantage 
To this end, we present a personalized deep neural network approach which can predict the probabilities of where a specific batsman will hit a specific bowler and bowl type, in a specific game-scenario We demonstrate how our personalized predictions provide vital information to inform the decision-making of coaches and captains, both in terms of pre-match and in-game tactical choices, using the 2019 World Cup final between England and New Zealand as a case study example

Patrick Lucey

Papers

Using a Free-Parts Representation for Visual Speech Recognition

LipreadingUsing ProfileVersusFrontal Views

Improved speech reading through a free-parts representation.

System and method for generating trackable video frames from broadcast video

You Cannot Do That Ben Stokes: Dynamically Predicting Shot Type in Cricket Using a Personalized Deep Neural Network.