On discrimination and classification with multivariate repeated measures data

Automated assessment of affect and arousal level can help psychologists and psychiatrists in clinical diagnoses; and may enable affect-aware robot-human interaction. This work identifies major difficulties in automating affect and arousal assessment and attempts to overcome some of them. We first analyze thermal infrared images and examine how changes in affect and/or arousal level would cause haemodynamic variations, concentrated along certain facial muscles. These concentrations are used to measure affect/arousal induced facial thermal variations. In step-1 of a 2-step pattern recognition schema, ‘between-affect’ and ‘between-arousal-level’ variations are used to derive facial thermal features as Principal Components (PCs) of the facial thermal measurements. The most influential of these PCs are used to cluster the feature space for different affects and subsequently assign a set of thermal features to an affect cluster. In step-2, affect clusters are partitioned into high, medium and mild arousal levels. The distance between a test face vector and the centroids of sub-clusters at three arousal levels belonging to a single affective state, identified from step-1, is used to determine the arousal level of the identified affective state.

Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level

A mixed effects model, enhanced by a Kronecker product structure for the residual variance-covariance matrix, is used in conjunction with a discriminant analysis technique, to devise a new statistical classification method on incomplete doubly multivariate data. The proposed method is efficient in small scale clinical trials that use relatively few patients. The new classification method is also applied to multiply imputed data sets. The misclassification error rates (MERs) are compared in order to investigate the effectiveness of the new classification rule on an incomplete data set. The classification method is applied to a real data set. The error rates on the incomplete data set are found to be much less than the median error rate on the multiply imputed data sets. Non-parametric methods, such as kernel method and k-nearest neighbourhood method, are also applied to multiply imputed data sets. Results illustrating the advantages of the new classification method over classic non-parametric classification methods are presented.

A new classification rule for incomplete doubly multivariate data using mixed effects model with performance comparisons on the imputed data

In previous research, scientists were able to use transient facial thermal features extracted from
Thermal Infra-Red Images (TIRIs) for making binary distinction between the affective states.
For example, thermal asymmetries localised in facial TIRIs have been used to distinguish
anxiety and deceit. Since affective human-computer interaction would require machines to
distinguish between the subtle facial expressions of affective states, computers’ able to make
such binary distinctions would not suffice a robust human-computer interaction. This work, for
the first time, uses affective-state-specific transient facial thermal features extracted from TIRIs
to recognise a much wider range of facial expressions under a much wider range of conditions.
Using infrared thermal imaging within the 8-14 μm, a database of 324 discrete, time-sequential,
visible-spectrum and thermal facial images was acquired, representing different facial
expressions from 23 participants in different situations. A facial thermal feature extraction and
pattern classification approach was developed, refined and tested on various Gaussian mixture
models constructed using the image database. Attempts were made to classify: neutral and
pretended happy and sad faces; multiple positive and negative facial expressions; six
(pretended) basic facial expressions; partially covered or occluded faces; and faces with evoked
happiness, sadness, disgust and anger.
The cluster-analytic classification in this work began by segmentation and detection of
thermal faces in the acquired TIRIs. The affective-state-specific temperature distributions on the
facial skin surface were realised through the pixel grey-level analysis. Examining the affectivestate-
specific temperature variations within the selected regions of interest in the TIRIs led to
the discovery of some significant Facial Thermal Feature Points (FTFPs) along the major facial
muscles. Following a multivariate analysis of the Thermal Intensity values (TIVs) measured at
the FTFPs, the TIRIs were represented along the Principal Components (PCs) of a covariance
matrix. The resulting PCs were ranked in the order of their effectiveness in the between-cluster
separation. Only the most effective PCs were retained to construct an optimised eigenspace. A
supervised learning algorithm was invoked for linear subdivision of the optimised eigenspace.
The statistical significance levels of the classification results were estimated for validating the
discriminant functions.
The main contribution of this research has been to show that: the infrared imaging of facial
thermal features within the 8-14 μm bandwidth may be used to observe affective-state-specific
thermal variations on the face; the pixel-grey level analysis of TIRIs can help localise FTFPs
along the major facial muscles of the face; cluster-analytic classification of transient thermal
features may help distinguish between the facial expressions of affective states in an optimized
eigenspace of input thermal feature vectors. The Gaussian mixture model with one cluster per
affect worked better for some facial expressions than others. This made the influence of the
Gaussian mixture model structure on the accuracy of the classification results obvious.
However, the linear discrimination and confusion patterns observed in this work were consistent
with the ones reported in several earlier studies.
This investigation also unveiled some important dimensions of the future research on use of
facial thermal features in affective human-computer interaction.

Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features

The multiple observations model is appropriate when multiple p-variate observations are available on each individual to be classified. The classical(probability density ratio)approach to this problem has been studied by Gupta and Logan(1990). This paper concerns Bayesian predictive discrimination using a diffuse prior for the mean vector and two dispersion matrices for each of two populations. Training samples are used to classify an unknown observation into one of the two populations. This extends the work of Geisser(1964)to a model that includes the earlier work as a special case. Simulation is used to demonstrate the behavior of the procedure, and a two-stage modification is proposed which guards against some undesirable behavior when the number of multiple observations becomes large.

Bayesian discrimination using multiple observations

In this paper the multiple observations, or two factor mixed hierarchical, model for the classification problem has been studied. Under this model the Bayes classification statistic and some of its properties are discussed. It will be seen that the multiple observations model inclueds the fixed effects model as a special case and bears an interesting relationship to the random effects model. The multiple observations model has a potentialk application in a variety of fields. In medicine, for example, a patient to be classified as “healthy” versus “non—healthy” might be observed on several attributes at multiple points in time. If individuals from the two populations differ in the variability of these measurements over time, this model can exploit that difference as an aid to classification. Multivariate normality is assumed throughout this paper.

A.K. Gupta

Papers

On a multiple observations model in discriminant analysis