Being bored? Recognising natural interest by extensive audiovisual integration for real-life application

TLDR: A fully automatic processing combination of Active-Appearance-Model-based facial expression, vision-based eye-activity estimation, acoustic features, linguistic analysis, non-linguistic vocalisations, and temporal context information in an early feature fusion process is introduced.

About: This article is published in Image and Vision Computing.The article was published on 2009-11-01 and is currently open access. It has received 193 citations till now. The article focuses on the topics: Affective computing.

Figures

Table 1: AV IC database recording parameters.

Table 3: Distribution of non-linguistic vocalisation by type in the AV IC database across the 5 frequently occurring types (only instances longer than 100ms).

Figure 6: Effect of the first shape model components.

Figure 3: Mean Level of Interest (LOI) histogram for the full LOI scale [-2,2] as used for regression experiments.

Figure 2: Distribution of the Level of Interest (LOI) over various labelers; LOI-distribution Inter Labeler Agreement (ILA): 316 510 170, min/mean/max Inter Labeler Agreement confidence: 0.75/0.89/1.0, κ = 0.66

Table 13: Statistical significance of the accuracy improvement for the different modality combinations (rows) when compared to the accuracy of other modality combinations (columns) at a significance level α of 0.05 (SVM classification, balanced training). “+”: accuracy of the modality combination (row) significantly better than performance of the modality combination in the corresponding column; “o”: no statistical significant difference in performance; “-”: accuracy of the modality combination (row) worse than accuracy of the modality combination in the corresponding column; “*”: late fusion; “dummy”: a single dummy feature for chance reference resulting in picking the majority class (LOI1) at any time; “F”: facial expression; “E”: eye activity; “L”: linguistic information; “A”: acoustics; “C”: contextual information

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Being bored? recognising natural interest by extensive audiovisual integration for real-life application" ?

Herein the authors introduce a fully automatic processing combination of Active-Appearance-Model-based facial expression, vision-based eyeactivity estimation, acoustic features, linguistic analysis, non-linguistic vocalisations, and temporal context information in an early feature fusion process. The authors provide detailed subject-independent results for classification and regression of the Level of Interest using Support-Vector Machines on an audiovisual interest corpus ( AV IC ) consisting of spontaneous, conversational speech demonstrating “ theoretical ” effectiveness of the approach. Further, to evaluate the approach with regards to real-life usability a user-study is conducted for proof of “ practical ” effectiveness. 

Q2. What are the future works mentioned in the paper "Being bored? recognising natural interest by extensive audiovisual integration for real-life application" ?

Future works will have to deal with improved discrimination of the subtle difference of the border class between strong interest and boredom. Automatically noticing such events and performance with automatic modalitiy selection will be one future research issue. Also, in this respect more instances of strongly expressed boredom should be recorded in future efforts to broaden the scope of use-cases: in the face-to-face communication captured herein, these did not occur sufficiently often - potentially due to subject ’ s minimum politeness. For many applications detection of boredom or high interest moments may be sufficient. 

Q3. What is the first step of building an active appearance model?

The first step of building an Active Appearance Model is the independent application of a Principal Component Analysis to the aligned and normalised shapes in S and the shape-free textures in T, thus generating a shape and a texture model. 

Q4. How is the spotting of non-linguistic vocalisations achieved?

For spotting of non-linguistic vocalisations in the first decoding pass as described in the previous section with best parameters a recall rate of 55% and a precision rate of 46% is achieved. 

Q5. What is the main reason for the face analysis system?

Their face analysis system is capable of such pattern recognition tasks due to multiple evaluations of the influence of algorithmic parameters and their optimisation. 

Q6. What is the importance of considering non-linguistic vocalisations for correct recognition of spontaneous speech?

considering non-linguistic vocalisations is important for correct recognition of spontaneous speech since they are an essential part of natural speech and also carry meaningful information [56, 57, 58]. 

Q7. How is the level of interest integrated into the feature space?

contextual interest information is integrated in the feature space by using the last estimate of the Level of Interest as feature. 

Q8. What is the downside of the regression approach?

The higher resolution of the regression approach (providing “inbetween” LOI values such as 1.5) has the downside of yielding a slightly lower accuracy: if the authors discretise the regression output into the discrete classes {LOI0, LOI1, LOI2} and compare it with the discrete master LOI, an F1 measure of 69.1% is obtained for the optimal case of fusion of all information instead of 76.0% for the directly discrete classification. 

Q9. What is the effect of diffusion by word errors?

this diffusion by word errors also leads to fewer observations of the same terms: already at a minimum term frequency of two within the database the annotation based level overtakes. 

Q10. What was the subject asked to do?

The subject was explicitly asked not to worry about being polite to the experimenter, e.g. by always showing a certain level of “polite” attention. 

Q11. What are the vocalisations that are referred to as garbage in the ongoing?

These vocalisations are breathing, consent, coughing, hesitation, laughter, long pause, short pause, and other human noise (referred to as garbage in the ongoing). 

Q12. What is the significance of the non-linguistic vocalisation coughing?

Note that the non-linguistic vocalisation coughing could not be detected automatically (cf. sec. 2.2.5) despite its high relevance for two-fold reasons: its occurrences are mostly shorter than 100 ms, which violates their HMM topology, and too few instances are contained for reliable training - IGR does not take overall occurrence into account but measures predictive ability of e.g. coughing when it appears. 

Q13. How is the performance of the fusion of all information sources achieved?

The results presented, show that by early fusion of all information sources the maximum accuracy is obtained: a remarkable subject-independent F1-measure of 72.2% is achieved for unbalanced training. 

Q14. What topics are used in the virtual product and company tour?

Nine topics are used in a virtual product and company tour (Toyota Museum, Safety, Intelligent Transport System, Toyota Production System, Environment, Motor sports, Toyota History, Toyota Partner Robot, and Toyota Prius).