Distracted and drowsy driving are two very common causes of car accidents as they contribute to 2.3% of all the fatalities caused on the US roads. Therefore, in the era of smart driving there is an increased need of technologies able to monitor driver's alertness and provide timely alerts to the driver. In this paper, we conduct as pilot study and we present a preliminary, yet novel multimodal dataset, collected from 10 subjects using three different modalities. Our modalities include a thermal camera, an RGB camera, and four physiological indicators. The dataset consists of two recording sessions for each subject, thus, offering in total 20 multimodal driving sessions. We propose a machine learning framework aiming to investigate the hypothesis that multimodal features have higher potential towards driver alertness detection. Our dataset and analysis focus on exploring the differences between alertness and drowsiness as they intersect with the presence of different distractions. The results highlight the validity of our hypothesis and introduce interesting future directions for research.

Towards detecting levels of alertness in drivers using multiple modalities

We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims where each claim is annotated with a truthfulness label and a ruling statement, and 33,880 textual paragraphs and 12,112 images in total as evidence. To establish baseline performances on Mocheg, we experiment with several state-of-the-art neural architectures on the three pipelined subtasks: multimodal evidence retrieval, claim verification, and explanation generation, and demonstrate that the performance of the state-of-the-art end-to-end multimodal fact-checking does not provide satisfactory outcomes. To the best of our knowledge, we are the first to build the benchmark dataset and solutions for end-to-end multimodal fact-checking and explanation generation. The dataset, source code and model checkpoints are available at https://github.com/VT-NLP/Mocheg.

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

Several studies applying Machine Learning to deception detection have been published in the last decade. A rich and complex set of settings, approaches, theories, and results is now available. Therefore, one may find it difficult to identify trends, successful paths, gaps, and opportunities for contribution. The present literature review aims to provide the state of research regarding deception detection with Machine Learning. We followed the PRISMA protocol and retrieved 648 articles from ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. 540 of them were screened (108 were duplicates). A final corpus of 81 documents has been summarized as mind maps. Metadata was extracted and has been encoded as Python dictionaries to support a statistical analysis scripted in Python programming language, and available as a collection of Jupyter Lab Notebooks in a GitHub repository. All are available as Jupyter Lab Notebooks. Neural Networks, Support Vector Machines, Random Forest, Decision Tree and K-nearest Neighbor are the five most explored techniques. The studies report a detection performance ranging from 51% to 100%, with 19 works reaching accuracy rate above 0.9. Monomodal, Bimodal, and Multimodal approaches were exploited and achieved various accuracy levels for detection. Bimodal and Multimodal approaches have become a trend over Monomodal ones, although there are high-performance examples of the latter. Studies that exploit language and linguistic features, 75% are dedicated to English. The findings include observations of the following: language and culture, emotional features, psychological traits, cognitive load, facial cues, complexity, performance, and Machine Learning topics. We also present a dataset benchmark. Main conclusions are that labeled datasets from real-life data are scarce. Also, there is still room for new approaches for deception detection with Machine Learning, especially if focused on languages and cultures other than English-based. Further research would greatly contribute by providing new labeled and multimodal datasets for deception detection, both for English and other languages.

/pdf/deception-detection-with-machine-learning-a-systematic-2zkebx7f.pdf

Deception detection with machine learning: A systematic review and statistical analysis

Significant research is currently carried out with a focus on autonomous vehicles; research is starting to focus on areas such as the modeling of occupant states and behavioral elements. This paper contributes to this line of research by developing a pipeline that extracts physiological signals from thermal imagery and modeling occupant enervation using a fully non-contact based approach. These signals are obtained via a multimodal dataset of 36 subjects across multiple channels, including the thermal and physiological modalities. Moreover, we provide a comparative analysis of non-contact and contact based channels to model the enervation state of individuals. Our analysis indicates that non-contact physiological signals extracted from thermal imagery can reach and exceed the performance of contact-based physiological signals. In addition, modeling of enervation is possible using said non-contact physiological signals and thermal features, with an accuracy of up to 70% in identifying energized and enervated occupant states. Our findings provide a novel approach for future research and opens the possibility for integration of unrestrictive sensors in future automobiles. 

Non-Contact Based Modeling of Enervation

In this paper, we consider the possible set of modalities for UAVs systems for finding people in case of emergency. Modalities allow to identify and monitor the physiological state of a found person. We consider modalities from acoustic and visual communication channels such as speech, facial structure and skin temperature. We also outline the possible signals of input modalities and describe the interchange format for them.

Analysis of Multimodal Information for Multi-robot System

Political statements are carefully crafted to garner public support for a particular ideology. These statements are often biased and sometimes misleading. Separating fact from fiction has proven to be a difficult task, generally accomplished by cross-checking political statements against an impartial and trustworthy news source. In this article, we make three contributions. First, we compile a novel multimodal dataset, which consists of 180 videos with accompanying audio recordings and transcripts, featuring 88 politicians categorized by political party. To our knowledge, this is the second multimodal deception detection dataset from real-life data and the first in the political field. Second, we extract features from the linguistic, visual, and acoustic modalities to develop a system capable of discriminating between truthful and deceptive political statements. Finally, we perform an extensive analysis on different multimodal features to identify the behavioral patterns used by politicians when it comes to deception.

Multimodal Political Deception Detection

Physiological signals provide a reliable method to identify the physical and mental state of a person at any given point in time. Accordingly, there is a myriad of techniques used to extract physiological signals from the human body. However, these techniques often require direct contact with the body. This demands the cooperation of the individual as well as the human effort required to connect devices and collect measurements.In this paper, we propose reliable, non-contact based methods for extracting respiration rate and heart rate from thermal images using a large dataset of human thermal recordings. These methods leverage a combination of image and signal processing techniques in order to extract and filter physiological signals from the thermal domain. Our results evidently show that features extracted from thermal images highly correlate with the ground truth measurements as well as indicate the feasibility of developing non-contact based methods to extract physiological signals.

A Non-contact Method for Extracting Heart and Respiration Rates

Multiple techniques are used to extract physiological signals from the human body. These signals provide a reliable method to identify the physical and mental state of a person at any given point in time. However, these techniques require contact and cooperation of the individual as well as human effort for connecting the devices and collecting the needed measurement. Moreover, these methods can be invasive, time-consuming, and infeasible in many cases. Recent efforts have been made in order to find alternatives to extract these measurements using non-contact and efficient techniques. In this paper we provide a survey that explores different approaches for extracting vital signs from thermal images as well as review applications that could potentially leverage these techniques.

A Survey on Extracting Physiological Measurements from Thermal Images

Physiological signals provide a reliable method to identify the physical and mental state of a person at any given point in time. Multiple techniques are used to extract physiological signals from the human body. However, these techniques require contact and cooperation of the individual as well as human effort for connecting the devices and collecting the needed measurement. Thermal imaging provides a non-contact approach for acquiring these signals. New applications are exploring ways to utilize physiological features extracted from thermal images to detect subtle changes in human physiology. In this paper, we provide a review of applications, which propose a variety of innovative techniques to model human behavior by analyzing thermal videos.

Christian Hessler

Papers

Multimodal Political Deception Detection

A Non-contact Method for Extracting Heart and Respiration Rates

A Survey on Extracting Physiological Measurements from Thermal Images

Using Thermal Images and Physiological Features to Model Human Behavior: A Survey