Several hypovigilance detection systems (HDx) were developed to avoid road-side accidents due to driver fatigue. They have suffered from several limitations. Notably many of these are focused on center-head position to define an area of interest (often referred to as PERCLOS (percentage eye closure)) without considering the face occlusion problem, light illumination, and suffer poor response time. These HDx systems mostly depend on image processing, vision-based, and multisensor-based features. To address these problems, the author utilized vision, sensors, environmental, and vehicular-based features that integrated together by fusion to predict multistage of HDx. Lately, few studies have utilized the combination of multimodal features and deep learning (DL) architectures. Those multimodal-based features (M-HDx) were feasible to predict stages of driver fatigue (multi-stage). However, there is a need to critically measure the performance of these M-HDx by carrying out a comparative analysis to recognize multi-stage of fatigue in terms of hardware-based benchmarks. Moreover, it is important to evaluate the M-HDx systems using different features-set with respect to traditional and advanced machine learning techniques. Therefore, the primary aim of this work is in algorithm and feature modeling, then compare the advantages and differences with other work. In this paper, a different study is conducted compare to state-of-the-art survey articles by statistically measuring the performance. After experiments on M-HDx systems, this paper concludes that there is still a research gap to real-time development of multistage M-HDx systems. In the end, the paper summarizes the directions, challenges, and applications in the development of HDx systems to assist other researchers for further research.

/pdf/a-methodological-review-on-prediction-of-multi-stage-19ppxu0q2x.pdf

A Methodological Review on Prediction of Multi-Stage Hypovigilance Detection Systems Using Multimodal Features

Considering the ever-growing presence of automobiles around the world, ensuring the safety of those on and near roadways is of great importance. From the causes of accidents, drowsiness and distractedness are among the most consequential. In this paper, we use a multimodal dataset consisting of 11 recorded channels over 45 subjects to model driver’s drowsiness and distraction. Our work puts forward the application of this dataset by using segmented windows as features, resulting in four main contributions. We explore the performance of each individual modality and specify which signals and features have a better capability of detecting drowsiness and different kinds of distractions. In addition, we analyze the effects of early fusion on the classification of the driver’s state using multiple physiological and thermal channels. Finally, we use cascaded late fusion and test three voting strategies to evaluate the performance of our proposed approach. Our results confirm the effectiveness of utilizing a multimodal approach in detecting both drowsiness and distraction as two separate factors influencing the driver and provide guidelines on which signals are appropriate for detecting different driver’s states.

Multimodal Detection of Drivers Drowsiness and Distraction

We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims where each claim is annotated with a truthfulness label and a ruling statement, and 33,880 textual paragraphs and 12,112 images in total as evidence. To establish baseline performances on Mocheg, we experiment with several state-of-the-art neural architectures on the three pipelined subtasks: multimodal evidence retrieval, claim verification, and explanation generation, and demonstrate that the performance of the state-of-the-art end-to-end multimodal fact-checking does not provide satisfactory outcomes. To the best of our knowledge, we are the first to build the benchmark dataset and solutions for end-to-end multimodal fact-checking and explanation generation. The dataset, source code and model checkpoints are available at https://github.com/VT-NLP/Mocheg.

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we experiment with visual and physiological information and explore the potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual and physiological groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the two modalities on identifying different types of driving distractions.

Understanding Driving Distractions: A Multimodal Analysis on Distraction Characterization

https://www.techscience.com/cmc/v71n1/45469/pdf

Hypo-Driver: A Multiview Driver Fatigue and Distraction Level Detection System

Distracted and drowsy driving are two very common causes of car accidents as they contribute to 2.3% of all the fatalities caused on the US roads. Therefore, in the era of smart driving there is an increased need of technologies able to monitor driver's alertness and provide timely alerts to the driver. In this paper, we conduct as pilot study and we present a preliminary, yet novel multimodal dataset, collected from 10 subjects using three different modalities. Our modalities include a thermal camera, an RGB camera, and four physiological indicators. The dataset consists of two recording sessions for each subject, thus, offering in total 20 multimodal driving sessions. We propose a machine learning framework aiming to investigate the hypothesis that multimodal features have higher potential towards driver alertness detection. Our dataset and analysis focus on exploring the differences between alertness and drowsiness as they intersect with the presence of different distractions. The results highlight the validity of our hypothesis and introduce interesting future directions for research.

Towards detecting levels of alertness in drivers using multiple modalities

Political statements are carefully crafted to garner public support for a particular ideology. These statements are often biased and sometimes misleading. Separating fact from fiction has proven to be a difficult task, generally accomplished by cross-checking political statements against an impartial and trustworthy news source. In this article, we make three contributions. First, we compile a novel multimodal dataset, which consists of 180 videos with accompanying audio recordings and transcripts, featuring 88 politicians categorized by political party. To our knowledge, this is the second multimodal deception detection dataset from real-life data and the first in the political field. Second, we extract features from the linguistic, visual, and acoustic modalities to develop a system capable of discriminating between truthful and deceptive political statements. Finally, we perform an extensive analysis on different multimodal features to identify the behavioral patterns used by politicians when it comes to deception.

Multimodal Political Deception Detection

Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we performed experiments with visual, physiological, and thermal information to explore potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual, physiological, and thermal groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the three modalities on identifying different types of driving distractions.

Kais Riani

Papers

Towards detecting levels of alertness in drivers using multiple modalities

Multimodal Detection of Drivers Drowsiness and Distraction

Multimodal Political Deception Detection

Understanding Driving Distractions: A Multimodal Analysis on Distraction Characterization

Detection and Recognition of Driver Distraction Using Multimodal Signals