HAMS: Driver and Driving Monitoring using a Smartphone
15 Oct 2018-pp 840-842
TL;DR: The objective of HAMS is to provide ADAS-like functionality with low-cost devices that can be retrofitted onto the large installed base of vehicles that lack specialized and expensive sensors such as LIDAR and RADAR.
Abstract: Road safety is a major public health issue the world over. Many studies have found that the primary factors responsible for road accidents center on the driver and her/his driving. Hence, there is the need to monitor driver's state and her/his driving, with a view to providing effective feedback. Our proposed demo is of HAMS, a windshield-mounted, smartphone-based system that uses the front camera to monitor the driver and back camera to monitor her/his driving behaviour. The objective of HAMS is to provide ADAS-like functionality with low-cost devices that can be retrofitted onto the large installed base of vehicles that lack specialized and expensive sensors such as LIDAR and RADAR. Our demo would show HAMS in action on an Android smartphone to monitor the state of the driver, specifically such as drowsiness, distraction and gaze, and vehicle ranging, lane detection running on pre-recorded videos from drives.
Citations
More filters
TL;DR: In this paper, the authors present a framework for detecting the three main distraction detection approaches: manual distraction, visual distraction, and cognitive distraction, which can also combine different approaches for higher detection quality.
Abstract: Driver inattention and distraction are the main causes of road accidents, many of which result in fatalities. To reduce road accidents, the development of information systems to detect driver inattention and distraction is essential. Currently, distraction detection systems for road vehicles are not yet widely available or are limited to specific causes of driver inattention such as driver fatigue. Despite the increasing automation of driving due to the availability of increasingly sophisticated assistance systems, the human driver will continue to play a longer role as supervisor of vehicle automation. With this in mind, we review the published scientific literature on driver distraction detection methods and integrate the identified approaches into a holistic framework that is the main contribution of the paper. Based on published scientific work, our driver distraction detection framework contains a structured summary of reviewed approaches for detecting the three main distraction detection approaches: manual distraction, visual distraction, and cognitive distraction. Our framework visualizes the whole detection information chain from used sensors, measured data, computed data, computed events, inferred behavior, and inferred distraction type. Besides providing a sound summary for researchers interested in distracted driving, we discuss several practical implications for the development of driver distraction detection systems that can also combine different approaches for higher detection quality. We think our research can be useful despite - or even because of - the great developments in automated driving.
42 citations
TL;DR: In this article, the authors proposed a short survey on the use of smartphone sensors in the detection of various kinds of anomalies in several fields namely environment, agriculture, healthcare and road/traffic conditions.
Abstract: With smartphones being so ubiquitous, more connected and largely fitted with several types of sensors such as GPS, microphones, cameras, magnetometers, accelerometers, etc; there is an increasing opportunity in the development of smartphone-based sensor systems. In this article, we propose a short survey on the use of smartphone sensors in the detection of various kinds of anomalies in several fields namely environment, agriculture, healthcare and road/traffic conditions. We also list the main advantages and limitations of the use of smartphone sensors systems in such fields.
36 citations
TL;DR: A novel concept of a driver digital twin (DDT) is proposed in this study to bridge the gap between existing automated driving systems and fully digitized ones and aid in the development of a complete driving human cyber-physical system (H-CPS).
Abstract: Digital Twin (DT) is an emerging technology and has been introduced into intelligent driving and transportation systems to digitize and synergize connected automated vehicles. However, existing studies focus on the design of the automated vehicle, whereas the digitization of the human driver, who plays an important role in driving, is largely ignored. Furthermore, previous driver-related tasks are limited to specific scenarios and have limited applicability. Thus, a novel concept of a driver digital twin (DDT) is proposed in this study to bridge the gap between existing automated driving systems and fully digitized ones and aid in the development of a complete driving human cyber-physical system (H-CPS). This concept is essential for constructing a harmonious human-centric intelligent driving system that considers the proactivity and sensitivity of the human driver. The primary characteristics of the DDT include multimodal state fusion, personalized modeling, and time variance. Compared with the original DT, the proposed DDT emphasizes on internal personality and capability with respect to the external physiological-level state. This study systematically illustrates the DDT and outlines its key enabling aspects. The related technologies are comprehensively reviewed and discussed with a view to improving them by leveraging the DDT. In addition, the potential applications and unsettled challenges are considered. This study aims to provide fundamental theoretical support to researchers in determining the future scope of the DDT system
27 citations
01 May 2019
TL;DR: This paper presents AutoRate, a system that leverages front camera of a windshield-mounted smartphone to monitor driver’s attention by combining several features that derive a driver attention rating by fusing spatio-temporal features based on the driver state and behavior such as head pose, eye gaze, eye closure, yawns and use of cellphones.
Abstract: Driver inattention is one of the leading causes of vehicle crashes and incidents worldwide. Driver inattention includes driver fatigue leading to drowsiness and driver distraction, say due to use of cellphone or rubbernecking, all of which leads to a lack of situational awareness. Hitherto, techniques presented to monitor driver attention evaluated factors such as fatigue and distraction independently. However, in order to develop a robust driver attention monitoring system all the factors affecting driver’s attention needs to be analyzed holistically. In this paper, we present AutoRate, a system that leverages front camera of a windshield-mounted smartphone to monitor driver’s attention by combining several features. We derive a driver attention rating by fusing spatio-temporal features based on the driver state and behavior such as head pose, eye gaze, eye closure, yawns, use of cellphones, etc.We perform extensive evaluation of AutoRateon real-world driving data and also data from controlled, static vehicle settings with 30 drivers in a large city. We compare AutoRate’s automatically-generated rating with the scores given by 5 human annotators. Further, we compute the agreement between AutoRate’s rating and human annotator rating using kappa coefficient. AutoRate’s automatically-generated rating has an overall agreement of 0.87 with the ratings provided by 5 human annotators on the static dataset.
26 citations
Cites background or methods from "HAMS: Driver and Driving Monitoring..."
...We considered two datasets [25]: (i) Driving dataset, where we collected data from real driving scenarios, and (ii) Static dataset, where we collected data in a static vehicle setting....
[...]
...Hence, eye gaze information is important to determine where the driver is looking [25]....
[...]
...AutoRate leverages facial landmarks to detect eye closure and yawns [25]....
[...]
04 Sep 2020
TL;DR: InSight is presented, a windshield-mounted smartphone-based system that can be retrofitted to the vehicle to monitor the state of the driver, specifically driver fatigue ( based on frequent yawning and eye closure) and driver distraction (based on their direction of gaze).
Abstract: Road safety is a major public health issue across the globe and over two-thirds of the road accidents occur at nighttime under low-light conditions or darkness. The state of the driver and her/his actions are the key factors impacting road safety. How can we monitor these in a cost-effective manner and in low-light conditions? RGB cameras present in smartphones perform poorly in low-lighting conditions due to lack of information captured. Hence, existing monitoring solutions rely upon specialized hardware such as infrared cameras or thermal cameras in low-light conditions, but are limited to only high-end vehicles owing to the cost of the hardware. We present InSight, a windshield-mounted smartphone-based system that can be retrofitted to the vehicle to monitor the state of the driver, specifically driver fatigue (based on frequent yawning and eye closure) and driver distraction (based on their direction of gaze). Challenges arise from designing an accurate, yet low-cost and non-intrusive system to continuously monitor the state of the driver. In this paper, we present two novel and practical approaches for continuous driver monitoring in low-light conditions: (i) Image synthesis: enabling monitoring in low-light conditions using just the smartphone RGB camera by synthesizing a thermal image from RGB with a Generative Adversarial Network, and (ii) Near-IR LED: using a low-cost near-IR (NIR) LED attachment to the smartphone, where the NIR LED acts as a light source to illuminate the driver's face, which is not visible to the human eyes, but can be captured by standard smartphone cameras without any specialized hardware. We show that the proposed techniques can capture the driver's face accurately in low-lighting conditions to monitor driver's state. Further, since NIR and thermal imagery is significantly different than RGB images, we present a systematic approach to generate labelled data, which is used to train existing computer vision models. We present an extensive evaluation of both the approaches with data collected from 15 drivers in controlled basement area and on real roads in low-light conditions. The proposed NIR LED setup has an accuracy (Fl-score) of 85% and 93.8% in detecting driver fatigue and distraction, respectively in low-light.
20 citations
References
More filters
Posted Content•
TL;DR: YOLO9000, a state-of-the-art, real-time object detection system that can detect over 9000 object categories, is introduced and a method to jointly train on object detection and classification is proposed, both novel and drawn from prior work.
Abstract: We introduce YOLO9000, a state-of-the-art, real-time object detection system that can detect over 9000 object categories. First we propose various improvements to the YOLO detection method, both novel and drawn from prior work. The improved model, YOLOv2, is state-of-the-art on standard detection tasks like PASCAL VOC and COCO. At 67 FPS, YOLOv2 gets 76.8 mAP on VOC 2007. At 40 FPS, YOLOv2 gets 78.6 mAP, outperforming state-of-the-art methods like Faster RCNN with ResNet and SSD while still running significantly faster. Finally we propose a method to jointly train on object detection and classification. Using this method we train YOLO9000 simultaneously on the COCO detection dataset and the ImageNet classification dataset. Our joint training allows YOLO9000 to predict detections for object classes that don't have labelled detection data. We validate our approach on the ImageNet detection task. YOLO9000 gets 19.7 mAP on the ImageNet detection validation set despite only having detection data for 44 of the 200 classes. On the 156 classes not in COCO, YOLO9000 gets 16.0 mAP. But YOLO can detect more than just 200 classes; it predicts detections for more than 9000 different object categories. And it still runs in real-time.
8,505 citations
TL;DR: It is concluded that by designing a hybrid drowsiness detection system that combines non-intusive physiological measures with other measures one would accurately determine the drowsy level of a driver.
Abstract: In recent years, driver drowsiness has been one of the major causes of road accidents and can lead to severe physical injuries, deaths and significant economic losses. Statistics indicate the need of a reliable driver drowsiness detection system which could alert the driver before a mishap happens. Researchers have attempted to determine driver drowsiness using the following measures: (1) vehicle-based measures; (2) behavioral measures and (3) physiological measures. A detailed review on these measures will provide insight on the present systems, issues associated with them and the enhancements that need to be done to make a robust system. In this paper, we review these three measures as to the sensors used and discuss the advantages and limitations of each. The various ways through which drowsiness has been experimentally manipulated is also discussed. We conclude that by designing a hybrid drowsiness detection system that combines non-intusive physiological measures with other measures one would accurately determine the drowsiness level of a driver. A number of road accidents might then be avoided if an alert is sent to a driver that is deemed drowsy.
583 citations
"HAMS: Driver and Driving Monitoring..." refers background in this paper
...We consider frequent yawning and eye closure as signs of drowsiness [6]....
[...]
01 Jan 2008
TL;DR: Prevention of road traffic injuries is best managed by a systems approach, which addresses the behavioral, road-related, and vehicle-related factors that affect the number and severity of road-traffic-related injuries.
Abstract: Road traffic injuries are a major source of death and injury worldwide, with more than 1.2 million people killed each year and estimated costs of $518 billion. Much of the burden is sustained by vulnerable road users in low- and middle-income countries and this is expected to increase, with road traffic injuries projected to become the second leading cause of death worldwide by 2020. Prevention of road traffic injuries is best managed by a systems approach, which addresses the behavioral, road-related, and vehicle-related factors that affect the number and severity of road-traffic-related injuries.
220 citations
07 Nov 2018
TL;DR: DeepLane is a system that leverages the back camera of a windshield-mounted smartphone to provide an accurate estimate of the vehicle's current lane and is implemented as an Android-app that runs at 5 fps on CPU and upto 15 fps on smart-phone's GPU and can also assist existing navigation applications with lane-level information.
Abstract: Current smartphone-based navigation applications fail to provide lane-level information due to poor GPS accuracy. Detecting and tracking a vehicle's lane position on the road assists in lane-level navigation. For instance, it would be important to know whether a vehicle is in the correct lane for safely making a turn, perhaps even alerting the driver in advance if it is not, or whether the vehicle's speed is compliant with a lane-specific speed limit. Recent efforts have used road network information and inertial sensors to estimate lane position. While inertial sensors can detect lane shifts over short windows, it would suffer from error accumulation over time. In this paper we present DeepLane, a system that leverages the back camera of a windshield-mounted smartphone to provide an accurate estimate of the vehicle's current lane. We employ a deep learning based technique to classify the vehicle's lane position. DeepLane does not depend on any infrastructure support such as lane markings and works even when there are no lane markings, a characteristic of many roads in developing regions.We perform extensive evaluation of DeepLane on real world datasets collected in developed and developing regions. DeepLane can detect vehicle's lane position with an accuracy of over 90% in both day and night conditions. We have implemented DeepLane as an Android-app that runs at 5 fps on CPU and upto 15 fps on smart-phone's GPU and can also assist existing navigation applications with lane-level information.
12 citations
"HAMS: Driver and Driving Monitoring..." refers methods in this paper
...We have built a three-way lane classifier to detect whether the vehicle is in the left, right, or middle lane [4]....
[...]