This article presents a comprehensive survey of deep learning applications for object detection and scene perception in autonomous vehicles. Unlike existing review papers, we examine the theory underlying self-driving vehicles from deep learning perspective and current implementations, followed by their critical evaluations. Deep learning is one potential solution for object detection and scene perception problems, which can enable algorithm-driven and data-driven cars. In this article, we aim to bridge the gap between deep learning and self-driving cars through a comprehensive survey. We begin with an introduction to self-driving cars, deep learning, and computer vision followed by an overview of artificial general intelligence. Then, we classify existing powerful deep learning libraries and their role and significance in the growth of deep learning. Finally, we discuss several techniques that address the image perception issues in real-time driving, and critically evaluate recent implementations and tests conducted on self-driving cars. The findings and practices at various stages are summarized to correlate prevalent and futuristic techniques, and the applicability, scalability and feasibility of deep learning to self-driving cars for achieving safe driving without human intervention. Based on the current survey, several recommendations for further research are discussed at the end of this article.

Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues

Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop.

Objective:The objective for this study was to investigate the effects of prior familiarization with takeover requests (TORs) during conditional automated driving on drivers’ initial takeover performance and automation trust.Background:System-initiated TORs are one of the biggest concerns for conditional automated driving and have been studied extensively in the past. Most, but not all, of these studies have included training sessions to familiarize participants with TORs. This makes them hard to compare and might obscure first-failure-like effects on takeover performance and automation trust formation.Method:A driving simulator study compared drivers’ takeover performance in two takeover situations across four prior familiarization groups (no familiarization, description, experience, description and experience) and automation trust before and after experiencing the system.Results:As hypothesized, prior familiarization with TORs had a more positive effect on takeover performance in the first than in a subs...

Prior Familiarization With Takeover Requests Affects Drivers’ Takeover Performance and Automation Trust

The topic of transitions in automated driving is becoming important now that cars are automated to ever greater extents. This paper proposes a theoretical framework to support and align human factors research on transitions in automated driving. Driving states are defined based on the allocation of primary driving tasks (i.e., lateral control, longitudinal control, and monitoring) between the driver and the automation. A transition in automated driving is defined as the process during which the human-automation system changes from one driving state to another, with transitions of monitoring activity and transitions of control being among the possibilities. Based on ‘Is the transition required?’, ‘Who initiates the transition?’, and ‘Who is in control after the transition?’, we define six types of control transitions between the driver and automation: (1) Optional Driver-Initiated Driver-in-Control, (2) Mandatory Driver-Initiated Driver-in-Control, (3) Optional Driver-Initiated Automation-in-Control, (4) Mandatory Driver-Initiated Automation-in-Control, (5) Automation-Initiated Driver-in-Control, and (6) Automation-Initiated Automation-in-Control. Use cases per transition type are introduced. Finally, we interpret previous experimental studies on transitions using our framework and identify areas for future research. We conclude that our framework of driving states and transitions is an important complement to the levels of automation proposed by transportation agencies, because it describes what the driver and automation are doing, rather than should be doing, at a moment of time.

/pdf/human-factors-of-transitions-in-automated-driving-a-general-bqe8v0aywt.pdf

Human factors of transitions in automated driving: A general framework and literature survey

Objective:This article provides a review of empirical studies of automated vehicle takeovers and driver modeling to identify influential factors and their impacts on takeover performance and sugges...

/pdf/toward-computational-simulations-of-behavior-during-2m8jj6tdf2.pdf

Toward Computational Simulations of Behavior During Automated Driving Takeovers: A Review of the Empirical and Modeling Literatures.

Different motor vehicle manufacturers have recently introduced assistance systems that are capable of both longitudinal and lateral vehicle control, while the driver still has to be able to take over the vehicle control at all times (so-called Partial Automation). While these systems usually allow hands-free driving only for short time periods (e.g., 10 s), there has been little research whether allowing longer time periods of hands-off driving actually has a negative impact on driving safety in situations that the automation cannot handle alone. Altogether, two partially automated assistance systems, differing in the permitted hands-off intervals (Hands-off system vs. Hands-on system, n = 20 participants per assistance condition, age 25–70 years) were implemented in the driving simulation with a realistic take-over concept. The Hands-off system is defined by having a permitted hands-off interval of 120 s, while the Hands-on system is defined by a permitted hands-off interval of 10 s. Drivers’ reactions at a functional system limit were tested under conditions of high ecological validity: while driving in a traffic jam, participants unexpectedly encountered a time-critical situation, consisting of a vehicle at standstill that appeared suddenly and required immediate action. A visual-auditory take-over request was issued to the drivers. Regardless of the hands-off interval, all participants brought the vehicle to a safe stop. In spite of a stronger brake reaction with the Hands-on system, no significant differences between assistance levels were found in brake reaction times and the criticality of the situation. The reason for this may be that most of the drivers kept contact with the steering wheel, even in the Hands-off condition. Neither age nor prior experience with ACC was found to impact the results. The study thus demonstrates that permitting longer periods of hands-off driving does not necessarily lead to performance deficits of the driver in the case of take-over situations, if a comprehensive take-over concept is implemented.

Controllability of partially automated driving functions - does it matter whether drivers are allowed to take their hands off the steering wheel?

Driving performance at lateral system limits during partially automated driving.

An imaging processing system adapted for use with a vehicle is provided in one aspect of the present invention. The system generally includes an image capturing device and an image processing device. The image capturing device is operable to capture an original image of a scene exterior to the vehicle. The captured original image is generally defined by a group of pixels. The image processing device is operable to receive the captured original image and morphologically process the captured original image to remove distortion from the captured original image to produce a final filtered image.

Fog isolation and rejection filter

A vehicle may be steered without a driver's hands being on a vehicle steering control mechanism. A presence of an object within a predetermined distance of the vehicle may be detected using data from at least one object detection sensor that provides data to at least one of a passive safety system, a lane control system, a speed control system, and a brake control system. A steering control mechanism hands-on mode can then be enabled based at least in part on the presence of the object.

Hands-off steering wheel governed by pedestrian detection

This paper is from the SAE World Congress & Exhibition, held in April 2007 in Detroit, Michigan, USA. Part of the Pedestrian Safety session, this paper describes the optical-based sensor systems that are underdevelopment for vehicle based detection and warning systems. These sensor systems are part of technological improvements developed to reduce accidents and limit injuries caused by accidents. The authors note that most sensor systems must go through a series of real world tests, a problematic issue in the case of pedestrian detection systems because of safety reasons. Therefore, simulated tests must be used. The authors describe their work, undertaken to understand the effectiveness of using virtual pedestrians as surrogates for real world pedestrian detection.

Reid Steiger

Papers

Controllability of partially automated driving functions - does it matter whether drivers are allowed to take their hands off the steering wheel?

Driving performance at lateral system limits during partially automated driving.

Fog isolation and rejection filter

Hands-off steering wheel governed by pedestrian detection

Real Time Simulation of Virtual Pedestrians for Development of Pedestrian Detection Systems