Semantic segmentation is a challenging task that addresses most of the perception needs of intelligent vehicles (IVs) in an unified way. Deep neural networks excel at this task, as they can be trained end-to-end to accurately classify multiple object categories in an image at pixel level. However, a good tradeoff between high quality and computational resources is yet not present in the state-of-the-art semantic segmentation approaches, limiting their application in real vehicles. In this paper, we propose a deep architecture that is able to run in real time while providing accurate semantic segmentation. The core of our architecture is a novel layer that uses residual connections and factorized convolutions in order to remain efficient while retaining remarkable accuracy. Our approach is able to run at over 83 FPS in a single Titan X, and 7 FPS in a Jetson TX1 (embedded device). A comprehensive set of experiments on the publicly available Cityscapes data set demonstrates that our system achieves an accuracy that is similar to the state of the art, while being orders of magnitude faster to compute than other architectures that achieve top precision. The resulting tradeoff makes our model an ideal approach for scene understanding in IV applications. The code is publicly available at: https://github.com/Eromera/erfnet

ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation

We focus on the challenging task of real-time semantic segmentation in this paper. It finds many practical applications and yet is with fundamental difficulty of reducing a large portion of computation for pixel-wise label inference. We propose an image cascade network (ICNet) that incorporates multi-resolution branches under proper label guidance to address this challenge. We provide in-depth analysis of our framework and introduce the cascade feature fusion unit to quickly achieve high-quality segmentation. Our system yields real-time inference on a single GPU card with decent quality results evaluated on challenging datasets like Cityscapes, CamVid and COCO-Stuff.

ICNet for Real-Time Semantic Segmentation on High-Resolution Images.

/pdf/icnet-for-real-time-semantic-segmentation-on-high-resolution-1lq2pmm1a5.pdf

ICNet for Real-Time Semantic Segmentation on High-Resolution Images

The goal of this paper is to generate high-quality 3D object proposals in the context of autonomous driving. Our method exploits stereo imagery to place proposals in the form of 3D bounding boxes. We formulate the problem as minimizing an energy function encoding object size priors, ground plane as well as several depth informed features that reason about free space, point cloud densities and distance to the ground. Our experiments show significant performance gains over existing RGB and RGB-D object proposal methods on the challenging KITTI benchmark. Combined with convolutional neural net (CNN) scoring, our approach outperforms all existing results on all three KITTI object classes.

/pdf/3d-object-proposals-for-accurate-object-class-detection-t06ijhccg7.pdf

3D object proposals for accurate object class detection

Perception in autonomous vehicles is often carried out through a suite of different sensing modalities. Given the massive amount of openly available labeled RGB data and the advent of high-quality deep learning algorithms for image-based recognition, high-level semantic perception tasks are pre-dominantly solved using high-resolution cameras. As a result of that, other sensor modalities potentially useful for this task are often ignored. In this paper, we push the state of the art in LiDAR-only semantic segmentation forward in order to provide another independent source of semantic information to the vehicle. Our approach can accurately perform full semantic segmentation of LiDAR point clouds at sensor frame rate. We exploit range images as an intermediate representation in combination with a Convolutional Neural Network (CNN) exploiting the rotating LiDAR sensor model. To obtain accurate results, we propose a novel post-processing algorithm that deals with problems arising from this intermediate representation such as discretization errors and blurry CNN outputs. We implemented and thoroughly evaluated our approach including several comparisons to the state of the art. Our experiments show that our approach outperforms state-of-the-art approaches, while still running online on a single embedded GPU. The code can be accessed at https://github.com/PRBonn/lidar-bonnetal.

RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation

We propose a system for performing structural change detection in street-view videos captured by a vehicle-mounted monocular camera over time. Our approach is motivated by the need for more frequent and efficient updates in the large-scale maps used in autonomous vehicle navigation. Our method chains a multi-sensor fusion SLAM and fast dense 3D reconstruction pipeline, which provide coarsely registered image pairs to a deep Deconvolutional Network (DN) for pixel-wise change detection. We investigate two DN architectures for change detection, the first one is based on the idea of stacking contraction and expansion blocks while the second one is based on the idea of Fully Convolutional Networks. To train and evaluate our networks we introduce a new urban change detection dataset which is an order of magnitude larger than existing datasets and contains challenging changes due to seasonal and lighting variations. Our method outperforms existing literature on this dataset, which we make available to the community, and an existing panoramic change detection dataset, demonstrating its wide applicability.

Street-view change detection with deconvolutional networks

This paper presents DriveSafe, a new driver safety app for iPhones that detects inattentive driving behaviors and gives corresponding feedback to drivers, scoring their driving and alerting them in case their behaviors are unsafe. It uses computer vision and pattern recognition techniques on the iPhone to assess whether the driver is drowsy or distracted using the rear-camera, the microphone, the inertial sensors and the GPS. We present the general architecture of DriveSafe and evaluate its performance using data from 12 drivers in two different studies. The first one evaluates the detection of some inattentive driving behaviors obtaining an overall precision of 82% at 92% of recall. The second one compares the scores between DriveSafe vs the commercial AXA Drive app obtaining a better valuation to its operation. DriveSafe is the first app for smartphones based on inbuilt sensors able to detect inattentive behaviors evaluating the quality of the driving at the same time. It represents a new disruptive technology because, on the one hand, it provides similar ADAS features that found in luxury cars, and on the other hand, it presents a viable alternative for the “blackboxes” installed in vehicles by the insurance companies.

DriveSafe: An app for alerting inattentive drivers and scoring driving behaviors

Driving analysis is a recent topic of interest due to the growing safety concerns in vehicles. However, the lack of publicly available driving data currently limits the progress on this field. Machine learning techniques could highly enhance research, but they rely on large amounts of data which are difficult and very costly to obtain through Naturalistic Driving Studies (NDSs), resulting in limited accessibility to the general research community. Additionally, the proliferation of smartphones has provided a cheap and easy-to-deploy platform for driver behavior sensing, but existing applications do not provide open access to their data. For these reasons, this paper presents the UAH-DriveSet, a public dataset that allows deep driving analysis by providing a large amount of data captured by our driving monitoring app DriveSafe. The application is run by 6 different drivers and vehicles, performing 3 different behaviors (normal, drowsy and aggressive) on two types of roads (motorway and secondary road), resulting in more than 500 minutes of naturalistic driving with its associated raw data and processed semantic information, together with the video recordings of the trips. This work also introduces a tool that helps to plot the data and display the trip videos simultaneously, in order to ease data analytics. The UAH-DriveSet is available at: http:// www.robesafe.com/personal/eduardo.romera/uah-driveset

Need data for driver behaviour analysis? Presenting the public UAH-DriveSet

In this paper a new vehicle logo recognition approach is presented using Histograms of Oriented Gradients (HOG) and Support Vector Machines (SVM). The system is specifically devised to work with images supplied by traffic cameras where the logos appear with low resolution. A sliding-window technique combined with a majority voting scheme are used to provide the estimated car manufacturer. The proposed approach is assessed on a set of 3.579 vehicle images, captured by two different traffic cameras that belong to 27 distinctive vehicle manufacturers. The reported results show an overall recognition rate of 92.59%, which supports the use of the system for real applications.

/pdf/vehicle-logo-recognition-in-traffic-images-using-hog-2e17teepfj.pdf

Roberto Arroyo

Papers

ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation

Street-view change detection with deconvolutional networks

DriveSafe: An app for alerting inattentive drivers and scoring driving behaviors

Need data for driver behaviour analysis? Presenting the public UAH-DriveSet

Vehicle logo recognition in traffic images using HOG features and SVM