Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This monograph attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.

/pdf/computer-vision-for-autonomous-vehicles-problems-datasets-557e7liccp.pdf

Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

In the last few decades, Structure from Motion (SfM) and visual Simultaneous Localization and Mapping (visual SLAM) techniques have gained significant interest from both the computer vision and robotic communities. Many variants of these techniques have started to make an impact in a wide range of applications, including robot navigation and augmented reality. However, despite some remarkable results in these areas, most SfM and visual SLAM techniques operate based on the assumption that the observed environment is static. However, when faced with moving objects, overall system accuracy can be jeopardized. In this article, we present for the first time a survey of visual SLAM and SfM techniques that are targeted toward operation in dynamic environments. We identify three main problems: how to perform reconstruction (robust visual SLAM), how to segment and track dynamic objects, and how to achieve joint motion segmentation and reconstruction. Based on this categorization, we provide a comprehensive taxonomy of existing approaches. Finally, the advantages and disadvantages of each solution class are critically discussed from the perspective of practicality and robustness.

Visual SLAM and Structure from Motion in Dynamic Environments: A Survey

This work investigates the estimation of dense three-dimensional motion fields, commonly referred to as scene flow. While great progress has been made in recent years, large displacements and adverse imaging conditions as observed in natural outdoor environments are still very challenging for current approaches to reconstruction and motion estimation. In this paper, we propose a unified random field model which reasons jointly about 3D scene flow as well as the location, shape and motion of vehicles in the observed scene. We formulate the problem as the task of decomposing the scene into a small number of rigidly moving objects sharing the same motion parameters. Thus, our formulation effectively introduces long-range spatial dependencies which commonly employed local rigidity priors are lacking. Our inference algorithm then estimates the association of image segments and object hypotheses together with their three-dimensional shape and motion. We demonstrate the potential of the proposed approach by introducing a novel challenging scene flow benchmark which allows for a thorough comparison of the proposed scene flow approach with respect to various baseline models. In contrast to previous benchmarks, our evaluation is the first to provide stereo and optical flow ground truth for dynamic real-world urban scenes at large scale. Our experiments reveal that rigid motion segmentation can be utilized as an effective regularizer for the scene flow problem, improving upon existing two-frame scene flow methods. At the same time, our method yields plausible object segmentations without requiring an explicitly trained recognition model for a specific object class.

Object Scene Flow

КРИЗИС ВЛАСТИ В КАБАРДИНО-БАЛКАРИИ В АВГУСТЕ 1991 ГОДА Статья посвящена исследованию влияния путча 1991 г. на этнополитическую обстановку в Кабардино-Балкарской Республике. Анализируется процесс "включения" национального фактора в разгоревшееся политическое противоборство, вылившееся в кризис власти. Авторами впервые сформулирована и обоснована мысль о том, что события августа 1991 г. привели к объединению к тому времени уже нелояльных по отношению друг к другу национальных движений "титульных" народов Кабардино-Балкарии, что нашло выражение в выдвижении общего требования отставки первых лиц республиканской власти. Адрес статьи: www.gramota.net/materials/3/2017/9/58.html

1991년도 사업경과 보고

The fusion of light detection and ranging (LiDAR) and camera data in real-time is known to be a crucial process in many applications, such as in autonomous driving, industrial automation, and robotics. Especially in the case of autonomous vehicles, the efficient fusion of data from these two types of sensors is important to enabling the depth of objects as well as the detection of objects at short and long distances. As both the sensors are capable of capturing the different attributes of the environment simultaneously, the integration of those attributes with an efficient fusion approach greatly benefits the reliable and consistent perception of the environment. This paper presents a method to estimate the distance (depth) between a self-driving car and other vehicles, objects, and signboards on its path using the accurate fusion approach. Based on the geometrical transformation and projection, low-level sensor fusion was performed between a camera and LiDAR using a 3D marker. Further, the fusion information is utilized to estimate the distance of objects detected by the RefineDet detector. Finally, the accuracy and performance of the sensor fusion and distance estimation approach were evaluated in terms of quantitative and qualitative analysis by considering real road and simulation environment scenarios. Thus the proposed low-level sensor fusion, based on the computational geometric transformation and projection for object distance estimation proves to be a promising solution for enabling reliable and consistent environment perception ability for autonomous vehicles.

LiDAR and Camera Fusion Approach for Object Distance Estimation in Self-Driving Vehicles

This paper presents a mobile object detection algorithm which performs with two consecutive stereo images. Like most motion detection methods, the proposed one is based on dense stereo matching and optical flow (OF) estimation. Noting that the main computational cost of existing methods is related to the estimation of OF, we propose to use a fast algorithm based on Lucas–Kanade paradigm. We then derive a comprehensive uncertainty model by taking into account all the estimation errors occurring during the process. In contrast with most previous works, we rigorously expand the error related to vision based ego-motion estimation. Finally, we present a comparative study of performance on the challenging KITTI dataset which demonstrates the effectiveness of the proposed approach.

https://hal.archives-ouvertes.fr/hal-01393423/document

Moving Object Detection in Real-Time Using Stereo from a Mobile Platform

Estimating the optical flow robustly in real-time is still a challenging issue as revealed by current KITTI benchmarks. We propose an original two-step method for fast and performant optical flow estimation from stereo vision. The first step is the prediction of the flow due to the ego-motion, efficiently conducted by stereo-matching and visual odometry. The correction step estimates the motion of mobile objects. Algorithmic choices are justified by empirical studies on real datasets. Our method achieves framerate processing on images of realistic size, and provides results comparable or better than methods having computation times one or two orders of magnitude higher.

A Prediction-Correction Approach for Real-Time Optical Flow Computation Using Stereo

This paper considers passive vision for robotics and focuses on devising a real-time process for moving object detection using a stereo rig. As several previous works, our method relies on the use of dense stereo and of optical flow. Observing that the main computational load of existing methods is related to the estimation of the optical flow, we propose to use a fast algorithm based on Lucas-Kanade's paradigm. We derive a new uncertainty model which explicitly takes into account all errors originating from each estimation step of the process. In contrast with most previous works, we describe a rigorous expansion of the error related to vision based ego-motion estimation. Finally, we present a comparative study of performance on the KITTI dataset, which demonstrates the effectiveness of the proposed approach.

/pdf/real-time-mobile-object-detection-using-stereo-3i5tlar9xo.pdf

Real-time mobile object detection using stereo

L'objectif de cette these est de concevoir un systeme de perception stereoscopique embarque, permettant une navigation robotique autonome en environnement dynamique (i.e. comportant des objets mobiles). Pour cela, nous nous sommes impose plusieurs contraintes : 1) Puisque l'on souhaite pouvoir naviguer en terrain inconnu et en presence de tout type d'objets mobiles, nous avons adopte une approche purement geometrique. 2) Pour assurer une couverture maximale du champ visuel nous avons choisi d'employer des methodes d'estimation denses qui traitent chaque pixel de l'image. 3) Puisque les algorithmes utilises doivent pouvoir s'executer en embarque sur un robot, nous avons attache le plus grand soin a selectionner ou concevoir des algorithmes particulierement rapides, pour nuire au minimum a la reactivite du systeme. La demarche presentee dans ce manuscrit et les contributions qui sont faites sont les suivantes. Dans un premier temps, nous etudions plusieurs algorithmes d’appariement stereo qui permettent d'estimer une carte de disparite dont on peut deduire, par triangulation, une carte de profondeur. Grâce a cette evaluation nous mettons en evidence un algorithme qui ne figure pas sur les benchmarks KITTI, mais qui offre un excellent compromis precision/temps de calcul. Nous proposons egalement une methode pour filtrer les cartes de disparite. En codant ces algorithmes en CUDA pour profiter de l’acceleration des calculs sur cartes graphiques (GPU), nous montrons qu’ils s’executent tres rapidement (19ms sur les images KITTI, sur GPU GeForce GTX Titan).Dans un deuxieme temps, nous souhaitons percevoir les objets mobiles et estimer leur mouvement. Pour cela nous calculons le deplacement du banc stereo par odometrie visuelle pour pouvoir isoler dans le mouvement apparent 2D ou 3D (estime par des algorithmes de flot optique ou de flot de scene) la part induite par le mouvement propre a chaque objet. Partant du constat que seul l'algorithme d'estimation du flot optique FOLKI permet un calcul en temps-reel, nous proposons plusieurs modifications de celui-ci qui ameliorent legerement ses performances au prix d'une augmentation de son temps de calcul. Concernant le flot de scene, aucun algorithme existant ne permet d'atteindre la vitesse d'execution souhaitee, nous proposons donc une nouvelle approche decouplant structure et mouvement pour estimer rapidement le flot de scene. Trois algorithmes sont proposes pour exploiter cette decomposition structure-mouvement et l’un d’eux, particulierement efficace, permet d'estimer tres rapidement le flot de scene avec une precision relativement bonne. A notre connaissance, il s'agit du seul algorithme publie de calcul du flot de scene capable de s'executer a cadence video sur les donnees KITTI (10Hz).Dans un troisieme temps, pour detecter les objets en mouvement et les segmenter dans l'image, nous presentons differents modeles statistiques et differents residus sur lesquels fonder une detection par seuillage d'un critere chi2. Nous proposons une modelisation statistique rigoureuse qui tient compte de toutes les incertitudes d'estimation, notamment celles de l'odometrie visuelle, ce qui n'avait pas ete fait a notre connaissance dans le contexte de la detection d'objets mobiles. Nous proposons aussi un nouveau residu pour la detection, en utilisant la methode par prediction d’image qui permet de faciliter la propagation des incertitudes et l'obtention du critere chi2. Le gain apporte par le residu et le modele d'erreur proposes est demontre par une evaluation des algorithmes de detection sur des exemples tires de la base KITTI. Enfin, pour valider experimentalement notre systeme de perception en embarque sur une plateforme robotique, nous implementons nos codes sous ROS et certains codes en CUDA pour une acceleration sur GPU. Nous decrivons le systeme de perception et de navigation utilise pour la preuve de concept qui montre que notre systeme de perception, convient a une application embarquee.

Maxime Derome

Papers

Moving Object Detection in Real-Time Using Stereo from a Mobile Platform

A Prediction-Correction Approach for Real-Time Optical Flow Computation Using Stereo

Real-time mobile object detection using stereo

Vision stéréoscopique temps-réel pour la navigation autonome d'un robot en environnement dynamique