Advanced driver assistance systems (ADASs), and particularly pedestrian protection systems (PPSs), have become an active research area aimed at improving traffic safety. The major challenge of PPSs is the development of reliable on-board pedestrian detection systems. Due to the varying appearance of pedestrians (e.g., different clothes, changing size, aspect ratio, and dynamic shape) and the unstructured environment, it is very difficult to cope with the demanded robustness of this kind of system. Two problems arising in this research area are the lack of public benchmarks and the difficulty in reproducing many of the proposed methods, which makes it difficult to compare the approaches. As a result, surveying the literature by enumerating the proposals one--after-another is not the most useful way to provide a comparative point of view. Accordingly, we present a more convenient strategy to survey the different approaches. We divide the problem of detecting pedestrians from images into different processing steps, each with attached responsibilities. Then, the different proposed methods are analyzed and classified with respect to each processing stage, favoring a comparative viewpoint. Finally, discussion of the important topics is presented, putting special emphasis on the future needs and challenges.

http://www.cvc.uab.es/~asappa/publications/J__IEEE_PAMI_vol_32_Issue_7_July_2010.pdf

Survey of Pedestrian Detection for Advanced Driver Assistance Systems

Infrared and visible image fusion methods and applications: A survey

With the increasing interest in pedestrian detection, pedestrian datasets have also been the subject of research in the past decades. However, most existing datasets focus on a color channel, while a thermal channel is helpful for detection even in a dark environment. With this in mind, we propose a multispectral pedestrian dataset which provides well aligned color-thermal image pairs, captured by beam splitter-based special hardware. The color-thermal dataset is as large as previous color-based datasets and provides dense annotations including temporal correspondences. With this dataset, we introduce multispectral ACF, which is an extension of aggregated channel features (ACF) to simultaneously handle color-thermal image pairs. Multi-spectral ACF reduces the average miss rate of ACF by 15%, and achieves another breakthrough in the pedestrian detection task.

/pdf/multispectral-pedestrian-detection-benchmark-dataset-and-3nq4ppahmf.pdf

Multispectral pedestrian detection: Benchmark dataset and baseline

Thermal cameras are passive sensors that capture the infrared radiation emitted by all objects with a temperature above absolute zero. This type of camera was originally developed as a surveillance and night vision tool for the military, but recently the price has dropped, significantly opening up a broader field of applications. Deploying this type of sensor in vision systems eliminates the illumination problems of normal greyscale and RGB cameras. This survey provides an overview of the current applications of thermal cameras. Applications include animals, agriculture, buildings, gas detection, industrial, and military applications, as well as detection, tracking, and recognition of humans. Moreover, this survey describes the nature of thermal radiation and the technology of thermal cameras.

/pdf/thermal-cameras-and-applications-a-survey-4wd1xvg4uy.pdf

Thermal cameras and applications: a survey

This paper describes the recent research on the enhancement of pedestrian safety to help develop a better understanding of the nature, issues, approaches, and challenges surrounding the problem. It presents a comprehensive review of research efforts underway dealing with pedestrian safety and collision avoidance. The importance of pedestrian protection is emphasized in a global context, discussing the research programs and efforts in various countries. Pedestrian safety measures, including infrastructure enhancements and passive safety features in vehicles, are described, followed by a systematic description of active safety systems based on pedestrian detection using sensors in vehicle and infrastructure. The pedestrian detection approaches are classified according to various criteria such as the type and configuration of sensors, as well as the video cues and classifiers used in detection algorithms. It is noted that collision avoidance not only requires detection of pedestrians but also requires collision prediction using pedestrian dynamics and behavior analysis. Hence, this paper includes research dealing with probabilistic modeling of pedestrian behavior for predicting collisions between pedestrians and vehicles.

https://cvrr.ucsd.edu/publications/2007/TGandhi_ITS07_pedestrian.pdf

Pedestrian Protection Systems: Issues, Survey, and Challenges

This paper presents an analysis of color-, infrared-, and multimodal-stereo approaches to pedestrian detection. We design a four-camera experimental testbed consisting of two color and two infrared cameras for capturing and analyzing various configuration permutations for pedestrian detection. We incorporate this four-camera system in a test vehicle and conduct comparative experiments of stereo-based approaches to obstacle detection using unimodal color and infrared imageries. A detailed analysis of the color and infrared features used to classify detected obstacles into pedestrian regions is used to motivate the development of a multimodal solution to pedestrian detection. We propose a multimodal trifocal framework consisting of a stereo pair of color cameras coupled with an infrared camera. We use this framework to combine multimodal-image features for pedestrian detection and to demonstrate that the detection performance is significantly higher when color, disparity, and infrared features are used together. This result motivates experiments and discussion toward achieving multimodal-feature combination using a single color and a single infrared camera arranged in a cross-spectral stereo pair. We demonstrate an approach to registering multiple objects across modalities and provide an experimental analysis that highlights issues and challenges of pursuing the cross-spectral approach to multimodal and multiperspective pedestrian analysis.

On Color-, Infrared-, and Multimodal-Stereo Approaches to Pedestrian Detection

http://cvrr.ucsd.edu/publications/2007/SKrotosky_CVIU07_mutualinfo.pdf

Mutual information based registration of multimodal stereo videos for person tracking

This paper presents an analysis of color and infrared stereo approaches to pedestrian detection. We design a four camera experimental testbed consisting of two color and two infrared cameras that allows for synchronous capture and direct frame-by-frame comparison of pedestrian detection approaches. We incorporate this four camera system in a test vehicle and conduct comparative experiments of stereo-based approaches to obstacle detection using color and infrared imagery. A detailed analysis of these experiments shows the robustness of both color and infrared stereo imagery to generate the dense stereo maps necessary for robust object detection and motivates investigation of color and infrared features that can be used to further classify detected obstacles into pedestrian regions. The complementary nature of color and infrared features gives rise to a discussion of a feature fusion techniques, including a cross-spectral stereo solution to pedestrian detection.

http://cvrr.ucsd.edu/publications/2007/SKrotosky_IV2007_colorirstereo.pdf

A Comparison of Color and Infrared Stereo Approaches to Pedestrian Detection

Vision based systems for "smart" airbag systems aim to give precise information about occupant pose and location. This information can be used to make intelligent airbag deployment decisions. This paper reviews both a stereo-based and long wave infrared-based system for "smart" airbag deployment. The algorithms are systematically evaluated through extensive real-time occupant tests. Data for both algorithms is simultaneously collected and evaluated for viability in an intelligent airbag system. Results of these experimental trials show the feasibility of each video based occupant position analysis system. Advantages and drawbacks of each method are discussed and potential solutions are suggested.

http://cvrr.ucsd.edu/publications/2004/itsc2004_krotosky.pdf

Face detection and head tracking using stereo and thermal infrared cameras for "smart" airbags: a comparative analysis

This paper presents an approach for the registration of multimodal imagery for pedestrian detection when the significant depth differences of objects in the scene preclude a global alignment assumption. Using maximization-of-mutual-information matching techniques and sliding correspondence windows over calibrated image pairs, we demonstrate successful registration of color and thermal data. We develop a robust method using disparity voting for determining the registration of each object in the scene and provide a statistically based measure for evaluating the match confidence. Testing shows successful registration in complex scenes with multiple people at different depths and levels of occlusion

http://cvrr.ucsd.edu/publications/2006/Krotosky_ITSC2006_MultiModalRegistration.pdf

S.J. Krotosky

Papers

On Color-, Infrared-, and Multimodal-Stereo Approaches to Pedestrian Detection

Mutual information based registration of multimodal stereo videos for person tracking

A Comparison of Color and Infrared Stereo Approaches to Pedestrian Detection

Face detection and head tracking using stereo and thermal infrared cameras for "smart" airbags: a comparative analysis

Multimodal Stereo Image Registration for Pedestrian Detection