Pedestrian detection has never been an easy task for computer vision and the automotive industry. Systems like the advanced driver-assistance system (ADAS) highly rely on far-infrared (FIR) data captured to detect pedestrians at nighttime. The recent development of deep learning-based detectors has proven the excellent results of pedestrian detection in perfect weather conditions. However, it is still unknown what the performance in adverse weather conditions is. In this paper, we introduce a 16-bit thermal data dataset called ZUT (Zachodniopomorski Uniwersytet Technologiczny) as having the widest variety of fine-grained annotated images captured in the four biggest European Union countries captured during severe weather conditions. We also provide a synchronized Controller Area Network (CAN bus) data, including driving speed, brake pedal status, and outside temperature for future ADAS system development. Furthermore, we have tested and provided 16-bit depth modifications for the YOLOv3 deep neural network (DNN) based detector, reaching a mean Average Precision (mAP) up to 89.1%. The ZUT dataset is published and publicly available at IEEE Dataport and Github.

/pdf/pedestrian-detection-in-severe-weather-conditions-3td7ndihfi.pdf

Pedestrian Detection in Severe Weather Conditions

Pedestrian detection using a moving camera: A novel framework for foreground detection

The early recognition and understanding of the actions performed by pedestrians in traffic scenes leads to an anticipation of pedestrian intentions in advance and helps in the process of collision warning and avoidance in the context of autonomous vehicles. An environment with low visibility conditions such as night-time, fog, heavy rain or smoke increases the number of difficult situations in traffic. A complete and original model for assessing if a pedestrian is engaged in a street cross action using only infrared monocular scene perception is proposed in this paper. The assessment of a street cross action is done by the time series analysis of features like: pedestrian motion, position of pedestrians with respect to the drivable area and their distance with respect to the ego-vehicle. The extraction of these features emerges from the combination of a deep learning based pedestrian detector with an original tracking algorithm, a semantic segmentation of the road surface and a time series long-short term memory network based action recognition. In order to validate the proposed method we introduce a new dataset named CROSSIR. It is formed of pedestrian annotations, action annotations and semantic labels for the road. The CROSSIR dataset is suitable for several common computer vision algorithms: (1) pedestrian detection and tracking algorithms because each pedestrian has a unique identifier over the frames in which it appears; (2) pedestrian action recognition; (3) semantic segmentation of the road pixels in the infrared image.

/pdf/pedestrian-street-cross-action-recognition-in-monocular-far-49ckept66d.pdf

Pedestrian Street-Cross Action Recognition in Monocular Far Infrared Sequences

Object detection is one of the most crucial tasks in computer vision and remote sensing to identify specific categories of various objects in images. The unmanned aerial vehicle (UAV)-based thermal infrared (TIR) remote sensing multi-scenario images and videos are two important data sources in public security. However, their object detection process is still challenging because of the complicated scene information, coarse resolution compared with the visible videos and lack of public labelled datasets and training models. This study proposed a UAV TIR object detection framework for images and videos. The You Only Look Once (YOLO) models based on Convolutional Neural Network (CNN) architecture were designed to extract features from ground-based TIR images and videos, which were captured by Forward-looking Infrared (FLIR) cameras. The most effective algorithm was finally identified by evaluation metrics and then applied to detect objects on TIR videos from UAVs. Results showed that the highest mean average precision (mAP) of the person and car instances was 88.69% in the validating task. The fastest detection speed achieved 50 frames per second (FPS), and the smallest model size was observed in YOLOv5-s. In the application, the cross-detection performance on persons and cars in UAV TIR videos under a YOLOv5-s model was discussed in terms of the different UAVs’ observation angles and the effectiveness of the YOLO architecture was revealed. This study provides positive support for the qualitative and quantitative evaluation of objection detection from TIR images and videos using deep-learning models. 

Object detection from UAV thermal infrared images and videos using YOLO models

MCNet: Multi-level Correction Network for thermal image semantic segmentation of nighttime driving scene

Benchmarking a large-scale FIR dataset for on-road pedestrian detection

Controlled table-to-text generation seeks to generate natural language descriptions for highlighted subparts of a table. Previous SOTA systems still employ a sequence-to-sequence generation method, which merely captures the table as a linear structure and is brittle when table layouts change. We seek to go beyond this paradigm by (1) effectively expressing the relations of content pieces in the table, and (2) making our model robust to content-invariant structural transformations. Accordingly, we propose an equivariance learning framework, which encodes tables with a structure-aware self-attention mechanism. This prunes the full self-attention structure into an order-invariant graph attention that captures the connected graph structure of cells belonging to the same row or column, and it differentiates between relevant cells and irrelevant cells from the structural perspective. Our framework also modifies the positional encoding mechanism to preserve the relative position of tokens in the same cell but enforce position invariance among different cells. Our technology is free to be plugged into existing table-to-text generation models, and has improved T5-based models to offer better performance on ToTTo and HiTab. Moreover, on a harder version of ToTTo, we preserve promising performance, while previous SOTA systems, even with transformation-based data augmentation, have seen significant performance drops.

Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning

Far infrared (FIR) pedestrian detection is an essential module of the advanced driver assistance system (ADAS) at nighttime. Recently, a wave of deep convolutional neural networks (CNN) has taken the visible spectrum pedestrian detection benchmarks top ranks. However, due to the lack of dataset, we could not evaluate the performance of CNN methods on FIR images. In this paper, we introduce a nighttime FIR pedestrian dataset, which is the largest nighttime FIR pedestrian dataset. The dataset contains fine-grained annotated video, recorded from diverse road scenes and we provide detailed statistical analysis. We selected three kinds of advanced pedestrian detection methods as the baseline and evaluated their performance. Benefit from training data volume and diversity, the experimental results show that CNN-based detectors obtained good performance on FIR image. We also propose three suggestions for improving performance, which reduces the average miss rate of the vanilla Faster R-CNN by 12.97% and 9.77% on KAIST and our dataset respectively. The dataset will be public online.

Nighttime FIR Pedestrian Detection Benchmark Dataset for ADAS

Far-infrared pedestrian detection onboard is more challenging compared to pedestrian detection of visible light. Existing works proved that the output of RoIs extraction, named as the proposal, is significantly related to recall rate and computational cost for pedestrian detection. However, it is non-trivial for RoIs extraction due to low resolution, blurred details, and pedestrian morphological features in far-infrared scenes. The paper proposes a novel RoIs extraction framework for far-infrared pedestrian detection on Board, named FIR-RoIEF, by using edge to obtain pedestrian contour feature and cascade filtering to gain valuable RoIs. Given pedestrian morphological features, we further present a vertical edge strategy to enhance pedestrian vertical features. A T-shaped template and RoIs reordering are used in the bounding box evaluation process to output pure and high-quality RoIs. Under basic and standard metrics, we perform experiments on public datasets, SCUT and KAIST, which both contain large far-infrared pedestrian objects. As for the result, given 100, 400, 1000 proposals, we both achieve the best recall in 1000 proposals, 94%, and 87% respectively. By the way, it can be proved that our method keeps a good balance between computational cost and good real-time.

Zhewei Xu

Papers

Benchmarking a large-scale FIR dataset for on-road pedestrian detection

Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning

Nighttime FIR Pedestrian Detection Benchmark Dataset for ADAS

Extracting RoIs for Robust Far-infrared Pedestrian Detection on Board

Self-training involving semantic-space finetuning for semi-supervised multi-label document classification