Data has become the most valuable resource in today’s world. With the massive proliferation of data-driven algorithms, such as deep learning-based approaches, the availability of data is of great interest. In this context, high-quality training, validation and testing datasets are particularly needed. Volumetric data is a very important resource in medicine, as it ranges from disease diagnoses to therapy monitoring. When the dataset is sufficient, models can be trained to help doctors with these tasks. Unfortunately, there are scenarios and applications where large amounts of data is unavailable. For example, in the medical field, rare diseases and privacy issues can lead to restricted data availability. In non-medical fields, the high cost of obtaining a sufficient amount of high-quality data can also be a concern. A solution to these problems can be the generation of synthetic data to perform data augmentation in combination with other more traditional methods of data augmentation. Therefore, most of the publications on 3D Generative Adversarial Networks (GANs) arewithin themedical domain. The existence ofmechanisms to generate realistic synthetic data is a good asset to overcome this challenge, especially in healthcare, as the data must be of good quality and close to reality, i.e. realistic, and without privacy issues. In this review, we provide a summary of works that generate realistic 3D synthetic data using GANs. We therefore outline GAN-based methods in these areas with common architectures, advantages and disadvantages. We present a novel taxonomy, evaluations, challenges and research opportunities to provide a holistic overview of the current state of GANs inmedicine and other fields.

GAN-based generation of realistic 3D data: A systematic review and taxonomy

For decades, the number of automobiles in urban areas around the world has been increasing. It causes serious challenges such as traffic congestion, accidents, and pollution, which have a social, economic, and environmental impact on widespread urban cities. To overcome these challenges, we need to explore smart AI-based perception systems for vehicular applications. Such types of systems can provide improved situational awareness to the driver and generate early alarm about upcoming obstacles and road incidents. In this study, we have presented the effective use of uncooled thermal IR sensors for designing smart thermal perception systems as an alternative to CMOS visible imaging by presenting state-of-the-art studies for in-cabin and out-cabin vehicular applications with potential long-term benefits for the automotive industry. The key rationale for selecting thermal IR sensors over conventional image sensors is that visible cameras are highly dependent on lighting conditions and performance is degraded significantly in low-lighting scenarios and harsh weather conditions. Contrary to this, thermal sensors remain largely unaffected by external lighting conditions or most environmental conditions, making it a perfect optical sensor choice for all-weather and harsh environmental conditions. This study presents a review of the current state of the art for automotive thermal imaging with a focus on the contributions and advances achieved by the EU-funded project ‘HELIAUS’ in the domain of AI-based thermal imaging pipelines for safer and reliable road journeys. 

On the Role of Thermal Imaging in Automotive Applications: A Critical Review

For decades, the number of automobiles in urban areas around the world has been increasing. It causes serious challenges such as traffic congestion, accidents, and pollution, which have a social, economic, and environmental impact on widespread urban cities. To overcome these challenges, we need to explore smart AI-based perception systems for vehicular applications. Such types of systems can provide improved situational awareness to the driver and generate early alarm about upcoming obstacles and road incidents. In this study, we have presented the effective use of uncooled thermal IR sensors for designing smart thermal perception systems as an alternative to CMOS visible imaging by presenting state-of-the-art studies for in-cabin and out-cabin vehicular applications with potential long-term benefits for the automotive industry. The key rationale for selecting thermal IR sensors over conventional image sensors is that visible cameras are highly dependent on lighting conditions and performance is degraded significantly in low-lighting scenarios and harsh weather conditions. Contrary to this, thermal sensors remain largely unaffected by external lighting conditions or most environmental conditions, making them a perfect optical sensor choice for all-weather and harsh environmental conditions. This study presents a review of the current state of the art for automotive thermal imaging with a focus on the contributions and advances achieved by the EU-funded project ‘HELIAUS’ in the domain of AI-based thermal imaging pipelines for safer and reliable road journeys.

In this paper, we propose ThermalSynth, a novel approach for creating synthetic thermal images by mixing 3D characters generated using the Unity game engine with real thermal backgrounds. We use a shader based on the Stefan-Boltzmann law [18] to approximate the appearance in the thermal domain of the synthetic characters. Additionally, we provide a post-processing pipeline to better blend the high-fidelity synthetic data with the lower-resolution real thermal surveillance one. The proposed approach is used to create a dataset for people falling into water near a harbor front. Diverse scenarios of such falls are generated with an ample amount of data to enable the use of deep learning algorithms. To demonstrate the effectiveness of the generated data, we train two standard deep neural networks (AlexNet and ResNet-18) on our synthetic thermal dataset using a supervised learning approach. We test our system on small datasets containing real video footage of actual falls. We observe that training these simple classification networks yields an accuracy of 98.70% at a sensitivity of 100% on the real-world voluntary fall dataset. The code for Ther-malSynth and the dataset is publically available at https://github.com/NeeluMadan/Thermal-Synth.

ThermalSynth: A Novel Approach for Generating Synthetic Thermal Human Scenarios

In this paper we propose a novel data augmentation approach for visual content domains that have scarce training datasets, compositing synthetic 3D objects within real scenes. We show the performance of the proposed system in the context of object detection in thermal videos, a domain where i) training datasets are very limited compared to visible spectrum datasets and ii) creating full realistic synthetic scenes is extremely cumbersome and expensive due to the difficulty in modeling the thermal properties of the materials of the scene. We compare different augmentation strategies, including state of the art approaches obtained through RL techniques, the injection of simulated data and the employment of a generative model, and study how to best combine our proposed augmentation with these other techniques. Experimental results demonstrate the effectiveness of our approach, and our single-modality detector achieves state-of-the-art results on the FLIR ADAS dataset.

https://dl.acm.org/doi/pdf/10.1145/3474085.3475679

Partially Fake it Till you Make It: Mixing Real and Fake Thermal Images for Improved Object Detection

In this paper we propose a novel data augmentation approach for visual content domains that have scarce training datasets, compositing synthetic 3D objects within real scenes. We show the performance of the proposed system in the context of object detection in thermal videos, a domain where 1) training datasets are very limited compared to visible spectrum datasets and 2) creating full realistic synthetic scenes is extremely cumbersome and expensive due to the difficulty in modeling the thermal properties of the materials of the scene. We compare different augmentation strategies, including state of the art approaches obtained through RL techniques, the injection of simulated data and the employment of a generative model, and study how to best combine our proposed augmentation with these other techniques.Experimental results demonstrate the effectiveness of our approach, and our single-modality detector achieves state-of-the-art results on the FLIR ADAS dataset.

Francesco Bongini

Papers

Partially Fake it Till you Make It: Mixing Real and Fake Thermal Images for Improved Object Detection

Partially fake it till you make it: mixing real and fake thermal images for improved object detection