In this paper, we proposed a new end-to-end model, termed as dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Our method establishes an adversarial game between a generator and two discriminators. The generator aims to generate a real-like fused image based on a specifically designed content loss to fool the two discriminators, while the two discriminators aim to distinguish the structure differences between the fused image and two source images, respectively, in addition to the content loss. Consequently, the fused image is forced to simultaneously keep the thermal radiation in the infrared image and the texture details in the visible image. Moreover, to fuse source images of different resolutions, e.g. , a low-resolution infrared image and a high-resolution visible image, our DDcGAN constrains the downsampled fused image to have similar property with the infrared image. This can avoid causing thermal radiation information blurring or visible texture detail loss, which typically happens in traditional methods. In addition, we also apply our DDcGAN to fusing multi-modality medical images of different resolutions, e.g. , a low-resolution positron emission tomography image and a high-resolution magnetic resonance image. The qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our DDcGAN over the state-of-the-art, in terms of both visual effect and quantitative metrics. Our code is publicly available at https://github.com/jiayi-ma/DDcGAN .

DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion

Autonomous vehicles (AV) are expected to improve, reshape, and revolutionize the future of ground transportation. It is anticipated that ordinary vehicles will one day be replaced with smart vehicles that are able to make decisions and perform driving tasks on their own. In order to achieve this objective, self-driving vehicles are equipped with sensors that are used to sense and perceive both their surroundings and the faraway environment, using further advances in communication technologies, such as 5G. In the meantime, local perception, as with human beings, will continue to be an effective means for controlling the vehicle at short range. In the other hand, extended perception allows for anticipation of distant events and produces smarter behavior to guide the vehicle to its destination while respecting a set of criteria (safety, energy management, traffic optimization, comfort). In spite of the remarkable advancements of sensor technologies in terms of their effectiveness and applicability for AV systems in recent years, sensors can still fail because of noise, ambient conditions, or manufacturing defects, among other factors; hence, it is not advisable to rely on a single sensor for any of the autonomous driving tasks. The practical solution is to incorporate multiple competitive and complementary sensors that work synergistically to overcome their individual shortcomings. This article provides a comprehensive review of the state-of-the-art methods utilized to improve the performance of AV systems in short-range or local vehicle environments. Specifically, it focuses on recent studies that use deep learning sensor fusion algorithms for perception, localization, and mapping. The article concludes by highlighting some of the current trends and possible future research directions.

https://hal.archives-ouvertes.fr/hal-02942600/document

Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review.

Advances in Multimodal Data Fusion in Neuroimaging: Overview, Challenges, and Novel Orientation

Multimodal medical image fusion is effectuated to minimize the redundancy while augmenting the necessary information from the input images acquired using different medical imaging sensors. The sole aim is to yield a single fused image, which could be more informative for an efficient clinical analysis. This paper presents a two-stage multimodal fusion framework using the cascaded combination of stationary wavelet transform (SWT) and non sub-sampled Contourlet transform (NSCT) domains for images acquired using two distinct medical imaging sensor modalities (i.e., magnetic resonance imaging and computed tomography scan). The major advantage of using a cascaded combination of SWT and NSCT is to improve upon the shift variance, directionality, and phase information in the finally fused image. The first stage employs a principal component analysis algorithm in SWT domain to minimize the redundancy. Maximum fusion rule is then applied in NSCT domain at second stage to enhance the contrast of the diagnostic features. A quantitative analysis of fused images is carried out using dedicated fusion metrics. The fusion responses of the proposed approach are also compared with other state-of-the-art fusion approaches; depicting the superiority of the obtained fusion results.

Multimodal Medical Image Sensor Fusion Framework Using Cascade of Wavelet and Contourlet Transform Domains

The following work presents an overview of smart sensors and sensor fusion targeted at biomedical applications and sports areas. In this work, the integration of these areas is demonstrated, promoting a reflection about techniques and applications to collect, quantify and qualify some physical variables associated with the human body. These techniques are presented in various biomedical and sports applications, which cover areas related to diagnostics, rehabilitation, physical monitoring, and the development of performance in athletes, among others. Although some applications are described in only one of two fields of study (biomedicine and sports), it is very likely that the same application fits in both, with small peculiarities or adaptations. To illustrate the contemporaneity of applications, an analysis of specialized papers published in the last six years has been made. In this context, the main characteristic of this review is to present the largest quantity of relevant examples of sensor fusion and smart sensors focusing on their utilization and proposals, without deeply addressing one specific system or technique, to the detriment of the others.

https://www.mdpi.com/1424-8220/16/10/1569/pdf

Sensor Fusion and Smart Sensor in Sports and Biomedical Applications.

Fusion of various images aids the rejuvenation of complementary attributes of the images. Similarly, medical image fusion constructs a composite image comprehending significant traits from multimodal source images. Current work exhibits medical image fusion utilizing Laplacian Pyramid (LP) employing DCT. LP decomposes the source medical images as different low pass filtered images, resembling a pyramidal structure. As the pyramidal level of decomposition increases, the quality of the fused image also increases. The proposed technique provides a fused image with better edges and information content from human visual system (HVS) point of view. Qualitative and quantitative analysis of the proposed technique is found to be superior than that of Daubechies complex wavelet transform (DCxWT).

Medical image fusion with Laplacian Pyramids

Medical image fusion for merging of complementary diagnostic content has been carried out in this paper using Principal Component Analysis (PCA) and Wavelets. The proposed fusion approach involves sub-band decomposition using 2D-Discrete Wavelet Transform (DWT) in order to preserve both spectral and spatial information. Further, PCA is applied on the decomposed coefficients to maximize the spatial resolution. An optimal variant of the daubechies wavelet family has been selected experimentally for better fusion results. Simulation results demonstrate an improvement in visual quality of the fused image in comparison to other state-of-art fusion approaches.

Medical image fusion using combination of PCA and wavelet analysis

Curvelet transform achieves a compact representation of edges and curved shapes in the image, which other techniques like wavelets and ridgelets are not able to represent. This property of curvelet transform facilitates the retrieval of complementary information from medical images for precise and efficient clinical diagnosis. This paper presents a combination of curvelet transform along with principal component analysis (PCA) and maximum selection rule as an improved fusion approach for MRI and CT-scan. The proposed fusion approach involves image decomposition using curvelet transform followed by application of PCA for dimensionality reduction and the selection of maximum matrix to select only the relevant information in the images. Fusion factor (FF) and structural similarity index (SSIM) are used for performance evaluation of the proposed approach. Simulation results demonstrate an improvement in visual quality of the fused image supported by higher values of fusion metrics.

Medical Image Fusion in Curvelet Domain Employing PCA and Maximum Selection Rule

Medical image fusion facilitates the retrieval of complementary information from medical images and has been employed diversely for computer-aided diagnosis of life threatening diseases. Fusion has been performed using various approaches such as Pyramidal, Multi-resolution, multi-scale etc. Each and every approach of fusion depicts only a particular feature i.e. the information content or the structural properties of an image. Therefore, this paper presents a comparative analysis and evaluation of multi-modal medical image fusion methodologies employing wavelet as a multi-resolution approach and ridgelet as a multi-scale approach. The current work tends to highlight upon the utility of these approaches according to the requirement of features in the fused image. Principal Component Analysis PCA based fusion algorithm has been employed in both ridgelet and wavelet domains for purpose of minimisation of redundancies. Simulations have been performed for different sets of MR and CT-scan images taken from 'The Whole Brain Atlas'. The performance evaluation has been carried out using different parameters of image quality evaluation like: Entropy E, Fusion Factor FF, Structural Similarity Index SSIM and Edge Strength QFAB. The outcome of this analysis highlights the trade-off between the retrieval of information content and the morphological details in finally fused image in wavelet and ridgelet domains.

Akanksha Sahu

Papers

Multimodal Medical Image Sensor Fusion Framework Using Cascade of Wavelet and Contourlet Transform Domains

Medical image fusion with Laplacian Pyramids

Medical image fusion using combination of PCA and wavelet analysis

Medical Image Fusion in Curvelet Domain Employing PCA and Maximum Selection Rule

Medical Image Fusion in Wavelet and Ridgelet Domains: A Comparative Evaluation