Visible and infrared image fusion (VIF) has attracted a lot of interest in recent years due to its application in many tasks, such as object detection, object tracking, scene segmentation, and crowd counting. In addition to conventional VIF methods, an increasing number of deep learning-based VIF methods have been proposed in the last five years. Different types of methods, such as CNN-based, autoencoder-based, GAN-based, and transformer-based methods, have been proposed. Deep learning-based methods have undoubtedly become dominant methods for the VIF task. However, while much progress has been made, the field will benefit from a systematic review of these deep learning-based methods. In this paper we present a comprehensive review of deep learning-based VIF methods. We discuss motivation, taxonomy, recent development characteristics, datasets, and performance evaluation methods in detail. We also discuss future prospects of the VIF field. This paper can serve as a reference for VIF researchers and those interested in entering this fast-developing field.

Visible and Infrared Image Fusion Using Deep Learning

Alzheimer's disease (AD) is the most general type of dementia, which concludes in memory-related problems. For researchers, the AD diagnosis at an early stage is a complex task. Alzheimer's disease is a chronic neurodegenerative disease. The risk of further degeneration will be decreased by early diagnosis of the disease. Other than predicting the possible growth of the disease, the recent studies concentrate on disease classification state. In this research, a Siamese-LSTM (long short-term memory) model is proposed for the enhanced multiple stages of Alzheimer's disease classification. The BoG bag of visual word method is utilized in order to improve the potency of texture based features (i.e., GLCM). The dataset used in this research work is ADNI. For multi-class classification, the samples are classified into three classes such as normal, AD, and MCI. The performance of the Alzheimer's disease stage is calculated in terms of accuracy, precision, and recall. Moreover, in addition, the sensitivity and specificity are evaluated.

A Hybrid Siamese-LSTM (Long Short-Term Memory) for Classification of Alzheimer's Disease

A Comparative Review on Multi-modal Sensors Fusion based on Deep Learning

Multimodal biometrics fusion plays an important role in the field of biometrics. Therefore, this paper presents a multimodal biometrics fusion algorithm using deep reinforcement learning. In order to reduce the influence of user behavior, user’s personal characteristics, and environmental light on image data quality, data preprocessing is realized through data transformation and single-mode biometric image region segmentation. A two-dimensional Gobar filter was used to analyze the texture of local sub-blocks, qualitatively describe the similarity between the filter and the sub-blocks and extract the phase information and local amplitude information of multimodal biometrics features. Deep reinforcement learning was used to construct the classifier of different modal biometrics, and the weighted sum fusion of different modal biometrics was implemented by fractional information. The multimodal biometrics fusion algorithm was designed. The Casia-iris-interval-v4 and NFBS datasets were used to test the performance of the proposed algorithm. The results show that the fused image quality is better, the feature extraction accuracy is between 84% and 93%, the average accuracy of feature classification is 97%, the multimodal biometric classification time is only 110 ms, the multimodal biometric fusion time is only 550 ms, the effect is good, and the practicability is strong.

Multimodal Biometrics Fusion Algorithm Using Deep Reinforcement Learning

Visible and infrared image fusion (VIF) has attracted a lot of interest in recent years due to its application in many tasks, such as object detection, object tracking, scene segmentation, and crowd counting. In addition to conventional VIF methods, an increasing number of deep learning-based VIF methods have been proposed in the last five years. Different types of methods, such as CNN-based, autoencoder-based, GAN-based, and transformer-based methods, have been proposed. Deep learning-based methods have undoubtedly become dominant methods for the VIF task. However, while much progress has been made, the field will benefit from a systematic review of these deep learning-based methods. In this paper we present a comprehensive review of deep learning-based VIF methods. We discuss motivation, taxonomy, recent development characteristics, datasets, and performance evaluation methods in detail. We also discuss future prospects of the VIF field. This paper can serve as a reference for VIF researchers and those interested in entering this fast-developing field. 

The fusion of the spatial characteristics, of visual image, and spectral aspects, of infrared image, is of immense practical importance. In this work, we propose a novel spatially constrained adversarial autoencoder that extracts deep features from the infrared and visible images to obtain a more exhaustive and global representation. A residual autoencoder architecture, regularised by a residual adversarial network has been employed, to generate a more realistic fused image. The residual module serves as a primary building block for the encoder, decoder, and adversarial network, as an add on the symmetric skip connections, perform the functionality of embedding the spatial characteristics directly from the initial layers of encoder structure to the decoder part of the network. The spectral information in the infrared image is incorporated by adding the feature maps over several layers in the encoder part of the fusion structure. The encoder section is made up of two separate branches to carry out independent inference on both the visual as well as infrared images. The loss function has been designed to incorporate the characteristics of both the modalities by optimizing over the textural content of the visible image and the spectral content of its infrared counterpart. In order to efficiently optimize the network's parameters, an adversarial regulariser network has been proposed that would perform supervised learning on the fused image and the original visual image since the visual image contains most of the structural content in comparison to the infrared image. The adversarial game has been incorporated in the structure by the addition of classification loss in the generator and discriminator loss functions in addition to the content loss.

/pdf/multimodal-sensor-fusion-using-symmetric-skip-autoencoder-4u3efjgg36.pdf

Multimodal Sensor Fusion Using Symmetric Skip Autoencoder Via an Adversarial Regulariser

Bayesian Sparse Signal Recovery (SSR) for Multiple Measurement Vectors, when elements of each row of solution matrix are correlated, is addressed in the paper. We propose a standard linear Gaussian observation model and a three-level hierarchical estimation framework, based on Gaussian Scale Mixture (GSM) model with some random and deterministic parameters, to model each row of the unknown solution matrix. This hierarchical model induces heavy-tailed marginal distribution over each row which encompasses several choices of distributions viz. Laplace distribution, Student's t distribution and Jeffery prior. Automatic Relevance Determination (ARD) phenomenon introduces sparsity in the model. It is interesting to see that Block Sparse Bayesian Learning framework is a special case of the proposed framework when induced marginal is Jeffrey prior. Experimental results for synthetic signals are provided to demonstrate its effectiveness. We also explore the possibility of using Multiple Measurement Vectors to model Dynamic Hand Posture Database which consists of sequence of temporally correlated hand posture sequence. It can be seen that by exploiting temporal correlation information present in the successive image samples, the proposed framework can reconstruct the data with less linear random measurements with high fidelity.

Sparse Signal Recovery for Multiple Measurement Vectors with Temporally Correlated Entries: A Bayesian Perspective

It is a challenging task to extract the best of both worlds by combining the spatial characteristics of a visible image and the spectral content of an infrared image. In this work, we propose a spatially constrained adversarial autoencoder that extracts deep features from the infrared and visible images to obtain a more exhaustive and global representation. In this paper, we propose a residual autoencoder architecture, regularised by a residual adversarial network, to generate a more realistic fused image. The residual module serves as primary building for the encoder, decoder and adversarial network, as an add on the symmetric skip connections perform the functionality of embedding the spatial characteristics directly from the initial layers of encoder structure to the decoder part of the network. The spectral information in the infrared image is incorporated by adding the feature maps over several layers in the encoder part of the fusion structure, which makes inference on both the visual and infrared images separately. In order to efficiently optimize the parameters of the network, we propose an adversarial regulariser network which would perform supervised learning on the fused image and the original visual image.

Image fusion using symmetric skip autoencodervia an Adversarial Regulariser.

Sign language recognition through image processing presents challenges related to the requirement of real time applicability and high accuracy. Though previous work adopting methodologies from deep convolutional neural network architectures have shown to achieve good performance, they lack a consummate solution in terms of accuracy due to consideration of word based recognition. Recent development of Inception Network based architectures have shown promising classification accuracy with relatively less computational demand. Hence in this paper we propose a methodology that adopts Inception Network for the task of Sign Language Recognition. We considered the American Sign Language Recognition and Correction model. The correction and suggestion tools are implemented in the model to rectify any incorrect sign detection. The results from our approach achieves accuracy in the order of 99 percent.

Snigdha Bhagat

Papers

Multimodal Sensor Fusion Using Symmetric Skip Autoencoder Via an Adversarial Regulariser

Sparse Signal Recovery for Multiple Measurement Vectors with Temporally Correlated Entries: A Bayesian Perspective

Image fusion using symmetric skip autoencodervia an Adversarial Regulariser.

Improved American Sign Language Recognition and Correction Using Inception Network, MediaPipe and PyEnchant

Quantification of Differential Information Using Matrix Pencil and Its Applications