With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity is readily available nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyse and interpret these strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the ﬁeld of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More speciﬁcally, some essential knowledge about this topic is ﬁrst given. Subsequently, a literature survey is conducted to analyse the trends of this ﬁeld. Some prevalent sub-ﬁelds in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Fusing low spatial resolution (LR) hyperspectral (HS) data and high spatial resolution (HR) multispectral (MS) data aims to obtain HR HS data. However, due to bad weather and aging of sensor equipment, HS images usually contain a lot of noise, e.g., Gaussian noise, strip noise and mixed noise, which would make the fused image have low quality. To solve this problem, we propose the multi-scale low rank deep back projection fusion network (MLR-DBPFN). First, HS and MS are superimposed, and multi-scale spectral features of the stacked image are extracted through multi-scale low-rank decomposition and convolution operation, which effectively remove noisy spectral features. Second, the up-sampling and down-sampling network mechanisms are used to extract the multi-scale spatial features from each layer of spectral features. Finally, the multi-scale spectral features and multi-scale spatial features are combined for network training, and the weight of the noisy spectrum features is reduced through the network feedback mechanism, which suppresses the noisy spectrum and improves the noisy HS fusion performance. Experimental results on datasets of different noise demonstrate that MLR-DBPFN has superior spatial and spectral fidelity, comparative fusion quality, and robust anti-noise performance compared with state-of-the-art methods.

MLR-DBPFN: A Multi-scale Low Rank Deep Back Projection Fusion Network for Anti-noise Hyperspectral and Multispectral Image Fusion

Fusing low spatial resolution (LR) hyperspectral (HS) data and high spatial resolution (HR) multispectral (MS) data aims to obtain HR HS data. However, due to bad weather and the aging of sensor equipment, HS images usually contain a lot of noise, e.g., Gaussian noise, strip noise, and mixed noise, which would make the fused image have low quality. To solve this problem, we propose the multiscale low-rank deep back projection fusion network (MLR-DBPFN). First, HS and MS are superimposed, and multiscale spectral features of the stacked image are extracted through multiscale low-rank decomposition and convolution operation, which effectively removes noisy spectral features. Second, the upsampling and downsampling network mechanisms are used to extract the multiscale spatial features from each layer of spectral features. Finally, the multiscale spectral features and multiscale spatial features are combined for network training, and the weight of the noisy spectrum features is reduced through the network feedback mechanism, which suppresses the noisy spectrum and improves the noisy HS fusion performance. Experimental results on datasets of different noise demonstrate that MLR-DBPFN has superior spatial and spectral fidelity, comparative fusion quality, and robust antinoise performance compared with state-of-the-art methods. 

MLR-DBPFN: A Multi-Scale Low Rank Deep Back Projection Fusion Network for Anti-Noise Hyperspectral and Multispectral Image Fusion

Deep learning in multimodal remote sensing data fusion: A comprehensive review

As a kind of non-invasive, low-cost, and readily available brain examination, EEG has attached significance to the means of clinical diagnosis of epilepsy. However, the reading of long-term EEG records has brought a heavy burden to neurologists and experts. Therefore, automatic EEG classification for epileptic patients plays an essential role in epilepsy diagnosis and treatment. This paper proposes an Attention Mechanism-based Wavelet Convolution Neural Network for epilepsy EEG classification. Attention Mechanism-based Wavelet Convolution Neural Network firstly uses multi-scale wavelet analysis to decompose the input EEGs to obtain their components in different frequency bands. Then, these decomposed multi-scale EEGs are input into the Convolution Neural Network with an attention mechanism for further feature extraction and classification. The proposed algorithm achieves 98.89% triple classification accuracy on the Bonn EEG database and 99.70% binary classification accuracy on the Bern-Barcelona EEG database. Our experiments prove that the proposed algorithm achieves a state-of-the-art classification effect on epilepsy EEG. 

/pdf/an-attention-based-wavelet-convolution-neural-network-for-m91i10zf.pdf

An Attention-Based Wavelet Convolution Neural Network for Epilepsy EEG Classification

To reconstruct images with high spatial resolution and high spectral resolution, one of the most common methods is to fuse a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multispectral image (MSI) of the same scene. Deep learning has been widely applied in the field of HSI-MSI fusion, which is limited with hardware. In order to break the limits, we construct an unsupervised multiattention-guided network named UMAG-Net without training data to better accomplish HSI-MSI fusion. UMAG-Net first extracts deep multiscale features of MSI by using a multiattention encoding network. Then, a loss function containing a pair of HSI and MSI is used to iteratively update parameters of UMAG-Net and learn prior knowledge of the fused image. Finally, a multiscale feature-guided network is constructed to generate an HR-HSI. The experimental results show the visual and quantitative superiority of the proposed method compared to other methods.

/pdf/umag-net-a-new-unsupervised-multiattention-guided-network-3i3l13iqkw.pdf

Siyu Miao

Papers

UMAG-Net: A New Unsupervised Multiattention-Guided Network for Hyperspectral and Multispectral Image Fusion