This paper presents a mosaic convolution-attention network (MCAN) for demosaicing spectral mosaic images captured using multispectral filter array (MSFA) imaging sensors. MSFA-based multispectral imaging systems acquire multispectral information of a scene in a single snap-shot operation. A complete multispectral image is reconstructed by demosaicing an MSFA-based spectral mosaic image. To avoid aliasing and artifacts in demosaicing, we utilize joint spatial-spectral correlation in a raw mosaic image. The proposed MCAN includes a mosaic convolution module (MCM) and a mosaic attention module (MAM). The MCM extracts features via a learning approach with a margin between splitting the periodic spectral mosaic and keeping the underlying spatial information of the raw image. Based on the strategy of position-sensitive weight sharing, MCM assigns the same weight to pixels with the same relative position in an MSFA. The MAM uses a position-sensitive feature aggregation strategy to describe the loading of mosaic patterns within the feature maps, which gradually reduces mosaic distortion through the attention mechanism. The experimental results on synthetic as well as real-world data show that the proposed scheme outperforms state-of-the-art methods in terms of spatial details and spectral fidelity.

Mosaic Convolution-Attention Network for Demosaicing Multispectral Filter Array Images

Hyperspectral video camera captures spatial, spectral and temporal information of moving objects. Traditional object tracking methods developed for color videos have been applied to hyperspectral videos after compressing hundreds of spectral bands into three, which does not fully utilize the wealth spectral information. In order to address this issue, we present a tensor sparse correlation filter with a spatial-spectral weighted regularizer for object tracking. First, tensor processing is employed to reduce the spectral differences of homogeneous background, thereby producing robust spectral structure features. Second, a spatial-spectral weighted regularizer is designed in the correlation filter framework to penalize filter template by suppressing spectral features dissimilar to the center pixel in tracking. Third, a sparse constraint term and tracking context information are incorporated to suppress unexpected peaks in the response map. Finally, a reformulated stacked HOG feature extractor and a two-dimensional adaptive scale search strategy are developed to further improve the tracker’s feature discrimination and scale adaptation capability. Experimental results demonstrate that the proposed method achieves superior tracking performance than traditional correlation filter-based trackers. 

Spatial–Spectral Weighted and Regularized Tensor Sparse Correlation Filter for Object Tracking in Hyperspectral Videos

This paper presents a feature descriptor using Histogram of Oriented Mosaic Gradient (HOMG) that extracts spatial-spectral features directly from mosaic spectral images. Spectral imaging utilizes unique spectral signatures to distinguish objects of interest in the scene more discriminatively. Snapshot spectral cameras equipped with spectral filter arrays (SFAs) capture spectral videos in real time, making it possible to detect/track fast moving targets based on spectral imaging. How to effectively extract the spatial-spectral feature directly from the mosaic spectral images acquired by snapshot spectral cameras is a core issue for detection/tracking. So far, there is a lack of comprehensive and in-depth research on this issue. To this end, this paper proposed a new spatial-spectral feature extractor for mosaic spectral images. The proposed scheme finds two forms of SFA neighborhood (SFAN) to construct a feature extractor suitable for any SFA structure. Exploiting the spatial-spectral correlation in two SFANs, we design six mosaic spatial-spectral gradient operators to compute spatial-spectral gradient maps (SGMs). HOMG descriptors are constructed using the magnitude and orientation of SGMs. The effectiveness and generalizability of the proposed method have been verified with object tracking experiments. Compared to the state-of-the-art feature descriptors, HOMG ranked first on two datasets captured with snapshot spectral camera with different SFAs, achieving a gain of 3.9% and 5.9% in average success rate over the second-ranked feature.

Histograms of oriented mosaic gradients for snapshot spectral image description

This paper presents a feature descriptor using Histogram of Oriented Mosaic Gradient (HOMG) that extracts spatial-spectral features directly from mosaic spectral images. Spectral imaging utilizes unique spectral signatures to distinguish objects of interest in the scene more discriminatively. Snapshot spectral cameras equipped with spectral filter arrays (SFAs) capture spectral videos in real time, making it possible to detect/track fast moving targets based on spectral imaging. How to effectively extract the spatial-spectral feature directly from the mosaic spectral images acquired by snapshot spectral cameras is a core issue for detection/tracking. So far, there is a lack of comprehensive and in-depth research on this issue. To this end, this paper proposed a new spatial-spectral feature extractor for mosaic spectral images. The proposed scheme finds two forms of SFA neighborhood (SFAN) to construct a feature extractor suitable for any SFA structure. Exploiting the spatial-spectral correlation in two SFANs, we design six mosaic spatial-spectral gradient operators to compute spatial-spectral gradient maps (SGMs). HOMG descriptors are constructed using the magnitude and orientation of SGMs. The effectiveness and generalizability of the proposed method have been verified with object tracking experiments. Compared to the state-of-the-art feature descriptors, HOMG ranked first on two datasets captured with snapshot spectral camera with different SFAs, achieving a gain of 3.9% and 5.9% in average success rate over the second-ranked feature. 

Hyperspectral videos (HSVs) have more potential in target tracking than color videos thanks to the material identification capability provided by abundant spectral bands. Due to limited HSVs for training, most current hyperspectral trackers are based on hand-crafted features rather than deeply learned ones, resulting in poor tracking performance. This paper introduces a material-guided Siamese fusion network (SiamF) for hyperspectral object tracking to make up this gap. Belonging to the Siamese tracker family and SiamF aims to model the appearance of hyperspectral objects using backbone networks trained on color images. Specifically, SiamF splits each hyperspectral frame into multiple groups of false-color images according to their band importance. Then SiamF employs a hyperspectral feature fusion (HFF) module with a dense connection architecture to integrate the extracted features from different layers and band groups, producing a multi-scale multilevel spatial-spectral representation of the targets. Instead of direct addition or concatenation, HFF employs global-local channel attention for feature fusion, so that yielded features capture the global and local structure of a specific object. Moreover, online spatial and material classifiers are developed to inject spatial and material appearance changes information into SiamF for adaptively online tracking. Experimental results demonstrate our tracker outperforms alternative methods.

Material-Guided Siamese Fusion Network for Hyperspectral Object Tracking

This paper presents a correlation filter object tracker based on fast spatial-spectral features (FSSF) to realize robust, real-time object tracking in hyperspectral surveillance video. Traditional object tracking in surveillance video based only on appearance information often fails in the presence of background clutter, low resolution, and appearance changes. Hyperspectral imaging uses unique spectral properties as well as spatial information to improve tracking accuracy in such challenging environments. However, the high-dimensionality of hyperspectral images causes high computational costs and difficulties for discriminative feature extraction. In FSSF, the real-time spatial-spectral convolution (RSSC) kernel is updated in real time in the Fourier transform domain without offline training to quickly extract discriminative spatial-spectral features. The spatial-spectral features are integrated into correlation filters to complete the hyperspectral tracking. To validate the proposed scheme, we collected a hyperspectral surveillance video (HSSV) dataset consisting of 70 sequences in 25 bands. Extensive experiments confirm the advantages and the efficiency of the proposed FSSF for object tracking in hyperspectral video tracking in challenging conditions of background clutter, low resolution, and appearance changes.

Chen Jiaxin

Papers

Object Tracking in Hyperspectral-Oriented Video with Fast Spatial-Spectral Features