The goal of infrared and visible image fusion is to generate an integrated image which can simultaneously preserve more visible detail information and prominent information from the input images. In order to achieve this goal, a novel infrared and visible fusion method based on visual saliency sparse representation and detail injection model (DIM) is presented. The proposed fusion method contains four steps. The first step is to decompose the source images into base layers and detail layers through the proposed multiscale decomposition method, which has the advantages of scale awareness and high-edge-preservation efficiency. Then, we design a novel subfusion rule called visual saliency sparse representation to get the fused base layer. Third, a DIM is developed to fuse the detail layers. In this model, we first employ a “max-absolute” scheme to obtain prefused images, which are then used as the model items for participating in the fusion process of the detail layers. Finally, the target-merged image is achieved through an inverse multiscale decomposition method. The experimental results demonstrate that the proposed method can not only preserve the details and significant information of source images but also enhance the brightness of the fused image, when compared to other state-of-the-art fusion methods.

Infrared and Visible Image Fusion Using Visual Saliency Sparse Representation and Detail Injection Model

Recently, convolutional neural networks (CNNs) have brought impressive improvements for object detection. However, detecting targets in infrared images still remains challenging, because the poor texture information, low resolution and high noise levels of the thermal imagery restrict the feature extraction ability of CNNs. In order to deal with these difficulties in the feature extraction, we propose a novel backbone network named Deep-IRTarget, composing of a frequency feature extractor, a spatial feature extractor and a dual-domain feature resource allocation model. Hypercomplex Infrared Fourier Transform is developed to calculate the infrared intensity saliency by designing hypercomplex representations in the frequency domain, while a convolutional neural network is invoked to extract feature maps in the spatial domain. Features from the frequency domain and spatial domain are stacked to construct Dual-domain features. To efficiently integrate and recalibrate them, we propose a Resource Allocation model for Features (RAF). The well-designed channel attention block and position attention block are used in RAF to respectively extract interdependent relationships among channel and position dimensions, and capture channel-wise and position-wise contextual information. Extensive experiments are conducted on three challenging infrared imagery databases. We achieve 10.14&#x0025;, 9.1&#x0025; and 8.05&#x0025; improvement on mAP scores, compared to the current state of the art method on MWIR, BITIR and WCIR respectively. 

Deep-IRTarget: An Automatic Target Detector in Infrared Imagery Using Dual-Domain Feature Extraction and Allocation

In this paper, we present a label transfer model from texts to images for image classification tasks. The problem of image classification is often much more challenging than text classification. On one hand, labeled text data is more widely available than the labeled images for classification tasks. On the other hand, text data tends to have natural semantic interpretability, and they are often more directly related to class labels. On the contrary, the image features are not directly related to concepts inherent in class labels. One of our goals in this paper is to develop a model for revealing the functional relationships between text and image features as to directly transfer intermodal and intramodal labels to annotate the images. This is implemented by learning a transfer function as a bridge to propagate the labels between two multimodal spaces. However, the intermodal label transfers could be undermined by blindly transferring the labels of noisy texts to annotate images. To mitigate this problem, we present an intramodal label transfer process, which complements the intermodal label transfer by transferring the image labels instead when relevant text is absent from the source corpus. In addition, we generalize the inter-modal label transfer to zero-shot learning scenario where there are only text examples available to label unseen classes of images without any positive image examples. We evaluate our algorithm on an image classification task and show the effectiveness with respect to the other compared algorithms.

Joint Intermodal and Intramodal Label Transfers for Extremely Rare or Unseen Classes

Exploiting complementary advantages of different reflection and scattering properties of near-infrared (NIR) images and visible (VIS) images, this letter first proposes a defogging model for single image input and then develops an extended model, a fusion model for NIR and VIS color images, to enhance the visibility of image objects in scattering environments. Our fusion model enhances the extracted details of NIR and VIS images by filtering with our defogging model in the frequency domain that takes into account the energy preservation of these two types of images in addition to the high resolution of the fused results. Finally, based on the initial fusion, we propose a color retention mapping method to keep the fusion results free of color distortion. Experimental results demonstrate that our proposed method not only achieves good defogging effect, but also can effectively combine the complementary NIR and VIS information in image color and visibility. 

Global Filter of Fusing Near-Infrared and Visible Images in Frequency Domain for Defogging

Identifying agricultural fields that grow cabbage in the highlands of South Korea is critical for accurate crop yield estimation. Only grown for a limited time during the summer, highland cabbage accounts for a significant proportion of South Korea’s annual cabbage production. Thus, it has a profound effect on the formation of cabbage prices. Traditionally, labor-extensive and time-consuming field surveys are manually carried out to derive agricultural field maps of the highlands. Recently, high-resolution overhead images of the highlands have become readily available with the rapid development of unmanned aerial vehicles (UAV) and remote sensing technology. In addition, deep learning-based semantic segmentation models have quickly advanced by recent improvements in algorithms and computational resources. In this study, we propose a semantic segmentation framework based on state-of-the-art deep learning techniques to automate the process of identifying cabbage cultivation fields. We operated UAVs and collected 2010 multispectral images under different spatiotemporal conditions to measure how well semantic segmentation models generalize. Next, we manually labeled these images at a pixel-level to obtain ground truth labels for training. Our results demonstrate that our framework performs well in detecting cabbage fields not only in areas included in the training data but also in unseen areas not included in the training data. Moreover, we analyzed the effects of infrared wavelengths on the performance of identifying cabbage fields. Based on the results of our framework, we expect agricultural officials to reduce time and manpower when identifying information about highlands cabbage fields by replacing field surveys.

Semantic segmentation of cabbage in the south korea highlands with images by unmanned aerial vehicles

The visible and near-infrared images fusion aims at utilizing their spectrum characteristics to enhance visibility. However, the current visible and near-infrared fusion algorithms cannot well preserve spectrum characteristics, which results in color distortion and halo artifacts. Therefore, this paper proposes a new visible and near infrared images fusion algorithm by fully considering their different reflection and scattering characteristics. According to image degradation model, the reflection weight model and the transmission weight model are established, respectively. The reflection weight model is established by calculating the difference between the visible (red, green, and blue) spectra and the near-infrared spectrum, while maintaining the correlation of the visible spectra. The proposed reflection weight model can preserve the original reflection characteristic of objects in natural scenes. On the other hand, the transmission weight model is explicitly proposed by calculating the gradient ratio of the visible spectra to the near-infrared spectrum. The proposed transmission weight model intends to make full use of the strong transmission performance of the near-infrared spectrum, which can complement the details loss of the visible spectra caused by light scattering. Moreover, the fused image based on two models is further enhanced according to the reflection characteristics of near-infrared spectrum in case of the non-uniform illumination. The experimental results demonstrate that the proposed algorithm can not only well preserve spectrum characteristics, but also avoid color distortion while maintaining the naturalness, which outperforms the state-of-the-art.

Li Zhuo

Papers

Spectrum Characteristics Preserved Visible and Near-Infrared Image Fusion Algorithm