scispace - formally typeset
Search or ask a question
Journal Article•DOI•

Anatomical-Functional Image Fusion by Information of Interest in Local Laplacian Filtering Domain

Jiao Du1, Weisheng Li2, Bin Xiao2•
Chongqing Technology and Business University1, Chongqing University of Posts and Telecommunications2
25 Aug 2017-IEEE Transactions on Image Processing (IEEE)-Vol. 26, Iss: 12, pp 5855-5866
TL;DR: A novel method for performing anatomical magnetic resonance imaging-functional (positron emission tomography or single photon emission computed tomography) image fusion is presented and can obtain better performance, compared with the state-of-the-art fusion methods.
Abstract: A novel method for performing anatomical magnetic resonance imaging-functional (positron emission tomography or single photon emission computed tomography) image fusion is presented. The method merges specific feature information from input image signals of a single or multiple medical imaging modalities into a single fused image, while preserving more information and generating less distortion. The proposed method uses a local Laplacian filtering-based technique realized through a novel multi-scale system architecture. First, the input images are generated in a multi-scale image representation and are processed using local Laplacian filtering. Second, at each scale, the decomposed images are combined to produce fused approximate images using a local energy maximum scheme and produce the fused residual images using an information of interest-based scheme. Finally, a fused image is obtained using a reconstruction process that is analogous to that of conventional Laplacian pyramid transform. Experimental results computed using individual multi-scale analysis-based decomposition schemes or fusion rules clearly demonstrate the superiority of the proposed method through subjective observation as well as objective metrics. Furthermore, the proposed method can obtain better performance, compared with the state-of-the-art fusion methods.
Citations
More filters
Journal Article•DOI•
Medical Image Fusion With Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain

[...]

Ming Yin1, Xiaoning Liu1, Yu Liu1, Xun Chen2•
Hefei University of Technology1, University of Science and Technology of China2
01 Jan 2019-IEEE Transactions on Instrumentation and Measurement
TL;DR: Experimental results demonstrate that the proposed method can obtain more competitive performance in comparison to nine representative medical image fusion methods, leading to state-of-the-art results on both visual quality and objective assessment.
Abstract: As an effective way to integrate the information contained in multiple medical images with different modalities, medical image fusion has emerged as a powerful technique in various clinical applications such as disease diagnosis and treatment planning. In this paper, a new multimodal medical image fusion method in nonsubsampled shearlet transform (NSST) domain is proposed. In the proposed method, the NSST decomposition is first performed on the source images to obtain their multiscale and multidirection representations. The high-frequency bands are fused by a parameter-adaptive pulse-coupled neural network (PA-PCNN) model, in which all the PCNN parameters can be adaptively estimated by the input band. The low-frequency bands are merged by a novel strategy that simultaneously addresses two crucial issues in medical image fusion, namely, energy preservation and detail extraction. Finally, the fused image is reconstructed by performing inverse NSST on the fused high-frequency and low-frequency bands. The effectiveness of the proposed method is verified by four different categories of medical image fusion problems [computed tomography (CT) and magnetic resonance (MR), MR-T1 and MR-T2, MR and positron emission tomography, and MR and single-photon emission CT] with more than 80 pairs of source images in total. Experimental results demonstrate that the proposed method can obtain more competitive performance in comparison to nine representative medical image fusion methods, leading to state-of-the-art results on both visual quality and objective assessment.

381 citations

Cites background or methods from "Anatomical-Functional Image Fusion ..."

  • ...onal matching pursuit (SR-SOMP) method [4], the guided filtering (GF) method [11], the phase congruency and directive contrast in NSCT domain (NSCT-PCDC) method [22], the NSCT-RPCNN method [23], the LP-SR method [8], the CTD-SR method [28], the LLF-IOI method [29], and the LP-CNN method [30]....

    [...]

  • ...[29] proposed a multiscale decomposition approach based on local Laplacian filtering (LLF) for medical image fusion and introduced an information of interest (IOI)-based strategy to fuse the highfrequency components....

    [...]

  • ...For the sake of clarity, we just show one set of results for each parameter, in which the other two parameters are set to well-performed values (this is a commonly used manner in the literature [9], [11], [29])....

    [...]

  • ...A popular approach for investigating the impacts of multiple parameters is known as controlling for a variable, which has also been widely adopted in the study of image fusion [9], [11], [29]....

    [...]

Journal Article•DOI•
A Phase Congruency and Local Laplacian Energy Based Multi-Modality Medical Image Fusion Method in NSCT Domain

[...]

Zhiqin Zhu1, Mingyao Zheng1, Guanqiu Qi2, Di Wang3, Yan Xiang1 •
Chongqing University of Posts and Telecommunications1, Mansfield University of Pennsylvania2, Chongqing University3
07 Feb 2019-IEEE Access
TL;DR: A novel multi-modality medical image fusion method based on phase congruency and local Laplacian energy that achieves competitive performance in both the image quantity and computational costs is presented.
Abstract: Multi-modality image fusion provides more comprehensive and sophisticated information in modern medical diagnosis, remote sensing, video surveillance, and so on. This paper presents a novel multi-modality medical image fusion method based on phase congruency and local Laplacian energy. In the proposed method, the non-subsampled contourlet transform is performed on medical image pairs to decompose the source images into high-pass and low-pass subbands. The high-pass subbands are integrated by a phase congruency-based fusion rule that can enhance the detailed features of the fused image for medical diagnosis. A local Laplacian energy-based fusion rule is proposed for low-pass subbands. The local Laplacian energy consists of weighted local energy and the weighted sum of Laplacian coefficients that describe the structured information and the detailed features of source image pairs, respectively. Thus, the proposed fusion rule can simultaneously integrate two key components for the fusion of low-pass subbands. The fused high-pass and low-pass subbands are inversely transformed to obtain the fused image. In the comparative experiments, three categories of multi-modality medical image pairs are used to verify the effectiveness of the proposed method. The experiment results show that the proposed method achieves competitive performance in both the image quantity and computational costs.

220 citations

Cites methods from "Anatomical-Functional Image Fusion ..."

  • ...[9] proposed a method of fuse anatomical images and functional images....

    [...]

Journal Article•DOI•
Medical Image Fusion via Convolutional Sparsity Based Morphological Component Analysis

[...]

Yu Liu1, Xun Chen2, Rabab K. Ward3, Z. Jane Wang3•
Hefei University of Technology1, University of Science and Technology of China2, University of British Columbia3
28 Jan 2019-IEEE Signal Processing Letters
TL;DR: Experimental results demonstrate that the proposed CS-MCA model can outperform some benchmarking and state-of-the-art SR-based fusion methods in terms of both visual perception and objective assessment.
Abstract: In this letter, a sparse representation (SR) model named convolutional sparsity based morphological component analysis (CS-MCA) is introduced for pixel-level medical image fusion. Unlike the standard SR model, which is based on single image component and overlapping patches, the CS-MCA model can simultaneously achieve multi-component and global SRs of source images, by integrating MCA and convolutional sparse representation (CSR) into a unified optimization framework. For each source image, in the proposed fusion method, the CSRs of its cartoon and texture components are first obtained by the CS-MCA model using pre-learned dictionaries. Then, for each image component, the sparse coefficients of all the source images are merged and the fused component is accordingly reconstructed using the corresponding dictionary. Finally, the fused image is calculated as the superposition of the fused cartoon and texture components. Experimental results demonstrate that the proposed method can outperform some benchmarking and state-of-the-art SR-based fusion methods in terms of both visual perception and objective assessment.

190 citations

Cites methods from "Anatomical-Functional Image Fusion ..."

  • ...According to a recent survey [2], these methods can be generally grouped into four categories based on the image transform strategy adopted: multi-scale decomposition (MSD)-based methods [3]–[12], sparse representation (SR)-based methods [13]–[25], methods performed in other domains [26]–[34] and methods based on combination of different transforms [35], [36]....

    [...]

Journal Article•DOI•
Unsupervised Deep Image Fusion With Structure Tensor Representations

[...]

Hyungjoo Jung1, Youngjung Kim2, Hyunsung Jang1, Namkoo Ha, Kwanghoon Sohn1 •
Yonsei University1, Agency for Defense Development2
17 Jan 2020-IEEE Transactions on Image Processing
TL;DR: Deep Image Fusion Network (DIF-Net) as discussed by the authors proposes an unsupervised loss function using the structure tensor representation of the multi-channel image contrasts, which is minimized by a stochastic deep learning solver with large-scale examples.
Abstract: Convolutional neural networks (CNNs) have facilitated substantial progress on various problems in computer vision and image processing. However, applying them to image fusion has remained challenging due to the lack of the labelled data for supervised learning. This paper introduces a deep image fusion network (DIF-Net), an unsupervised deep learning framework for image fusion. The DIF-Net parameterizes the entire processes of image fusion, comprising of feature extraction, feature fusion, and image reconstruction, using a CNN. The purpose of DIF-Net is to generate an output image which has an identical contrast to high-dimensional input images. To realize this, we propose an unsupervised loss function using the structure tensor representation of the multi-channel image contrasts. Different from traditional fusion methods that involve time-consuming optimization or iterative procedures to obtain the results, our loss function is minimized by a stochastic deep learning solver with large-scale examples. Consequently, the proposed method can produce fused images that preserve source image details through a single forward network trained without reference ground-truth labels. The proposed method has broad applicability to various image fusion problems, including multi-spectral, multi-focus, and multi-exposure image fusions. Quantitative and qualitative evaluations show that the proposed technique outperforms existing state-of-the-art approaches for various applications.

98 citations

Cites background from "Anatomical-Functional Image Fusion ..."

  • ...IMAGE fusion has remained an active research topic for multiple imaging applications, including computational photography [1], [2], multi-spectral imaging [3], [4], and medical imaging [5], [6]....

    [...]

Journal Article•DOI•
Laplacian Redecomposition for Multimodal Medical Image Fusion

[...]

Xiaoxiao Li1, Xiaopeng Guo2, Pengfei Han, Xiang Wang3, Huaguang Li1, Tao Luo •
Yunnan University1, Peking University2, Tsinghua University3
20 Feb 2020-IEEE Transactions on Instrumentation and Measurement
TL;DR: A novel Laplacian redecomposition (LRD) framework tailored to multimodal medical image fusion that outperforms other current popular fusion methods qualitatively and quantitatively.
Abstract: The field of multimodal medical image fusion has made huge progress in the past decade. However, previous methods always suffer from color distortion , blurring , and noise . To address these problems, we propose a novel Laplacian redecomposition (LRD) framework tailored to multimodal medical image fusion in this article. The proposed LRD has two technical innovations. First, we present a Laplacian decision graph decomposition scheme with image enhancement to obtain complementary information, redundant information, and low-frequency subband images. Second, considering the heterogeneous characteristics of redundant and complementary information, we introduce the concept of the overlapping domain (OD) and non-OD (NOD), where the OD contributes to fuse redundant information while the NOD is responsible for fusing complementary information. In addition, an inverse redecomposition scheme is given by leveraging the global decision graph and local mean to reconstruct high-frequency subband fusion images. Finally, the inverse Laplacian transform is applied to generate the fusion result. Experimental results demonstrate that the proposal outperforms other current popular fusion methods qualitatively and quantitatively.

93 citations

Cites background or methods from "Anatomical-Functional Image Fusion ..."

  • ...In order to validate the effectiveness of our method, we compare with seven representative algorithms, including CVT-SR [5], DTCWT-SR [5], DWT [8], NSCT [12], CNN [17], LLFIOI [22], and NSST-PAPCNN [24]....

    [...]

  • ...In addition, the optimization (7) reduces the calculation amount compared with the version in the literature [22]....

    [...]

  • ...[22] presented the local Laplacian filtering scheme to decompose the source image and used the information of interest (LLF-IOI)-based strategy to fuse the HSI....

    [...]

  • ...[22] defines LEM as the square of the sum of local window pixels....

    [...]

  • ...On the other hand, we use the LEM fusion rule [22] to fuse LSI....

    [...]

References
More filters
Journal Article•DOI•
Image quality assessment: from error visibility to structural similarity

[...]

Zhou Wang1, Alan C. Bovik2, Hamid R. Sheikh2, Eero P. Simoncelli3•
Center for Neural Science1, University of Texas at Austin2, Howard Hughes Medical Institute3
01 Apr 2004-IEEE Transactions on Image Processing
TL;DR: In this article, a structural similarity index is proposed for image quality assessment based on the degradation of structural information, which can be applied to both subjective ratings and objective methods on a database of images compressed with JPEG and JPEG2000.
Abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

40,609 citations

Journal Article•DOI•
The Laplacian Pyramid as a Compact Image Code

[...]

Peter J. Burt1, Edward H. Adelson2•
Rensselaer Polytechnic Institute1, Sarnoff Corporation2
01 Apr 1983-IEEE Transactions on Communications
TL;DR: A technique for image encoding in which local operators of many scales but identical shape serve as the basis functions, which tends to enhance salient image features and is well suited for many image analysis tasks as well as for image compression.
Abstract: We describe a technique for image encoding in which local operators of many scales but identical shape serve as the basis functions. The representation differs from established techniques in that the code elements are localized in spatial frequency as well as in space. Pixel-to-pixel correlations are first removed by subtracting a lowpass filtered copy of the image from the image itself. The result is a net data compression since the difference, or error, image has low variance and entropy, and the low-pass filtered image may represented at reduced sample density. Further data compression is achieved by quantizing the difference image. These steps are then repeated to compress the low-pass image. Iteration of the process at appropriately expanded scales generates a pyramid data structure. The encoding process is equivalent to sampling the image with Laplacian operators of many scales. Thus, the code tends to enhance salient image features. A further advantage of the present code is that it is well suited for many image analysis tasks as well as for image compression. Fast algorithms are described for coding and decoding.

6,975 citations

"Anatomical-Functional Image Fusion ..." refers background or methods in this paper

  • ...In the first class of fusion methods such as the Laplacian pyramid transform (LAP) [10], gradient pyramid transform (GRP) [11], curvelet transform (CVT) [12], contourlet transform (COT) [13] and shearlet transform (ST) [14]), each level of the sub-band image results from subsampling the corresponding level....

    [...]

  • ...Traditionally, image decomposition in MSA can be divided into two categories: the pyramid transform [10]–[16] and the parallelepiped transform [17]–[19]....

    [...]

  • ...The proposed image decomposition scheme, LLF, is compared to LAP [10], COT [13], CVT [12], NSCT [17], and ST [14] using the same image fusion rule: Average-maximum....

    [...]

  • ...Since LAP [10] is the difference between the successive levels of the low-...

    [...]

Journal Article•DOI•
Image information and visual quality

[...]

Hamid R. Sheikh1, Alan C. Bovik1•
University of Texas at Austin1
01 Feb 2006-IEEE Transactions on Image Processing
TL;DR: An image information measure is proposed that quantifies the information that is present in the reference image and how much of this reference information can be extracted from the distorted image and combined these two quantities form a visual information fidelity measure for image QA.
Abstract: Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Image QA algorithms generally interpret image quality as fidelity or similarity with a "reference" or "perfect" image in some perceptual space. Such "full-reference" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by signal fidelity measures. In this paper, we approach the image QA problem as an information fidelity problem. Specifically, we propose to quantify the loss of image information to the distortion process and explore the relationship between image information and visual quality. QA systems are invariably involved with judging the visual quality of "natural" images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of such natural signals. Using these models, we previously presented an information fidelity criterion for image QA that related image quality with the amount of information shared between a reference and a distorted image. In this paper, we propose an image information measure that quantifies the information that is present in the reference image and how much of this reference information can be extracted from the distorted image. Combining these two quantities, we propose a visual information fidelity measure for image QA. We validate the performance of our algorithm with an extensive subjective study involving 779 images and show that our method outperforms recent state-of-the-art image QA algorithms by a sizeable margin in our simulations. The code and the data from the subjective study are available at the LIVE website.

3,146 citations

Additional excerpts

  • ...To access the fused images resulting from different methods, ten objective image quality metrics (i.e., structural similarity (SSIM) [42], standard (STD) [24], edge intensity (EI) [43], [44], SF [29], average gradient (AG) [45], [46], mutual information (MI) [47], gradient magnitude similarity (GMS) [48], visual information fidelity (VIF) [49], tone-mapped image quality index (TMQI) [50], and spatialspectral entropy-based quality index (SSEQ) [51]) are adopted....

    [...]

  • ..., structural similarity (SSIM) [42], standard (STD) [24], edge intensity (EI) [43], [44], SF [29], average gradient (AG) [45], [46], mutual information (MI) [47], gradient magnitude similarity (GMS) [48], visual information fidelity (VIF) [49], tone-mapped image quality index (TMQI) [50], and spatialspectral entropy-based quality index (SSEQ) [51]) are adopted....

    [...]

  • ...The larger the metrices including SSIM, STD, EI, SF, AG, MI, GMS, VIF, and TMQI, the better the quality of the fused image is....

    [...]

Journal Article•DOI•
Fast Discrete Curvelet Transforms

[...]

Emmanuel J. Candès, Laurent Demanet, David L. Donoho, Lexing Ying
26 Sep 2006-Multiscale Modeling & Simulation
TL;DR: This paper describes two digital implementations of a new mathematical transform, namely, the second generation curvelet transform in two and three dimensions, based on unequally spaced fast Fourier transforms, while the second is based on the wrapping of specially selected Fourier samples.
Abstract: This paper describes two digital implementations of a new mathematical transform, namely, the second generation curvelet transform in two and three dimensions. The first digital transformation is based on unequally spaced fast Fourier transforms, while the second is based on the wrapping of specially selected Fourier samples. The two implementations essentially differ by the choice of spatial grid used to translate curvelets at each scale and angle. Both digital transformations return a table of digital curvelet coefficients indexed by a scale parameter, an orientation parameter, and a spatial location parameter. And both implementations are fast in the sense that they run in O(n^2 log n) flops for n by n Cartesian arrays; in addition, they are also invertible, with rapid inversion algorithms of about the same complexity. Our digital transformations improve upon earlier implementations—based upon the first generation of curvelets—in the sense that they are conceptually simpler, faster, and far less redundant. The software CurveLab, which implements both transforms presented in this paper, is available at http://www.curvelet.org.

2,603 citations

"Anatomical-Functional Image Fusion ..." refers methods in this paper

  • ...However, the CVT and ST methods are inferior with regards to detail information of structure while the ST method is superior with regards to luminance from the input functional PET image....

    [...]

  • ...In the first class of fusion methods such as the Laplacian pyramid transform (LAP) [10], gradient pyramid transform (GRP) [11], curvelet transform (CVT) [12], contourlet transform (COT) [13] and shearlet transform (ST) [14]), each level of the sub-band image results from subsampling the corresponding level....

    [...]

  • ...The proposed image decomposition scheme, LLF, is compared to LAP [10], COT [13], CVT [12], NSCT [17], and ST [14] using the same image fusion rule: Average-maximum....

    [...]

Journal Article•DOI•
The Nonsubsampled Contourlet Transform: Theory, Design, and Applications

[...]

A.L. da Cunha1, Jianping Zhou1, Minh N. Do1•
University of Illinois at Urbana–Champaign1
01 Oct 2006-IEEE Transactions on Image Processing
TL;DR: This paper proposes a design framework based on the mapping approach, that allows for a fast implementation based on a lifting or ladder structure, and only uses one-dimensional filtering in some cases.
Abstract: In this paper, we develop the nonsubsampled contourlet transform (NSCT) and study its applications. The construction proposed in this paper is based on a nonsubsampled pyramid structure and nonsubsampled directional filter banks. The result is a flexible multiscale, multidirection, and shift-invariant image decomposition that can be efficiently implemented via the a trous algorithm. At the core of the proposed scheme is the nonseparable two-channel nonsubsampled filter bank (NSFB). We exploit the less stringent design condition of the NSFB to design filters that lead to a NSCT with better frequency selectivity and regularity when compared to the contourlet transform. We propose a design framework based on the mapping approach, that allows for a fast implementation based on a lifting or ladder structure, and only uses one-dimensional filtering in some cases. In addition, our design ensures that the corresponding frame elements are regular, symmetric, and the frame is close to a tight one. We assess the performance of the NSCT in image denoising and enhancement applications. In both applications the NSCT compares favorably to other existing methods in the literature

1,900 citations

"Anatomical-Functional Image Fusion ..." refers methods in this paper

  • ...However, there is almost no color information in the results using the NSCT+PCNN method....

    [...]

  • ...The proposed image decomposition scheme, LLF, is compared to LAP [10], COT [13], CVT [12], NSCT [17], and ST [14] using the same image fusion rule: Average-maximum....

    [...]

  • ...5) Overall Comparison: To demonstrate the effectiveness of the proposed method: LLF+IOI with the parameters (σr = 0.4, α = 0.25, and β = 1), the experiments are compared to three state-of-the-art methods: NSCT+PCNN [23], LAP+SR [24], and LES+LEM [55]....

    [...]

  • ...NSCT [15] was developed from the nonsubsampled LAP and the nonsubsampled directional filter bank....

    [...]

  • ...(3) Compared to the proposed method, the images generated using the NSCT+PCNN method are closely related to the decomposition scheme NSCT....

    [...]

Related Papers (5)
A general framework for image fusion based on multi-scale transform and sparse representation

[...]

01 Jul 2015-Information Fusion
Yu Liu, Shuping Liu, Zengfu Wang
Image Fusion With Guided Filtering

[...]

30 Jan 2013-IEEE Transactions on Image Processing
Shutao Li, Xudong Kang, Jianwen Hu
Medical image fusion: A survey of the state of the art

[...]

01 Sep 2014-Information Fusion
Alex Pappachen James, Belur V. Dasarathy
Image quality assessment: from error visibility to structural similarity

[...]

01 Apr 2004-IEEE Transactions on Image Processing
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, Eero P. Simoncelli 
Objective image fusion performance measure

[...]

17 Feb 2000-Electronics Letters
Costas Xydeas, Vladimir Petrovic