Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods. 

MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module

In this paper, we reported a novel Multi-Focus Image Fusion (MFIF) method based on regularization optimization and sparse representation model, which takes full advantages of the imaging characteristics of multi-focus images. First, a multi-source regularization optimization model was proposed to divide source images into a common background layer and respective detail layers jointly. This makes the focused details scattered in different sources are extracted in concert. Then the resulting detail layers are fused in sparse representation domain. To be more task-specific for MFIF, an atom focus degree measurement based fusion rule was proposed to highlight the focused atoms. Finally, the fusion result is reconstructed by combining the fused detail layer and the common background layer. Thorough experimental evaluations confirm the effectiveness of the proposed fusion approach both visually and quantitatively, and show that our approach outperforms some state-of-the-art fusion methods.

A novel multi-focus image fusion method based on joint regularization optimization layering and sparse representation

Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods.

https://ieeexplore.ieee.org/ielx7/6287639/6514899/09745028.pdf

A new multi-focus image fusion method based on multi-classification focus learning and multi-scale decomposition

Integrated MPCAM: Multi-PSF learning for large depth-of-field computational imaging

Previous Convolutional Neural Networks (CNNs) based multi-focus image fusion methods rely primarily on local information of images. In this paper, we propose a novel deep network architecture for multi-focus image fusion that is based on a non-local image model. The motivation of this paper stems from local and non-local self-similarity widely shown in nature images. We build on this concept and introduce a recurrent neural network (RNN) that performs non-local processing. The RNN captures global and local information by retrieving long distant dependencies, hence augmenting the representation of each pixel with contextual representations. The augmented representation is beneficial to detect accurately focused and defocused pixels. In addition, we design a regression loss to address the influences of texture information. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods, both qualitatively and quantitatively.

/pdf/non-local-multi-focus-image-fusion-with-recurrent-neural-3kd6kaa41i.pdf

Non-Local Multi-Focus Image Fusion With Recurrent Neural Networks

DCKN: Multi-focus image fusion via dynamic convolutional kernel network

Intermediate prototype network for few-shot segmentation

Few-shot segmentation has got a lot of concerns recently. Existing methods mainly locate and recognize the target object based on a cross-guided way that applies masked target object features of support(query) images to make a feature matching with query(support) images. However, there are some differences between support images and query images because of large appearance and scale variation, which will lead to inaccurate and incomplete segmentation. This problem inspired us to explore the local coherence of the image to guide the segmentation. We try to get some target pixels in the query image and apply these pixels to search for more target pixels in the query image. In this work, we propose a novel network that combines a universal cross-guided branch with a new pseudo-interacting guided branch. Specifically, we first employ the universal cross-guided branch to produce a pseudo-labeling that represents the probability of each pixel belonging to the target object. Then we design a pseudo-interacting guided branch, which applies some pixels with high probabilities based on generated pseudo-labeling to segment the target object in the query image and revises the results of the cross-guided branch simultaneously. Extensive experiments show that our approach outperforms state-of-the-art methods on both PASCAL-5i and COCO-20i datasets.

Pseudo-Interacting Guided Network for Few-Shot Segmentation

In this work, we address the challenging issue of few-shot segmentation. Existing methods mainly explore the target object through the semantic similarity between the query and support pixels. However, the semantic similarity often fails to deal well with the target objects with large variations in appearance and the error predictions along the boundary. To this end, we propose a novel spatial similarity guidance network (S2GNet), which adaptively integrates spatial information with semantic information for building a target-aware correlation region to enhance the target object localization. To promote the overall spatial position understanding of the target object, we exploit boundaries as crucial guidance for spatial information. Thus we jointly train a boundary detection task and a segmentation task in an end-to-end way. With that, a target-aware attention module is further proposed to capture the target correlation region by combining the spatial similarity with the semantic similarity for each pair of pixels in the query image, which refines the location of the target object effectively and improves the segmentation performance. Extensive experiments on both PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> datasets show that our approach can achieve state-of-the-art performances. 

Zhao Duan

Papers

Non-Local Multi-Focus Image Fusion With Recurrent Neural Networks

DCKN: Multi-focus image fusion via dynamic convolutional kernel network

Intermediate prototype network for few-shot segmentation

Pseudo-Interacting Guided Network for Few-Shot Segmentation

Spatial Similarity Guidance for Few-Shot Segmentation