Swin Transformer for Fast MRI

The performance of artificial intelligence (AI) for brain MRI can improve if enough data are made available. Generative adversarial networks (GANs) showed a lot of potential to generate synthetic MRI data that can capture the distribution of real MRI. Besides, GANs are also popular for segmentation, noise removal, and super-resolution of brain MRI images. This scoping review aims to explore how GANs methods are being used on brain MRI data, as reported in the literature. The review describes the different applications of GANs for brain MRI, presents the most commonly used GANs architectures, and summarizes the publicly available brain MRI datasets for advancing the research and development of GANs-based approaches. This review followed the guidelines of PRISMA-ScR to perform the study search and selection. The search was conducted on five popular scientific databases. The screening and selection of studies were performed by two independent reviewers, followed by validation by a third reviewer. Finally, the data were synthesized using a narrative approach. This review included 139 studies out of 789 search results. The most common use case of GANs was the synthesis of brain MRI images for data augmentation. GANs were also used to segment brain tumors and translate healthy images to diseased images or CT to MRI and vice versa. The included studies showed that GANs could enhance the performance of AI methods used on brain MRI imaging data. However, more efforts are needed to transform the GANs-based methods in clinical applications. 

/pdf/the-role-of-generative-adversarial-networks-in-brain-mri-a-rbi1mtmr.pdf

The role of generative adversarial networks in brain MRI: a scoping review

The small size of labelled samples is one of the challenging problems in identifying early lung nodules from CT images using deep learning methods. Recent literature on the topic shows that deep convolutional generative adversarial network (DCGAN) has been used in medical data synthesis and gained some success, but does not demonstrate satisfactory results in synthesizing CT images. It primarily suffers from the problem of model convergence and is prone to mode collapse. In this paper, we propose a generative adversarial network (GAN) model with prior knowledge to generate CT images of early lung nodules from a small-size of labelled samples, i.e., SLS-PriGAN. Particularly, a knowledge acquisition network and a sharpening network are designed for priori knowledge learning and acquisition, and then, a GAN model is developed to produce CT images of early lung nodules. To validate our method, a general fast R-CNN network is trained using the CT images generated by SLS-PriGAN. The experiment result shows that it achieved a recognizing accuracy of 91%, a recall rate of 81%, and F1 score of 0.85 in identifying clinic CT images of early lung nodules. This provides a promising way of identifying early lung nodules from CT images using deep learning with small-size labelled samples.

/pdf/an-enhanced-priori-knowledge-gan-for-ct-images-generation-of-1o7jjhyx.pdf

An Enhanced Priori Knowledge GAN for CT Images Generation of Early Lung Nodules with Small-Size Labelled Samples

Background It is unclear whether deep-learning–based super-resolution technology (SR) or compressed sensing technology (CS) can accelerate magnetic resonance imaging (MRI) . Purpose To compare SR accelerated images with CS images regarding the image similarity to reference 2D- and 3D gradient-echo sequence (GRE) brain MRI. Material and Methods We prospectively acquired 1.3× and 2.0× faster 2D and 3D GRE images of 20 volunteers from the reference time by reducing the matrix size or increasing the CS factor. For SR, we trained the generative adversarial network (GAN), upscaling the low-resolution images to the reference images with twofold cross-validation. We compared the structural similarity (SSIM) index of accelerated images to the reference image. The rate of incorrect answers of a radiologist discriminating faster and reference image was used as a subjective image similarity (ISM) index. Results The SR demonstrated significantly higher SSIM than the CS (SSIM=0.9993–0.999 vs. 0.9947–0.9986; P < 0.001). In 2D GRE, it was challenging to discriminate the SR image from the reference image, compared to the CS (ISM index 40% vs. 17.5% in 1.3×; P = 0.039 and 17.5% vs. 2.5% in 2.0×; P = 0.034). In 3D GRE, the CS revealed a significantly higher ISM index than the SR (22.5% vs. 2.5%; P = 0.011) in 2.0 × faster images. However, the ISM index was identical for the 2.0× CS and 1.3× SR (22.5% vs. 27.5%; P = 0.62) with comparable time costs. Conclusion The GAN-based SR outperformed CS in image similarity with 2D GRE for MRI acceleration. In addition, CS was more advantageous in 3D GRE than SR.

Generative adversarial network-based post-processed image super-resolution technology for accelerating brain MRI: comparison with compressed sensing

Magnetic resonance imaging (MRI) is a widely used non-radiative and non-invasive method for clinical interro-gation of organ structures and metabolism, with an inherently long scanning time. Methods by k-space undersampling and deep learning based reconstruction have been popularised to accelerate the scanning process. This work focuses on investigating how powerful transformers are for fast MRI by exploiting and comparing different novel network architectures. In particular, a generative adversarial network (GAN) based Swin transformer (ST-GAN) was introduced for the fast MRI reconstruction. To further preserve the edge and texture information, edge enhanced GAN based Swin transformer (EES-GAN) and texture enhanced GAN based Swin transformer (TES-GAN) were also developed, where a dual-discriminator GAN structure was applied. We compared our proposed GAN based transformers, standalone Swin transformer and other convolutional neural networks based GAN model in terms of the evaluation metrics PSNR, SSIM and FID. We showed that transformers work well for the MRI reconstruction from different undersampling conditions. The utilisation of GAN's adversarial structure improves the quality of images reconstructed when undersampled for 30% or higher. The code is publicly available at https://github.comJayanglab/SwinGANMR. 

/pdf/fast-mri-reconstruction-how-powerful-transformers-are-lnvbfjmo.pdf

Fast MRI Reconstruction: How Powerful Transformers Are?

Transfer learning enhanced generative adversarial networks for multi-channel MRI reconstruction

Deep learning based generative adversarial networks (GAN) can effectively perform image reconstruction with under-sampled MR data. In general, a large number of training samples are required to improve the reconstruction performance of a certain model. However, in real clinical applications, it is difficult to obtain tens of thousands of raw patient data to train the model since saving k-space data is not in the routine clinical flow. Therefore, enhancing the generalizability of a network based on small samples is urgently needed. In this study, three novel applications were explored based on parallel imaging combined with the GAN model (PI-GAN) and transfer learning. The model was pre-trained with public Calgary brain images and then fine-tuned for use in (1) patients with tumors in our center; (2) different anatomies, including knee and liver; (3) different k-space sampling masks with acceleration factors (AFs) of 2 and 6. As for the brain tumor dataset, the transfer learning results could remove the artifacts found in PI-GAN and yield smoother brain edges. The transfer learning results for the knee and liver were superior to those of the PI-GAN model trained with its own dataset using a smaller number of training cases. However, the learning procedure converged more slowly in the knee datasets compared to the learning in the brain tumor datasets. The reconstruction performance was improved by transfer learning both in the models with AFs of 2 and 6. Of these two models, the one with AF=2 showed better results. The results also showed that transfer learning with the pre-trained model could solve the problem of inconsistency between the training and test datasets and facilitate generalization to unseen data.

Weibo Chen

Papers

Transfer learning enhanced generative adversarial networks for multi-channel MRI reconstruction

Transfer Learning Enhanced Generative Adversarial Networks for Multi-Channel MRI Reconstruction