This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs, and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were then summarized with discussion.

A review on medical imaging synthesis using deep learning and its clinical applications.

This article reviews the use of a subdiscipline of artificial intelligence (AI), deep learning , for the reconstruction of images in positron emission tomography (PET). Deep learning can be used either directly or as a component of conventional reconstruction, in order to reconstruct images from noisy PET data. The review starts with an overview of conventional PET image reconstruction and then covers the principles of general linear and convolution-based mappings from data to images, and proceeds to consider nonlinearities, as used in convolutional neural networks (CNNs). The direct deep-learning methodology is then reviewed in the context of PET reconstruction. Direct methods learn the imaging physics and statistics from scratch, not relying on a priori knowledge of these models of the data. In contrast, model-based or physics-informed deep-learning uses existing advances in PET image reconstruction, replacing conventional components with deep-learning data-driven alternatives, such as for the regularization. These methods use trusted models of the imaging physics and noise distribution, while relying on training data examples to learn deep mappings for regularization and resolution recovery. After reviewing the main examples of these approaches in the literature, the review finishes with a brief look ahead to future directions.

/pdf/deep-learning-for-pet-image-reconstruction-48vhnvchzh.pdf

Deep Learning for PET Image Reconstruction

The promise of artificial intelligence and deep learning in PET and SPECT imaging.

This brief review summarizes the major applications of artificial intelligence (AI), in particular deep learning approaches, in molecular imaging and radiation therapy research. To this end, the applications of artificial intelligence in five generic fields of molecular imaging and radiation therapy, including PET instrumentation design, PET image reconstruction quantification and segmentation, image denoising (low-dose imaging), radiation dosimetry and computer-aided diagnosis, and outcome prediction are discussed. This review sets out to cover briefly the fundamental concepts of AI and deep learning followed by a presentation of seminal achievements and the challenges facing their adoption in clinical setting.

/pdf/applications-of-artificial-intelligence-and-deep-learning-in-176plkcprp.pdf

Applications of artificial intelligence and deep learning in molecular imaging and radiotherapy.

Machine learning has found unique applications in nuclear medicine from photon detection to quantitative image reconstruction. Although there have been impressive strides in detector development for time-of-flight positron emission tomography (PET), most detectors still make use of simple signal processing methods to extract the time and position information from the detector signals. Now, with the availability of fast waveform digitizers, machine learning techniques have been applied to estimate the position and arrival time of high-energy photons. In quantitative image reconstruction, machine learning has been used to estimate various corrections factors, including scattered events and attenuation images, as well as to reduce statistical noise in reconstructed images. Here, machine learning either provides a faster alternative to an existing time-consuming computation, such as in the case of scatter estimation, or creates a data-driven approach to map an implicitly defined function, such as in the case of estimating the attenuation map for PET/MR scans. In this article, we will review the above-mentioned applications of machine learning in nuclear medicine.

Machine Learning in PET: From Photon Detection to Quantitative Image Reconstruction

Reducing radiation dose is important for PET imaging. However, reducing injection doses causes increased image noise and low signal-to-noise ratio (SNR), subsequently affecting diagnostic and quantitative accuracies. Deep learning methods have shown a great potential to reduce the noise and improve the SNR in low dose PET data. In this work, we comprehensively investigated the quantitative accuracy of small lung nodules, in addition to visual image quality, using deep learning based denoising methods for oncological PET imaging. We applied and optimized an advanced deep learning method based on the U-net architecture to predict the standard dose PET image from 10% low-dose PET data. We also investigated the effect of different network architectures, image dimensions, labels and inputs for deep learning methods with respect to both noise reduction performance and quantitative accuracy. Normalized mean square error (NMSE), SNR, and standard uptake value (SUV) bias of different nodule regions of interest (ROIs) were used for evaluation. Our results showed that U-net and GAN are superior to CAE with smaller SUVmean and SUVmax bias at the expense of inferior SNR. A fully 3D U-net has optimal quantitative performance compared to 2D and 2.5D U-net with less than 15% SUVmean bias for all the ten patients. U-net outperforms Residual U-net (r-U-net) in general with smaller NMSE, higher SNR and lower SUVmax bias. Fully 3D U-net is superior to several existing denoising methods, including Gaussian filter, anatomical-guided non-local mean (NLM) filter, and MAP reconstruction with Quadratic prior and relative difference prior, in terms of superior image quality and trade-off between noise and bias. Furthermore, incorporating aligned CT images has the potential to further improve the quantitative accuracy in multi-channel U-net. We found the optimal architectures and parameters of deep learning based methods are different for absolute quantitative accuracy and visual image quality. Our quantitative results demonstrated that fully 3D U-net can both effectively reduce image noise and control bias even for sub-centimeter small lung nodules when generating standard dose PET using 10% low count down-sampled data.

https://iopscience.iop.org/article/10.1088/1361-6560/ab3242/pdf

An investigation of quantitative accuracy for deep learning based denoising in oncological PET

Previous studies have demonstrated the feasibility of reducing noise with deep learning-based methods for low-dose fluorodeoxyglucose (FDG) positron emission tomography (PET). This work aimed to investigate the feasibility of noise reduction for tracers without sufficient training datasets using a deep transfer learning approach, which can utilize existing networks trained by the widely available FDG datasets. In this study, the deep transfer learning strategy based on a fully 3D patch-based U-Net was investigated on a 18F-fluoromisonidazole (18F-FMISO) dataset using single-bed scanning and a 68Ga-DOTATATE dataset using whole-body scanning. The datasets of 18F-FDG by single-bed scanning and whole-body scanning were used to obtain pre-trained U-Nets separately for subsequent cross-tracer and cross-protocol transfer learning. The full-dose PET images were used as the labels while low-dose PET images from 10% counts were used as the inputs. Three types of U-Nets were obtained: a U-Net trained by FDG dataset, a pre-trained FDG U-Net fine-tuned by another less-available tracer (FMISO/DOATATE), and a U-Net completely trained by a large number of less-available tracer datasets (FMISO/DOATATE), used as the reference U-Net. The denoising performance of the three types of U-Nets was evaluated on single-bed 18F-FMISO and whole-body 68Ga-DOTATATE separately and compared using normalized root-mean-square error (NRMSE), signal-to-noise ratio (SNR), and relative bias of region of interest (ROI). For cross-tracer transfer learning, all the U-Nets provided denoised images with similar quality for both tracers. There was no significant difference in terms of NRMSE and SNR when comparing the former two U-Nets with the reference U-Net. The ROI biases for these U-Nets were similar. For cross-tracer and cross-protocol transfer learning, the pre-trained single-bed FDG U-Net fine-tuned by whole-body DOTATATE data provided the most consistent images with the reference U-Net. Fine-tuning significantly reduced the NRMSE and the ROI bias and improved the SNR when comparing the fine-tuned U-Net with the U-Net trained by single-bed FDG only (NRMSE: 96.3% ± 21.1% versus 120.6% ± 18.5%, ROI bias: -10.5% ± 13.0% versus -14.7% ± 6.4%, SNR: 4.2 ± 1.4 versus 3.9 ± 1.6, for fine-tuned U-Net and the U-Net trained by single-bed FDG, respectively, with p < 0.01 in all cases). This work demonstrated that it is feasible to utilize existing networks well-trained by FDG datasets to reduce the noise for other less-available tracers and other scanning protocols by using the fine-tuning strategy.

Noise reduction with cross-tracer and cross-protocol deep transfer learning for low-dose PET.

108 Objectives: Deep convolutional neural networks can be robust and effective in noise reduction for low-dose FDG PET thanks to large amount of training datasets. However, clinically, PET datasets from new or uncommonly used tracers may not be adequately available and acquisitions of high-count images from tracers with long half-lives such as Zr-89 are difficult. Therefore, image de-noising using a deep learning method for these tracers could be challenging. In this work, we investigated the feasibility of transferring a pre-trained neural network from widely-used FDG tracer to a less frequently used tracer, fluoromisonidazole (FMISO), for image noise reduction.
 Methods: A 3D deep convolutional neural network using a U-Net architecture was adopted for noise reduction. The network was trained to minimize an L2 loss function using the Adam optimizer. A patch size of 64x64x16 was used for training and testing. Our datasets consisted of two groups of PET lung studies acquired with a Siemens Biograph mCT (9 scans for the FDG group and 11 scans for the FMISO group). The scans were performed for 90 min after 10 mCi FDG injection for the FDG group and were for 120 min with 5 mCi FMISO injection for the FMISO group. The list-mode data ranging from 60 to 80 min for the FDG group and from 70 to 120 min for the FMISO group were extracted to generate full-count sinograms. Subsequently, 10 samples of low-count sinograms were obtained by independent 10% down-sampling of the full-count list-mode data. The extracted low-dose sinograms had similar noise levels between both groups. The resulting sinograms were reconstructed using the OSEM algorithm with 3 iterations and 21 subsets. The image size was 400×400×109 with 2.04×2.04×2.03 mm3 voxel size, and was cropped to 300×180×70 for training. In 3D U-Net training, low-count images were used as the input with full-count images as the target. Three U-Net networks were trained and obtained: FDG U-Net trained with the FDG group data from scratch, fine-tuned U-Net based on pre-trained FDG U-Net with fine-tuning using 1 scan of the FMISO group as transfer learning, and FMISO U-Net trained with 9 scans of the FMISO group from scratch. The remaining 2 scans of the FMISO group were used to generate the predicted de-noised images for testing. The quantification results inside nodule region of interest (ROI) were calculated, and the predicted ROI mean values obtained from FDG U-Net and fine-tuned U-Net were compared with those from FMISO U-Net across 10 samples for each scan. The ROI relative bias (using full-count image as ground truth) and background noise for the predicted images of the three U-Nets were also calculated and compared with those obtained from post-reconstruction anatomical-guided non-local mean (ANLM) filtering. Results: All the three U-Nets were capable of reducing the noise of low-count images to the level similar to the full-count images. For both FMISO test scans, there was no significant difference in the profiles of the predicted images obtained from FDG U-Net, fine-tuned U-Net and FMISO U-Net. Furthermore, when quantitatively comparing FDG U-Net, fine-tuned U-Net with FMISO U-Net, the average correlation coefficient of the predicted ROI means across 10 samples was higher than 0.96 for the two scans and the regression line was close to the line of identity in the linear regression analysis. There was underestimation for the predicted ROI mean value in the de-noised low-count images when compared with full-count images, whereas the underestimations were similar for all the three U-Nets. Relative bias versus background curves demonstrated that all three U-Nets performed better than the ANLM method in terms of lower background noise at the same bias level, indicating that the feasibility of both FDG U-Net and fine-tuned U-Net for FMISO image noise reduction.
 Conclusions: Transfer deep learning across different tracers for low-dose PET noise reduction is feasible, which may be beneficial for tracers that are not widely used and with long half-lives.

Noise reduction with cross-tracer transfer deep learning for low-dose oncological PET

Proton therapy is attracting increasing attention nowadays owing to its accurate dose delivery to the target tumor. However, this advanced treatment modality is subject to proton beam range uncertainty, which could lead to overdose in healthy tissues. In the past decades, prompt gamma imaging (PGI) has gained increasing interest and been proved promising for real-time proton beam monitoring. The detection of 2–8 MeV high energy prompt gamma, however, poses a serious challenge for PGI system design, especially, for the detector design. In this article, we propose a prompt gamma detector design for a PGI system under development in our lab. The detector was composed of a 12 $\times $ 12 BGO block coupled to an 8 $\times $ 8 SiPM array with a crystal pixel size of 3.5 mm $\times $ 3.5 mm $\times $ 30 mm. A numerical simulation model was developed to evaluate the SiPM saturation effect caused by the finite number of microcells in each SiPM pixel while a huge number of optical photons might impinge the SiPM pixel under high energy gamma ray irradiation. Experiments under high energy gamma isotopes (22Na – 511 keV, 1.275 MeV, 88Y – 898 keV, 1.836 MeV, and 232Th – 2.6 MeV) irradiation were conducted to evaluate positioning performance, energy response linearity, and energy resolution (ER) of the detector. Root mean squared (RMS) metric was defined to assess the detector’s positioning performance quantitatively. The numerical simulation results demonstrated that the SiPM saturation effect of the proposed detector design is negligible up to 10-MeV gamma photon detection. In the experimental studies, the obtained energy response linearity was 0.9997± 0.0005, indicating almost no SiPM saturation effect for up to 2.6-MeV gamma photon detection with the proposed detector design. ER of 31.96% (511 keV), 26.67% (898 keV), 20.19% (1.275 MeV), 17.38% (1.836 MeV), and 15.88% (2.6 MeV) were achieved. With increased gamma photon energy, the crystal elements in the flood maps became clearer with reduced RMS value of the flood maps, indicating improved positioning performance. Based on the simulation and experimental results, we conclude that the proposed detector design is feasible for the PGI system with good energy and positioning performance as well as excellent anti-saturation capability for high energy gamma ray detection.

Design and Performance Evaluation of a BGO + SiPM Detector for High-Energy Prompt Gamma Imaging in Proton Therapy Monitoring

Proton therapy is superior to conventional radiation therapy due to dose deposition sharply increasing at Bragg peak. However, an uncertainty in Bragg peak position as large as 15 mm could occur due to a variety of uncertainties in proton therapy treatment delivery. One solution is to monitor the treatment during delivery by prompt gamma imaging. Pencil beam scanning mode is one of the widely adopted modes in proton therapy treatment In this mode, two sets of magnets scan the proton beams in a 2D pattern, and therefore the emitted gamma photons are focused to a small region, and the position of this region are rapidly changing with time. This is different with conventional nuclear medicine imaging, when the gamma source are distributed over the entire patient body, and rapidly changing with time. To meet the requirement of 40 cm × 40 cm × 30 cm FOV for proton pencil beam scanning mode, we propose a prompt gamma imaging system design with a multi-knife-edge slit collimator based on the optimization of a single-knife-edge slit collimator design. Monte Carlo simulation was performed to evaluate the positioning accuracy of Bragg peak and the spatial resolution of the imaging system. The prompt gamma distribution was generated via Monte Carlo simulation of proton interaction with water in GATE. The generated prompt gamma distribution was subsequently used to evaluate the system performance. For the multi-knife-edge slits collimator-based imaging system, preliminary evaluation results show that it can provide a large FOV size of 40×40×30 cm3, ∼0.17% detection efficiency at FOV center, <2mm special resolution and <3 mm positioning accuracy along Bragg peak direction. Based on the preliminary results, we conclude that the proposed multi-knife-edge slits collimator-based prompt gamma imaging system design is very promising and could potentially facilitate precise proton therapy.

Wenzhuo Lu

Papers

An investigation of quantitative accuracy for deep learning based denoising in oncological PET

Noise reduction with cross-tracer and cross-protocol deep transfer learning for low-dose PET.

Noise reduction with cross-tracer transfer deep learning for low-dose oncological PET

Design and Performance Evaluation of a BGO + SiPM Detector for High-Energy Prompt Gamma Imaging in Proton Therapy Monitoring

Prompt gamma imaging with a multi-knife-edge slit collimator for large FOV monitoring of scanned proton pencil beams