Since its renaissance, deep learning (DL) has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high-performance computing. However, medical imaging presents unique challenges that confront DL approaches. In this survey article, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in DL are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, and so on. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies With Progress Highlights, and Future Promises

Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises

Automated synthetic computed tomography (sCT) generation based on magnetic resonance imaging (MRI) images would allow for MRI-only based treatment planning in radiation therapy, eliminating the need for CT simulation and simplifying the patient treatment workflow. In this work, the authors propose a novel method for generation of sCT based on dense cycle-consistent generative adversarial networks (cycle GAN), a deep-learning based model that trains two transformation mappings (MRI to CT and CT to MRI) simultaneously.
The cycle GAN-based model was developed to generate sCT images in a patch-based framework. Cycle GAN was applied to this problem because it includes an inverse transformation from CT to MRI, which helps constrain the model to learn a one-to-one mapping. Dense block-based networks were used to construct generator of cycle GAN. The network weights and variables were optimized via a gradient difference (GD) loss and a novel distance loss metric between sCT and original CT.
Leave-one-out cross-validation was performed to validate the proposed model. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and normalized cross correlation (NCC) indexes were used to quantify the differences between the sCT and original planning CT images. For the proposed method, the mean MAE between sCT and CT were 55.7 Hounsfield units (HU) for 24 brain cancer patients and 50.8HU for 20 prostate cancer patients. The mean PSNR and NCC were 26.6dB and 0.963 in the brain cases, and 24.5dB and 0.929 in the pelvis.
We developed and validated a novel learning-based approach to generate CT images from routine MRIs based on dense cycle GAN model to effectively capture the relationship between the CT and MRIs. The proposed method can generate robust, high-quality sCT in minutes. The proposed method offers strong potential for supporting near real-time MRI-only treatment planning in the brain and pelvis.

MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks.

To enable magnetic resonance (MR)-only radiotherapy and facilitate modelling of radiation attenuation in humans, synthetic-CT (sCT) images need to be generated. Considering the application of MR-guided radiotherapy and online adaptive replanning, sCT generation should occur within minutes. This work aims at assessing whether an existing deep learning network can rapidly generate sCT images to be used for accurate MR-based dose calculations in the entire pelvis. A study was conducted on data of 91 patients with prostate, rectal and cervical cancer who underwent external beam radiotherapy acquiring both CT and MRI for patients' simulation. Dixon reconstructed water, fat and in-phase images obtained from a conventional dual gradient-recalled echo sequence were used to generate sCT images. A conditional generative adversarial network (cGAN) was trained in a paired fashion on 2D transverse slices of 32 prostate cancer patients. The trained network was tested on the remaining patients to generate sCT images. For 30 patients in the test set, dose recalculations of the clinical plan were performed on sCT images. Dose distributions were evaluated comparing voxel-based dose differences, gamma and dose-volume histogram (DVH) analysis. The sCT generation required 5.6 s and 21 s for a single patient volume on a GPU and CPU, respectively. On average, sCT images resulted in a higher dose to the target of maximum 0.3%. Results suggest that accurate MR-based dose calculation using sCT images generated with a cGAN trained on prostate cancer patients is feasible for the entire pelvis. The sCT generation was sufficiently fast to be integrated into an MR-guided radiotherapy workflow.

Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy

This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs, and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were then summarized with discussion.

A review on medical imaging synthesis using deep learning and its clinical applications.

MRI-only brain radiotherapy: Assessing the dosimetric accuracy of synthetic CT images generated using a deep learning approach

Inter-and intra-observer variation in delineating regions of interest (ROIs) occurs because of differences in expertise level and preferences of the radiation oncologists. We evaluated the accuracy of a segmentation model using the U-Net structure to delineate the prostate, bladder, and rectum in male pelvic CT images. The dataset used for training and testing the model consisted of raw CT scan images of 85 prostate cancer patients. We designed a 2D U-Net model to directly learn a mapping function that converts a 2D CT grayscale image to its corresponding 2D OAR segmented image. Our network contains blocks of convolution 2D layers with variable kernel sizes, channel number, and activation functions. On the left side of the U-Net model, we used three 3x3 convolutions, each followed by a rectified linear unit (ReLu) (activation function), and one max pooling operation. On the right side of the U-Net model, we used a 2x2 transposed convolution and two 3x3 convolution networks followed by a ReLu activation function. The automatic segmentation using the U-Net generated an average dice similarity coefficient (DC) and standard deviation (SD) of the following: DC +- SD (0.88 +- 0.12), (0.95 +- 0.04), and (0.92 +- 0.06) for the prostate, bladder, and rectum, respectively. Furthermore, the mean of average surface Hausdorff distance (ASHD) and SD were 1.2 +- 0.9 mm, 1.08 +- 0.8 mm, and 0.8 +- 0.6 mm for the prostate, bladder, and rectum, respectively. Our proposed method, which employs the U-Net structure, is highly accurate and reproducible for automated ROI segmentation. This provides a foundation to improve automatic delineation of the boundaries between the target and surrounding normal soft tissues on a standard radiation therapy planning CT scan.

https://iopscience.iop.org/article/10.1088/2057-1976/aad100/pdf

Sarah McGuire

Papers

MRI-only brain radiotherapy: Assessing the dosimetric accuracy of synthetic CT images generated using a deep learning approach

Segmentation of the prostate and organs at risk in male pelvic ct images using deep learning

Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning