Deep learning methods have achieved significant results in various fields. Due to the success of these methods, many researchers have used deep learning algorithms in medical analyses. Using multimodal data to achieve more accurate results is a successful strategy because multimodal data provide complementary information. This paper first introduces the most popular modalities, fusion strategies, and deep learning architectures. We also explain learning strategies, including transfer learning, end-to-end learning, and multitask learning. Then, we give an overview of deep learning methods for multimodal medical data analysis. We have focused on articles published over the last four years. We end with a summary of the current state-of-the-art, common problems, and directions for future research. 

An overview of deep learning methods for multimodal medical data mining

Machine learning-based radiomic, clinical and semantic feature analysis for predicting overall survival and MGMT promoter methylation status in patients with glioblastoma.

To develop a weakly supervised deep learning (WSDL) method that could utilize incomplete/missing survival data to predict the prognosis of extranodal natural killer/T cell lymphoma, nasal type (ENKTL) based on pretreatment 18F-FDG PET/CT results. One hundred and sixty-seven patients with ENKTL who underwent pretreatment 18F-FDG PET/CT were retrospectively collected. Eighty-four patients were followed up for at least 2 years (training set = 64, test set = 20). A WSDL method was developed to enable the integration of the remaining 83 patients with incomplete/missing follow-up information in the training set. To test generalization, these data were derived from three types of scanners. Prediction similarity index (PSI) was derived from deep learning features of images. Its discriminative ability was calculated and compared with that of a conventional deep learning (CDL) method. Univariate and multivariate analyses helped explore the significance of PSI and clinical features. PSI achieved area under the curve scores of 0.9858 and 0.9946 (training set) and 0.8750 and 0.7344 (test set) in the prediction of progression-free survival (PFS) with the WSDL and CDL methods, respectively. PSI threshold of 1.0 could significantly differentiate the prognosis. In the test set, WSDL and CDL achieved prediction sensitivity, specificity, and accuracy of 87.50% and 62.50%, 83.33% and 83.33%, and 85.00% and 75.00%, respectively. Multivariate analysis confirmed PSI to be an independent significant predictor of PFS in both the methods. The WSDL-based framework was more effective for extracting 18F-FDG PET/CT features and predicting the prognosis of ENKTL than the CDL method.

/pdf/weakly-supervised-deep-learning-for-determining-the-4flynhnki1.pdf

Weakly supervised deep learning for determining the prognostic value of 18 F-FDG PET/CT in extranodal natural killer/T cell lymphoma, nasal type

Abstract Background Accurate characterization of glioma is crucial for clinical decision making. A delineation of the tumor is also desirable in the initial decision stages but is time-consuming. Previously, deep learning methods have been developed that can either non-invasively predict the genetic or histological features of glioma, or that can automatically delineate the tumor, but not both tasks at the same time. Here, we present our method that can predict the molecular subtype and grade, while simultaneously providing a delineation of the tumor. Methods We developed a single multi-task convolutional neural network that uses the full 3D, structural, preoperative MRI scans to predict the IDH mutation status, the 1p/19q co-deletion status, and the grade of a tumor, while simultaneously segmenting the tumor. We trained our method using a patient cohort containing 1508 glioma patients from 16 institutes. We tested our method on an independent dataset of 240 patients from 13 different institutes. Results In the independent test set, we achieved an IDH-AUC of 0.90, an 1p/19q co-deletion AUC of 0.85, and a grade AUC of 0.81 (grade II/III/IV). For the tumor delineation, we achieved a mean whole tumor Dice score of 0.84. Conclusions We developed a method that non-invasively predicts multiple, clinically relevant features of glioma. Evaluation in an independent dataset shows that the method achieves a high performance and that it generalizes well to the broader clinical population. This first-of-its-kind method opens the door to more generalizable, instead of hyper-specialized, AI methods.

/pdf/combined-molecular-subtyping-grading-and-segmentation-of-2jezl56g.pdf

Combined molecular subtyping, grading, and segmentation of glioma using multi-task deep learning

Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers and carries a dismal prognosis. Surgery remains the best chance of a potential cure for patients who are eligible for initial resection of PDAC. However, outcomes vary significantly even among the resected patients of the same stage and received similar treatments. Accurate preoperative prognosis of resectable PDACs for personalized treatment is thus highly desired. Nevertheless, there are no automated methods yet to fully exploit the contrast-enhanced computed tomography (CE-CT) imaging for PDAC. Tumor attenuation changes across different CT phases can reflect the tumor internal stromal fractions and vascularization of individual tumors that may impact the clinical outcomes. In this work, we propose a novel deep neural network for the survival prediction of resectable PDAC patients, named as 3D Contrast-Enhanced Convolutional Long Short-Term Memory network (CE-ConvLSTM), which can derive the tumor attenuation signatures or patterns from CE-CT imaging studies. We present a multi-task CNN to accomplish both tasks of outcome and margin prediction where the network benefits from learning the tumor resection margin related features to improve survival prediction. The proposed framework can improve the prediction performances compared with existing state-of-the-art survival analysis approaches. The tumor signature built from our model has evidently added values to be combined with the existing clinical staging system.

DeepPrognosis: Preoperative Prediction of Pancreatic Cancer Survival and Surgical Margin via Contrast-Enhanced CT Imaging

Glioblastoma (GBM) is the most common and deadly malignant brain tumor. For personalized treatment, an accurate pre-operative prognosis for GBM patients is highly desired. Recently, many machine learning-based methods have been adopted to predict overall survival (OS) time based on the pre-operative mono- or multi-modal imaging phenotype. The genotypic information of GBM has been proven to be strongly indicative of the prognosis; however, this has not been considered in the existing imaging-based OS prediction methods. The main reason is that the tumor genotype is unavailable pre-operatively unless deriving from craniotomy. In this paper, we propose a new deep learning-based OS prediction method for GBM patients, which can derive tumor genotype-related features from pre-operative multimodal magnetic resonance imaging (MRI) brain data and feed them to OS prediction. Specifically, we propose a multi-task convolutional neural network (CNN) to accomplish both tumor genotype and OS prediction tasks jointly . As the network can benefit from learning tumor genotype-related features for genotype prediction, the accuracy of predicting OS time can be prominently improved. In the experiments, multimodal MRI brain dataset of 120 GBM patients, with as many as four different genotypic/molecular biomarkers, are used to evaluate our method. Our method achieves the highest OS prediction accuracy compared to other state-of-the-art methods.

Deep Learning of Imaging Phenotype and Genotype for Predicting Overall Survival Time of Glioblastoma Patients

Glioblastoma (GBM) is the most common and deadly malignant brain tumor with short yet varied overall survival (OS) time. Per request of personalized treatment, accurate pre-operative prognosis for GBM patients is highly desired. Currently, many machine learning-based studies have been conducted to predict OS time based on pre-operative multimodal MR images of brain tumor patients. However, tumor genotype, such as MGMT and IDH, which has been proven to have strong relationship with OS, is completely not considered in pre-operative prognosis as the genotype information is unavailable until craniotomy. In this paper, we propose a new deep learning based method for OS time prediction. It can derive genotype related features from pre-operative multimodal MR images of brain tumor patients to guide OS time prediction. Particularly, we propose a multi-task convolutional neural network (CNN) to accomplish tumor genotype and OS time prediction tasks. As the network can benefit from learning genotype related features toward genotype prediction, we verify upon a dataset of 120 GBM patients and conclude that the multi-task learning can effectively improve the accuracy of predicting OS time in personalized prognosis.

Yuyun Xu

Papers

Deep Learning of Imaging Phenotype and Genotype for Predicting Overall Survival Time of Glioblastoma Patients

Pre-operative Overall Survival Time Prediction for Glioblastoma Patients Using Deep Learning on Both Imaging Phenotype and Genotype