U-net is an image segmentation technique developed primarily for image segmentation tasks. These traits provide U-net with a high utility within the medical imaging community and have resulted in extensive adoption of U-net as the primary tool for segmentation tasks in medical imaging. The success of U-net is evident in its widespread use in nearly all major image modalities, from CT scans and MRI to X-rays and microscopy. Furthermore, while U-net is largely a segmentation tool, there have been instances of the use of U-net in other applications. Given that U-net’s potential is still increasing, this narrative literature review examines the numerous developments and breakthroughs in the U-net architecture and provides observations on recent trends. We also discuss the many innovations that have advanced in deep learning and discuss how these tools facilitate U-net. In addition, we review the different image modalities and application areas that have been enhanced by U-net.

U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications

Over half a million individuals are diagnosed with head and neck cancer each year worldwide Radiotherapy is an important curative treatment for this disease, but it requires manual time consuming delineation of radio-sensitive organs at risk (OARs) This planning process can delay treatment, while also introducing inter-operator variability with resulting downstream radiation dose differences While auto-segmentation algorithms offer a potentially time-saving solution, the challenges in defining, quantifying and achieving expert performance remain Adopting a deep learning approach, we demonstrate a 3D U-Net architecture that achieves expert-level performance in delineating 21 distinct head and neck OARs commonly segmented in clinical practice The model was trained on a dataset of 663 deidentified computed tomography (CT) scans acquired in routine clinical practice and with both segmentations taken from clinical practice and segmentations created by experienced radiographers as part of this research, all in accordance with consensus OAR definitions We demonstrate the model's clinical applicability by assessing its performance on a test set of 21 CT scans from clinical practice, each with the 21 OARs segmented by two independent experts We also introduce surface Dice similarity coefficient (surface DSC), a new metric for the comparison of organ delineation, to quantify deviation between OAR surface contours rather than volumes, better reflecting the clinical task of correcting errors in the automated organ segmentations The model's generalisability is then demonstrated on two distinct open source datasets, reflecting different centres and countries to model training With appropriate validation studies and regulatory approvals, this system could improve the efficiency, consistency, and safety of radiotherapy pathways

Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy

Background: Over half a million individuals are diagnosed with head and neck cancer each year globally. Radiotherapy is an important curative treatment for this disease, but it requires manual time to delineate radiosensitive organs at risk. This planning process can delay treatment while also introducing interoperator variability, resulting in downstream radiation dose differences. Although auto-segmentation algorithms offer a potentially time-saving solution, the challenges in defining, quantifying, and achieving expert performance remain.
Objective: Adopting a deep learning approach, we aim to demonstrate a 3D U-Net architecture that achieves expert-level performance in delineating 21 distinct head and neck organs at risk commonly segmented in clinical practice.
Methods: The model was trained on a data set of 663 deidentified computed tomography scans acquired in routine clinical practice and with both segmentations taken from clinical practice and segmentations created by experienced radiographers as part of this research, all in accordance with consensus organ at risk definitions.
Results: We demonstrated the model’s clinical applicability by assessing its performance on a test set of 21 computed tomography scans from clinical practice, each with 21 organs at risk segmented by 2 independent experts. We also introduced surface Dice similarity coefficient, a new metric for the comparison of organ delineation, to quantify the deviation between organ at risk surface contours rather than volumes, better reflecting the clinical task of correcting errors in automated organ segmentations. The model’s generalizability was then demonstrated on 2 distinct open-source data sets, reflecting different centers and countries to model training.
Conclusions: Deep learning is an effective and clinically applicable technique for the segmentation of the head and neck anatomy for radiotherapy. With appropriate validation studies and regulatory approvals, this system could improve the efficiency, consistency, and safety of radiotherapy pathways.

/pdf/clinically-applicable-segmentation-of-head-and-neck-anatomy-4giajeog60.pdf

Clinically Applicable Segmentation of Head and Neck Anatomy for Radiotherapy: Deep Learning Algorithm Development and Validation Study

Deep learning has become the mainstream technology in computer vision, and it has received extensive research interest in developing new medical image processing algorithms to support disease detection and diagnosis. As compared to conventional machine learning technologies, the major advantage of deep learning is that models can automatically identify and recognize representative features through the hierarchal model architecture, while avoiding the laborious development of hand-crafted features. In this paper, we reviewed and summarized more than 200 recently published papers to provide a comprehensive overview of applying deep learning methods in various medical image analysis tasks. Especially, we emphasize the latest progress and contributions of state-of-the-art unsupervised and semi-supervised deep learning in medical images, which are summarized based on different application scenarios, including lesion classification, segmentation, detection, and image registration. Additionally, we also discussed the major technical challenges and suggested the possible solutions in future research efforts.

Recent advances and clinical applications of deep learning in medical image analysis.

Radiotherapy (RT) is one of the basic treatment modalities for cancer of the head and neck (HN OAR - the spinal cord, brainstem, and major salivary glands are the most studied OARs, but additional experiments should be conducted for several less studied soft tissue structures; image database - several image databases with the corresponding ground truth are currently available for methodology evaluation, but should be augmented with data from multiple observers and multiple institutions; methodology - current methods have shifted from atlas-based to deep learning auto-segmentation, which is expected to become even more sophisticated; ground truth - delineation guidelines should be followed and participation of multiple experts from multiple institutions is recommended; performance metrics - the Dice coefficient as the standard volumetric overlap metrics should be accompanied with at least one distance metrics, and combined with clinical acceptability scores and risk assessments; segmentation performance - the best performing methods achieve clinically acceptable auto-segmentation for several OARs, however, the dosimetric impact should be also studied to provide clinically relevant endpoints for RT planning.

Auto-segmentation of organs at risk for head and neck radiotherapy planning: From atlas-based to deep learning methods.

Accurate segmentation of organs at risk (OARs) plays a critical role in the treatment planning of image-guided radiotherapy of head and neck cancer. This segmentation task is challenging for both humans and automated algorithms because of the relatively large number of OARs to be segmented, the large variability in size and morphology across different OARs, and the low contrast between some OARs and the background. In this study, we propose a two-stage segmentation framework based on 3D U-Net. In this framework, the segmentation of each OAR is decomposed into two subtasks: locating a bounding box of the OAR and segmenting the OAR from a small volume within the bounding box, and each subtask is fulfilled by a dedicated 3D U-Net. The decomposition makes each subtask much easier so that it can be better completed. We evaluated the proposed method and compared it to state-of-the-art methods using the Medical Image Computing and Computer-Assisted Intervention 2015 Challenge dataset. In terms of the boundary-based metric 95% Hausdorff distance, the proposed method ranked first for seven of nine OARs and ranked second for the other OARs. In terms of the area-based metric dice similarity coefficient, the proposed method ranked first for five of nine OARs and ranked second for the other three OARs with a small difference from the method that ranked first.

Organ at Risk Segmentation in Head and Neck CT Images Using a Two-Stage Segmentation Framework Based on 3D U-Net

Microvascular invasion (MVI) is a critical determinant of the early recurrence and poor prognosis of patients with hepatocellular carcinoma (HCC). Prediction of MVI status is clinically significant for the decision of treatment strategies and the assessment of patient’s prognosis. A deep learning (DL) model was developed to predict the MVI status and grade in HCC patients based on preoperative dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) and clinical parameters. HCC patients with pathologically confirmed MVI status from January to December 2016 were enrolled and preoperative DCE-MRI of these patients were collected in this study. Then they were randomly divided into the training and testing cohorts. A DL model with eight conventional neural network (CNN) branches for eight MRI sequences was built to predict the presence of MVI, and further combined with clinical parameters for better prediction. Among 601 HCC patients, 376 patients were pathologically MVI absent, and 225 patients were MVI present. To predict the presence of MVI, the DL model based only on images achieved an area under curve (AUC) of 0.915 in the testing cohort as compared to the radiomics model with an AUC of 0.731. The DL combined with clinical parameters (DLC) model yielded the best predictive performance with an AUC of 0.931. For the MVI-grade stratification, the DLC models achieved an overall accuracy of 0.793. Survival analysis demonstrated that the patients with DLC-predicted MVI status were associated with the poor overall survival (OS) and recurrence-free survival (RFS). Further investigation showed that hepatectomy with the wide resection margin contributes to better OS and RFS in the DLC-predicted MVI present patients. The proposed DLC model can provide a non-invasive approach to evaluate MVI before surgery, which can help surgeons make decisions of surgical strategies and assess patient’s prognosis.

/pdf/using-deep-learning-to-predict-microvascular-invasion-in-50m9e8sc19.pdf

Using deep learning to predict microvascular invasion in hepatocellular carcinoma based on dynamic contrast-enhanced MRI combined with clinical parameters.

In the past few years, convolutional neural networks (CNNs) have achieved milestones in medical image analysis. Especially, the deep neural networks based on U-shaped architecture and skip-connections have been widely applied in a variety of medical image tasks. However, although CNN has achieved excellent performance, it cannot learn global and long-range semantic information interaction well due to the locality of the convolution operation. In this paper, we propose Swin-Unet, which is an Unet-like pure Transformer for medical image segmentation. The tokenized image patches are fed into the Transformer-based U-shaped Encoder-Decoder architecture with skip-connections for local-global semantic feature learning. Specifically, we use hierarchical Swin Transformer with shifted windows as the encoder to extract context features. And a symmetric Swin Transformer-based decoder with patch expanding layer is designed to perform the up-sampling operation to restore the spatial resolution of the feature maps. Under the direct down-sampling and up-sampling of the inputs and outputs by 4x, experiments on multi-organ and cardiac segmentation tasks demonstrate that the pure Transformer-based U-shaped Encoder-Decoder network outperforms those methods with full-convolution or the combination of transformer and convolution. The codes and trained models will be publicly available at this https URL.

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation.

Selecting proper combination of mpMRI sequences for prostate cancer classification using multi-input convolutional neuronal network

Most traffic accidents are caused by bad driving habits. Online monitoring of the abnormal driving behaviors of drivers can help reduce traffic accidents. Recently, abnormal driving behavior recognition based on the sensors' data embedded in commodity smartphones has attracted much attention. Though much progress has been made about driving behavior recognition, the existing works cannot achieve high recognition accuracy and show poor robustness. To improve the driving behaviors recognition accuracy and robustness, we propose an algorithm based on Soft Thresholding and Temporal Convolutional Network (S‐TCN) for driving behavior recognition. In this algorithm, we first introduce a soft attention mechanism to learn the importance of different sensors. The TCN has the advantages of small memory requirement and high computational efficiency. And the soft thresholding can further filter the redundant features and extract the main features. So, we fuse the TCN and soft thresholding to improve the model's stability and accuracy. Our proposed model is extensively evaluated on four real public data sets. The experimental results show that our proposed model outperforms best state‐of‐the‐art baselines by 2.24%.

Yueyue Wang

Papers

Organ at Risk Segmentation in Head and Neck CT Images Using a Two-Stage Segmentation Framework Based on 3D U-Net

Using deep learning to predict microvascular invasion in hepatocellular carcinoma based on dynamic contrast-enhanced MRI combined with clinical parameters.

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation.

Selecting proper combination of mpMRI sequences for prostate cancer classification using multi-input convolutional neuronal network

An abnormal driving behavior recognition algorithm based on the temporal convolutional network and soft thresholding