U-net is an image segmentation technique developed primarily for image segmentation tasks. These traits provide U-net with a high utility within the medical imaging community and have resulted in extensive adoption of U-net as the primary tool for segmentation tasks in medical imaging. The success of U-net is evident in its widespread use in nearly all major image modalities, from CT scans and MRI to X-rays and microscopy. Furthermore, while U-net is largely a segmentation tool, there have been instances of the use of U-net in other applications. Given that U-net’s potential is still increasing, this narrative literature review examines the numerous developments and breakthroughs in the U-net architecture and provides observations on recent trends. We also discuss the many innovations that have advanced in deep learning and discuss how these tools facilitate U-net. In addition, we review the different image modalities and application areas that have been enhanced by U-net.

U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications

In the context of analyzing neck vascular morphology, this work formulates and compares Mask R-CNN and U-Net-based algorithms to automatically segment the carotid artery (CA) and internal jugular vein (IJV) from transverse neck ultrasound (US).
 US scans of the neck vasculature were collected to produce a dataset of 2439 images and their respective manual segmentations. Fourfold cross-validation was employed to train and evaluate Mask RCNN and U-Net models. The U-Net algorithm includes a post-processing step that selects the largest connected segmentation for each class. A Mask R-CNN-based vascular reconstruction pipeline was validated by performing a surface-to-surface distance comparison between US and CT reconstructions from the same patient.
 The average CA and IJV Dice scores produced by the Mask R-CNN across the evaluation data from all four sets were $$0.90\pm 0.08$$
 and $$0.88\pm 0.14$$
 . The average Dice scores produced by the post-processed U-Net were $$0.81\pm 0.21$$
 and $$0.71\pm 0.23$$
 , for the CA and IJV, respectively. The reconstruction algorithm utilizing the Mask R-CNN was capable of producing accurate 3D reconstructions with majority of US reconstruction surface points being within 2 mm of the CT equivalent. On average, the Mask R-CNN produced more accurate vascular segmentations compared to U-Net. The Mask R-CNN models were used to produce 3D reconstructed vasculature with a similar accuracy to that of a manually segmented CT scan. This implementation of the Mask R-CNN network enables automatic analysis of the neck vasculature and facilitates 3D vascular reconstruction.

Automatic segmentation of the carotid artery and internal jugular vein from 2D ultrasound images for 3D vascular reconstruction

Carotid ultrasound is a screening modality used by physicians to direct treatment in the prevention of ischemic stroke in high-risk patients. It is a time intensive process that requires highly trained technicians and physicians. Evaluation of a carotid ultrasound requires segmentation of the vessel wall, lumen, and plaque of the carotid artery. Convolutional neural networks are state of the art in image segmentation yet there are no previous methods to solve this problem on carotid ultrasounds. We introduce two novel convolutional U-net models for both vessel and plaque from ultrasound images of the entire carotid system. We obtained de-identified images under IRB approval from 226 patients. We isolated a total of 500 ultrasound images spanning the internal, external, and common carotid arteries. We manually segmented the vessel lumen and plaque in each image that we then use as ground truth. In 10-fold cross-validation all models attain over 90% accuracy for vessel segmentation. With a basic convolutional U-Net we obtained an accuracy of 66.8% for plaque segmentation. With our dual-decoder model we see an improvement to 68.8% whereas our two-stage model falls behind at 65.1% accuracy. However, if we gave our two-stage model the true correct vessel as input its plaque accuracy rises to 81.7% suggesting that the method has potential and needs more work. We ensemble our U-Net and dual decoder U-Net models to obtain confidence scores for segmentations. By considering high confidence outputs above the 60% and 80% thresholds the accuracy of our dual decoder U-Net rises to 75.2% and 87.3% respectively. Our work here shows the potential of dual and two-stage methods for vessel and plaque segmentation in carotid artery ultrasound images and is an important first step in creating a system that can independently evaluate carotid ultrasounds.

Two-stage and dual-decoder convolutional U-Net ensembles for reliable vessel and plaque segmentation in carotid ultrasound images

Medical imaging refers to several different technologies that are used to view the human body to diagnose, monitor, or treat medical conditions. It requires significant expertise to efficiently and correctly interpret the images generated by each of these technologies, which among others include radiography, ultrasound, and magnetic resonance imaging. Deep learning and machine learning techniques provide different solutions for medical image interpretation including those associated with detection and diagnosis. Despite the huge success of deep learning algorithms in image analysis, training algorithms to reach human-level performance in these tasks depends on the availability of large amounts of high-quality training data, including high-quality annotations to serve as ground-truth. Different annotation tools have been developed to assist with the annotation process. In this survey, we present the currently available annotation tools for medical imaging, including descriptions of graphical user interfaces (GUI) and supporting instruments. The main contribution of this study is to provide an intensive review of the popular annotation tools and show their successful usage in annotating medical imaging dataset to guide researchers in this area. 

/pdf/towards-a-better-understanding-of-annotation-tools-for-3adz6ugx.pdf

Towards a better understanding of annotation tools for medical imaging: a survey

APRIL: Anatomical prior-guided reinforcement learning for accurate carotid lumen diameter and intima-media thickness measurement.

Carotid ultrasound is a screening modality used by physicians to direct treatment in the prevention of ischemic stroke in high-risk patients. It is a time intensive process that requires highly trained technicians and physicians. Evaluation of a carotid ultrasound requires identification of the vessel wall, lumen, and plaque of the carotid artery. Automated machine learning methods for these tasks are highly limited. We propose and evaluate here single and multi-path convolutional U-neural network for lumen identification from ultrasound images. We obtained de-identified images under IRB approval from 98 patients. We isolated just the internal carotid artery ultrasound images for these patients giving us a total of 302 images. We manually segmented the vessel lumen, which we use as ground truth to develop and validate our model. With a basic simple convolutional U-Net we obtained a 10-fold cross-validation accuracy of 95%. We also evaluated a dual-path U-Net where we modified the original image and used it as a synthetic modality but we found no improvement in accuracy. We found that the sample size made a considerable difference and thus expect our accuracy to rise as we add more training samples to the model. Our work here represents a first successful step towards the automated identification of the vessel lumen in carotid artery ultrasound images and is an important first step in creating a system that can independently evaluate carotid ultrasounds.

Vessel lumen segmentation in internal carotid artery ultrasounds with deep convolutional neural networks

Carotid ultrasound is a screening modality used by physicians to direct treatment in the prevention of ischemic stroke in high-risk patients. It is a time intensive process that requires highly trained technicians and physicians. Evaluation of a carotid ultrasound requires segmentation of the vessel wall, lumen, and plaque of the carotid artery. Convolutional neural networks are state of the art in image segmentation yet there are no previous methods to solve this problem on carotid ultrasounds. We evaluate here a U-net convolutional neural network for lumen segmentation from ultrasound images of the entire carotid system. We obtained de-identified images under IRB approval from 226 patients. We isolated the internal, external, and common carotid artery ultrasound images for these patients giving us a total of 2156 images. We manually segmented the vessel lumen in each image that we then use as ground truth. With our convolutional U-Net we obtained a 10-fold cross-validation accuracy of 94.3%. We see that the U-Net correctly segments the lumen even in the presence of significant plaque, calcified wall, and ultrasound shadowing, all of which make it difficult to outline the vessel. We also see that the common carotid artery vessels are easiest to segment with a 96.6% cross-validation accuracy whereas internal and external carotid are harder both with 92.7% and 91.9% cross-validation accuracies respectively. Our work here represents a first successful step towards the automated segmentation of the vessel lumen in carotid artery ultrasound images and is an important first step in creating a system that can independently evaluate carotid ultrasounds.

Vessel lumen segmentation in carotid artery ultrasounds with the U-Net convolutional neural network

Adversarial attacks in medical AI imaging systems can lead to misdiagnosis and insurance fraud as recently highlighted by Finlayson et al in Science 2019 They can also be carried out on widely used ECG time-series data as shown in Han et al in Nature Medicine 2020 At the heart of adversarial attacks are imperceptible distortions that are visually and statistically undetectable but cause the machine learning model to misclassify data Recent empirical studies have shown that a gradient-free trained sign activation neural network ensemble model requires a larger distortion than state of the art models We apply them on medical data in this study as a potential solution to detect and deter adversarial attacks We show on chest X-ray and histopathology images, and on two ECG datasets that this model requires a greater distortion to be fooled than full-precision, binary, and convolutional neural networks, and random forests We show that adversaries targeting the gradient-free sign networks are visually distinguishable from the original data and thus likely to be detected by human inspection Since the sign network distortions are higher we expect an automated method could be developed to detect and deter attacks in advance Our work here is a significant step towards safe and secure medical machine learning

Justin Ady

Papers

Two-stage and dual-decoder convolutional U-Net ensembles for reliable vessel and plaque segmentation in carotid ultrasound images

Vessel lumen segmentation in internal carotid artery ultrasounds with deep convolutional neural networks

Vessel lumen segmentation in carotid artery ultrasounds with the U-Net convolutional neural network

Accurate and adversarially robust classification of medical images and ECG time-series with gradient-free trained sign activation neural networks

Modern mortality risk stratification scores accurately and equally predict real-world postoperative mortality after ruptured abdominal aortic aneurysm