Automatic Carotid ultrasound segmentation using deep Convolutional Neural Networks and phase congruency maps
18 Apr 2017-pp 624-628
TL;DR: Deep networks are proposed to be employed for automated segmentation of the media-adventitia boundary in transverse and longitudinal sections of carotid ultrasound images using an encoder-decoder convolutional structure which allows the network to be trained end-to-end for pixel-wise classification.
Abstract: The segmentation of media-adventitia and lumen-intima boundaries of the Carotid Artery forms an essential part in assessing plaque morphology in Ultrasound Imaging. Manual methods are tedious and prone to variability and thus, developing automated segmentation algorithms is preferable. In this paper, we propose to use deep convolutional networks for automated segmentation of the media-adventitia boundary in transverse and longitudinal sections of carotid ultrasound images. Deep networks have recently been employed with good success on image segmentation tasks, and we thus propose their application on ultrasound data, using an encoder-decoder convolutional structure which allows the network to be trained end-to-end for pixel-wise classification. Concurrently, we evaluate the performance for various configurations, depths and filter sizes within the network. In addition, we further propose a novel fusion of envelope and phase congruency data as an input to the network, as the latter provides an intensity-invariant data source to the network. We show that this data fusion and the proposed network structure yields higher segmentation performance than the state-of-the-art techniques.
Citations
More filters
TL;DR: A semiautomatic segmentation method based on deep learning to segment the MAB and LIB from carotid three‐dimensional ultrasound (3DUS) images shows high accuracy and repeatability with less user interactions, suggesting that the method could be used in clinical practice to measure VWV and monitor the progression and regression ofcarotid plaques.
Abstract: Purpose Quantification of carotid plaques has been shown to be important for assessing as well as monitoring the progression and regression of carotid atherosclerosis. Various metrics have been proposed and methods of measurements ranging from manual tracing to automated segmentations have also been investigated. Of those metrics, quantification of carotid plaques by measuring vessel-wall-volume (VWV) using the segmented media-adventitia (MAB) and lumen-intima (LIB) boundaries has been shown to be sensitive to temporal changes in carotid plaque burden. Thus, semi-automatic MAB and LIB segmentation methods are required to help generate VWV measurements with high accuracy and less user interaction. Methods In this paper, we propose a semiautomatic segmentation method based on deep learning to segment the MAB and LIB from carotid three-dimensional ultrasound (3DUS) images. For the MAB segmentation, we convert the segmentation problem to a pixel-by-pixel classification problem. A dynamic convolutional neural network (Dynamic CNN) is proposed to classify the patches generated by sliding a window along the norm line of the initial contour where the CNN model is fine-tuned dynamically in each test task. The LIB is segmented by applying a region-of-interest of carotid images to a U-Net model, which allows the network to be trained end-to-end for pixel-wise classification. Results A total of 144 3DUS images were used in this development, and a threefold cross-validation technique was used for evaluation of the proposed algorithm. The proposed algorithm-generated accuracy was significantly higher than the previous methods but with less user interactions. Comparing the algorithm segmentation results with manual segmentations by an expert showed that the average Dice similarity coefficients (DSC) were 96.46 ± 2.22% and 92.84 ± 4.46% for the MAB and LIB, respectively, while only an average of 34 s (vs 1.13, 2.8 and 4.4 min in previous methods) was required to segment a 3DUS image. The interobserver experiment indicated that the DSC was 96.14 ± 1.87% between algorithm-generated MAB contours of two observers' initialization. Conclusions Our results showed that the proposed carotid plaque segmentation method obtains high accuracy and repeatability with less user interactions, suggesting that the method could be used in clinical practice to measure VWV and monitor the progression and regression of carotid plaques.
68 citations
Cites methods from "Automatic Carotid ultrasound segmen..."
...Shin et al used a CNN to segment the intima and media boundaries automatically from videos of the carotid artery.27 For both these methods, a sliding window approach was applied to the whole image to detect the intima and media boundaries and both were applied to longitudinal US images of CCA....
[...]
...Azzopardi et al used deep CNN and phase congruency maps to segment MAB from B-mode 2DUS images of CCA.28 However, there was a correlation among images used in training and testing since all images were obtained from only five subjects....
[...]
...Some recent studies used deep learning methods to segment B-mode 2DUS images of CCA....
[...]
...Azzopardi et al used deep CNN and phase congruency maps to segment MAB from B-mode 2DUS images of CCA.(28) However, there was a correlation among images used in training and testing since all images were obtained from only five subjects....
[...]
...Rosa-Maria et al. applied extreme learning machine (ELM)-based autoencoders to segment the intima and media boundaries to obtain an IMT measurement from longitudinal B-mode 2DUS images of CCA.26 Although, the ELM-based autoencoders achieved a good performance to measure the IMT, it can only be used in the early-stage atherosclerosis detection....
[...]
TL;DR: The proposed DeepMAD network automatically segments the lumen and the outer wall together and diagnoses the carotid atherosclerosis with high performances and can be used in clinical trials to help radiologists get rid of tedious reading tasks.
Abstract: Purpose Early detection of carotid atherosclerosis on the vessel wall (VW) magnetic resonance imaging (MRI) (VW-MRI) images can prevent the progression of cardiovascular disease. However, the manual inspection process of the VW-MRI images is cumbersome and has low reproducibility. Therefore in this paper, by using the convolutional neural networks (CNNs), we develop a deep morphology aided diagnosis (DeepMAD) network for automated segmentation of the VW of carotid artery and for automated diagnosis of the carotid atherosclerosis with the black-blood (BB) VW-MRI (i.e., the T1-weighted MRI) in a slice-by-slice manner. Methods The proposed DeepMAD network consists of a segmentation subnetwork and a diagnosis subnetwork for performing the segmentation and diagnosis tasks on the BB-VW-MRI images, where the manual labeled lumen area, the manual labeled outer wall area and the manual labeled lesion Types based on the modified American Heart Association (AHA) criteria are used as the ground-truth. Specifically, a deep U-shape CNN with a weighted fusion layer is designed as the segmentation subnetwork, where the lumen area and the outer wall area can be simultaneously segmented under the supervision of the triple Dice loss to provide the vessel wall map as morphological information. Then, the image stream from the BB-VWMRI image and the morphology stream from the obtained vessel wall map are extracted from two deep CNNs and combined to obtain the diagnosis results of atherosclerosis in the diagnosis subnetwork. In addition, the triple input set is formed by three carotid regions of interest (ROIs) from three consecutive slices of the MRI sequence and input to the DeepMAD network, where the first and last slices used as additional adjacent slices to provide 2.5D spatial information along the carotid artery centerline for the intermediate slice, which is the target slice for segmentation and diagnosis in the study. Results Compared to other existing methods, the DeepMAD network can achieve promising segmentation performances (0.9594 Dice for the lumen and 0.9657 Dice for the outer wall) and better diagnosis Accuracy of the carotid atherosclerosis (0.9503 AUC and 0.8916 Accuracy) in the test dataset (including invisible subjects) from same source as the training dataset. In addition, the trained DeepMAD model can be successfully transferred to another test dataset for segmentation and diagnosis tasks with remarkable performance (0.9475 Dice for the lumen and 0.9542 Dice for the outer wall, 0. 9227 AUC and 0.8679 Accuracy for diagnosis). Conclusions Even without the intervention of reviewers required for previous works, the proposed DeepMAD network automatically segments the lumen and the outer wall together and diagnoses the carotid atherosclerosis with high performances. The DeepMAD network can be used in clinical trials to help radiologists get rid of tedious reading tasks, such as screening review to separate the normal carotid from the atherosclerotic arteries and outlining the vessel wall contours.
45 citations
TL;DR: A novel multi-output Convolutional neural network algorithm with dilated convolutional layers is presented to segment thyroid nodules, cystic components inside the nodules and normal thyroid gland from clinical ultrasound B-mode scans, eliminating the need for a seed in the segmentation process.
Abstract: Sonographic features associated with margins, shape, size, and volume of thyroid nodules are used to assess their risk of malignancy. Automatically segmenting nodules from normal thyroid gland would enable an automated estimation of these features. A novel multi-output convolutional neural network algorithm with dilated convolutional layers is presented to segment thyroid nodules, cystic components inside the nodules, and normal thyroid gland from clinical ultrasound B-mode scans. A prospective study was conducted, collecting data from 234 patients undergoing a thyroid ultrasound exam before biopsy. The training and validation sets encompassed 188 patients total; the testing set consisted of 48 patients. The algorithm effectively segmented thyroid anatomy into nodules, normal gland, and cystic components. The algorithm achieved a mean Dice coefficient of 0.76, a mean true positive fraction of 0.90, and a mean false positive fraction of $1.61\times 10^{-6}$ . The values are on par with a conventional seeded algorithm. The proposed algorithm eliminates the need for a seed in the segmentation process, thus automatically detecting and segmenting the thyroid nodules and cystic components. The detection rate for thyroid nodules and cystic components was 82% and 44%, respectively. The inference time per image, per fold was 107ms. The mean error in volume estimation of thyroid nodules for five select cases was 7.47%. The algorithm can be used for detection, segmentation, size estimation, volume estimation, and generating thyroid maps for thyroid nodules. The algorithm has applications in point of care, mobile health monitoring, improving workflow, reducing localization time, and assisting sonographers with limited expertise.
36 citations
TL;DR: The proposed method could be used clinically for generating VWV to monitor progression and regression of carotid plaques and required less than 1s, which will not alter the clinical work-flow as 10s is required to image one side of the neck.
Abstract: Vessel-wall-volume (VWV) is an important three-dimensional ultrasound (3DUS) metric used in the assessment of carotid plaque burden and monitoring changes in carotid atherosclerosis in response to medical treatment. To generate the VWV measurement, we proposed an approach that combined a voxel-based fully convolution network (Voxel-FCN) and a continuous max-flow module to automatically segment the carotid media-adventitia (MAB) and lumen-intima boundaries (LIB) from 3DUS images. Voxel-FCN includes an encoder consisting of a general 3D CNN and a 3D pyramid pooling module to extract spatial and contextual information, and a decoder using a concatenating module with an attention mechanism to fuse multi-level features extracted by the encoder. A continuous max-flow algorithm is used to improve the coarse segmentation provided by the Voxel-FCN. Using 1007 3DUS images, our approach yielded a Dice-similarity-coefficient (DSC) of 93.2±3.0% for the MAB in the common carotid artery (CCA), and 91.9±5.0% in the bifurcation by comparing algorithm and expert manual segmentations. We achieved a DSC of 89.5±6.7% and 89.3±6.8% for the LIB in the CCA and the bifurcation respectively. The mean errors between the algorithm-and manually-generated VWVs were 0.2±51.2 mm3 for the CCA and −4.0±98.2 mm3 for the bifurcation. The algorithm segmentation accuracy was comparable to intra-observer manual segmentation but our approach required less than 1s, which will not alter the clinical work-flow as 10s is required to image one side of the neck. Therefore, we believe that the proposed method could be used clinically for generating VWV to monitor progression and regression of carotid plaques.
30 citations
TL;DR: It is concluded that Deep Neural Networks provide a reliable trained manner in which carotid ultrasound images may be automatically segmented, using amplitude data and intensity invariant phase congruency maps as a data source.
Abstract: For asymptomatic patients suffering from carotid stenosis, the assessment of plaque morphology is an important clinical task which allows monitoring of the risk of plaque rupture and future incidents of stroke. Ultrasound Imaging provides a safe and non-invasive modality for this, and the segmentation of media-adventitia boundaries and lumen-intima boundaries of the Carotid artery form an essential part in this monitoring process. In this paper, we propose a novel Deep Neural Network as a fully automated segmentation tool, and its application in delineating both the media-adventitia boundary and the lumen-intima boundary. We develop a new geometrically constrained objective function as part of the Network's Stochastic Gradient Descent optimisation, thus tuning it to the problem at hand. Furthermore, we also apply a bimodal fusion of amplitude and phase congruency data proposed by us in previous work, as an input to the network, as the latter provides an intensity-invariant data source to the network. We finally report the segmentation performance of the network on transverse sections of the carotid. Tests are carried out on an augmented dataset of 81,000 images, and the results are compared to other studies by reporting the DICE coefficient of similarity, modified Hausdorff Distance, sensitivity and specificity. Our proposed modification is shown to yield improved results on the standard network over this larger dataset, with the advantage of it being fully automated. We conclude that Deep Neural Networks provide a reliable trained manner in which carotid ultrasound images may be automatically segmented, using amplitude data and intensity invariant phase congruency maps as a data source.
24 citations
Cites methods from "Automatic Carotid ultrasound segmen..."
...As a novel application over our previous work in [29], the network was trained to identify the contours of both the media...
[...]
...In our own previous preliminary work in 2017 [29], we segmented the MAB interface in transverse and longitudinal carotid images using deep convolutional neural networks....
[...]
...The depth of the U-Net structure was decreased from the original structure proposed in [42] given the smaller size of our input data, and given also the good performance-to-depth ratio achieved with the similar CNN tested in [29]....
[...]
...Our previous work in [29] carried out a similar ablation study but on MAB contours only and with a much smaller dataset....
[...]
References
More filters
Proceedings Article•
07 Dec 2015TL;DR: The highway networks as mentioned in this paper allow unimpeded information flow across many layers on information highways, inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow.
Abstract: Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.
1,218 citations
01 Jan 1995
TL;DR: Videre: Journal of Computer Vision Research is a quarterly journal published electronically on the Internet by The MIT Press, Cambridge, Massachusetts, 02142 and prices subject to change without notice.
Abstract: Videre: Journal of Computer Vision Research (ISSN 1089-2788) is a quarterly journal published electronically on the Internet by The MIT Press, Cambridge, Massachusetts, 02142. Subscriptions and address changes should be addressed to MIT Press Journals, Five Cambridge Center, Cambridge, MA 02142; phone: (617) 253-2889; fax: (617) 577-1545; e-mail: journals-orders@mit.edu. Subscription rates are: Individuals $30.00, Institutions $125.00. Canadians add additional 7% GST. Prices subject to change without notice.
1,186 citations
"Automatic Carotid ultrasound segmen..." refers methods in this paper
...In our case, we compute the 2D PC maps for each image using a more convenient method by Kovesi in [19], whereby the PC maps are obtained from the even and odd responses of a filter bank of quadrature logarithmic Gabor filters convolved with the signal....
[...]
TL;DR: A more general definition of features such as edges, shadows and bars is developed, based on an analysis of the phase of the harmonic components, showing that these features always occur at points of maximum phase congruency.
673 citations
"Automatic Carotid ultrasound segmen..." refers background in this paper
...The authors in [18] define phase congruency as:...
[...]
...The local energy model, proposed by [18], postulates that when Fourier components of an image are maximally in phase, and thus phase congruency (PC) reaches a maximum, features may be perceived, thus yielding an interesting alternative data representation....
[...]
22 Sep 2013
TL;DR: A novel system for voxel classification integrating three 2D CNNs, which have a one-to-one association with the xy, yz and zx planes of 3D image, respectively, which performs better than a state-of-the-art method using 3D multi-scale features.
Abstract: Segmentation of anatomical structures in medical images is often based on a voxel/pixel classification approach. Deep learning systems, such as convolutional neural networks (CNNs), can infer a hierarchical representation of images that fosters categorization. We propose a novel system for voxel classification integrating three 2D CNNs, which have a one-to-one association with the xy, yz and zx planes of 3D image, respectively. We applied our method to the segmentation of tibial cartilage in low field knee MRI scans and tested it on 114 unseen scans. Although our method uses only 2D features at a single scale, it performs better than a state-of-the-art method using 3D multi-scale features. In the latter approach, the features and the classifier have been carefully adapted to the problem at hand. That we were able to get better results by a deep learning architecture that autonomously learns the features from the images is the main insight of this study.
578 citations
"Automatic Carotid ultrasound segmen..." refers background or methods in this paper
...The sub-sampling layers are used to reduce the computational complexity by summarising the statistics of a feature over a region in the image [13]....
[...]
...can provide such a classification by constructing hierarchies of features from training data [13]....
[...]
...Previous work exists that applies DCNNs for medical image segmentation, although these focus on different applications [13, 14], or else use different structures to classify pixels within sub-image patches to estimate the I-M Thickness in longitudinal images [15]....
[...]
...At the end of the decoder network, the number of output maps are reduced to two, and fed into a softmax classifier, which provides logistic regression for a two class problem [13]....
[...]
...The convolutional layers learn small features from small image patches sampled from the whole image [13]....
[...]
TL;DR: In vitro studies suggest that macrophages and T cells release cytokines and proteinase, which stimulate breakdown of cap collagen and smooth muscle cell apoptosis and thereby promote plaque rupture, may be a critical step in promoting plaque rupture and resultant embolization or carotid occlusion.
Abstract: Background—The natural histories of equally severe symptomatic and asymptomatic carotid stenoses are very different, which suggests dichotomy in plaque behavior. The vascular biology of the symptomatic carotid plaque is presented in this review. Summary of Review—Histology studies comparing asymptomatic and symptomatic plaques were identified from MEDLINE. Reports in which stenosis severity was not stated or not similar for symptomatic and asymptomatic patients were excluded. In vitro studies and reports from the coronary circulation were reviewed with regard to the vascular biology of the plaque. Histology studies comparing carotid plaques removed from symptomatic and asymptomatic patients reveal characteristic features of unstable plaques: surface ulceration and plaque rupture (48% of symptomatic compared with 31% of asymptomatic, P<0.001), thinning of the fibrous cap, and infiltration of the cap by greater numbers of macrophages and T cells. In vitro studies suggest that macrophages and T cells release...
429 citations
"Automatic Carotid ultrasound segmen..." refers background in this paper
...When plaque ruptures in the carotid artery, there is a significant risk that the blood clot which forms will travel upstream to occlude a narrower vessel in the brain - ultimately leading to a stroke [3, 4]....
[...]