Showing papers in "International Journal of Computer Assisted Radiology and Surgery in 2020"

DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images

Abstract: Gliomas are the most common and aggressive type of brain tumors due to their infiltrative nature and rapid progression. The process of distinguishing tumor boundaries from healthy cells is still a challenging task in the clinical routine. Fluid-attenuated inversion recovery (FLAIR) MRI modality can provide the physician with information about tumor infiltration. Therefore, this paper proposes a new generic deep learning architecture, namely DeepSeg, for fully automated detection and segmentation of the brain lesion using FLAIR MRI data. The developed DeepSeg is a modular decoupling framework. It consists of two connected core parts based on an encoding and decoding relationship. The encoder part is a convolutional neural network (CNN) responsible for spatial information extraction. The resulting semantic map is inserted into the decoder part to get the full-resolution probability map. Based on modified U-Net architecture, different CNN models such as residual neural network (ResNet), dense convolutional network (DenseNet), and NASNet have been utilized in this study. The proposed deep learning architectures have been successfully tested and evaluated on-line based on MRI datasets of brain tumor segmentation (BraTS 2019) challenge, including s336 cases as training data and 125 cases for validation data. The dice and Hausdorff distance scores of obtained segmentation results are about 0.81 to 0.84 and 9.8 to 19.7 correspondingly. This study showed successful feasibility and comparative performance of applying different deep learning models in a new DeepSeg framework for automated brain tumor segmentation in FLAIR MR images. The proposed DeepSeg is open source and freely available at https://github.com/razeineldin/DeepSeg/.

Two-stage ultrasound image segmentation using U-Net and test time augmentation.

Abstract: Detecting breast lesions using ultrasound imaging is an important application of computer-aided diagnosis systems. Several automatic methods have been proposed for breast lesion detection and segmentation; however, due to the ultrasound artefacts, and to the complexity of lesion shapes and locations, lesion or tumor segmentation from ultrasound breast images is still an open problem. In this paper, we propose using a lesion detection stage prior to the segmentation stage in order to improve the accuracy of the segmentation. We used a breast ultrasound imaging dataset which contained 163 images of the breast with either benign lesions or malignant tumors. First, we used a U-Net to detect the lesions and then used another U-Net to segment the detected region. We could show when the lesion is precisely detected, the segmentation performance substantially improves; however, if the detection stage is not precise enough, the segmentation stage also fails. Therefore, we developed a test-time augmentation technique to assess the detection stage performance. By using the proposed two-stage approach, we could improve the average Dice score by 1.8% overall. The improvement was substantially more for images wherein the original Dice score was less than 70%, where average Dice score was improved by 14.5%. The proposed two-stage technique shows promising results for segmentation of breast US images and has a much smaller chance of failure.

Multiplanar analysis for pulmonary nodule classification in CT images using deep convolutional neural network and generative adversarial networks

Abstract: Early detection and treatment of lung cancer holds great importance. However, pulmonary-nodule classification using CT images alone is difficult to realize. To address this concern, a method for pulmonary-nodule classification based on a deep convolutional neural network (DCNN) and generative adversarial networks (GAN) has previously been proposed by the authors. In that method, the said classification was performed exclusively using axial cross sections of pulmonary nodules. During actual medical-examination procedures, however, a comprehensive judgment can only be made via observation of various pulmonary-nodule cross sections. In the present study, a comprehensive analysis was performed by extending the application of the previously proposed DCNN- and GAN-based automatic classification method to multiple cross sections of pulmonary nodules. Using the proposed method, CT images of 60 cases with confirmed pathological diagnosis by biopsy are analyzed. Firstly, multiplanar images of the pulmonary nodule are generated. Classification training was performed for three DCNNs. A certain pretraining was initially performed using GAN-generated nodule images. This was followed by fine-tuning of each pretrained DCNN using original nodule images provided as input. As a result of the evaluation, the specificity was 77.8% and the sensitivity was 93.9%. Additionally, the specificity was observed to have improved by 11.1% without any reduction in the sensitivity, compared to our previous report. This study reports development of a comprehensive analysis method to classify pulmonary nodules at multiple sections using GAN and DCNN. The effectiveness of the proposed discrimination method based on use of multiplanar images has been demonstrated to be improved compared to that realized in a previous study reported by the authors. In addition, the possibility of enhancing classification accuracy via application of GAN-generated images, instead of data augmentation, for pretraining even for medical datasets that contain relatively few images has also been demonstrated.

A manifold learning regularization approach to enhance 3D CT image-based lung nodule classification

Abstract: Diagnosis of lung cancer requires radiologists to review every lung nodule in CT images. Such a process can be very time-consuming, and the accuracy is affected by many factors, such as experience of radiologists and available diagnosis time. To address this problem, we proposed to develop a deep learning-based system to automatically classify benign and malignant lung nodules. The proposed method automatically determines benignity or malignancy given the 3D CT image patch of a lung nodule to assist diagnosis process. Motivated by the fact that real structure among data is often embedded on a low-dimensional manifold, we developed a novel manifold regularized classification deep neural network (MRC-DNN) to perform classification directly based on the manifold representation of lung nodule images. The concise manifold representation revealing important data structure is expected to benefit the classification, while the manifold regularization enforces strong, but natural constraints on network training, preventing over-fitting. The proposed method achieves accurate manifold learning with reconstruction error of ~ 30 HU on real lung nodule CT image data. In addition, the classification accuracy on testing data is 0.90 with sensitivity of 0.81 and specificity of 0.95, which outperforms state-of-the-art deep learning methods. The proposed MRC-DNN facilitates an accurate manifold learning approach for lung nodule classification based on 3D CT images. More importantly, MRC-DNN suggests a new and effective idea of enforcing regularization for network training, possessing the potential impact to a board range of applications.

Deep learning for automated cerebral aneurysm detection on computed tomography images

Abstract: Cerebrovascular aneurysms are being observed with rapidly increasing incidence. Therefore, tools are needed for accurate and efficient detection of aneurysms. We used deep learning techniques with CT angiography acquired from multiple medical centers and different machines to develop and evaluate an automatic detection model. In this study, we have introduced a deep learning model, the faster RCNN model, in order to develop a tool for automatic detection of aneurysms from medical images. The inputs of the model were 2D nearby projection (NP) images from 3D CTA, which were made by the NP method proposed in this study. This method made aneurysms clearly visible on images and improved the model’s performance. The study included 311 patients with 352 aneurysms, selected from three hospitals, and 208 and 103 of these patients, respectively, were randomly selected to train and test the models. The sensitivity of the trained model was 91.8%. For aneurysm sizes larger than 3 mm, the sensitivity of successful aneurysm detection was 96.7%. We achieved state-of-the-art sensitivity for > 3 mm aneurysms. The sensitivities also indicated that there was no significant difference among aneurysms at different locations in the body. Computing time for the detection process was less than 25 s per case. We successfully developed a deep learning model that can automatically detect aneurysms. The model performed well for aneurysms of different sizes or in different locations. This finding indicates that the deep learning model has the potential to vastly improve clinician performance by providing automated aneurysm detection.

Toward automatic quantification of knee osteoarthritis severity using improved Faster R-CNN

Abstract: Knee osteoarthritis (OA) is a common disease that impairs knee function and causes pain. Radiologists usually review knee X-ray images and grade the severity of the impairments according to the Kellgren–Lawrence grading scheme. However, this approach becomes inefficient in hospitals with high throughput as it is time-consuming, tedious and also subjective. This paper introduces a model for automatic diagnosis of knee OA based on an end-to-end deep learning method. In order to process the input images with location and classification simultaneously, we use Faster R-CNN as baseline, which consists of region proposal network (RPN) and Fast R-CNN. The RPN is trained to generate region proposals, which contain knee joint and then be used by Fast R-CNN for classification. Due to the localized classification via CNNs, the useless information in X-ray images can be filtered and we can extract clinically relevant features. For the further improvement in the model’s performance, we use a novel loss function whose weighting scheme allows us to address the class imbalance. Besides, larger anchors are used to overcome the problem that anchors don’t match the object when increasing the input size of X-ray images. The performance of the proposed model is thoroughly assessed using various measures. The results show that our adjusted model outperforms the Faster R-CNN, achieving a mean average precision nearly 0.82 with a sensitivity above 78% and a specificity above 94%. It takes 0.33 s to test each image, which achieves a better trade-off between accuracy and speed. The proposed end-to-end fully automatic model which is computationally efficient has the potential to achieve the real automatic diagnosis of knee OA and be used as computer-aided diagnosis tools in clinical applications.

Multimodal mixed reality visualisation for intraoperative surgical guidance.

Abstract: In the last decade, there has been a great effort to bring mixed reality (MR) into the operating room to assist surgeons intraoperatively. However, progress towards this goal is still at an early stage. The aim of this paper is to propose a MR visualisation platform which projects multiple imaging modalities to assist intraoperative surgical guidance. In this work, a MR visualisation platform has been developed for the Microsoft HoloLens. The platform contains three visualisation components, namely a 3D organ model, volumetric data, and tissue morphology captured with intraoperative imaging modalities. Furthermore, a set of novel interactive functionalities have been designed including scrolling through volumetric data and adjustment of the virtual objects’ transparency. A pilot user study has been conducted to evaluate the usability of the proposed platform in the operating room. The participants were allowed to interact with the visualisation components and test the different functionalities. Each surgeon answered a questionnaire on the usability of the platform and provided their feedback and suggestions. The analysis of the surgeons’ scores showed that the 3D model is the most popular MR visualisation component and neurosurgery is the most relevant speciality for this platform. The majority of the surgeons found the proposed visualisation platform intuitive and would use it in their operating rooms for intraoperative surgical guidance. Our platform has several promising potential clinical applications, including vascular neurosurgery. The presented pilot study verified the potential of the proposed visualisation platform and its usability in the operating room. Our future work will focus on enhancing the platform by incorporating the surgeons’ suggestions and conducting extensive evaluation on a large group of surgeons.

Normative ranges of nasal airflow variables in healthy adults

Abstract: Virtual surgery planning based on computational fluid dynamics (CFD) simulations of nasal airflow has the potential to improve surgical outcomes for patients with nasal airway obstruction (NAO). Virtual surgery planning requires normative ranges of airflow variables, but few studies to date have quantified inter-individual variability of nasal airflow among healthy subjects. This study reports CFD simulations of nasal airflow in 47 healthy adults. Anatomically accurate three-dimensional nasal models were reconstructed from cone beam computed tomography scans and used for steady-state inspiratory airflow simulations with a bilateral flowrate of 250 ml/s. Normal subjective sensation of nasal patency was confirmed using the nasal obstruction symptom evaluation and visual analog scale. Healthy ranges for several CFD variables known to correlate with subjective nasal patency were computed, including unilateral airflow, nasal resistance, airspace minimal cross-sectional area (mCSA), heat flux (HF), and surface area stimulated by mucosal cooling (defined as the area where HF > 50 W/m2). The normative ranges were targeted to contain 95% of the healthy population and computed using a nonparametric method based on order statistics. A wide range of inter-individual variability in nasal airflow was observed among healthy subjects. Unilateral airflow varied from 60 to 191 ml/s, airflow partitioning ranged from 23.8 to 76.2%, and unilateral mCSA varied from 0.24 to 1.21 cm2. These ranges are in good agreement with rhinomanometry and acoustic rhinometry data from the literature. A key innovation of this study are the normative ranges of flow variables associated with mucosal cooling, which recent research suggests is the primary physiological mechanism of nasal airflow sensation. Unilateral HF ranged from 94 to 281 W/m2, while the surface area stimulated by cooling ranged from 27.4 to 64.3 cm2. These normative ranges may serve as targets in future virtual surgery planning for patients with NAO.

Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration

Abstract: Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. In this paper, we propose a method for fully automatic registration using anatomical annotations produced by a neural network. Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data are obtained using a computationally intensive, intraoperatively incompatible, 2D/3D registration of the pelvis and each femur. Ground truth 2D segmentation labels and anatomical landmark locations are established using projected 3D annotations. Intraoperative registration couples a traditional intensity-based strategy with annotations inferred by the network and requires no human assistance. Ground truth segmentation labels and anatomical landmarks were obtained in 366 fluoroscopic images across 6 cadaveric specimens. In a leave-one-subject-out experiment, networks trained on these data obtained mean dice coefficients for left and right hemipelves, left and right femurs of 0.86, 0.87, 0.90, and 0.84, respectively. The mean 2D landmark localization error was 5.0 mm. The pelvis was registered within $$1^{\circ }$$ for 86% of the images when using the proposed intraoperative approach with an average runtime of 7 s. In comparison, an intensity-only approach without manual initialization registered the pelvis to $$1^{\circ }$$ in 18% of images. We have created the first accurately annotated, non-synthetic, dataset of hip fluoroscopy. By using these annotations as training data for neural networks, state-of-the-art performance in fluoroscopic segmentation and landmark localization was achieved. Integrating these annotations allows for a robust, fully automatic, and efficient intraoperative registration during fluoroscopic navigation of the hip.

Contour-aware multi-label chest X-ray organ segmentation

Abstract: Segmentation of organs from chest X-ray images is an essential task for an accurate and reliable diagnosis of lung diseases and chest organ morphometry. In this study, we investigated the benefits of augmenting state-of-the-art deep convolutional neural networks (CNNs) for image segmentation with organ contour information and evaluated the performance of such augmentation on segmentation of lung fields, heart, and clavicles from chest X-ray images. Three state-of-the-art CNNs were augmented, namely the UNet and LinkNet architecture with the ResNeXt feature extraction backbone, and the Tiramisu architecture with the DenseNet. All CNN architectures were trained on ground-truth segmentation masks and additionally on the corresponding contours. The contribution of such contour-based augmentation was evaluated against the contour-free architectures, and 20 existing algorithms for lung field segmentation. The proposed contour-aware segmentation improved the segmentation performance, and when compared against existing algorithms on the same publicly available database of 247 chest X-ray images, the UNet architecture with the ResNeXt50 encoder combined with the contour-aware approach resulted in the best overall segmentation performance, achieving a Jaccard overlap coefficient of 0.971, 0.933, and 0.903 for the lung fields, heart, and clavicles, respectively. In this study, we proposed to augment CNN architectures for CXR segmentation with organ contour information and were able to significantly improve segmentation accuracy and outperform all existing solution using a public chest X-ray database.

Multi-input deep learning architecture for predicting breast tumor response to chemotherapy using quantitative MR images.

Abstract: Neoadjuvant chemotherapy (NAC) aims to minimize the tumor size before surgery Predicting response to NAC could reduce toxicity and delays to effective intervention Computational analysis of dynamic contrast-enhanced magnetic resonance images (DCE-MRI) through deep convolution neural network (CNN) has shown a significant performance to distinguish responders and no responder’s patients This study intends to present a new deep learning (DL) model predicting the breast cancer response to NAC based on multiple MRI inputs A cohort of 723 axial slices extracted from 42 breast cancer patients who underwent NAC therapy was used to train and validate the developed DL model This dataset was provided by our collaborator institute of radiology in Brussels Fourteen external cases were used to validate the best obtained model to predict pCR based on pre- and post-chemotherapy DCE-MRI The model performance was assessed by area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and feature map visualization The developed multi-inputs deep learning architecture was able to predict the pCR to NAC treatment in the validation dataset with an AUC of 091 using combined pre- and post-NAC images The visual results showed that the most important extracted features from non-pCR tumors are in the peripheral region The proposed method was more productive than the previous ones Even with a limited training dataset size, the proposed and developed CNN model using DCE-MR images acquired before and after the first chemotherapy was able to classify pCR and non-pCR patients with substantial accuracy This model could be used hereafter in clinical analysis after its evaluation based on more extra data

CT-based radiomics for prediction of histologic subtype and metastatic disease in primary malignant lung neoplasms.

Abstract: As some of the most important factors for treatment decision of lung cancer (which is the deadliest neoplasm) are staging and histology, this work aimed to associate quantitative contrast-enhanced computed tomography (CT) features from malignant lung tumors with distant and nodal metastases (according to clinical TNM staging) and histopathology (according to biopsy and surgical resection) using radiomics assessment. A local cohort of 85 patients were retrospectively (2010–2017) analyzed after approval by the institutional research review board. CT images acquired with the same protocol were semiautomatically segmented by a volumetric segmentation method. Tumors were characterized by quantitative CT features of shape, first-order, second-order, and higher-order textures. Statistical and machine learning analyses assessed the features individually and combined with clinical data. Univariate and multivariate analyses identified 40, 2003, and 45 quantitative features associated with distant metastasis, nodal metastasis, and histopathology (adenocarcinoma and squamous cell carcinoma), respectively. A machine learning model yielded the highest areas under the receiver operating characteristic curves of 0.92, 0.84, and 0.88 to predict the same previous patterns. Several radiomic features (including wavelet energies, information measures of correlation and maximum probability from co-occurrence matrix, busyness from neighborhood intensity-difference matrix, directionalities from Tamura’s texture, and fractal dimension estimation) significantly associated with distant metastasis, nodal metastasis, and histology were discovered in this work, presenting great potential as imaging biomarkers for pathological diagnosis and target therapy decision.

Assisted phase and step annotation for surgical videos

Abstract: Annotation of surgical videos is a time-consuming task which requires specific knowledge. In this paper, we present and evaluate a deep learning-based method that includes pre-annotation of the phases and steps in surgical videos and user assistance in the annotation process. We propose a classification function that automatically detects errors and infers temporal coherence in predictions made by a convolutional neural network. First, we trained three different architectures of neural networks to assess the method on two surgical procedures: cholecystectomy and cataract surgery. The proposed method was then implemented in an annotation software to test its ability to assist surgical video annotation. A user study was conducted to validate our approach, in which participants had to annotate the phases and the steps of a cataract surgery video. The annotation and the completion time were recorded. The participants who used the assistance system were 7% more accurate on the step annotation and 10 min faster than the participants who used the manual system. The results of the questionnaire showed that the assistance system did not disturb the participants and did not complicate the task. The annotation process is a difficult and time-consuming task essential to train deep learning algorithms. In this publication, we propose a method to assist the annotation of surgical workflows which was validated through a user study. The proposed assistance system significantly improved annotation duration and accuracy.

Fast and accurate online calibration of optical see-through head-mounted display for AR-based surgical navigation using Microsoft HoloLens

Abstract: The use of an optical see-through head-mounted display (OST-HMD) in augmented reality (AR) has significantly increased in recent years, but the alignment between the virtual scene and physical reality is still a challenge. A fast and accurate calibration method of OST-HMD is important for augmented reality in the medical field. We proposed a fast online calibration procedure for OST-HMD with the aid of an optical tracking system. Two 3D datasets are collected in this procedure: the virtual points rendered in front of the observer’s eyes and the corresponding points in optical tracking space. The transformation between these two 3D coordinates is solved to build the connection between virtual and real space. An AR-based surgical navigation system is developed with the help of this procedure, which is used for experiment verification and clinical trial. Phantom experiment based on the 3D-printed skull is performed, and the average root-mean-square error of control points between rendered object and skull model is $$1.30 \pm 0.39$$ mm, and the time consumption of the calibration procedure can be less than 30 s. A clinical trial is also conducted to show the feasibility in real surgery theatre. The proposed calibration method does not rely on the camera of the OST-HMD and is very easy to operate. Phantom experiment and clinical case demonstrated the feasibility of our AR-based surgical navigation system and indicated it has the potential in clinical application.

Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs.

Abstract: In this paper, we propose to apply generative adversarial neural networks trained with a cycle consistency loss, or CycleGANs, to improve realism in ultrasound (US) simulation from computed tomography (CT) scans. A ray-casting US simulation approach is used to generate intermediate synthetic images from abdominal CT scans. Then, an unpaired set of these synthetic and real US images is used to train CycleGANs with two alternative architectures for the generator, a U-Net and a ResNet. These networks are finally used to translate ray-casting based simulations into more realistic synthetic US images. Our approach was evaluated both qualitatively and quantitatively. A user study performed by 21 experts in US imaging shows that both networks significantly improve realism with respect to the original ray-casting algorithm ($$p \ll 0.0001$$), with the ResNet model performing better than the U-Net ($$p \ll 0.0001$$). Applying CycleGANs allows to obtain better synthetic US images of the abdomen. These results can contribute to reduce the gap between artificially generated and real US scans, which might positively impact in applications such as semi-supervised training of machine learning algorithms and low-cost training of medical doctors and radiologists in US image interpretation.

SpeckleGAN: a generative adversarial network with an adaptive speckle layer to augment limited training data for ultrasound image processing

Abstract: In the field of medical image analysis, deep learning methods gained huge attention over the last years This can be explained by their often improved performance compared to classic explicit algorithms In order to work well, they need large amounts of annotated data for supervised learning, but these are often not available in the case of medical image data One way to overcome this limitation is to generate synthetic training data, eg, by performing simulations to artificially augment the dataset However, simulations require domain knowledge and are limited by the complexity of the underlying physical model Another method to perform data augmentation is the generation of images by means of neural networks We developed a new algorithm for generation of synthetic medical images exhibiting speckle noise via generative adversarial networks (GANs) Key ingredient is a speckle layer, which can be incorporated into a neural network in order to add realistic and domain-dependent speckle We call the resulting GAN architecture SpeckleGAN We compared our new approach to an equivalent GAN without speckle layer SpeckleGAN was able to generate ultrasound images with very crisp speckle patterns in contrast to the baseline GAN, even for small datasets of 50 images SpeckleGAN outperformed the baseline GAN by up to 165 % with respect to the Frechet Inception distance For artery layer and lumen segmentation, a performance improvement of up to 4 % was obtained for small datasets, when these were augmented with images by SpeckleGAN SpeckleGAN facilitates the generation of realistic synthetic ultrasound images to augment small training sets for deep learning based image processing Its application is not restricted to ultrasound images but could be used for every imaging methodology that produces images with speckle such as optical coherence tomography or radar

A microsurgical robot research platform for robot-assisted microsurgery research and training

Abstract: Ocular surgery, ear, nose and throat surgery and neurosurgery are typical types of microsurgery. A versatile training platform can assist microsurgical skills development and accelerate the uptake of robot-assisted microsurgery (RAMS). However, the currently available platforms are mainly designed for macro-scale minimally invasive surgery. There is a need to develop a dedicated microsurgical robot research platform for both research and clinical training. A microsurgical robot research platform (MRRP) is introduced in this paper. The hardware system includes a slave robot with bimanual manipulators, two master controllers and a vision system. It is flexible to support multiple microsurgical tools. The software architecture is developed based on the robot operating system, which is extensible at high-level control. The selection of master–slave mapping strategy was explored, while comparisons were made between different interfaces. Experimental verification was conducted based on two microsurgical tasks for training evaluation, i.e. trajectory following and targeting. User study results indicated that the proposed hybrid interface is more effective than the traditional approach in terms of frequency of clutching, task completion time and ease of control. Results indicated that the MRRP can be utilized for microsurgical skills training, since motion kinematic data and vision data can provide objective means of verification and scoring. The proposed system can further be used for verifying high-level control algorithms and task automation for RAMS research.

Robotic fetal ultrasonography platform with a passive scan mechanism

Abstract: The shortage of obstetricians and gynecologists has intensified in developed countries. Our long-term goal is to develop a robotic prenatal care platform for automatic ultrasound (US) scanning to improve the workflow efficiency of obstetricians and gynecologists. This paper develops a hardware platform for the positioning of the US probe to obtain diagnostic US images while satisfying safety requirements of the fetus and pregnant woman. The proposed system includes a mechanism that maintains the contact force in a certain range and passively adjusts the US probe posture relative to the body surface. The system is designed according to clinical survey data. For proof of concept, we conducted a robotic US scan with an agar phantom and three pregnant women under the operation of a physician. Experimental results show the passive US scan motion followed the phantom surface with an acceptable contact force (< 15 N). Clinical trials were safely carried out with observations of fetal body parts. Our proposed platform acquired US images with satisfactory contact forces in the phantom study. The feasibility of the platform was demonstrated in a clinical study.

Automated measurement of bone scan index from a whole-body bone scintigram

Abstract: We propose a deep learning-based image interpretation system for skeleton segmentation and extraction of hot spots of bone metastatic lesion from a whole-body bone scintigram followed by automated measurement of a bone scan index (BSI), which will be clinically useful. The proposed system employs butterfly-type networks (BtrflyNets) for skeleton segmentation and extraction of hot spots of bone metastatic lesions, in which a pair of anterior and posterior images are processed simultaneously. BSI is then measured using the segmented bones and extracted hot spots. To further improve the networks, deep supervision (DSV) and residual learning technologies were introduced. We evaluated the performance of the proposed system using 246 bone scintigrams of prostate cancer in terms of accuracy of skeleton segmentation, hot spot extraction, and BSI measurement, as well as computational cost. In a threefold cross-validation experiment, the best performance was achieved by BtrflyNet with DSV for skeleton segmentation and BtrflyNet with residual blocks. The cross-correlation between the measured and true BSI was 0.9337, and the computational time for a case was 112.0 s. We proposed a deep learning-based BSI measurement system for a whole-body bone scintigram and proved its effectiveness by threefold cross-validation study using 246 whole-body bone scintigrams. The automatically measured BSI and computational time for a case are deemed clinically acceptable and reliable.

Toward automatic C-arm positioning for standard projections in orthopedic surgery.

Abstract: Guidance and quality control in orthopedic surgery increasingly rely on intra-operative fluoroscopy using a mobile C-arm. The accurate acquisition of standardized and anatomy-specific projections is essential in this process. The corresponding iterative positioning of the C-arm is error prone and involves repeated manual acquisitions or even continuous fluoroscopy. To reduce time and radiation exposure for patients and clinical staff and to avoid errors in fracture reduction or implant placement, we aim at guiding—and in the long-run automating—this procedure. In contrast to the state of the art, we tackle this inherently ill-posed problem without requiring patient-individual prior information like preoperative computed tomography (CT) scans, without the need of registration and without requiring additional technical equipment besides the projection images themselves. We propose learning the necessary anatomical hints for efficient C-arm positioning from in silico simulations, leveraging masses of 3D CTs. Specifically, we propose a convolutional neural network regression model that predicts 5 degrees of freedom pose updates directly from a first X-ray image. The method is generalizable to different anatomical regions and standard projections. Quantitative and qualitative validation was performed for two clinical applications involving two highly dissimilar anatomies, namely the lumbar spine and the proximal femur. Starting from one initial projection, the mean absolute pose error to the desired standard pose is iteratively reduced across different anatomy-specific standard projections. Acquisitions of both hip joints on 4 cadavers allowed for an evaluation on clinical data, demonstrating that the approach generalizes without retraining. Overall, the results suggest the feasibility of an efficient deep learning-based automated positioning procedure, which is trained on simulations. Our proposed 2-stage approach for C-arm positioning significantly improves accuracy on synthetic images. In addition, we demonstrated that learning based on simulations translates to acceptable performance on real X-rays.

Ultrasound needle segmentation and trajectory prediction using excitation network

Abstract: Ultrasound (US)-guided percutaneous kidney biopsy is a challenge for interventionists as US artefacts prevent accurate viewing of the biopsy needle tip. Automatic needle tracking and trajectory prediction can increase operator confidence in performing biopsies, reduce procedure time, minimize the risk of inadvertent biopsy bleedings, and enable future image-guided robotic procedures. In this paper, we propose a tracking-by-segmentation model with spatial and channel “Squeeze and Excitation” (scSE) for US needle detection and trajectory prediction. We adopt a light deep learning architecture (e.g. LinkNet) as our segmentation baseline network and integrate the scSE module to learn spatial information for better prediction. The proposed model is trained with the US images of anonymized kidney biopsy clips from 8 patients. The contour is obtained using the border-following algorithm and area calculated using Green formula. Trajectory prediction is made by extrapolating from the smallest bounding box that can capture the contour. We train and test our model on a total of 996 images extracted from 102 short videos at a rate of 3 frames per second from each video. A set of 794 images is used for training and 202 images for testing. Our model has achieved IOU of 41.01%, dice accuracy of 56.65%, F1-score of 36.61%, and root-mean-square angle error of 13.3$$^{\circ }$$. We are thus able to predict and extrapolate the trajectory of the biopsy needle with decent accuracy for interventionists to better perform biopsies. Our novel model combining LinkNet and scSE shows a promising result for kidney biopsy application, which implies potential to other similar ultrasound-guided biopsies that require needle tracking and trajectory prediction.

Fast interactive medical image segmentation with weakly supervised deep learning method

Abstract: To achieve accurate image segmentation, which is the first critical step in medical image analysis and interventions, using deep neural networks seems a promising approach provided sufficiently large and diverse annotated data from experts. However, annotated datasets are often limited because it is prone to variations in acquisition parameters and require high-level expert’s knowledge, and manually labeling targets by tracing their contour is often laborious. Developing fast, interactive, and weakly supervised deep learning methods is thus highly desirable. We propose a new efficient deep learning method to accurately segment targets from images while generating an annotated dataset for deep learning methods. It involves a generative neural network-based prior-knowledge prediction from pseudo-contour landmarks. The predicted prior knowledge (i.e., contour proposal) is then refined using a convolutional neural network that leverages the information from the predicted prior knowledge and the raw input image. Our method was evaluated on a clinical database of 145 intraoperative ultrasound and 78 postoperative CT images of image-guided prostate brachytherapy. It was also evaluated on a cardiac multi-structure segmentation from 450 2D echocardiographic images. Experimental results show that our model can segment the prostate clinical target volume in 0.499 s (i.e., 7.79 milliseconds per image) with an average Dice coefficient of 96.9 ± 0.9% and 95.4 ± 0.9%, 3D Hausdorff distance of 4.25 ± 4.58 and 5.17 ± 1.41 mm, and volumetric overlap ratio of 93.9 ± 1.80% and 91.3 ± 1.70 from TRUS and CT images, respectively. It also yielded an average Dice coefficient of 96.3 ± 1.3% on echocardiographic images. We proposed and evaluated a fast, interactive deep learning method for accurate medical image segmentation. Moreover, our approach has the potential to solve the bottleneck of deep learning methods in adapting to inter-clinical variations and speed up the annotation processes.

Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches.

Abstract: The determination of surgeons’ psychomotor skills in minimally invasive surgery techniques is one of the major concerns of the programs of surgical training in several hospitals. Therefore, it is important to assess and classify objectively the level of experience of surgeons and residents during their training process. The aim of this study was to investigate three classification methods for establishing automatically the level of surgical competence of the surgeons based on their psychomotor laparoscopic skills. A total of 43 participants, divided into an experienced surgeons group with ten experts (> 100 laparoscopic procedures performed) and non-experienced surgeons group with 24 residents and nine medical students (< 10 laparoscopic procedures performed), performed three tasks in the EndoViS training system. Motion data of the instruments were captured with a video-tracking system built into the EndoViS simulator and analyzed using 13 motion analysis parameters (MAPs). Radial basis function networks (RBFNets), K-star (K*), and random forest (RF) were used for classifying surgeons based on the MAPs’ scores of all participants. The performance of the three classifiers was examined using hold-out and leave-one-out validation techniques. For all three tasks, the K-star method was superior in terms of accuracy and AUC in both validation techniques. The mean accuracy of the classifiers was 93.33% for K-star, 87.58% for RBFNets, and 84.85% for RF in hold-out validation, and 91.47% for K-star, 89.92% for RBFNets, and 83.72% for RF in leave-one-out cross-validation. The three proposed methods demonstrated high performance in the classification of laparoscopic surgeons, according to their level of psychomotor skills. Together with motion analysis and three laparoscopic tasks of the Fundamental Laparoscopic Surgery Program, these classifiers provide a means for objectively classifying surgical competence of the surgeons for existing laparoscopic box trainers.

Automatic cancer tissue detection using multispectral photoacoustic imaging

Abstract: In the case of multispecimen study to locate cancer regions, such as in thyroidectomy and prostatectomy, a significant labor-intensive processing is required at a high cost. Pathology diagnosis is usually done by a pathologist observing tissue-stained glass slide under a microscope. Multispectral photoacoustic (MPA) specimen imaging has proven successful in differentiating photoacoustic (PA) signal characteristics between a histopathology-defined cancer region and normal tissue. This is mainly due to its ability to efficiently map oxyhemoglobin and deoxyhemoglobin contents from MPA images and key features for cancer detection. A fully automated deep learning algorithm is purposed, which learns to detect the presence of malignant tissue in freshly excised ex vivo human thyroid and prostate tissue specimens using the three-dimensional MPA dataset. The proposed automated deep learning model consisted of the convolutional neural network architecture, which extracts spatially colocated features, and a softmax function, which detects thyroid and prostate cancer tissue at once. This is one of the first deep learning models, to the best of our knowledge, to detect the presence of cancer in excised thyroid and prostate tissue of humans at once based on PA imaging. The area under the curve (AUC) was used as a metric to evaluate the predictive performance of the classifier. The proposed model detected the cancer tissue with the AUC of 0.96, which is very promising. This model is an improvement over the previous work using machine learning and deep learning algorithms. This model may have immediate application in cancer screening of the numerous sliced specimens that result from thyroidectomy and prostatectomy. Since the instrument that was used to capture the ex vivo PA images is now being developed for in vivo use, this model may also prove to be a starting point for in vivo PA image analysis for cancer diagnosis.

A learning-based method for online adjustment of C-arm Cone-beam CT source trajectories for artifact avoidance.

Abstract: During spinal fusion surgery, screws are placed close to critical nerves suggesting the need for highly accurate screw placement. Verifying screw placement on high-quality tomographic imaging is essential. C-arm cone-beam CT (CBCT) provides intraoperative 3D tomographic imaging which would allow for immediate verification and, if needed, revision. However, the reconstruction quality attainable with commercial CBCT devices is insufficient, predominantly due to severe metal artifacts in the presence of pedicle screws. These artifacts arise from a mismatch between the true physics of image formation and an idealized model thereof assumed during reconstruction. Prospectively acquiring views onto anatomy that are least affected by this mismatch can, therefore, improve reconstruction quality. We propose to adjust the C-arm CBCT source trajectory during the scan to optimize reconstruction quality with respect to a certain task, i.e., verification of screw placement. Adjustments are performed on-the-fly using a convolutional neural network that regresses a quality index over all possible next views given the current X-ray image. Adjusting the CBCT trajectory to acquire the recommended views results in non-circular source orbits that avoid poor images, and thus, data inconsistencies. We demonstrate that convolutional neural networks trained on realistically simulated data are capable of predicting quality metrics that enable scene-specific adjustments of the CBCT source trajectory. Using both realistically simulated data as well as real CBCT acquisitions of a semianthropomorphic phantom, we show that tomographic reconstructions of the resulting scene-specific CBCT acquisitions exhibit improved image quality particularly in terms of metal artifacts. The proposed method is a step toward online patient-specific C-arm CBCT source trajectories that enable high-quality tomographic imaging in the operating room. Since the optimization objective is implicitly encoded in a neural network trained on large amounts of well-annotated projection images, the proposed approach overcomes the need for 3D information at run-time.

Virtual septoplasty: a method to predict surgical outcomes for patients with nasal airway obstruction.

Abstract: A deviated nasal septum is the most common etiology for nasal airway obstruction (NAO), and septoplasty is the most common surgical procedure performed by ear–nose–throat surgeons in adults. However, quantitative criteria are rarely adopted to select patients for surgery, which may explain why up to 50% of patients report persistent or recurrent symptoms of nasal obstruction postoperatively. This study reports a systematic virtual surgery method to identify patients who may benefit from septoplasty. One patient with symptoms of NAO due to a septal deviation was selected to illustrate the virtual surgery concept. Virtual septoplasty was implemented in three steps: (1) determining if septal geometry is abnormal preoperatively, (2) virtually correcting the deviation while preserving the anatomical shape of the septum, and (3) estimating the post-surgical improvement in airflow using computational fluid dynamics. Anatomical and functional changes predicted by the virtual surgery method were compared to a standard septoplasty performed independently from the computational analysis. A benchmark healthy nasal septum geometry was obtained by averaging the septum dimensions of 47 healthy individuals. A comparison of the nasal septum geometry in the NAO patient with the benchmark geometry identified the precise locations where septal deviation and thickness exceeded the healthy range. Good agreement was found between the virtual surgery predictions and the actual surgical outcomes for both airspace minimal cross-sectional area (0.05 cm2 pre-surgery, 0.54 cm2 virtual surgery, 0.50 cm2 actual surgery) and nasal resistance (0.91 Pa.s/ml pre-surgery, 0.08 Pa.s/ml virtual surgery, 0.08 Pa.s/ml actual surgery). Previous virtual surgery methods for NAO were based on manual edits and subjective criteria. The virtual septoplasty method proposed in this study is objective and has the potential to be fully automated. Future implementation of this method in virtual surgery planning software has the potential to improve septoplasty outcomes.

Toward interprofessional team training for surgeons and anesthesiologists using virtual reality.

Abstract: In this work, a virtual environment for interprofessional team training in laparoscopic surgery is proposed. Our objective is to provide a tool to train and improve intraoperative communication between anesthesiologists and surgeons during laparoscopic procedures. An anesthesia simulation software and laparoscopic simulation software are combined within a multi-user virtual reality (VR) environment. Furthermore, two medical training scenarios for communication training between anesthesiologists and surgeons are proposed and evaluated. Testing was conducted and social presence was measured. In addition, clinical feedback from experts was collected by following a think-aloud protocol and through structured interviews. Our prototype is assessed as a reasonable basis for training and extensive clinical evaluation. Furthermore, the results of testing revealed a high degree of exhilaration and social presence of the involved physicians. Valuable insights were gained from the interviews and the think-aloud protocol with the experts of anesthesia and surgery that showed the feasibility of team training in VR, the usefulness of the system for medical training, and current limitations. The proposed VR prototype provides a new basis for interprofessional team training in surgery. It engages the training of problem-based communication during surgery and might open new directions for operating room training.

Tracking and visualization of the sensing area for a tethered laparoscopic gamma probe

Abstract: In surgical oncology, complete cancer resection and lymph node identification are challenging due to the lack of reliable intraoperative visualization Recently, endoscopic radio-guided cancer resection has been introduced where a novel tethered laparoscopic gamma detector can be used to determine the location of tracer activity, which can complement preoperative nuclear imaging data and endoscopic imaging However, these probes do not clearly indicate where on the tissue surface the activity originates, making localization of pathological sites difficult and increasing the mental workload of the surgeons Therefore, a robust real-time gamma probe tracking system integrated with augmented reality is proposed A dual-pattern marker has been attached to the gamma probe, which combines chessboard vertices and circular dots for higher detection accuracy Both patterns are detected simultaneously based on blob detection and the pixel intensity-based vertices detector and used to estimate the pose of the probe Temporal information is incorporated into the framework to reduce tracking failure Furthermore, we utilized the 3D point cloud generated from structure from motion to find the intersection between the probe axis and the tissue surface When presented as an augmented image, this can provide visual feedback to the surgeons The method has been validated with ground truth probe pose data generated using the OptiTrack system When detecting the orientation of the pose using circular dots and chessboard dots alone, the mean error obtained is $$005^{\circ }$$ and $$006^{\circ }$$ , respectively As for the translation, the mean error for each pattern is 178 mm and 181 mm The detection limits for pitch, roll and yaw are $$360^{\circ }, 360^{\circ }$$ and $$8^{\circ }$$ – $$82^{\circ }\cup 188^{\circ }$$ – $$352^{\circ }$$ The performance evaluation results show that this dual-pattern marker can provide high detection rates, as well as more accurate pose estimation and a larger workspace than the previously proposed hybrid markers The augmented reality will be used to provide visual feedback to the surgeons on the location of the affected lymph nodes or tumor

Body-mounted robotic assistant for MRI-guided low back pain injection

Abstract: This paper presents the development of a body-mounted robotic assistant for magnetic resonance imaging (MRI)-guided low back pain injection. Our goal was to eliminate the radiation exposure of traditional X-ray guided procedures while enabling the exquisite image quality available under MRI. The robot is designed with a compact and lightweight profile that can be mounted directly on the patient’s lower back via straps, thus minimizing the effect of patient motion by moving along with the patient. The robot was built with MR-conditional materials and actuated with piezoelectric motors so it can operate inside the MRI scanner bore during imaging and therefore streamline the clinical workflow by utilizing intraoperative MR images. The robot is designed with a four degrees of freedom parallel mechanism, stacking two identical Cartesian stages, to align the needle under intraoperative MRI-guidance. The system targeting accuracy was first evaluated in free space with an optical tracking system, and further assessed with a phantom study under live MRI-guidance. Qualitative imaging quality evaluation was performed on a human volunteer to assess the image quality degradation caused by the robotic assistant. Free space positioning accuracy study demonstrated that the mean error of the tip position to be $$0.51\pm 0.27$$ mm and needle angle to be $$0.70^\circ \pm 0.38^\circ $$. MRI-guided phantom study indicated the mean errors of the target to be $$1.70\pm 0.21$$ mm, entry point to be $$1.53\pm 0.19$$ mm, and needle angle to be $$0.66^\circ \pm 0.43^\circ $$. Qualitative imaging quality evaluation validated that the image degradation caused by the robotic assistant in the lumbar spine anatomy is negligible. The study demonstrates that the proposed body-mounted robotic system is able to perform MRI-guided low back injection in a phantom study with sufficient accuracy and with minimal visible image degradation that should not affect the procedure.

Comparison of statistical learning approaches for cerebral aneurysm rupture assessment

Abstract: Incidental aneurysms pose a challenge to physicians who need to decide whether or not to treat them. A statistical model could potentially support such treatment decisions. The aim of this study was to compare a previously developed aneurysm rupture logistic regression probability model (LRM) to other machine learning (ML) classifiers for discrimination of aneurysm rupture status. Hemodynamic, morphological, and patient-related information of 1631 cerebral aneurysms characterized by computational fluid dynamics simulations were used to train support vector machines (SVMs) with linear and RBF kernel (RBF-SVM), k-nearest neighbors (kNN), decision tree, random forest, and multilayer perceptron (MLP) neural network classifiers for predicting the aneurysm rupture status. The classifiers’ accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were evaluated and compared to the LRM using 249 test cases obtained from two external cohorts. Additionally, important variables were determined based on the random forest and weights of the linear SVM. The AUCs of the MLP, LRM, linear SVM, RBF-SVM, kNN, decision tree, and random forest were 0.83, 0.82, 0.80, 0.81, 0.76, 0.70, and 0.79, respectively. The accuracy ranged between 0.76 (decision tree,) and 0.79 (linear SVM, RBF-SVM, and MLP). Important variables for predicting the aneurysm rupture status included aneurysm location, the mean surface curvature, and maximum flow velocity. The performance of the LRM was overall comparable to that of the other ML classifiers, confirming its potential for aneurysm rupture assessment. To further improve the predictions, additional information, e.g., related to the aneurysm wall, might be needed.