Showing papers in "International Journal of Computer Assisted Radiology and Surgery in 2018"

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery

Abstract: With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1–3 second window, without needing an observation of entire training trial. This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

Transfer learning with deep convolutional neural network for liver steatosis assessment in ultrasound images.

Abstract: The nonalcoholic fatty liver disease is the most common liver abnormality. Up to date, liver biopsy is the reference standard for direct liver steatosis quantification in hepatic tissue samples. In this paper we propose a neural network-based approach for nonalcoholic fatty liver disease assessment in ultrasound. We used the Inception-ResNet-v2 deep convolutional neural network pre-trained on the ImageNet dataset to extract high-level features in liver B-mode ultrasound image sequences. The steatosis level of each liver was graded by wedge biopsy. The proposed approach was compared with the hepatorenal index technique and the gray-level co-occurrence matrix algorithm. After the feature extraction, we applied the support vector machine algorithm to classify images containing fatty liver. Based on liver biopsy, the fatty liver was defined to have more than 5% of hepatocytes with steatosis. Next, we used the features and the Lasso regression method to assess the steatosis level. The area under the receiver operating characteristics curve obtained using the proposed approach was equal to 0.977, being higher than the one obtained with the hepatorenal index method, 0.959, and much higher than in the case of the gray-level co-occurrence matrix algorithm, 0.893. For regression the Spearman correlation coefficients between the steatosis level and the proposed approach, the hepatorenal index and the gray-level co-occurrence matrix algorithm were equal to 0.78, 0.80 and 0.39, respectively. The proposed approach may help the sonographers automatically diagnose the amount of fat in the liver. The presented approach is efficient and in comparison with other methods does not require the sonographers to select the region of interest.

Agile convolutional neural network for pulmonary nodule classification using CT images

Abstract: To distinguish benign from malignant pulmonary nodules using CT images is critical for their precise diagnosis and treatment. A new Agile convolutional neural network (CNN) framework is proposed to conquer the challenges of a small-scale medical image database and the small size of the nodules, and it improves the performance of pulmonary nodule classification using CT images. A hybrid CNN of LeNet and AlexNet is constructed through combining the layer settings of LeNet and the parameter settings of AlexNet. A dataset with 743 CT image nodule samples is built up based on the 1018 CT scans of LIDC to train and evaluate the Agile CNN model. Through adjusting the parameters of the kernel size, learning rate, and other factors, the effect of these parameters on the performance of the CNN model is investigated, and an optimized setting of the CNN is obtained finally. After finely optimizing the settings of the CNN, the estimation accuracy and the area under the curve can reach 0.822 and 0.877, respectively. The accuracy of the CNN is significantly dependent on the kernel size, learning rate, training batch size, dropout, and weight initializations. The best performance is achieved when the kernel size is set to $$7\times 7$$ , the learning rate is 0.005, the batch size is 32, and dropout and Gaussian initialization are used. This competitive performance demonstrates that our proposed CNN framework and the optimization strategy of the CNN parameters are suitable for pulmonary nodule classification characterized by small medical datasets and small targets. The classification model might help diagnose and treat pulmonary nodules effectively.

Exploiting the potential of unlabeled endoscopic video data with self-supervised learning

Abstract: Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a conditional generative adversarial network (cGAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical (COCO) or medical data (MICCAI EndoVis2017 challenge) using the target task (in this instance: segmentation). As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.

Automated surgical skill assessment in RMIS training.

Abstract: Manual feedback in basic robot-assisted minimally invasive surgery (RMIS) training can consume a significant amount of time from expert surgeons’ schedule and is prone to subjectivity. In this paper, we explore the usage of different holistic features for automated skill assessment using only robot kinematic data and propose a weighted feature fusion technique for improving score prediction performance. Moreover, we also propose a method for generating ‘task highlights’ which can give surgeons a more directed feedback regarding which segments had the most effect on the final skill score. We perform our experiments on the publicly available JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and evaluate four different types of holistic features from robot kinematic data—sequential motion texture (SMT), discrete Fourier transform (DFT), discrete cosine transform (DCT) and approximate entropy (ApEn). The features are then used for skill classification and exact skill score prediction. Along with using these features individually, we also evaluate the performance using our proposed weighted combination technique. The task highlights are produced using DCT features. Our results demonstrate that these holistic features outperform all previous Hidden Markov Model (HMM)-based state-of-the-art methods for skill classification on the JIGSAWS dataset. Also, our proposed feature fusion strategy significantly improves performance for skill score predictions achieving up to 0.61 average spearman correlation coefficient. Moreover, we provide an analysis on how the proposed task highlights can relate to different surgical gestures within a task. Holistic features capturing global information from robot kinematic data can successfully be used for evaluating surgeon skill in basic surgical tasks on the da Vinci robot. Using the framework presented can potentially allow for real-time score feedback in RMIS training and help surgical trainees have more focused training.

Video and accelerometer-based motion analysis for automated surgical skills assessment

Abstract: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save experts time and improve training efficiency. There have been some recent attempts at automated surgical skills assessment using either video analysis or acceleration data. In this paper, we present a novel approach for automated assessment of OSATS-like surgical skills and provide an analysis of different features on multi-modal data (video and accelerometer data). We conduct a large study for basic surgical skill assessment on a dataset that contained video and accelerometer data for suturing and knot-tying tasks. We introduce “entropy-based” features—approximate entropy and cross-approximate entropy, which quantify the amount of predictability and regularity of fluctuations in time series data. The proposed features are compared to existing methods of Sequential Motion Texture, Discrete Cosine Transform and Discrete Fourier Transform, for surgical skills assessment. We report average performance of different features across all applicable OSATS-like criteria for suturing and knot-tying tasks. Our analysis shows that the proposed entropy-based features outperform previous state-of-the-art methods using video data, achieving average classification accuracies of 95.1 and 92.2% for suturing and knot tying, respectively. For accelerometer data, our method performs better for suturing achieving 86.8% average accuracy. We also show that fusion of video and acceleration features can improve overall performance for skill assessment. Automated surgical skills assessment can be achieved with high accuracy using the proposed entropy features. Such a system can significantly improve the efficiency of surgical training in medical schools and teaching hospitals.

Prostate segmentation in MRI using a convolutional neural network architecture and training strategy based on statistical shape models.

Abstract: Most of the existing convolutional neural network (CNN)-based medical image segmentation methods are based on methods that have originally been developed for segmentation of natural images. Therefore, they largely ignore the differences between the two domains, such as the smaller degree of variability in the shape and appearance of the target volume and the smaller amounts of training data in medical applications. We propose a CNN-based method for prostate segmentation in MRI that employs statistical shape models to address these issues. Our CNN predicts the location of the prostate center and the parameters of the shape model, which determine the position of prostate surface keypoints. To train such a large model for segmentation of 3D images using small data (1) we adopt a stage-wise training strategy by first training the network to predict the prostate center and subsequently adding modules for predicting the parameters of the shape model and prostate rotation, (2) we propose a data augmentation method whereby the training images and their prostate surface keypoints are deformed according to the displacements computed based on the shape model, and (3) we employ various regularization techniques. Our proposed method achieves a Dice score of 0.88, which is obtained by using both elastic-net and spectral dropout for regularization. Compared with a standard CNN-based method, our method shows significantly better segmentation performance on the prostate base and apex. Our experiments also show that data augmentation using the shape model significantly improves the segmentation results. Prior knowledge about the shape of the target organ can improve the performance of CNN-based segmentation methods, especially where image features are not sufficient for a precise segmentation. Statistical shape models can also be employed to synthesize additional training data that can ease the training of large CNNs.

MRI radiomics analysis of molecular alterations in low-grade gliomas.

Abstract: Low-grade gliomas (LGG) are classified into three distinct groups based on their IDH1 mutation and 1p/19q codeletion status, each of which is associated with a different clinical expression. The genomic sub-classification of LGG requires tumor sampling via neurosurgical procedures. The aim of this study was to evaluate the radiomics approach for noninvasive classification of patients with LGG and IDH mutation, based on their 1p/19q codeletion status, by testing different classifiers and assessing the contribution of the different MR contrasts. Preoperative MRI scans of 47 patients diagnosed with LGG with IDH1-mutated tumors and a genetic analysis for 1p/19q deletion status were included in this study. A total of 152 features, including size, location and texture, were extracted from fluid-attenuated inversion recovery images, $$\hbox {T}_{2}$$ -weighted images (WI) and post-contrast $$\hbox {T}_{1}\hbox {WI}$$ . Classification was performed using 17 machine learning classifiers. Results were evaluated by a fivefold cross-validation analysis. Radiomic analysis differentiated tumors with 1p/19q intact ( $$n=21$$ ; astrocytomas) from those with 1p/19q codeleted ( $$n=26$$ ; oligodendrogliomas). Best classification was obtained using the Ensemble Bagged Trees classifier, with sensitivity $$=$$ 92%, specificity $$=$$ 83% and accuracy $$=$$ 87%, and with area under the curve $$=$$ 0.87. Tumors with 1p/19q intact were larger than those with 1p/19q codeleted ( $$46.2\pm 30.0$$ vs. $$30.8\pm 16.8$$ cc, respectively; $$p=0.03$$ ) and predominantly located to the left insula ( $$p=0.04$$ ). The proposed method yielded good discrimination between LGG with and without 1p/19q codeletion. Results from this study demonstrate the great potential of this method to aid decision-making in the clinical management of patients with LGG.

Highly immersive virtual reality laparoscopy simulation: development and future aspects

Abstract: Virtual reality (VR) applications with head-mounted displays (HMDs) have had an impact on information and multimedia technologies. The current work aimed to describe the process of developing a highly immersive VR simulation for laparoscopic surgery. We combined a VR laparoscopy simulator (LapSim) and a VR-HMD to create a user-friendly VR simulation scenario. Continuous clinical feedback was an essential aspect of the development process. We created an artificial VR (AVR) scenario by integrating the simulator video output with VR game components of figures and equipment in an operating room. We also created a highly immersive VR surrounding (IVR) by integrating the simulator video output with a $$360{^{\circ }}$$ video of a standard laparoscopy scenario in the department’s operating room. Clinical feedback led to optimization of the visualization, synchronization, and resolution of the virtual operating rooms (in both the IVR and the AVR). Preliminary testing results revealed that individuals experienced a high degree of exhilaration and presence, with rare events of motion sickness. The technical performance showed no significant difference compared to that achieved with the standard LapSim. Our results provided a proof of concept for the technical feasibility of an custom highly immersive VR-HMD setup. Future technical research is needed to improve the visualization, immersion, and capability of interacting within the virtual scenario.

Toward a standard ontology of surgical process models

Abstract: Purpose : The development of common ontologies has recently been identified as one of the key challenges in the emerging field of surgical data science (SDS). However, past and existing initiatives in the domain of surgery have mainly been focussing on individual groups and failed to achieve widespread international acceptance by the research community. To address this challenge, the authors of this paper launched a European initiative—OntoSPM Collaborative Action—with the goal of establishing a framework for joint development of ontologies in the field of SDS. This manuscript summarizes the goals and the current status of the international initiative. Methods : A workshop was organized in 2016, gathering the main European research groups having experience in developing and using ontologies in this domain. It led to the conclusion that a common ontology for surgical process models (SPM) was absolutely needed, and that the existing OntoSPM ontology could provide a good starting point toward the collaborative design and promotion of common, standard ontologies on SPM. Results : The workshop led to the OntoSPM Collaborative Action—launched in mid-2016—with the objective to develop, maintain and promote the use of common ontologies of SPM relevant to the whole domain of SDS. The fundamental concept, the architecture, the management and curation of the common ontology have been established, making it ready for wider public use. Conclusion : The OntoSPM Collaborative Action has been in operation for 24 months, with a growing dedicated membership. Its main result is a modular ontology, undergoing constant updates and extensions, based on the experts’ suggestions. It remains an open collaborative action, which always welcomes new contributors and applications.

An advanced simulator for orthopedic surgical training.

Abstract: The purpose of creating the virtual reality (VR) simulator is to facilitate and supplement the training opportunities provided to orthopedic residents. The use of VR simulators has increased rapidly in the field of medical surgery for training purposes. This paper discusses the creation of the virtual surgical environment (VSE) for training residents in an orthopedic surgical process called less invasive stabilization system (LISS) surgery which is used to address fractures of the femur. The overall methodology included first obtaining an understanding of the LISS plating process through interactions with expert orthopedic surgeons and developing the information centric models. The information centric models provided a structured basis to design and build the simulator. Subsequently, the haptic-based simulator was built. Finally, the learning assessments were conducted in a medical school. The results from the learning assessments confirm the effectiveness of the VSE for teaching medical residents and students. The scope of the assessment was to ensure (1) the correctness and (2) the usefulness of the VSE. Out of 37 residents/students who participated in the test, 32 showed improvements in their understanding of the LISS plating surgical process. A majority of participants were satisfied with the use of teaching Avatars and haptic technology. A paired t test was conducted to test the statistical significance of the assessment data which showed that the data were statistically significant. This paper demonstrates the usefulness of adopting information centric modeling approach in the design and development of the simulator. The assessment results underscore the potential of using VR-based simulators in medical education especially in orthopedic surgery.

Planning of skull reconstruction based on a statistical shape model combined with geometric morphometrics

Abstract: Virtual reconstruction of large cranial defects is still a challenging task. The current reconstruction procedures depend on the surgeon’s experience and skills in planning the reconstruction based on mirroring and manual adaptation. The aim of this study is to propose and evaluate a computer-based approach employing a statistical shape model (SSM) of the cranial vault. An SSM was created based on 131 CT scans of pathologically unaffected adult crania. After segmentation, the resulting surface mesh of one patient was established as template and subsequently registered to the entire sample. Using the registered surface meshes, an SSM was generated capturing the shape variability of the cranial vault. The knowledge about this shape variation in healthy patients was used to estimate the missing parts. The accuracy of the reconstruction was evaluated by using 31 CT scans not included in the SSM. Both unilateral and bilateral bony defects were created on each skull. The reconstruction was performed using the current gold standard of mirroring the intact to the affected side, and the result was compared to the outcome of our proposed SSM-driven method. The accuracy of the reconstruction was determined by calculating the distances to the corresponding parts on the intact skull. While unilateral defects could be reconstructed with both methods, the reconstruction of bilateral defects was, for obvious reasons, only possible employing the SSM-based method. Comparing all groups, the analysis shows a significantly higher precision of the SSM group, with a mean error of 0.47 mm compared to the mirroring group which exhibited a mean error of 1.13 mm. Reconstructions of bilateral defects yielded only slightly higher estimation errors than those of unilateral defects. The presented computer-based approach using SSM is a precise and simple tool in the field of computer-assisted surgery. It helps to reconstruct large-size defects of the skull considering the natural asymmetry of the cranium and is not limited to unilateral defects.

Deep dense multi-path neural network for prostate segmentation in magnetic resonance imaging

Abstract: We propose an approach of 3D convolutional neural network to segment the prostate in MR images. A 3D deep dense multi-path convolutional neural network that follows the framework of the encoder–decoder design is proposed. The encoder is built based upon densely connected layers that learn the high-level feature representation of the prostate. The decoder interprets the features and predicts the whole prostate volume by utilizing a residual layout and grouped convolution. A set of sub-volumes of MR images, centered at the prostate, is generated and fed into the proposed network for training purpose. The performance of the proposed network is compared to previously reported approaches. Two independent datasets were employed to assess the proposed network. In quantitative evaluations, the proposed network achieved 95.11 and 89.01 Dice coefficients for the two datasets. The segmentation results were robust to variations in MR images. In comparison experiments, the segmentation performance of the proposed network was comparable to the previously reported approaches. In qualitative evaluations, the segmentation results by the proposed network were well matched to the ground truth provided by human experts. The proposed network is capable of segmenting the prostate in an accurate and robust manner. This approach can be applied to other types of medical images.

Convolution neural networks for real-time needle detection and localization in 2D ultrasound

Abstract: We propose a framework for automatic and accurate detection of steeply inserted needles in 2D ultrasound data using convolution neural networks. We demonstrate its application in needle trajectory estimation and tip localization. Our approach consists of a unified network, comprising a fully convolutional network (FCN) and a fast region-based convolutional neural network (R-CNN). The FCN proposes candidate regions, which are then fed to a fast R-CNN for finer needle detection. We leverage a transfer learning paradigm, where the network weights are initialized by training with non-medical images, and fine-tuned with ex vivo ultrasound scans collected during insertion of a 17G epidural needle into freshly excised porcine and bovine tissue at depth settings up to 9 cm and $$40^{\circ }$$ – $$75^{\circ }$$ insertion angles. Needle detection results are used to accurately estimate needle trajectory from intensity invariant needle features and perform needle tip localization from an intensity search along the needle trajectory. Our needle detection model was trained and validated on 2500 ex vivo ultrasound scans. The detection system has a frame rate of 25 fps on a GPU and achieves 99.6% precision, 99.78% recall rate and an $${F}_{1}$$ score of 0.99. Validation for needle localization was performed on 400 scans collected using a different imaging platform, over a bovine/porcine lumbosacral spine phantom. Shaft localization error of $$0.82^{\circ }\pm 0.4^{\circ }$$ , tip localization error of $$0.23\pm 0.05$$ mm, and a total processing time of 0.58 s were achieved. The proposed method is fully automatic and provides robust needle localization results in challenging scanning conditions. The accurate and robust results coupled with real-time detection and sub-second total processing make the proposed method promising in applications for needle detection and localization during challenging minimally invasive ultrasound-guided procedures.

Breast cancer cell nuclei classification in histopathology images using deep neural networks

Abstract: Cell nuclei classification in breast cancer histopathology images plays an important role in effective diagnose since breast cancer can often be characterized by its expression in cell nuclei. However, due to the small and variant sizes of cell nuclei, and heavy noise in histopathology images, traditional machine learning methods cannot achieve desirable recognition accuracy. To address this challenge, this paper aims to present a novel deep neural network which performs representation learning and cell nuclei recognition in an end-to-end manner. The proposed model hierarchically maps raw medical images into a latent space in which robustness is achieved by employing a stacked denoising autoencoder. A supervised classifier is further developed to improve the discrimination of the model by maximizing inter-subject separability in the latent space. The proposed method involves a cascade model which jointly learns a set of nonlinear mappings and a classifier from the given raw medical images. Such an on-the-shelf learning strategy makes obtaining discriminative features possible, thus leading to better recognition performance. Extensive experiments with benign and malignant breast cancer datasets are conducted to verify the effectiveness of the proposed method. Better performance was obtained when compared with other feature extraction methods, and higher recognition rate was achieved when compared with other seven classification methods. We propose an end-to-end DNN model for cell nuclei and non-nuclei classification of histopathology images. It demonstrates that the proposed method can achieve promising performance in cell nuclei classification, and the proposed method is suitable for the cell nuclei classification task.

Development and internal validation of an aneurysm rupture probability model based on patient characteristics and aneurysm location, morphology, and hemodynamics

Abstract: Unruptured cerebral aneurysms pose a dilemma for physicians who need to weigh the risk of a devastating subarachnoid hemorrhage against the risk of surgery or endovascular treatment and their complications when deciding on a treatment strategy. A prediction model could potentially support such treatment decisions. The aim of this study was to develop and internally validate a model for aneurysm rupture based on hemodynamic and geometric parameters, aneurysm location, and patient gender and age. Cross-sectional data from 1061 patients were used for image-based computational fluid dynamics and shape characterization of 1631 aneurysms for training an aneurysm rupture probability model using logistic group Lasso regression. The model’s discrimination and calibration were internally validated based on the area under the curve (AUC) of the receiver operating characteristic and calibration plots. The final model retained 11 hemodynamic and 12 morphological variables, aneurysm location, as well as patient age and gender. An adverse hemodynamic environment characterized by a higher maximum oscillatory shear index, higher kinetic energy and smaller low shear area as well as a more complex aneurysm shape, male gender and younger age were associated with an increased rupture risk. The corresponding AUC of the model was 0.86 (95% CI [0.85, 0.86], after correction for optimism 0.84). The model combining variables from various domains was able to discriminate between ruptured and unruptured aneurysms with an AUC of 86%. Internal validation indicated potential for the application of this model in clinical practice after evaluation with longitudinal data.

Multistage segmentation model and SVM-ensemble for precise lung nodule detection

Abstract: Lung cancer detection at its initial stages increases the survival chances of patients. Automatic detection of lung nodules facilitates radiologists during the diagnosis. However, there is a challenge of false positives in automated systems which may lead to wrong findings. Precise segmentation facilitates to accurately extract nodules from lung CT images in order to improve performance of the diagnostic method. A multistage segmentation model is presented in this study. The lung region is extracted by applying corner-seeded region growing combined with differential evolution-based optimal thresholding. In addition to this, morphological operations are applied in boundary smoothing, hole filling and juxtavascular nodule extraction. Geometric properties along with 3D edge information are applied to extract nodule candidates. Geometric texture features descriptor (GTFD) followed by support vector machine-based ensemble classification is employed to distinguish actual nodules from the candidate set. A publicly available dataset, namely lung image database consortium and image database resource initiative, is used to evaluate performance of the proposed method. The classification is performed over GTFD feature vector and the results show 99% accuracy, 98.6% sensitivity and 98.2% specificity with 3.4 false positives per scan (FPs/scan). A lung nodule detection method is presented to facilitate radiologists in accurately diagnosing cancer from CT images. Results indicate that the proposed method has not only reduced FPs/scan but also significantly improved sensitivity as compared to related studies.

Reliability and correlation analysis of computed methods to convert conventional 2D radiological hindfoot measurements to a 3D setting using weightbearing CT

Abstract: The exact radiographic assessment of the hindfoot alignment remains challenging. This is reflected in the different measurement methods available. Weightbearing CT (WBCT) has been demonstrated to be more accurate in hindfoot measurements. However, current measurements are still performed in 2D. This study wants to assess the use of computed methods to convert the former uniplanar hindfoot measurements obtained after WBCT towards a 3D setting. Forty-eight patients, mean age of 39.6 ± 13.2 years, with absence of hindfoot pathology were included. A WBCT was obtained, and images were subsequently segmented and analyzed using computer-aided design operations. In addition to the hindfoot angle (HA), other ankle and hindfoot parameters such as the anatomical tibia axis, talocalcaneal axis (TCA), talocrural angle, tibial inclination (TI), talar tilt, and subtalar vertical angle were determined in 2D and 3D. The mean $$\hbox {HA}_{2\mathrm{D}}$$ was $$0.79^{\circ }$$ of valgus ± 3.2 and the $$\hbox {HA}_{\mathrm{3D}}$$ was $$8.08^{\circ }$$ of valgus ± 6.5. These angles differed significantly from each other with a $$P<0.001$$ . The correlation between both showed to be good by $$\hbox {a}$$ Pearson correlation coefficient (r) of 0.72 ( $$P < 0.001$$ ). The $$\hbox {ICC}_{\mathrm{3D}}$$ showed to be excellent when compared to the $$\hbox {ICC}_{\mathrm{2D}}$$ , which was good. Similar findings were obtained in other angles. The highest correlation was seen between the $$\hbox {TI}_{\mathrm{2D}}$$ and $$\hbox {TI}_{\mathrm{3D}}$$ (r = 0.83, $$P < 0.001$$ ) and an almost perfect agreement in the $$\hbox {TCA}_\mathrm{3D}$$ ( $$\hbox {ICC}_{\mathrm{3D}}=0.99$$ ). This study shows a good and reliable correlation between the $$\hbox {HA}_{\mathrm{2D}}$$ and $$\hbox {HA}_{\mathrm{3D}}$$ . However, the $$\hbox {HA}_{\mathrm{3D}}$$ overcomes the shortcomings of inaccuracy and provides valuable spatial data that could be incorporated during computer-assisted surgery to assess the multiplanar correction of a hindfoot deformity.

Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization

Abstract: Endovascular intervention is limited by two-dimensional intraoperative imaging and prolonged procedure times in the presence of complex anatomies. Robotic catheter technology could offer benefits such as reduced radiation exposure to the clinician and improved intravascular navigation. Incorporating three-dimensional preoperative imaging into a semiautonomous robotic catheterization platform has the potential for safer and more precise navigation. This paper discusses a semiautonomous robotic catheter platform based on previous work (Rafii-Tari et al., in: MICCAI2013, pp 369–377. https://doi.org/10.1007/978-3-642-40763-5_46 , 2013) by proposing a method to address anatomical variability among aortic arches. It incorporates anatomical information in the process of catheter trajectories optimization, hence can adapt to the scale and orientation differences among patient-specific anatomies. Statistical modeling is implemented to encode the catheter motions of both proximal and distal sites based on cannulation data obtained from a single phantom by an expert operator. Non-rigid registration is applied to obtain a warping function to map catheter tip trajectories into other anatomically similar but shape/scale/orientation different models. The remapped trajectories were used to generate robot trajectories to conduct a collaborative cannulation task under flow simulations. Cross-validations were performed to test the performance of the non-rigid registration. Success rates of the cannulation task executed by the robotic platform were measured. The quality of the catheterization was also assessed using performance metrics for manual and robotic approaches. Furthermore, the contact forces between the instruments and the phantoms were measured and compared for both approaches. The success rate for semiautomatic cannulation is 98.1% under dry simulation and 94.4% under continuous flow simulation. The proposed robotic approach achieved smoother catheter paths than manual approach. The mean contact forces have been reduced by 33.3% with the robotic approach, and 70.6% less STDEV forces were observed with the robot. This work provides insights into catheter task planning and an improved design of hands-on ergonomic catheter navigation robots.

Non-rigid registration of 3D ultrasound for neurosurgery using automatic feature detection and matching

Abstract: The brain undergoes significant structural change over the course of neurosurgery, including highly nonlinear deformation and resection. It can be informative to recover the spatial mapping between structures identified in preoperative surgical planning and the intraoperative state of the brain. We present a novel feature-based method for achieving robust, fully automatic deformable registration of intraoperative neurosurgical ultrasound images. A sparse set of local image feature correspondences is first estimated between ultrasound image pairs, after which rigid, affine and thin-plate spline models are used to estimate dense mappings throughout the image. Correspondences are derived from 3D features, distinctive generic image patterns that are automatically extracted from 3D ultrasound images and characterized in terms of their geometry (i.e., location, scale, and orientation) and a descriptor of local image appearance. Feature correspondences between ultrasound images are achieved based on a nearest-neighbor descriptor matching and probabilistic voting model similar to the Hough transform. Experiments demonstrate our method on intraoperative ultrasound images acquired before and after opening of the dura mater, during resection and after resection in nine clinical cases. A total of 1620 automatically extracted 3D feature correspondences were manually validated by eleven experts and used to guide the registration. Then, using manually labeled corresponding landmarks in the pre- and post-resection ultrasound images, we show that our feature-based registration reduces the mean target registration error from an initial value of 3.3 to 1.5 mm. This result demonstrates that the 3D features promise to offer a robust and accurate solution for 3D ultrasound registration and to correct for brain shift in image-guided neurosurgery.

Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks

Abstract: During needle interventions, successful automated detection of the needle immediately after insertion is necessary to allow the physician identify and correct any misalignment of the needle and the target at early stages, which reduces needle passes and improves health outcomes. We present a novel approach to localize partially inserted needles in 3D ultrasound volume with high precision using convolutional neural networks. We propose two methods based on patch classification and semantic segmentation of the needle from orthogonal 2D cross-sections extracted from the volume. For patch classification, each voxel is classified from locally extracted raw data of three orthogonal planes centered on it. We propose a bootstrap resampling approach to enhance the training in our highly imbalanced data. For semantic segmentation, parts of a needle are detected in cross-sections perpendicular to the lateral and elevational axes. We propose to exploit the structural information in the data with a novel thick-slice processing approach for efficient modeling of the context. Our introduced methods successfully detect 17 and 22 G needles with a single trained network, showing a robust generalized approach. Extensive ex-vivo evaluations on datasets of chicken breast and porcine leg show 80 and 84% F1-scores, respectively. Furthermore, very short needles are detected with tip localization errors of less than 0.7 mm for lengths of only 5 and 10 mm at 0.2 and 0.36 mm voxel sizes, respectively. Our method is able to accurately detect even very short needles, ensuring that the needle and its tip are maximally visible in the visualized plane during the entire intervention, thereby eliminating the need for advanced bi-manual coordination of the needle and transducer.

From ideas to long-term studies: 3D printing clinical trials review

Abstract: Although high costs are often cited as the main limitation of 3D printing (3DP) in the medical field, current lack of clinical evidence is asserting itself as an impost as the field begins to mature. The aim is to review clinical trials in the field of 3DP, an area of research which has grown dramatically in recent years. We surveyed clinical trials registered in 15 primary registries worldwide, including ClinicalTrials.gov. All trials which utilized 3DP in a clinical setting were included in this review. Our search was performed on December 15, 2017. Data regarding the purpose of the study, inclusion criteria, number of patients enrolled, primary outcomes, centers, start and estimated completion dates were extracted. A total of 92 clinical trials with $${N}=6$$ 252 patients matched the criteria and were included in the study. A total of 42 (45.65%) studies cited China as their location. Only 10 trials were multicenter and 2 were registered as international. The discipline that most commonly utilized 3DP was Orthopedic Surgery, with 25 (27.17%) registered trials. At the time of data extraction, 17 (18.48%) clinical trials were complete. After several years of case reports, feasibility studies and technical reports in the field, larger-scale studies are beginning to emerge. There are almost no international register entries. Although there are new emerging areas of study in disciplines that may benefit from 3DP, it is likely to remain limited to very specific applications.

FCN-based approach for the automatic segmentation of bone surfaces in ultrasound images.

Abstract: A new algorithm, based on fully convolutional networks (FCN), is proposed for the automatic localization of the bone interface in ultrasound (US) images. The aim of this paper is to compare and validate this method with (1) a manual segmentation and (2) a state-of-the-art method called confidence in phase symmetry (CPS). The dataset used for this study was composed of 1738 US images collected from three volunteers and manually delineated by three experts. The inter- and intra-observer variabilities of this manual delineation were assessed. Images having annotations with an inter-observer variability higher than a confidence threshold were rejected, resulting in 1287 images. Both FCN-based and CPS approaches were studied and compared to the average inter-observer segmentation according to six criteria: recall, precision, F1 score, accuracy, specificity and root-mean-square error (RMSE). The intra- and inter-observer variabilities were inferior to 1 mm for 90% of manual annotations. The RMSE was 1.32 ± 3.70 mm and 5.00 ± 7.70 mm for, respectively, the FCN-based approach and the CPS algorithm. The mean recall, precision, F1 score, accuracy and specificity were, respectively, 62%, 64%, 57%, 80% and 83% for the FCN-based approach and 66%, 34%, 41%, 52% and 43% for the CPS algorithm. The FCN-based approach outperforms the CPS algorithm, and the obtained RMSE is similar to the manual segmentation uncertainty.

Texture-specific bag of visual words model and spatial cone matching-based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images.

Abstract: The bag of visual words (BoVW) model is a powerful tool for feature representation that can integrate various handcrafted features like intensity, texture, and spatial information. In this paper, we propose a novel BoVW-based method that incorporates texture and spatial information for the content-based image retrieval to assist radiologists in clinical diagnosis. This paper presents a texture-specific BoVW method to represent focal liver lesions (FLLs). Pixels in the region of interest (ROI) are classified into nine texture categories using the rotation-invariant uniform local binary pattern method. The BoVW-based features are calculated for each texture category. In addition, a spatial cone matching (SCM)-based representation strategy is proposed to describe the spatial information of the visual words in the ROI. In a pilot study, eight radiologists with different clinical experience performed diagnoses for 20 cases with and without the top six retrieved results. A total of 132 multiphase computed tomography volumes including five pathological types were collected. The texture-specific BoVW was compared to other BoVW-based methods using the constructed dataset of FLLs. The results show that our proposed model outperforms the other three BoVW methods in discriminating different lesions. The SCM method, which adds spatial information to the orderless BoVW model, impacted the retrieval performance. In the pilot trial, the average diagnosis accuracy of the radiologists was improved from 66 to 80% using the retrieval system. The preliminary results indicate that the texture-specific features and the SCM-based BoVW features can effectively characterize various liver lesions. The retrieval system has the potential to improve the diagnostic accuracy and the confidence of the radiologists.

3D/2D model-to-image registration by imitation learning for cardiac procedures

Abstract: In cardiac interventions, such as cardiac resynchronization therapy (CRT), image guidance can be enhanced by involving preoperative models. Multimodality 3D/2D registration for image guidance, however, remains a significant research challenge for fundamentally different image data, i.e., MR to X-ray. Registration methods must account for differences in intensity, contrast levels, resolution, dimensionality, field of view. Furthermore, same anatomical structures may not be visible in both modalities. Current approaches have focused on developing modality-specific solutions for individual clinical use cases, by introducing constraints, or identifying cross-modality information manually. Machine learning approaches have the potential to create more general registration platforms. However, training image to image methods would require large multimodal datasets and ground truth for each target application. This paper proposes a model-to-image registration approach instead, because it is common in image-guided interventions to create anatomical models for diagnosis, planning or guidance prior to procedures. An imitation learning-based method, trained on 702 datasets, is used to register preoperative models to intraoperative X-ray images. Accuracy is demonstrated on cardiac models and artificial X-rays generated from CTs. The registration error was $$2.92\pm 2.22\,\hbox { mm}$$ on 1000 test cases, superior to that of manual ( $$6.48\pm 5.6\,\hbox { mm}$$ ) and gradient-based ( $$6.79\pm 4.75\,\hbox { mm}$$ ) registration. High robustness is shown in 19 clinical CRT cases. Besides the proposed methods feasibility in a clinical environment, evaluation has shown good accuracy and high robustness indicating that it could be applied in image-guided interventions.

A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy

Abstract: Pedicle screw fixation is a challenging procedure with a concerning rates of reoperation. After insertion of the screws is completed, the most common intraoperative verification approach is to acquire anterior–posterior and lateral radiographic images, based on which the surgeons try to visually assess the correctness of insertion. Given the limited accuracy of the existing verification techniques, we identified the need for an accurate and automated pedicle screw assessment system that can verify the screw insertion intraoperatively. For doing so, this paper offers a framework for automatic segmentation and pose estimation of pedicle screws based on deep learning principles. Segmentation of pedicle screw X-ray projections was performed by a convolutional neural network. The network could isolate the input X-rays into three classes: screw head, screw shaft and background. Once all the screw shafts were segmented, knowledge about the spatial configuration of the acquired biplanar X-rays was used to identify the correspondence between the projections. Pose estimation was then performed to estimate the 6 degree-of-freedom pose of each screw. The performance of the proposed pose estimation method was tested on a porcine specimen. The developed machine learning framework was capable of segmenting the screw shafts with 93% and 83% accuracy when tested on synthetic X-rays and on clinically realistic X-rays, respectively. The pose estimation accuracy of this method was shown to be $$1.93^{\circ } \pm 0.64^{\circ }$$ and $$1.92 \pm 0.55\,\hbox {mm}$$ on clinically realistic X-rays. The proposed system offers an accurate and fully automatic pedicle screw segmentation and pose assessment framework. Such a system can help to provide an intraoperative pedicle screw insertion assessment protocol with minimal interference with the existing surgical routines.

Comparison of the application of B-mode and strain elastography ultrasound in the estimation of lymph node metastasis of papillary thyroid carcinoma based on a radiomics approach

Abstract: B-mode ultrasound (B-US) and strain elastography ultrasound (SE-US) images have a potential to distinguish thyroid tumor with different lymph node (LN) status. The purpose of our study is to investigate whether the application of multi-modality images including B-US and SE-US can improve the discriminability of thyroid tumor with LN metastasis based on a radiomics approach. Ultrasound (US) images including B-US and SE-US images of 75 papillary thyroid carcinoma (PTC) cases were retrospectively collected. A radiomics approach was developed in this study to estimate LNs status of PTC patients. The approach included image segmentation, quantitative feature extraction, feature selection and classification. Three feature sets were extracted from B-US, SE-US, and multi-modality containing B-US and SE-US. They were used to evaluate the contribution of different modalities. A total of 684 radiomics features have been extracted in our study. We used sparse representation coefficient-based feature selection method with 10-bootstrap to reduce the dimension of feature sets. Support vector machine with leave-one-out cross-validation was used to build the model for estimating LN status. Using features extracted from both B-US and SE-US, the radiomics-based model produced an area under the receiver operating characteristic curve (AUC) $$=$$ 0.90, accuracy (ACC) $$=$$ 0.85, sensitivity (SENS) $$=$$ 0.77 and specificity (SPEC) $$=$$ 0.88, which was better than using features extracted from B-US or SE-US separately. Multi-modality images provided more information in radiomics study. Combining use of B-US and SE-US could improve the LN metastasis estimation accuracy for PTC patients.

Automatic bladder segmentation from CT images using deep CNN and 3D fully connected CRF-RNN

Abstract: Automatic approach for bladder segmentation from computed tomography (CT) images is highly desirable in clinical practice. It is a challenging task since the bladder usually suffers large variations of appearance and low soft-tissue contrast in CT images. In this study, we present a deep learning-based approach which involves a convolutional neural network (CNN) and a 3D fully connected conditional random fields recurrent neural network (CRF-RNN) to perform accurate bladder segmentation. We also propose a novel preprocessing method, called dual-channel preprocessing, to further advance the segmentation performance of our approach. The presented approach works as following: first, we apply our proposed preprocessing method on the input CT image and obtain a dual-channel image which consists of the CT image and an enhanced bladder density map. Second, we exploit a CNN to predict a coarse voxel-wise bladder score map on this dual-channel image. Finally, a 3D fully connected CRF-RNN refines the coarse bladder score map and produce final fine-localized segmentation result. We compare our approach to the state-of-the-art V-net on a clinical dataset. Results show that our approach achieves superior segmentation accuracy, outperforming the V-net by a significant margin. The Dice Similarity Coefficient of our approach (92.24%) is 8.12% higher than that of the V-net. Moreover, the bladder probability maps performed by our approach present sharper boundaries and more accurate localizations compared with that of the V-net. Our approach achieves higher segmentation accuracy than the state-of-the-art method on clinical data. Both the dual-channel processing and the 3D fully connected CRF-RNN contribute to this improvement. The united deep network composed of the CNN and 3D CRF-RNN also outperforms a system where the CRF model acts as a post-processing method disconnected from the CNN.

Improved medical image fusion based on cascaded PCA and shift invariant wavelet transforms.

Abstract: In the medical field, radiologists need more informative and high-quality medical images to diagnose diseases. Image fusion plays a vital role in the field of biomedical image analysis. It aims to integrate the complementary information from multimodal images, producing a new composite image which is expected to be more informative for visual perception than any of the individual input images. The main objective of this paper is to improve the information, to preserve the edges and to enhance the quality of the fused image using cascaded principal component analysis (PCA) and shift invariant wavelet transforms. A novel image fusion technique based on cascaded PCA and shift invariant wavelet transforms is proposed in this paper. PCA in spatial domain extracts relevant information from the large dataset based on eigenvalue decomposition, and the wavelet transform operating in the complex domain with shift invariant properties brings out more directional and phase details of the image. The significance of maximum fusion rule applied in dual-tree complex wavelet transform domain enhances the average information and morphological details. The input images of the human brain of two different modalities (MRI and CT) are collected from whole brain atlas data distributed by Harvard University. Both MRI and CT images are fused using cascaded PCA and shift invariant wavelet transform method. The proposed method is evaluated based on three main key factors, namely structure preservation, edge preservation, contrast preservation. The experimental results and comparison with other existing fusion methods show the superior performance of the proposed image fusion framework in terms of visual and quantitative evaluations. In this paper, a complex wavelet-based image fusion has been discussed. The experimental results demonstrate that the proposed method enhances the directional features as well as fine edge details. Also, it reduces the redundant details, artifacts, distortions.

Automated muscle segmentation from CT images of the hip and thigh using a hierarchical multi-atlas method

Abstract: Patient-specific quantitative assessments of muscle mass and biomechanical musculoskeletal simulations require segmentation of the muscles from medical images. The objective of this work is to automate muscle segmentation from CT data of the hip and thigh. We propose a hierarchical multi-atlas method in which each hierarchy includes spatial normalization using simpler pre-segmented structures in order to reduce the inter-patient variability of more complex target structures. The proposed hierarchical method was evaluated with 19 muscles from 20 CT images of the hip and thigh using the manual segmentation by expert orthopedic surgeons as ground truth. The average symmetric surface distance was significantly reduced in the proposed method (1.53 mm) in comparison with the conventional method (2.65 mm). We demonstrated that the proposed hierarchical multi-atlas method improved the accuracy of muscle segmentation from CT images, in which large inter-patient variability and insufficient contrast were involved.