The challenge of vascular interventional surgery is that surgeons require to be exposed to X-ray for a long time, operating guidewires and catheters to complete the treatment. To reduce the burden of the surgeons, it is of great significance to develop a tactile sensing robot-assisted system for vascular interventional surgery. Therefore, a slave manipulator with the function of collaborative operating guidewires and catheters was developed to replace doctors to perform the surgery in the operating room. In addition, a master manipulator based on magnetorheological fluids was located on the master side, and the haptic force feedback of the system was realized by generating the tactile force acting on the doctor’s hand. To verify the proposed system, a series of experiments were carried out, the results of experiments in “Vitro” indicated that the proposed system has good performance in collaborative operating and can accurately deliver a guidewire and a catheter to the target position. The maximum tracking error of the axial motion was less than 2 mm, and the maximum tracking error of the radial motion was less than 2 degrees, which is acceptable. And under the guidance of the force feedback, the safety of the system was obviously higher than that of without force feedback, after the experiment was completed by 5 participants, the safety increased by 4.32% on average. So, we can get the results that our system is feasible and effective.

Development of a Tactile Sensing Robot-Assisted System for Vascular Interventional Surgery

Contextual information has been shown to be powerful for semantic segmentation. This work proposes a novel Context-based Tandem Network (CTNet) by interactively exploring the spatial contextual information and the channel contextual information, which can discover the semantic context for semantic segmentation. Specifically, the Spatial Contextual Module (SCM) is leveraged to uncover the spatial contextual dependency between pixels by exploring the correlation between pixels and categories. Meanwhile, the Channel Contextual Module (CCM) is introduced to learn the semantic features including the semantic feature maps and class-specific features by modeling the long-term semantic dependence between channels. The learned semantic features are utilized as the prior knowledge to guide the learning of SCM, which can make SCM obtain more accurate long-range spatial dependency. Finally, to further improve the performance of the learned representations for semantic segmentation, the results of the two context modules are adaptively integrated to achieve better results. Extensive experiments are conducted on four widely-used datasets, i.e., PASCAL-Context, Cityscapes, ADE20K and PASCAL VOC2012. The results demonstrate the superior performance of the proposed CTNet by comparison with several state-of-the-art methods. The source code and models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/syp2ysy/CTNet</uri> . 

/pdf/ctnet-context-based-tandem-network-for-semantic-segmentation-3cy0xuxn.pdf

CTNet: Context-Based Tandem Network for Semantic Segmentation

An accurate assessment of surgical operation skills is essential for improving the vascular intervention surgical outcome and the performance of endovascular surgery robots. In existing studies, subjective and objective assessments of surgical operation skills use a variety of indicators, such as the operation speed and operation smoothness. However, the vascular conditions of particular patients have not been considered in the assessment, leading to deviations in the evaluation. Therefore, in this paper, an operation skills assessment method including the vascular difficulty level index for catheter insertion at the aortic arch in endovascular surgery is proposed. First, the model describing the difficulty of the vascular anatomical structure is established with characteristics of different aortic arch branches based on machine learning. Afterwards, the vascular difficulty level is set as an objective index combined with operating characteristics extracted from the operations performed by surgeons to evaluate the surgical operation skills at the aortic arch using machine learning. The accuracy of the assessment improves from 86.67 to 96.67% after inclusion of the vascular difficulty as an evaluation indicator to more objectively and accurately evaluate skills. The method described in this paper can be adopted to train novice surgeons in endovascular surgery, and for studies of vascular interventional surgery robots. Graphical abstract Operation skill assessment with vascular difficulty for vascular interventional surgery.

Machine learning–based operation skills assessment with vascular difficulty index for vascular intervention surgery

The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion block, and transposed convolution. Depthwise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global contexts and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample feature maps for acquiring refined edges. LWANet can segment surgical instruments in real-time while takes little computational costs. Based on 960x544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the- art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with a 4.10% increase on mean IOU.

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Computer-aided medical image segmentation has been applied widely in diagnosis and treatment to obtain clinically useful information of shapes and volumes of target organs and tissues. In the past several years, convolutional neural network (CNN)-based methods (e.g., U-Net) have dominated this area, but still suffered from inadequate long-range information capturing. Hence, recent work presented computer vision Transformer variants for medical image segmentation tasks and obtained promising performances. Such Transformers modeled long-range dependency by computing pair-wise patch relations. However, they incurred prohibitive computational costs, especially on 3D medical images (e.g., CT and MRI). In this paper, we propose a new method called Dilated Transformer, which conducts self-attention alternately in local and global scopes for pair-wise patch relations capturing. Inspired by dilated convolution kernels, we conduct the global self-attention in a dilated manner, enlarging receptive fields without increasing the patches involved and thus reducing computational costs. Based on this design of Dilated Transformer, we construct a U-shaped encoder–decoder hierarchical architecture called D-Former for 3D medical image segmentation. Experiments on the Synapse and ACDC datasets show that our D-Former model, trained from scratch, outperforms various competitive CNN-based or Transformer-based segmentation models at a low computational cost without time-consuming per-training process. 

/pdf/d-former-a-u-shaped-dilated-transformer-for-3d-medical-image-17crwiox.pdf

D-former: a U-shaped Dilated Transformer for 3D medical image segmentation

Semantic segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, accurate segmentation of cataract surgical instruments is still a challenge due to specular reflection and class imbalance issues. In this paper, an attention-guided network is proposed to segment the cataract surgical instrument. A new attention module is designed to learn discriminative features and address the specular reflection issue. It captures global context and encodes semantic dependencies to emphasize key semantic features, boosting the feature representation. This attention module has very few parameters, which helps to save memory. Thus, it can be flexibly plugged into other networks. Besides, a hybrid loss is introduced to train our network for addressing the class imbalance issue, which merges cross entropy and logarithms of Dice loss. A new dataset named Cata7 is constructed to evaluate our network. To the best of our knowledge, this is the first cataract surgical instrument dataset for semantic segmentation. Based on this dataset, RAUNet achieves state-of-the-art performance 97.71$\%$ mean Dice and 95.62$\%$ mean IOU.

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

Objective: Technical skill assessment plays an important role in the professional development of an interventionalist in percutaneous coronary intervention (PCI). However, most of the traditional assessment methods are time consuming and subjective. This paper aims to develop objective assessment techniques. Methods: In this study, a natural-behavior-based assessment framework is proposed to qualitatively and quantitatively assess technical skills in PCI. In vivo porcine studies were conducted to deliver a medical guidewire to two target coronaries of left circumflex arteries by six novice and four expert interventionalists. Simultaneously, four types of natural behaviors (i.e., hand motion, proximal force, muscle activity, and finger motion) were acquired from the subjects’ dominant hand and arm. The features extracted from the behaviors of different skill-level groups were compared using the Mann–Whitney U-test for effective behavior selection. The effective ones were further applied in the Gaussian-mixture-model-based qualitative assessment and Mahalanobis-distance-based quantitative assessment. Results: The qualitative assessment achieves an accuracy of 92% to distinguish the novice and expert attempts, which is significantly higher than that of using single guidewire motions. Furthermore, the quantitative assessment can assign objective and effective scores for all attempts, indicating high correlation ( $R$ $=$ 0.9225) to those obtained by traditional methods. Conclusion: The objective, effective, and comprehensive assessment of technical skills can be provided by qualitatively and quantitatively analyzing interventionalists’ natural behaviors in PCI. Significance: This paper suggests a novel approach for the technical skill assessment and the promising results demonstrate the great importance and effectiveness of the proposed method for promoting the development of objective assessment techniques.

Qualitative and Quantitative Assessment of Technical Skills in Percutaneous Coronary Intervention: In Vivo Porcine Studies

Semantic segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, accurate segmentation of cataract surgical instruments is still a challenge due to specular reflection and class imbalance issues. In this paper, an attention-guided network is proposed to segment the cataract surgical instrument. A new attention module is designed to learn discriminative features and address the specular reflection issue. It captures global context and encodes semantic dependencies to emphasize key semantic features, boosting the feature representation. This attention module has very few parameters, which helps to save memory. Thus, it can be flexibly plugged into other networks. Besides, a hybrid loss is introduced to train our network for addressing the class imbalance issue, which merges cross entropy and logarithms of Dice loss. A new dataset named Cata7 is constructed to evaluate our network. To the best of our knowledge, this is the first cataract surgical instrument dataset for semantic segmentation. Based on this dataset, RAUNet achieves state-of-the-art performance 97.71% mean Dice and 95.62% mean IOU.

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K-means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

A Multilayer and Multimodal-Fusion Architecture for Simultaneous Recognition of Endovascular Manipulations and Assessment of Technical Skills

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Rui-Qi Li

Papers

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

Qualitative and Quantitative Assessment of Technical Skills in Percutaneous Coronary Intervention: In Vivo Porcine Studies

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

A Multilayer and Multimodal-Fusion Architecture for Simultaneous Recognition of Endovascular Manipulations and Assessment of Technical Skills

Real-Time Multi-Guidewire Endpoint Localization in Fluoroscopy Images