Showing papers by "Zeng-Guang Hou published in 2020"

Design of a Low-cost Miniature Robot to Assist the COVID-19 Nasopharyngeal Swab Sampling

Abstract: Nasopharyngeal (NP) swab sampling is an effective approach for the diagnosis of coronavirus disease 2019 (COVID-19). Medical staffs carrying out the task of collecting NP specimens are in close contact with the suspected patient, thereby posing a high risk of cross-infection. We propose a low-cost miniature robot that can be easily assembled and remotely controlled. The system includes an active end-effector, a passive positioning arm, and a detachable swab gripper with integrated force sensing capability. The cost of the materials for building this robot is 55 USD and the total weight of the functional part is 0.23kg. The design of the force sensing swab gripper was justified using Finite Element (FE) modeling and the performances of the robot were validated with a simulation phantom and three pig noses. FE analysis indicated a 0.5mm magnitude displacement of the gripper's sensing beam, which meets the ideal detecting range of the optoelectronic sensor. Studies on both the phantom and the pig nose demonstrated the successful operation of the robot during the collection task. The average forces were found to be 0.35N and 0.85N, respectively. It is concluded that the proposed robot is promising and could be further developed to be used in vivo.

Enhanced Motor Imagery Based Brain- Computer Interface via FES and VR for Lower Limbs

Abstract: Motor imagery based brain-computer interface (MI-BCI) has been studied for improvement of patients’ motor function in neurorehabilitation and motor assistance. However, the difficulties in performing imagery tasks limit its application. To overcome the limitation, an enhanced MI-BCI based on functional electrical stimulation (FES) and virtual reality (VR) is proposed in this study. On one hand, the FES is used to stimulate the subjects’ lower limbs before their imagination to make them experience the muscles’ contraction and improve their attention on the lower limbs, by which it is supposed that the subjects’ motor imagery (MI) abilities can be enhanced. On the other hand, a ball-kicking movement scenario from the first-person perspective is designed to provide visual guidance for performing MI tasks. The combination of FES and VR can be used to reduce the difficulties in performing MI tasks and improve classification accuracy. Finally, the comparison experiments were conducted on twelve healthy subjects to validate the performance of the enhanced MI-BCI. The results show that the classification performance can be improved significantly by using the proposed MI-BCI in terms of the classification accuracy (ACC), the area under the curve (AUC) and the F1 score (paired t-test, ${p} ).

Pyramid Attention Aggregation Network for Semantic Segmentation of Surgical Instruments

Abstract: Semantic segmentation of surgical instruments plays a critical role in computer-assisted surgery. However, specular reflection and scale variation of instruments are likely to occur in the surgical environment, undesirably altering visual features of instruments, such as color and shape. These issues make semantic segmentation of surgical instruments more challenging. In this paper, a novel network, Pyramid Attention Aggregation Network, is proposed to aggregate multi-scale attentive features for surgical instruments. It contains two critical modules: Double Attention Module and Pyramid Upsampling Module. Specifically, the Double Attention Module includes two attention blocks (i.e., position attention block and channel attention block), which model semantic dependencies between positions and channels by capturing joint semantic information and global contexts, respectively. The attentive features generated by the Double Attention Module can distinguish target regions, contributing to solving the specular reflection issue. Moreover, the Pyramid Upsampling Module extracts local details and global contexts by aggregating multi-scale attentive features. It learns the shape and size features of surgical instruments in different receptive fields and thus addresses the scale variation issue. The proposed network achieves state-of-the-art performance on various datasets. It achieves a new record of 97.10% mean IOU on Cata7. Besides, it comes first in the MICCAI EndoVis Challenge 2017 with 9.90% increase on mean IOU.

Robust Medical Instrument Segmentation Challenge 2019.

Abstract: Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Learning impedance control of robots with enhanced transient and steady-state control performances

Abstract: This study proposes a learning impedance controller comprising a proportional feedback control term, a composite-learning-based uncertainty estimation term, and a robot-environment interaction control term. The impedance control problem is converted into a particular reference-trajectory tracking problem based on a generated reference trajectory. The proposed controller ensures the exponential convergence of the auxiliary tracking error and the uncertainty estimation error. The interaction control term improves the transient control performance through suppression/encouragement of the incorrect/correct robot movements. The composite-learning update law enhances the transient and steady-state control performances based on the exponential convergence of the uncertainty estimation error and auxiliary tracking error. Finally, the effectiveness and advantages of the proposed impedance controller are validated by theoretical analysis and simulations on a parallel robot.

Toward Improving Engagement in Neural Rehabilitation: Attention Enhancement Based on Brain–Computer Interface and Audiovisual Feedback

Abstract: Both motor and cognitive function rehabilitation benefits can be improved significantly by patients’ active participation. To this goal, an attention enhancement system based on the brain–computer interface (BCI) and audiovisual feedback is proposed. First, an interactive position-tracking riding game is designed to increase the training challenge and neural engagement. Subjects were asked to drive one of the avatars to keep up with another by adjusting their riding speed and attention. Second, the subject’s electroencephalogram (EEG)-based attention level is divided into three regions (low, moderate, and high) by using the theta-to-beta ratio (TBR). According to the subject’s attention states, different speed adjustment strategies are adopted to adjust the tracking challenge and improve the subject’s attention. Besides, if the subject’s attention focused on the training is moderate or low, an auditory feedback will be given to remind the subject to pay more attention to the training. The contrast experimental results show that subjects’ performance indicated by overall attention level and average muscle activation can be improved significantly by using the attention enhancement system, which validates the feasibility of the proposed system for improving the neural and motor engagement.

Removing Feasibility Conditions on Adaptive Neural Tracking Control of Nonlinear Time-Delay Systems With Time-Varying Powers, Input, and Full-State Constraints.

Abstract: This article investigates the tracking control for input and full-state-constrained nonlinear time-delay systems with unknown time-varying powers, whose nonlinearities do not impose any growth assumption. By utilizing the auxiliary control signal and nonlinear state-dependent transformation (NSDT) to counteract the effect of input saturation and cope with full-state constraints, respectively, and then introducing lower and higher powers and Lyapunov-Krasovskii (L-K) functionals in control design together with the adaptive neural-networks (NNs) method, an adaptive neural tracking control design is provided without feasibility conditions. It is proved that NNs approximation is valid, all the closed-loop signals are semiglobally bounded, and input and full-state constraints are not violated.

Learning Regional Attention Convolutional Neural Network for Motion Intention Recognition Based on EEG Data.

Abstract: Recent deep learning-based Brain-Computer Interface (BCI) decoding algorithms mainly focus on spatial-temporal features, while failing to explicitly explore spectral information which is one of the most important cues for BCI. In this paper, we propose a novel regional attention convolutional neural network (RACNN) to take full advantage of spectral-spatial-temporal features for EEG motion intention recognition. Time-frequency based analysis is adopted to reveal spectral-temporal features in terms of neural oscillations of primary sensorimotor. The basic idea of RACNN is to identify the activated area of the primary sensorimotor adaptively. The RACNN aggregates a varied number of spectral-temporal features produced by a backbone convolutional neural network into a compact fixedlength representation. Inspired by the neuroscience findings that functional asymmetry of the cerebral hemisphere, we propose a region biased loss to encourage high attention weights for the most critical regions. Extensive evaluations on two benchmark datasets and real-world BCI dataset show that our approach significantly outperforms previous methods.

Modeling and Kinematics Analysis of a Novel 5-DOF Upper Limb Exoskeleton Rehabilitation Robot

Abstract: It has become increasingly popular that robot is applied to assist patients with central nervous system injury in rehabilitation exercise. This paper introduces a novel 5-DOF upper limb exoskeleton rehabilitation robot. In MATLAB, the forward kinematics expression of the robot is established by using the D-H method and the workspace of the robot is analyzed by Monte Carlo method. Considering the physiological characteristics of the human body and the practical requirement of rehabilitation movement, this paper uses the "Swivel Angle" to solve the redundancy solution problem of inverse kinematics of the robot on line. In order to verify the correctness of the kinematics model of the robot, a simulation experiment based on the minimum jerk trajectory planning is designed. This work lays the foundation for the future studies on dynamics, control and human-computer interaction strategy of the robot.

Lightweight Double Attention-Fused Networks for Intraoperative Stent Segmentation

Abstract: In endovascular interventional therapy, the fusion of preoperative data with intraoperative X-ray fluoroscopy has demonstrated the potential to reduce radiation dose, contrast agent and processing time. Real-time intraoperative stent segmentation is an important pre-requisite for accurate fusion. Nevertheless, this task often comes with the challenge of the thin stent wires with low contrast in noisy X-ray fluoroscopy. In this paper, a novel and efficient network, termed Lightweight Double Attention-fused Network (LDA-Net), is proposed for end-to-end stent segmentation in intraoperative X-ray fluoroscopy. The proposed LDA-Net consists of three major components, namely feature attention module, relevance attention module and pre-trained MobileNetV2 encoder. Besides, a hybrid loss function of both reinforced focal loss and dice loss is designed to better address the issues of class imbalance and misclassified examples. Quantitative and qualitative evaluations on 175 intraoperative X-ray sequences demonstrate that the proposed LDA-Net significantly outperforms simpler baselines as well as the best previously-published result for this task, achieving the state-of-the-art performance.

CAU-net: A Novel Convolutional Neural Network for Coronary Artery Segmentation in Digital Substraction Angiography

Abstract: Coronary artery analysis plays an important role in the diagnosis and treatment of coronary heart disease. Coronary artery segmentation, as an important part of quantitative researc h on coronary heart disease, has become the main topic in coronary artery analysis. In this paper, a deep convolutional neural network (CNN) based method called Coronary Artery U-net (CAU-net) is proposed for the automatic segmentation of coronary arteries in digital subtraction angiography (DSA) images. CAU-net is a variant of U-net. Based on the observation that coronary arteries are composed of many vessels with the same appearance but different thicknesses, a novel multi-scale feature fusion method is proposed in CAU-net. Besides, a new dataset is proposed to solve the problem of no available public dataset on coronary arteries segmentation, which is also one of our contributions. Our dataset contains 538 image samples, which is relatively large compared with the public datasets of other vessel segmentation tasks. In our dataset, a new labeling method is applied to ensure the purity of the labeling samples. From the experimental results, we prove that CAU-net can make significant improvements compared with the vanilla U-net, and achieve the state-of-the-art performance compared with other traditional segmentation methods and deep learning methods.

Engagement Enhancement Based on Human-in-the-Loop Optimization for Neural Rehabilitation.

Abstract: Enhancing patients' engagement is of great benefit for neural rehabilitation. However, physiological and neurological differences among individuals can cause divergent responses to the same task, and the responses can further change considerably during training; both of these factors make engagement enhancement a challenge. This challenge can be overcome by training task optimization based on subjects' responses. To this end, an engagement enhancement method based on human-in-the-loop optimization is proposed in this paper. Firstly, an interactive speed-tracking riding game is designed as the training task in which four reference speed curves (RSCs) are designed to construct the reference trajectory in each generation. Each RSC is modeled using a piecewise function, which is determined by the starting velocity, transient time, and end velocity. Based on the parameterized model, the difficulty of the training task, which is a key factor affecting the engagement, can be optimized. Then, the objective function is designed with consideration to the tracking accuracy and the surface electromyogram (sEMG)-based muscle activation, and the physical and physiological responses of the subjects can consequently be evaluated simultaneously. Moreover, a covariance matrix adaption evolution strategy, which is relatively tolerant of both measurement noises and human adaptation, is used to generate the optimal parameters of the RSCs periodically. By optimization of the RSCs persistently, the objective function can be maximized, and the subjects' engagement can be enhanced. Finally, the performance of the proposed method is demonstrated by the validation and comparison experiments. The results show that both subjects' sEMG-based motor engagement and electroencephalography based neural engagement can be improved significantly and maintained at a high level.

A Multilayer-Multimodal Fusion Architecture for Pattern Recognition of Natural Manipulations in Percutaneous Coronary Interventions

Abstract: The increasingly-used robotic systems can provide precise delivery and reduce X-ray radiation to medical staff in percutaneous coronary interventions (PCI), but natural manipulations of interventionalists are forgone in most robot-assisted procedures. Therefore, it is necessary to explore natural manipulations to design more advanced human-robot interfaces (HRI). In this study, a multilayer-multimodal fusion architecture is proposed to recognize six typical subpatterns of guidewire manipulations in conventional PCI. The synchronously acquired multimodal behaviors from ten subjects are used as the inputs of the fusion architecture. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. Experimental results indicate that the multimodal fusion brings significant accuracy improvement in comparison with single-modal schemes. Furthermore, the proposed architecture can achieve the overall accuracy of 96.90%, much higher than that of a singlelayer recognition architecture (92.56%). These results have indicated the potential of the proposed method for facilitating the development of HRI for robot-assisted PCI.

IoT-Based Remote Control Study of a Robotic Trans-Esophageal Ultrasound Probe via LAN and 5G

Abstract: A robotic trans-esophageal echocardiography (TEE) probe has been recently developed to address the problems with manual control in the X-ray environment when a conventional probe is used for interventional procedure guidance. However, the robot was exclusively to be used in local areas and the effectiveness of remote control has not been scientifically tested. In this study, we implemented an Internet-of-things (IoT)-based configuration to the TEE robot so the system can set up a local area network (LAN) or be configured to connect to an internet cloud over 5G. To investigate the remote control, backlash hysteresis effects were measured and analysed. A joystick-based device and a button-based gamepad were then employed and compared with the manual control in a target reaching experiment for the two steering axes. The results indicated different hysteresis curves for the left-right and up-down steering axes with the input wheel’s deadbands found to be 15° and 8°, respectively. Similar magnitudes of positioning errors at approximately 0.5° and maximum overshoots at around 2.5° were found when manually and robotically controlling the TEE probe. The amount of time to finish the task indicated a better performance using the button-based gamepad over joystick-based device, although both were worse than the manual control. It is concluded that the IoT-based remote control of the TEE probe is feasible and a trained user can accurately manipulate the probe. The main identified problem was the backlash hysteresis in the steering axes, which can result in continuous oscillations and overshoots.

A Novel Vascular Robotic System: Performance Evaluation

Abstract: Percutaneous coronary intervention (PCI) has become a common method for the treatment of cardiovascular diseases (CVDs). However, the accumulated X-ray radiation during the procedures greatly increases the probability of medical staff suffering from cataracts and brain tumors. This study bases on an existing vascular robotic system designed in our previous work. The main component of this robotic system is a bio-inspired Dual-finger Robotic Hand (DRH), which consists of a pair of bionic thumb and forefinger to imitate the surgical manipulations of interventionalists. This study is to evaluate the performance of the robotic system through a series of experiments: advancing a guidewire at different speeds and accelerations. The mean root mean square error (RMSe) of the actual and desired axial movements is 0.72 ± 0.49 mm, demonstrating the effectiveness and robustness of the robotic system.

IoT-based Remote Control Study of a Robotic Trans-esophageal Ultrasound Probe via LAN and 5G

Abstract: A robotic trans-esophageal echocardiography (TEE) probe has been recently developed to address the problems with manual control in the X-ray envi-ronment when a conventional probe is used for interventional procedure guidance. However, the robot was exclusively to be used in local areas and the effectiveness of remote control has not been scientifically tested. In this study, we implemented an Internet-of-things (IoT)-based configuration to the TEE robot so the system can set up a local area network (LAN) or be configured to connect to an internet cloud over 5G. To investigate the re-mote control, backlash hysteresis effects were measured and analysed. A joy-stick-based device and a button-based gamepad were then employed and compared with the manual control in a target reaching experiment for the two steering axes. The results indicated different hysteresis curves for the left-right and up-down steering axes with the input wheel's deadbands found to be 15 deg and deg, respectively. Similar magnitudes of positioning errors at approximately 0.5 deg and maximum overshoots at around 2.5 deg were found when manually and robotically controlling the TEE probe. The amount of time to finish the task indicated a better performance using the button-based gamepad over joystick-based device, although both were worse than the manual control. It is concluded that the IoT-based remote control of the TEE probe is feasible and a trained user can accurately manipulate the probe. The main identified problem was the backlash hysteresis in the steering axes, which can result in continuous oscillations and overshoots.