Learning manipulation skills from open surgery provides more flexible access to the organ targets in the abdomen cavity and this could make the surgical robot working in a highly intelligent and friendly manner. Teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. This work aims to transfer motion skills from multiple human demonstrations in open surgery to robot manipulators in robot-assisted minimally invasive surgery (RA-MIS) by using TbD. However, the kinematic constraint should be respected during the performing of the learned skills by using a robot for minimally invasive surgery. In this article, we propose a novel methodology by integrating the cognitive learning techniques and the developed control techniques, allowing the robot to be highly intelligent to learn senior surgeons’ skills and to perform the learned surgical operations in semiautonomous surgery in the future. Finally, experiments are performed to verify the efficiency of the proposed strategy, and the results demonstrate the ability of the system to transfer human manipulation skills to a robot in RA-MIS and also shows that the remote center of motion (RCM) constraint can be guaranteed simultaneously. Note to Practitioners —This article is inspired by limited access to the manipulation of laparoscopic surgery under a kinematic constraint at the point of incision. Current commercial surgical robots are mostly operated by teleoperation, which is representing less autonomy on surgery. Assisting and enhancing the surgeon’s performance by increasing the autonomy of surgical robots has fundamental importance. The technique of teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. With the improved ability to interact with humans, such as flexibility and compliance, the new generation of serial robots becomes more and more popular in nonclinical research. Thus, advanced control strategies are required by integrating cognitive functions and learning techniques into the processes of surgical operation between robots, surgeon, and minimally invasive surgery (MIS). In this article, we propose a novel methodology to model the manipulation skill from multiple demonstrations and execute the learned operations in robot-assisted minimally invasive surgery (RA-MIS) by using a decoupled controller to respect the remote center of motion (RCM) constraint exploiting the redundancy of the robot. The developed control scheme has the following functionalities: 1) it enables the 3-D manipulation skill modeling after multiple demonstrations of the surgical tasks in open surgery by integrating dynamic time warping (DTW) and Gaussian mixture model (GMM)-based dynamic movement primitive (DMP) and 2) it maintains the RCM constraint in a smaller safe area while performing the learned operation in RA-MIS. The developed control strategy can also be potentially used in other industrial applications with a similar scenario.

/pdf/toward-teaching-by-demonstration-for-robot-assisted-5816etqquy.pdf

Toward Teaching by Demonstration for Robot-Assisted Minimally Invasive Surgery

Accurate path tracking and stability are the main challenges of lateral motion control in mobile robots, especially under the situation with complex road conditions. The interaction force between robots and the external environment may cause interference, which should be considered to guarantee its path tracking performance in dynamic and uncertain environments. In this article, a flexible lateral control scheme is considered for the developed wheel-legged robot, which consists of a cubature Kalman algorithm to evaluate the centroid slip angle and the yaw rate. Furthermore, a fuzzy compensation and preview angle-enhanced sliding model controller to improve the tracking accuracy and robustness. Finally, some simulations and experimental demonstrations using the four-wheel-legged robot (BIT-NAZA) are carried out to illustrate the effectiveness and robustness, and the proposed method has achieved satisfactory results in high-precision trajectory tracking and stability control of the mobile robot. 

Fuzzy-Torque Approximation-Enhanced Sliding Mode Control for Lateral Stability of Mobile Robot

Skill transfer learning for autonomous robots and human-robot cooperation: A survey

Localization strategies for autonomous mobile robots: A review

With the rapid development of deep learning, more and more deep learning-based motor imagery electroencephalograph (EEG) decoding methods have emerged in recent years. However, the existing deep learning-based methods usually only adopt the constraint of classification loss, which hardly obtains the features with high discrimination and limits the improvement of EEG decoding accuracy. In this paper, a discriminative feature learning strategy is proposed to improve the discrimination of features, which includes the central distance loss (CD-loss), the central vector shift strategy, and the central vector update process. First, the CD-loss is proposed to make the same class of samples converge to the corresponding central vector. Then, the central vector shift strategy extends the distance between different classes of samples in the feature space. Finally, the central vector update process is adopted to avoid the non-convergence of CD-loss and weaken the influence of the initial value of central vectors on the final results. In addition, overfitting is another severe challenge for deep learning-based EEG decoding methods. To deal with this problem, a data augmentation method based on circular translation strategy is proposed to expand the experimental datasets without introducing any extra noise or losing any information of the original data. To validate the effectiveness of the proposed method, we conduct some experiments on two public motor imagery EEG datasets (BCI competition IV 2a and 2b dataset), respectively. The comparison with current state-of-the-art methods indicates that our method achieves the highest average accuracy and good stability on the two experimental datasets.

/pdf/motor-imagery-eeg-decoding-method-based-on-a-discriminative-l2dbcg6gve.pdf

Motor Imagery EEG Decoding Method Based on a Discriminative Feature Learning Strategy

This paper proposes a brain–computer interface (BCI)-based teleoperation strategy for a dual-arm robot carrying a common object by multifingered hands. The BCI is based on motor imagery of the human brain, which utilizes common spatial pattern method to analyze the filtered electroencephalograph signals. Human intentions can be recognized and classified into the corresponding reference commands in task space for the robot according to phenomena of event-related synchronization/desynchronization, such that the object manipulation tasks guided by human user’s mind can be achieved. Subsequently, a concise dynamics consisting of the dynamics of the robotic arms and the geometrical constraints between the end-effectors and the object is formulated for the coordinated dual arm. To achieve optimization motion in the task space, a redundancy resolution at velocity level has been implemented through neural-dynamics optimization. Extensive experiments have been made by a number of subjects, and the results were provided to demonstrate the effectiveness of the proposed control strategy.

Motor-Imagery-Based Teleoperation of a Dual-Arm Robot Performing Manipulation Tasks

To achieve the better navigation performance of a mobile robot in the unknown environments, a novel human–robot hybrid system incorporating a motor-imagery (MI)-based brain teleoperation control is presented in this paper, where a deep-learning-based active perception is developed in the simultaneous localization and mapping (SLAM) framework. Using the deep-learning-based object recognition in the red–green–blue-depth (RGB-D) data acquisition process, the designed SLAM approach can select the valid feature points effectively, and the speed of displacement tracking can be improved by combining the oriented FAST and rotated BRIEF (ORB) SLAM algorithm with the optical flow method. The global trajectory map can also be mended using graph-based nonlinear error optimization. In addition, to build the connection between human intentions and the robot control commands flexibly in the developed mobile robot, a common spatial pattern (CSP)-based support vector machine (SVM) classification algorithm is proposed so that the control commands can be obtained directly from the human electroencephalograph (EEG) signals, which are preanalyzed and classified using the phenomena of event-related synchronization/desynchronization (ERS/ERD). Experiments involving several operators have verified the effectiveness of the proposed framework in the actual unstructured environments. Note to Practitioners —This paper is motivated by the exploration of the mobile robots remotely controlled by the disables in an unstructured environment that includes some unknown moving objects in the background. In the conventional approaches, the image feedback and the brain–computer interface-based EEG signals invoked by the visual stimulus are used to guide the motion of the robots. In this paper, the MI of left or right hand is adopted as an input of the brain teleoperation system, and the EEG signals are classified into two categories representing two robot control commands through the designed algorithm. Besides, a deep-learning-based SLAM is designed to analyze the image and depth of the environmental information provided by the robot RGB-D sensor, and the results can be transferred into a 3-D map locating the robot and assisting the operator to understand the operating environment. It has been verified that the deep-learning-based SLAM presented is more efficient and robust than traditional SLAM. The feasibility of the system is demonstrated by a set of experiments in the corridor environment.

Development of a Human–Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM

In this paper, a brain–computer interface (BCI)-based navigation and control strategy is developed for a mobile robot in indoor environments. It combines the simultaneous localization and mapping to achieve the navigation and positioning for a mobile robot in indoor environments, where the RGB landmarks are regarded as the environmental features learned by the FastSLAM algorithm. The online BCI, based on steady-state visually evoked potentials, exploits multivariate synchronization index classification algorithm to analyze the human electroencephalograph (EEG) signals so that the human intention can be recognized accurately, and then the EEG-based motion commands are produced for the mobile robot. Probability potential field approach based on the probability density function of 2-D normal distribution is connected with the brain signals to generate a collision-free trajectory for the mobile robot. The entire system is semiautonomous, since the robot’s low level behaviors are autonomous and the stochastic navigation is executed by the BCI, and it is verified by the extensive experiments involving five volunteers. All the participants can successfully tele-operate the mobile robot, and the experimental results have verified the effectiveness of the proposed approach.

Guangming Shi

Papers

Motor-Imagery-Based Teleoperation of a Dual-Arm Robot Performing Manipulation Tasks

Development of a Human–Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM

Brain–Computer Interface-Based Stochastic Navigation and Control of a Semiautonomous Mobile Robot in Indoor Environments