Learning manipulation skills from open surgery provides more flexible access to the organ targets in the abdomen cavity and this could make the surgical robot working in a highly intelligent and friendly manner. Teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. This work aims to transfer motion skills from multiple human demonstrations in open surgery to robot manipulators in robot-assisted minimally invasive surgery (RA-MIS) by using TbD. However, the kinematic constraint should be respected during the performing of the learned skills by using a robot for minimally invasive surgery. In this article, we propose a novel methodology by integrating the cognitive learning techniques and the developed control techniques, allowing the robot to be highly intelligent to learn senior surgeons’ skills and to perform the learned surgical operations in semiautonomous surgery in the future. Finally, experiments are performed to verify the efficiency of the proposed strategy, and the results demonstrate the ability of the system to transfer human manipulation skills to a robot in RA-MIS and also shows that the remote center of motion (RCM) constraint can be guaranteed simultaneously. Note to Practitioners —This article is inspired by limited access to the manipulation of laparoscopic surgery under a kinematic constraint at the point of incision. Current commercial surgical robots are mostly operated by teleoperation, which is representing less autonomy on surgery. Assisting and enhancing the surgeon’s performance by increasing the autonomy of surgical robots has fundamental importance. The technique of teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. With the improved ability to interact with humans, such as flexibility and compliance, the new generation of serial robots becomes more and more popular in nonclinical research. Thus, advanced control strategies are required by integrating cognitive functions and learning techniques into the processes of surgical operation between robots, surgeon, and minimally invasive surgery (MIS). In this article, we propose a novel methodology to model the manipulation skill from multiple demonstrations and execute the learned operations in robot-assisted minimally invasive surgery (RA-MIS) by using a decoupled controller to respect the remote center of motion (RCM) constraint exploiting the redundancy of the robot. The developed control scheme has the following functionalities: 1) it enables the 3-D manipulation skill modeling after multiple demonstrations of the surgical tasks in open surgery by integrating dynamic time warping (DTW) and Gaussian mixture model (GMM)-based dynamic movement primitive (DMP) and 2) it maintains the RCM constraint in a smaller safe area while performing the learned operation in RA-MIS. The developed control strategy can also be potentially used in other industrial applications with a similar scenario.

/pdf/toward-teaching-by-demonstration-for-robot-assisted-5816etqquy.pdf

Toward Teaching by Demonstration for Robot-Assisted Minimally Invasive Surgery

Accurate path tracking and stability are the main challenges of lateral motion control in mobile robots, especially under the situation with complex road conditions. The interaction force between robots and the external environment may cause interference, which should be considered to guarantee its path tracking performance in dynamic and uncertain environments. In this article, a flexible lateral control scheme is considered for the developed wheel-legged robot, which consists of a cubature Kalman algorithm to evaluate the centroid slip angle and the yaw rate. Furthermore, a fuzzy compensation and preview angle-enhanced sliding model controller to improve the tracking accuracy and robustness. Finally, some simulations and experimental demonstrations using the four-wheel-legged robot (BIT-NAZA) are carried out to illustrate the effectiveness and robustness, and the proposed method has achieved satisfactory results in high-precision trajectory tracking and stability control of the mobile robot. 

Fuzzy-Torque Approximation-Enhanced Sliding Mode Control for Lateral Stability of Mobile Robot

Skill transfer learning for autonomous robots and human-robot cooperation: A survey

Localization strategies for autonomous mobile robots: A review

With the rapid development of deep learning, more and more deep learning-based motor imagery electroencephalograph (EEG) decoding methods have emerged in recent years. However, the existing deep learning-based methods usually only adopt the constraint of classification loss, which hardly obtains the features with high discrimination and limits the improvement of EEG decoding accuracy. In this paper, a discriminative feature learning strategy is proposed to improve the discrimination of features, which includes the central distance loss (CD-loss), the central vector shift strategy, and the central vector update process. First, the CD-loss is proposed to make the same class of samples converge to the corresponding central vector. Then, the central vector shift strategy extends the distance between different classes of samples in the feature space. Finally, the central vector update process is adopted to avoid the non-convergence of CD-loss and weaken the influence of the initial value of central vectors on the final results. In addition, overfitting is another severe challenge for deep learning-based EEG decoding methods. To deal with this problem, a data augmentation method based on circular translation strategy is proposed to expand the experimental datasets without introducing any extra noise or losing any information of the original data. To validate the effectiveness of the proposed method, we conduct some experiments on two public motor imagery EEG datasets (BCI competition IV 2a and 2b dataset), respectively. The comparison with current state-of-the-art methods indicates that our method achieves the highest average accuracy and good stability on the two experimental datasets.

/pdf/motor-imagery-eeg-decoding-method-based-on-a-discriminative-l2dbcg6gve.pdf

Motor Imagery EEG Decoding Method Based on a Discriminative Feature Learning Strategy

This paper proposes a brain–computer interface (BCI)-based teleoperation strategy for a dual-arm robot carrying a common object by multifingered hands. The BCI is based on motor imagery of the human brain, which utilizes common spatial pattern method to analyze the filtered electroencephalograph signals. Human intentions can be recognized and classified into the corresponding reference commands in task space for the robot according to phenomena of event-related synchronization/desynchronization, such that the object manipulation tasks guided by human user’s mind can be achieved. Subsequently, a concise dynamics consisting of the dynamics of the robotic arms and the geometrical constraints between the end-effectors and the object is formulated for the coordinated dual arm. To achieve optimization motion in the task space, a redundancy resolution at velocity level has been implemented through neural-dynamics optimization. Extensive experiments have been made by a number of subjects, and the results were provided to demonstrate the effectiveness of the proposed control strategy.

Motor-Imagery-Based Teleoperation of a Dual-Arm Robot Performing Manipulation Tasks

To achieve the better navigation performance of a mobile robot in the unknown environments, a novel human–robot hybrid system incorporating a motor-imagery (MI)-based brain teleoperation control is presented in this paper, where a deep-learning-based active perception is developed in the simultaneous localization and mapping (SLAM) framework. Using the deep-learning-based object recognition in the red–green–blue-depth (RGB-D) data acquisition process, the designed SLAM approach can select the valid feature points effectively, and the speed of displacement tracking can be improved by combining the oriented FAST and rotated BRIEF (ORB) SLAM algorithm with the optical flow method. The global trajectory map can also be mended using graph-based nonlinear error optimization. In addition, to build the connection between human intentions and the robot control commands flexibly in the developed mobile robot, a common spatial pattern (CSP)-based support vector machine (SVM) classification algorithm is proposed so that the control commands can be obtained directly from the human electroencephalograph (EEG) signals, which are preanalyzed and classified using the phenomena of event-related synchronization/desynchronization (ERS/ERD). Experiments involving several operators have verified the effectiveness of the proposed framework in the actual unstructured environments. Note to Practitioners —This paper is motivated by the exploration of the mobile robots remotely controlled by the disables in an unstructured environment that includes some unknown moving objects in the background. In the conventional approaches, the image feedback and the brain–computer interface-based EEG signals invoked by the visual stimulus are used to guide the motion of the robots. In this paper, the MI of left or right hand is adopted as an input of the brain teleoperation system, and the EEG signals are classified into two categories representing two robot control commands through the designed algorithm. Besides, a deep-learning-based SLAM is designed to analyze the image and depth of the environmental information provided by the robot RGB-D sensor, and the results can be transferred into a 3-D map locating the robot and assisting the operator to understand the operating environment. It has been verified that the deep-learning-based SLAM presented is more efficient and robust than traditional SLAM. The feasibility of the system is demonstrated by a set of experiments in the corridor environment.

Development of a Human–Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM

This paper proposes a brain-teleoperation control strategy that combines the deep learning technique (DLT) to realize the control and navigation of a mobile robot in unknown environments. The support vector machine (SVM) algorithm is utilized to recognize the human electroencephalograph (EEG) signals in the brain-computer interface (BCI) system which is based on steady state visually evoked potentials (SSVEP). In this way the intentions of human can be distinguished and control commands are generated for mobile robot. The DLT is used to recognize the type of environmental obstacles and environmental features by analysing the images that describe the environment. And then according to the classification of obstacle, various potential fields are built for the specific obstacles. By utilizing bottles as the features of environment, a whole map of the surroundings can be built through a sequential simultaneous localization and mapping (SLAM) algorithm. The main contribution of this paper is that the relationship between the potential field strength and classification of EEG signals is built up through the combination of multiple artificial potential fields with the brain signals, which produces the motion commands and designs a trajectory free of obstacles in un-structure environments. Three volunteer subjects are invited to test the entire system, and all operators can successfully complete experiments of manipulating the robot in corridor environments.

Brain Teleoperation of a Mobile Robot Using Deep Learning Technique

This paper presents the nonholonomic navigation and obstacle-avoidance control method for a developed wheeled chair The nonholonomic navigation technology is based on SLAM, in addition, a new method of collision avoidance control is proposed for this latter The method makes the wheeled chair move intelligently and avoid obstacle with more efficient and robust performance Extensive experiment research is implemented to proof the effectiveness of the method we proposed

http://ieeexplore.ieee.org/iel7/7592497/7606880/07606959.pdf

Nonholonomic navigation and control of a wheeled chair

In this paper, a brain-machine interface (BMI) with the capability of navigating a mobile robot in indoor environment, is proposed. The BMI, which is based on steady state visually evoked potentials (SSVEP), utilizes canonical correlation analysis (CCA) algorithm to classify electroencephalogram (EEG) signals and then translates the recognition results into motion commands for the trajectory planning based on artificial potential field (APF). In the navigation control strategy, the combination of particle filter-based simultaneous localization and mapping (S-LAM) and EEG-APF method is proposed to navigate the robot to a goal point with free obstacles, where SLAM algorithm would build a global map of the corridor and EEG-APF method would produce a trajectory free of obstacles. Extensive experiments have been conducted to verify the effectiveness of our system.

Yiliang Liu

Papers

Motor-Imagery-Based Teleoperation of a Dual-Arm Robot Performing Manipulation Tasks

Development of a Human–Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM

Brain Teleoperation of a Mobile Robot Using Deep Learning Technique

Nonholonomic navigation and control of a wheeled chair

An Indoor Navigation Control Strategy for a Brain-Actuated Mobile Robot