Reinforcement learning offers to robotics a framework and set of tools for the design of sophisticated and hard-to-engineer behaviors. Conversely, the challenges of robotic problems provide both inspiration, impact, and validation for developments in reinforcement learning. The relationship between disciplines has sufficient promise to be likened to that between physics and mathematics. In this article, we attempt to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots. We highlight both key challenges in robot reinforcement learning as well as notable successes. We discuss how contributions tamed the complexity of the domain and study the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, a particular focus of our paper lies on the choice between model-based and model-free as well as between value-function-based and policy-search methods. By analyzing a simple problem in some detail we demonstrate how reinforcement learning approaches may be profitably applied, and we note throughout open questions and the tremendous potential for future research.

/pdf/reinforcement-learning-in-robotics-a-survey-4w4397qx9i.pdf

Reinforcement learning in robotics: A survey

Summary) The abstract should follow the structure of the article (relevance, degree of exploration of the problem, the goal, the main results, conclusion) and characterize the theoretical and practical significance of the study results. The abstract should not contain wording echoing the title, cumbersome grammatical structures and abbreviations. The text should be written in scientific style. The volume of abstracts (summaries) depends on the content of the article, but should not be less than 250 words. All abbreviations must be disclosed in the summary (in spite of the fact that they will be disclosed in the main text of the article), references to the numbers of publications from reference list should not be made. The sentences of the abstract should constitute an integral text, which can be made by use of the words “consequently”, “for example”, “as a result”. Avoid the use of unnecessary introductory phrases (eg, “the author of the article considers...”, “The article presents...” and so on.)

/pdf/ministry-of-education-and-science-of-the-russian-federation-7m2jn0xnm1.pdf

Ministry of Education and Science of the Russian Federation

Presents an approach to movement planning, on-line trajectory modification, and imitation learning by representing movement plans based on a set of nonlinear differential equations with well-defined attractor dynamics. The resultant movement plan remains an autonomous set of nonlinear differential equations that forms a control policy (CP) which is robust to strong external perturbations and that can be modified on-line by additional perceptual variables. We evaluate the system with a humanoid robot simulation and an actual humanoid robot. Experiments are presented for the imitation of three types of movements: reaching movements with one arm, drawing movements of 2-D patterns, and tennis swings. Our results demonstrate (a) that multi-joint human movements can be encoded successfully by the CPs, (b) that a learned movement policy can readily be reused to produce robust trajectories towards different targets, (c) that a policy fitted for one particular target provides a good predictor of human reaching movements towards neighboring targets, and (d) that the parameter space which encodes a policy is suitable for measuring to which extent two trajectories are qualitatively similar.

/pdf/movement-imitation-with-nonlinear-dynamical-systems-in-3i2aovrlle.pdf

Movement imitation with nonlinear dynamical systems in humanoid robots

In this paper we introduce a new underlying probabilistic model for principal component analysis (PCA). Our formulation interprets PCA as a particular Gaussian process prior on a mapping from a latent space to the observed data-space. We show that if the prior's covariance function constrains the mappings to be linear the model is equivalent to PCA, we then extend the model by considering less restrictive covariance functions which allow non-linear mappings. This more general Gaussian process latent variable model (GPLVM) is then evaluated as an approach to the visualisation of high dimensional data for three different data-sets. Additionally our non-linear algorithm can be further kernelised leading to 'twin kernel PCA' in which a mapping between feature spaces occurs.

/pdf/gaussian-process-latent-variable-models-for-visualisation-of-26wk3yaplf.pdf

Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data

Related perceptual, motor, and cognitive performances were examined to reveal the accuracy of the properties of action spontaneously represented when mentally simulating moving one's hand. The kinematic configuration of the body represented and transformed in mental simulations was not fixed or canonical but corresponded to one's current configuration. Mental simulation time mimicked movement time for natural efficient movement from a posture midway between each of the hand's joint limits into many other postures. Equal time was required for simulated and real movements into more common, comfortable postures; shorter but proportional time was required for simulated movement than real movement into less common postures that involved longer trajectories, coordinated activity at more joints, motion near extremes of joint limits, and uncomfortable kinesthetic sensations. The findings suggest that sensorimotor structures support mental simulations of actions.

/pdf/temporal-and-kinematic-properties-of-motor-behavior-46arrhindi.pdf

Temporal and kinematic properties of motor behavior reflected in mentally simulated action.

Model-Based Control of a Robot Manipulator presents the first integrated treatment of many of the most important recent developments in using detailed dynamic models of robots to improve their control. The authors' work on automatic identification of kinematic and dynamic parameters, feedforward position control, stability in force control, and trajectory learning has significant implications for improving performance in future robot systems. All of the main ideas discussed in this book have been validated by experiments on a direct-drive robot arm.The book addresses the issues of building accurate robot models and of applying them for high performance control. It first describes how three sets of models - the kinematic model of the links and the inertial models of the links and of rigid-body loads - can be obtained automatically using experimental data. These models are then incorporated into position control, single trajectory learning, and force control. The MIT Serial Link Direct Drive Arm, on which these models were developed and applied to control, is one of the few manipulators currently suitable for testing such concepts.Contents: Introduction. Direct Drive Arms. Kinematic Calibration. Estimation of Load Inertial Parameters. Estimation of Link Inertial Parameters. Feedforward and Computed Torque Control. Model-Based Robot Learning. Dynamic Stability Issues in Force Control. Kinematic Stability Issues in Force Control. Conclusion.Chae An is Research Staff Member, IBM T.J. Watson Research Center, Christopher Atkeson is an Assistant Professor and John Hollerbach is an Associate Professor in the MIT Department of Brain and Cognitive Sciences and the MIT Artificial Intelligence Laboratory. Model-Based Control of a Robot Manipulator is included in the Artificial Intelligence Series edited by Patrick Winston and Michael Brady.

Model-Based Control of a Robot Manipulator

Trajectory tracking errors resulting from the application of various controllers have been experimentally determined on the MIT Serial Link Direct Drive Arm. The controllers range from simple analog PD (proportional-derivative) control applied independently at each joint to feedforward and computed torque methods incorporating full dynamics. It was found that trajectory tracking errors decreased as more dynamic compensation terms were incorporated. There was no significant difference in trajectory tracking performance between the feedforward controller using independent digital servos and the full computed torque controller. Implementing the model-based controller highlights the need for accurate control of joint torque, accurate joint position and velocity sensing, and adequate sampling rates. >

Experimental evaluation of feedforward and computed torque control

Robots in force control mode often become unstable during contact with stiff environments, due mainly to the high gain nature of wrist force-sensor feedback. Three methods are presented for achieving stable force control compliant coverings or soft sensors, sell-tuning of force gains after estimation of environmental impedance, and reliance on fast open-loop joint torque control and using tip force sensor feedback in a slow loop to maintain accuracy. The latter method is proposed as the best one. Dynamic stability is analysed with a simple model of the robot and its environment. The analyses are verified by single-link experiments on the MIT Serial Link Direct Drive Arm.

Dynamic Stability Issues in Force Control of Manipulators

Dynamic models are as important in Cartesian force control as they are in position control. A variety of Cartesian force control schemes are examined, comprising some that do incorporate a dynamic model into the control loop (resolved acceleration force control, operational space method, and impedance control) and some that do not (hybrid control and stiffness control). Stability analyses and experimental imple mentations are presented that demonstrate not only that using a dynamic model leads to more accurate control, but also that not using a model can in certain cases make force control unstable. Experiments on the MIT Serial Link Direct Drive Arm show that resolved acceleration force control is stable and accurate in producing force steps and sinusoidal force responses and in complying to sinusoidal position dis turbances.

The Role of Dynamic Models in Cartesian Force Control of Manipulators

The problem of controlling the end-point position of a redundant coarse-fine manipulator is investigated. Such a system typically consists of a coarse robot for approximate positioning and, fastened to the end of it, a fast and accurate fine-motion robotic device. For small motions for which a linearized model of the coarse-fine dynamics is assumed, it is shown that, under mild assumptions, a good disturbance rejection over a wide frequency range of the fine manipulator is sufficient to insure stability and good performance of the coarse-fine redundant manipulator. The fine robot controller can be designed by H/sup infinity / norm minimization, while the coarse robot controller need not be changed. >

C.H. An

Papers

Model-Based Control of a Robot Manipulator

Experimental evaluation of feedforward and computed torque control

Dynamic Stability Issues in Force Control of Manipulators

The Role of Dynamic Models in Cartesian Force Control of Manipulators

On the control of redundant coarse-fine manipulators