Learning robot tasks or controllers using deep reinforcement learning has been proven effective in simulations. Learning in simulation has several advantages. For example, one can fully control the simulated environment, including halting motions while performing computations. Another advantage when robots are involved, is that the amount of time a robot is occupied learning a task---rather than being productive---can be reduced by transferring the learned task to the real robot. Transfer learning requires some amount of fine-tuning on the real robot. For tasks which involve complex (non-linear) dynamics, the fine-tuning itself may take a substantial amount of time. In order to reduce the amount of fine-tuning we propose to learn robustified controllers in simulation. Robustified controllers are learned by exploiting the ability to change simulation parameters (both appearance and dynamics) for successive training episodes. An additional benefit for this approach is that it alleviates the precise determination of physics parameters for the simulator, which is a non-trivial task. We demonstrate our proposed approach on a real setup in which a robot aims to solve a maze game, which involves complex dynamics due to static friction and potentially large accelerations. We show that the amount of fine-tuning in transfer learning for a robustified controller is substantially reduced compared to a non-robustified controller.

Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics.

Learning robot tasks or controllers using deep reinforcement learning has been proven effective in simulations. Learning in simulation has several advantages. For example, one can fully control the simulated environment, including halting motions while performing computations. Another advantage when robots are involved, is that the amount of time a robot is occupied learning a task—rather than being productive—can be reduced by transferring the learned task to the real robot. Transfer learning requires some amount of fine-tuning on the real robot. For tasks which involve complex (non-linear) dynamics, the fine-tuning itself may take a substantial amount of time. In order to reduce the amount of fine-tuning we propose to learn robustified controllers in simulation. Robustified controllers are learned by exploiting the ability to change simulation parameters (both appearance and dynamics) for successive training episodes. An additional benefit for this approach is that it alleviates the precise determination of physics parameters for the simulator, which is a non-trivial task. We demonstrate our proposed approach on a real setup in which a robot aims to solve a maze game, which involves complex dynamics due to static friction and potentially large accelerations. We show that the amount of fine-tuning in transfer learning for a robustified controller is substantially reduced compared to a non-robustified controller.

Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics

Control of underactuated balance robot requires external subsystem trajectory tracking and internal unstable subsystem balancing with limited control authority. We present a learning-based control approach for underactuated balance robots. The tracking and balancing control is designed the controller in fast- and slow-time scales. In the slow-time scale, model predictive control is adopted to plan desired internal state profile to achieve external trajectory tracking task. The internal state is then stabilized around the planned profile in the fast-time scale. The control design is based on a learned Gaussian process (GP) regression model without need of a priori knowledge about the robot dynamics. The controller also incorporates the GP model predicted variance to enhance robustness to modeling errors. Experiments are presented using a Furuta pendulum system.

Gaussian Processes Model-Based Control of Underactuated Balance Robots

Active learning in robotics: A review of control principles

A robot's dynamic model depends on both the kinematic and mass-inertial parameters of a robot Robot model identification therefore typically begins with kinematic identification; the mass-inertial parameters are then identified with the identified kinematic parameters used in the dynamic model In this article we show that poorly identified kinematic parameters can lead to an uncorrectable bias in the dynamic model, leading to errors in the mass-inertial parameters that are many times that of kinematic parameter errors We instead argue that a unified kinodynamic identification leads to more accurate identification of both the kinematic and mass-inertial parameters A linearly weighted kinodynamic objective function is proposed, in which the weight can be interpreted from a maximum likelihood perspective as the relative accuracy of the kinematic sensors vis-a-vis the dynamic sensors Recursive algorithms for computing exact analytic gradients of the kinodynamic objective function are newly derived, leading to robust and fast-converging identification algorithms Extensive numerical and hardware experiments demonstrate the advantages of our unified kinodynamic identification procedure

Kinodynamic Model Identification: A Unified Geometric Approach

In this letter, we propose a derivative-free model learning framework for Reinforcement Learning (RL) algorithms based on Gaussian Process Regression (GPR). In many mechanical systems, only positions can be measured by the sensing instruments. Then, instead of representing the system state as suggested by the physics with a collection of positions, velocities, and accelerations, we define the state as the set of past position measurements. However, the equation of motions derived by physical first principles cannot be directly applied in this framework, being functions of velocities and accelerations. For this reason, we introduce a novel derivative-free physically-inspired kernel, which can be easily combined with nonparametric derivative-free Gaussian Process models. Tests performed on two real platforms show that the considered state definition combined with the proposed model improves estimation performance and data-efficiency w.r.t. traditional models based on GPR. Finally, we validate the proposed framework by solving two RL control problems for two real robotic systems.

Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements

A novel Multiplicative Polynomial Kernel for Volterra series identification

Robotics systems are becoming more and more autonomous and reconfigurable. In this context, the design of algorithms capable of deriving kinematics and dynamics models directly from data could be particularly useful. In this article, we present an algorithm that learns a forward kinematics model of a robot starting from a time series of visual observations. Our strategy can be applied to any robot with serial kinematics composed of revolute and prismatics joints. First, the algorithm identifies the robot kinematic structure, i.e., a high-level description of the robot geometry that defines the connections between the rigid-bodies composing the robot. Then, the algorithm derives the forward kinematics relying on a Gaussian process (GP) model. More precisely, the GP model is based on a polynomial kernel, defined exploiting the kinematic structure previously identified. The effectiveness of the proposed solution has been tested via extensive Monte Carlo simulations, as well as via experiments on a real UR10 robot.

Autonomous Learning of the Robot Kinematic Model

Volterra series are especially useful for nonlinear system identification, also thanks to their capability to approximate a broad range of input-output maps. However, their identification from a finite set of data is hard, due to the curse of dimensionality. Recent approaches have shown how regularized kernel-based methods can be useful for this task. In this paper, we propose a new regularization network for Volterra models identification. It relies on a new kernel given by the product of basic building blocks. Each block contains some unknown parameters that can be estimated from data using marginal likelihood optimization. In comparison with other algorithms proposed in the literature, numerical experiments show that our approach allows to better select the monomials that really influence the system output, much increasing the prediction capability of the model.

In this letter, we introduce a novel data-driven inverse dynamics estimator based on Gaussian Process Regression. Driven by the fact that the inverse dynamics can be described as a polynomial function on a suitable input space, we propose the use of a novel kernel, called Geometrically Inspired Polynomial Kernel (GIP). The resulting estimator behaves similarly to model-based approaches as concerns data efficiency. Indeed, we proved that the GIP kernel defines a finite-dimensional Reproducing Kernel Hilbert Space that contains the inverse dynamics function computed through the Rigid Body Dynamics. The proposed kernel is based on the recently introduced Multiplicative Polynomial Kernel , a redefinition of the classical polynomial kernel equipped with a set of parameters that allows for a higher regularization. We tested the proposed approach in a simulated environment, and also in real experiments with a UR10 robot. The obtained results confirm that, compared to other data-driven estimators, the proposed approach is more data-efficient and exhibits better generalization properties. Instead, with respect to model-based estimators, our approach requires less prior information and is not affected by model bias.

Alberto Dalla Libera

Papers

Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements

A novel Multiplicative Polynomial Kernel for Volterra series identification

Autonomous Learning of the Robot Kinematic Model

A novel Multiplicative Polynomial Kernel for Volterra series identification

A Data-Efficient Geometrically Inspired Polynomial Kernel for Robot Inverse Dynamic