Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

/pdf/end-to-end-training-of-deep-visuomotor-policies-1vqd5uo50d.pdf

End-to-end training of deep visuomotor policies

Shows the many advances in the field of cognitive neurosciences. From the molecular level up to that of human consciousness, the contributions cover one of the most fascinating areas of science - the relationship between the structural and physiological mechanisms of the brain/nervous system.

/pdf/the-new-cognitive-neurosciences-n72o5zqocv.pdf

The new cognitive neurosciences

New cognitive neurosciences

This paper presents an inverse kinematics system based on a learned model of human poses. Given a set of constraints, our system can produce the most likely pose satisfying those constraints, in real-time. Training the model on different input data leads to different styles of IK. The model is represented as a probability distribution over the space of all possible poses. This means that our IK system can generate any pose, but prefers poses that are most similar to the space of poses in the training data. We represent the probability with a novel model called a Scaled Gaussian Process Latent Variable Model. The parameters of the model are all learned automatically; no manual tuning is required for the learning component of the system. We additionally describe a novel procedure for interpolating between styles.Our style-based IK can replace conventional IK, wherever it is used in computer animation and computer vision. We demonstrate our system in the context of a number of applications: interactive character posing, trajectory keyframing, real-time motion capture with missing markers, and posing from a 2D image.

/pdf/style-based-inverse-kinematics-4ymoiudyta.pdf

Style-based inverse kinematics

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

End-to-End Training of Deep Visuomotor Policies

Recorded motion data is combined with interactive control techniques to manipulate the animation of articulated figures. The methods enable computer animated characters to produce fully interactive goal-directed behaviors, such as bipedal walking, through simultaneous satisfaction of position, alignment, posture, balance, obstacle avoidance, and joint limitation constraints while retaining qualitative characteristics of the original non-interactive motion data. Goal-based motion primitives, called synergies, are used to coordinate sets of joint movements that attempt to satisfy each of the above constraints. Non-interactive motion capture and keyframe data, representing examples of desired character movements, are accommodated in the present animation system in three ways: 1) direct approach--non-interactive motion data used directly to specify desired body posture synergy goals as a function of time, 2) hybrid approach--non-interactive motion data and program control commands blended to specify elements of desired position, alignment and/or balance synergy goals as a function of time, and 3) template approach--non-interactive motion data used to auto-tune adjustable parameters, enabling program control commands to generate fully interactive movements that qualitatively resemble the non-interactive motion data. The disclosed methods allow libraries of pre-configured goal-directed behaviors, such as reaching, sitting, walking, jumping, etc., to be constructed and used to animate a wide variety of characters.

Interactive motion data animation system

A method and apparatus for interactively controlling and coordinating the limb movements of computer-generated articulated characters with an arbitrary number of joints. On-line computational methods are used for animating limb movements of articulated characters by solving associated forward and inverse kinematics problems in real time subject to multiple goals and constraints. The methods provide computer animated characters with fully interactive goal-directed behaviors, such as bipedal walking, through simultaneous satisfaction of position, alignment, posture, balance, obstacle avoidance, and joint limitation constraints. Goal-based motion primitives, called synergies, coordinate sets of joint movements which separately attempt to satisfy each of the above constraints. The present methods adapt character movements on-line to accommodate uneven terrain, body modifications, or changes in the environment by automatically transforming and producing joint rotations relative to the instantaneous point of contact of the body with the world. Libraries of pre-configured goal-directed behaviors, such as reaching, sitting, walking, jumping, etc., can be constructed and used to animate a wide variety of characters.

Limb coordination system for interactive computer animation of articulated characters

Skillful control under uncertainty via direct reinforcement learning

Direct associative reinforcement learning methods for dynamic systems control

Describes an intelligent control architecture designed to endow human-like capabilities to robots and report experimental results that demonstrate the utility of this architecture in controlling a redundant dynamic manipulator in a hybrid position/force control task. Motor synergies that arise when control of a subset of the available degrees of freedom is coupled and coordinated to accomplish specific task sub-goals are used to simplify the problem, of controlling redundant systems by reducing the dimensionality of the control space. Using synergies as a basis control set gives the controller the general ability to execute novel tasks in unstructured environments. In addition, the rapid learning capabilities of the controller permit refinement of control through the acquisition of skilled control with practice.

Vijaykumar Gullapalli

Papers

Interactive motion data animation system

Limb coordination system for interactive computer animation of articulated characters

Skillful control under uncertainty via direct reinforcement learning

Direct associative reinforcement learning methods for dynamic systems control

Synergy-based learning of hybrid position/force control for redundant manipulators