We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies that can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed...

Learning dexterous in-hand manipulation:

Humans can feel, weigh and grasp diverse objects, and simultaneously infer their material properties while applying the right amount of force—a challenging set of tasks for a modern robot1. Mechanoreceptor networks that provide sensory feedback and enable the dexterity of the human grasp2 remain difficult to replicate in robots. Whereas computer-vision-based robot grasping strategies3–5 have progressed substantially with the abundance of visual data and emerging machine-learning tools, there are as yet no equivalent sensing platforms and large-scale datasets with which to probe the use of the tactile information that humans rely on when grasping objects. Studying the mechanics of how humans grasp objects will complement vision-based robotic object handling. Importantly, the inability to record and analyse tactile signals currently limits our understanding of the role of tactile information in the human grasp itself—for example, how tactile maps are used to identify objects and infer their properties is unknown6. Here we use a scalable tactile glove and deep convolutional neural networks to show that sensors uniformly distributed over the hand can be used to identify individual objects, estimate their weight and explore the typical tactile patterns that emerge while grasping objects. The sensor array (548 sensors) is assembled on a knitted glove, and consists of a piezoresistive film connected by a network of conductive thread electrodes that are passively probed. Using a low-cost (about US$10) scalable tactile glove sensor array, we record a large-scale tactile dataset with 135,000 frames, each covering the full hand, while interacting with 26 different objects. This set of interactions with different objects reveals the key correspondences between different regions of a human hand while it is manipulating objects. Insights from the tactile signatures of the human grasp—through the lens of an artificial analogue of the natural mechanoreceptor network—can thus aid the future design of prosthetics7, robot grasping tools and human–robot interactions1,8–10. Tactile patterns obtained from a scalable sensor-embedded glove and deep convolutional neural networks help to explain how the human hand can identify and grasp individual objects and estimate their weights.

Learning the signatures of the human grasp using a scalable tactile glove.

Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid defOrmations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes at http://mano.is.tue.mpg.de.

Embodied hands: modeling and capturing hands and bodies together

This work presents a novel technique for direct 3D printing of soft pneumatic actuators using 3D printers based on fused deposition modeling (FDM) technology. Existing fabrication techniques for soft pneumatic actuators with complex inner geometry are normally time-consuming and involve multistep processes. A low-cost open-source consumer 3D printer and a commercially available printing material were identified for printing soft pneumatic actuators with complex inner geometry and high degree of freedom. We investigated the material properties of the printing material, simulated the mechanical behavior of the printed actuators, characterized the performances of the actuators in terms of their bending capability, output forces, as well as durability, and demonstrated the potential soft robotic applications of the 3D printed actuators. Using the 3D printed actuators, we developed a soft gripper that was able to grasp and lift heavy objects with high pay–-to-weight ratio, which demonstrated that the ...

High-Force Soft Printable Pneumatics for Soft Robotic Applications

BACKGROUND Humans have a fantastic ability to manipulate objects of various shapes, sizes, and materials and can control the objects’ position in confined spaces with the advanced dexterity capabilities of our hands. Building machines inspired by human hands, with the functionality to autonomously pick up and manipulate objects, has always been an essential component of robotics. The first robot manipulators date back to the 1960s and are some of the first robotic devices ever constructed. In these early days, robotic manipulation consisted of carefully prescribed movement sequences that a robot would execute with no ability to adapt to a changing environment. As time passed, robots gradually gained the ability to automatically generate movement sequences, drawing on artificial intelligence and automated reasoning. Robots would stack boxes according to size, weight, and so forth, extending beyond geometric reasoning. This task also required robots to handle errors and uncertainty in sensing at run time, given that the slightest imprecision in the position and orientation of stacked boxes might cause the entire tower to topple. Methods from control theory also became instrumental for enabling robots to comply with the environment’s natural uncertainty by empowering them to adapt exerted forces upon contact. The ability to stably vary forces upon contact expanded robots’ manipulation repertoire to more-complex tasks, such as inserting pegs in holes or hammering. However, none of these actions truly demonstrated fine or in-hand manipulation capabilities, and they were commonly performed using simple two-fingered grippers. To enable multipurpose fine manipulation, roboticists focused their efforts on designing humanlike hands capable of using tools. Wielding a tool in-hand became a problem of its own, and a variety of advanced algorithms were developed to facilitate stable holding of objects and provide optimality guarantees. Because optimality was difficult to achieve in a stochastic environment, from the 1990s onward researchers aimed to increase the robustness of object manipulation at all levels. These efforts initiated the design of sensors and hardware for improved control of hand–object contacts. Studies that followed were focused on robust perception for coping with object occlusion and noisy measurements, as well as on adaptive control approaches to infer an object’s physical properties, so as to handle objects whose properties are unknown or change as a result of manipulation. ADVANCES Roboticists are still working to develop robots capable of sorting and packaging objects, chopping vegetables, and folding clothes in unstructured and dynamic environments. Robots used for modern manufacturing have accomplished some of these tasks in structured settings that still require fences between the robots and human operators to ensure safety. Ideally, robots should be able to work side by side with humans, offering their strength to carry heavy loads while presenting no danger. Over the past decade, robots have gained new levels of dexterity. This enhancement is due to breakthroughs in mechanics with sensors for perceiving touch along a robot’s body and new mechanics for soft actuation to offer natural compliance. Most notably, this development leverages the immense progress in machine learning to encapsulate models of uncertainty and support further advances in adaptive and robust control. Learning to manipulate in real-world settings is costly in terms of both time and hardware. To further elaborate on data-driven methods but avoid generating examples with real, physical systems, many researchers use simulation environments. Still, grasping and dexterous manipulation require a level of reality that existing simulators are not yet able to deliver—for example, in the case of modeling contacts for soft and deformable objects. Two roads are hence pursued: The first draws inspiration from the way humans acquire interaction skills and prompts robots to learn skills from observing humans performing complex manipulation. This allows robots to acquire manipulation capabilities in only a few trials. However, generalizing the acquired knowledge to apply to actions that differ from those previously demonstrated remains difficult. The second road constructs databases of real object manipulation, with the goal to better inform the simulators and generate examples that are as realistic as possible. Yet achieving realistic simulation of friction, material deformation, and other physical properties may not be possible anytime soon, and real experimental evaluation will be unavoidable for learning to manipulate highly deformable objects. OUTLOOK Despite many years of software and hardware development, achieving dexterous manipulation capabilities in robots remains an open problem—albeit an interesting one, given that it necessitates improved understanding of human grasping and manipulation techniques. We build robots to automate tasks but also to provide tools for humans to easily perform repetitive and dangerous tasks while avoiding harm. Achieving robust and flexible collaboration between humans and robots is hence the next major challenge. Fences that currently separate humans from robots will gradually disappear, and robots will start manipulating objects jointly with humans. To achieve this objective, robots must become smooth and trustable partners that interpret humans’ intentions and respond accordingly. Furthermore, robots must acquire a better understanding of how humans interact and must attain real-time adaptation capabilities. There is also a need to develop robots that are safe by design, with an emphasis on soft and lightweight structures as well as control and planning methodologies based on multisensory feedback.

Trends and challenges in robot manipulation

MATERIAL STRUCTURE.- Bionic (Nano)Membranes.- Tribology in Biology.- Python Regius (Ball Python) Shed Skin: Biomimetic Analogue for Function-Targeted Design of Tribo-Surfaces.- Poro-Micromechanics as Analysis Tool for Revealing Mechanical Design principles in Bone and Bone Biomaterials.- Bio-inspired Cellular Structures: Solid Freeform Fabrication and Mechanical Properties.- Bio-inspired Technology of Fassads in Architectur.- FORM AND CONSTRUCTION.- Architecture and Biomimetics - Design Methods and Case Studies.- Morphological Processes in Architecture.- Fractal Geometry and Architecture.- INFORMATION AND DYNAMICS.- Biomimetics in Automatisation.- Technical Rebuilding of Movement Function using Functional Electrostimulation FES.- Amelioration of Hearing by Disturbing Signals.- Biodynamics.- Bio-inspired Front Technology in Architecture.

Biomimetics - materials, structures and processes : examples, ideas and case studies

Carved turns with alpine skis are investigated. During the movement of a ski, snow is loaded and unloaded. Compacted snow is not elastic, i.e. deformations remain. Such effects are modeled by a hypoplastic constitutive equation. During a turn the shovel digs into the snow and the tail maintains nearly the same penetration depth as the part under maximum load. This results in a higher resistance against shearing for the afterbody of the ski. In the present work we investigated the benefits of the hypoplastic against the elastic forcepenetration relationship. Simulation results for a sledge on two skis are compared to experimental track data.

Modeling of the Ski-Snow Contact for a Carved Turn

Background In treatment of pertrochanteric femoral fractures with dynamic hip screws (DHSs) (135-degree, Synthes, Bettlach, Switzerland), damage was observed in removed lag screws, leading to the conclusion that the gliding mechanism must have been obstructed as a result of either inappropriate position of the implant or insufficient medial support in the fracture zone. Methods The forces and moments transmitted in the screw socket are calculated using a mathematical model to find the optimal position of the implant. Results The forces and moments depend on the position and orientation of the lag screw as well as on the position of the contact point between the two main fragments. By changing the point of contact, a better decrease of the load to the DHS can be achieved than by changing the position and orientation of the screw. For a low contact point, the model shows the lowest values for the forces in the socket. Conclusion Complete agreement was found between the results of the presented calculations and our own clinical experience in removed DHSs.

Mechanical factors responsible for the obstruction of the gliding mechanism of a dynamic hip screw for stabilizing pertrochanteric femoral fractures.

The interpretation of joint kinematics data in terms of displacements is a product of the type of movement, the measurement technique and the underlying model of the joint implemented in optimization procedures. Kinematic constraints reducing the number of degrees of freedom (DOFs) are expected to compensate for measurement errors and noise, thus, increasing the reproducibility of joint angles. One approach already successfully applied by several groups approximates the healthy human knee joint as a compound hinge joint with minimal varus/valgus rotation. Most of these optimizations involve an orthogonality constraint. This contribution compares the effect of a model with and without orthogonality constraint on the obtained joint rotation angles. For this purpose, knee joint motion is simulated to generate kinematic data without noise and with normally distributed noise of varying size. For small noise the unconstrained model provides more accurate results, whereas for larger noise this is the case for th...

/pdf/reconstructing-the-knee-joint-mechanism-from-kinematic-data-27o8xnl1mh.pdf

Reconstructing the knee joint mechanism from kinematic data

The interpretation of joint kinematics data in terms of displacements is a product of the type of movement, the measurement technique, and the underlying model of the joint implemented in optimization procedures. Kinematic constraints reducing the number of degrees of freedom are expected to compensate for measurement errors and noise, thus, increasing the reproducibility of joint angles. One approach already successfully applied by several groups approximates the healthy human knee joint as a compound hinge joint with minimal varus/valgus rotation. Most of these optimizations involve an orthogonality constraint. This contribution compares the efiect of a model with and without orthogonality constraint on the obtained joint rotation angles. For this purpose kinematic data is simulated without noise and with normally distributed noise of varying size. For small noise the unconstrained model provides more accurate results while for larger noise this is the case for the constrained model. This can be attributed to the shape of the objective function of the unconstrained model near its minimum.

Heinz-Bodo Schmiedmayer

Papers

Biomimetics - materials, structures and processes : examples, ideas and case studies

Modeling of the Ski-Snow Contact for a Carved Turn

Mechanical factors responsible for the obstruction of the gliding mechanism of a dynamic hip screw for stabilizing pertrochanteric femoral fractures.

Reconstructing the knee joint mechanism from kinematic data

RESEARCH ARTICLE Reconstructing the knee joint mechanism from kinematic data