There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

CNN based automatic detection of photovoltaic cell defects in electroluminescence images

Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at this https URL.

GRAB: A Dataset of Whole-Body Human Grasping of Objects

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer.github.io

RT-1: Robotics Transformer for Real-World Control at Scale

We propose an approach to estimate 3D human pose in real world units from a single RGBD image and show that it exceeds performance of monocular 3D pose estimation approaches from color as well as pose estimation exclusively from depth. Our approach builds on robust human keypoint detectors for color images and incorporates depth for lifting into 3D. We combine the system with our learning from demonstration framework to instruct a service robot without the need of markers. Experiments in real world settings demonstrate that our approach enables a PR2 robot to imitate manipulation actions observed from a human teacher.

3D Human Pose Estimation in RGBD Images for Robotic Task Learning

Microcracks in silicon wafers reduce the strength of the wafers and can lead to critical failure within the solar-cell production. Both detection of the microcracks and their impact on fracture strength of the wafers are addressed within this study. To improve the accuracy of the crack detection in photoluminescence (PL) and infrared transmission (IR) images of as-cut wafers, we introduce a pattern recognition approach based on local descriptors and support-vector classification. The learning model requires a set of labeled data generated by an artificial insertion of cracks. Within this evaluation, the algorithm detects 81% of the cracks for PL-images and 98% for IR-images at precision rates above 98% in each case, which outperforms the quality of pure IR-intensity-based crack-detection systems with a hit-rate of 65% at a precision of 59%. The proposed algorithm may be combined with the images of the grain structure to avoid the confusion of cracks and grain boundaries. Moreover, the comprehensive set of wafers allows the impact of crack morphology on wafer strength to be investigated. Despite complex crack morphologies, the theoretically expected dependence between crack length and fracture strength is confirmed. Therefore, sorting criteria are derived to rate the cracks with respect to the expected fracture strength of the wafer based on the measured crack length only.

Microcracks in Silicon Wafers I: Inline Detection and Implications of Crack Morphology on Wafer Strength

Over the past years learning from demonstration has become a popular method to intuitively teach new skills to service robots without explicit programming. However, most teaching approaches in literature use kinesthetic training and do not include mobile platforms. Here, we present a novel approach to learn joint robot base and gripper action models from observing demonstrations carried out by a human teacher. To achieve this we adapt RGBD observations of the human teacher to the capabilities of the robot. We formulate a graph optimization problem that links observations with robot grasping capabilities and kinematic constraints between co-occurring base and gripper poses. In real world experiments we show that the robot is able to learn complex mobile manipulation tasks such as opening and driving through a door.

Learning mobile manipulation actions from human demonstrations

Mobile manipulation tasks remain one of the critical challenges for the widespread adoption of autonomous robots in both service and industrial scenarios. While planning approaches are good at generating feasible whole-body robot trajectories, they struggle with dynamic environments as well as the incorporation of constraints given by the task and the environment. On the other hand, dynamic motion models in the action space struggle with generating kinematically feasible trajectories for mobile manipulation actions. We propose a deep reinforcement learning approach to learn feasible dynamic motions for a mobile base while the end-effector follows a trajectory in task space generated by an arbitrary system to fulfill the task at hand. This modular formulation has several benefits: it enables us to readily transform a broad range of end-effector motions into mobile applications, it allows us to use the kinematic feasibility of the end-effector trajectory as a dense reward signal and its modular formulation allows it to generalise to unseen end-effector motions at test time. We demonstrate the capabilities of our approach on multiple mobile robot platforms with different kinematic abilities and different types of wheeled platforms in extensive simulated as well as real-world experiments.

Tim Welschehold

Papers

3D Human Pose Estimation in RGBD Images for Robotic Task Learning

Microcracks in Silicon Wafers I: Inline Detection and Implications of Crack Morphology on Wafer Strength

Learning mobile manipulation actions from human demonstrations

Learning Kinematic Feasibility for Mobile Manipulation Through Deep Reinforcement Learning

3D Human Pose Estimation in RGBD Images for Robotic Task Learning