We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar, or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally, for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

/pdf/data-driven-grasp-synthesis-a-survey-3ahemc6ci5.pdf

Data-Driven Grasp Synthesis—A Survey

This work addresses the problem of real-time 3D shape based object class recognition, its scaling to many categories and the reliable perception of categories. A novel shape descriptor for partial point clouds based on shape functions is presented, capable of training on synthetic data and classifying objects from a depth sensor in a single partial view in a fast and robust manner. The classification task is stated as a 3D retrieval task finding the nearest neighbors from synthetically generated views of CAD-models to the sensed point cloud with a Kinect-style depth sensor. The presented shape descriptor shows that the combination of angle, point-distance and area shape functions gives a significant boost in recognition rate against the baseline descriptor and outperforms the state-of-the-art descriptors in our experimental evaluation on a publicly available dataset of real-world objects in table scene contexts with up to 200 categories.

Ensemble of shape functions for 3D object classification

Artificial skin and tactile sensing for socially interactive robots

Robots operating in domestic environments generally need to interact with articulated objects, such as doors, cabinets, dishwashers or fridges. In this work, we present a novel, probabilistic framework for modeling articulated objects as kinematic graphs. Vertices in this graph correspond to object parts, while edges between them model their kinematic relationship. In particular, we present a set of parametric and non-parametric edge models and how they can robustly be estimated from noisy pose observations. We furthermore describe how to estimate the kinematic structure and how to use the learned kinematic models for pose prediction and for robotic manipulation tasks. We finally present how the learned models can be generalized to new and previously unseen objects. In various experiments using real robots with different camera systems as well as in simulation, we show that our approach is valid, accurate and efficient. Further, we demonstrate that our approach has a broad set of applications, in particular for the emerging fields of mobile manipulation and service robotics.

/pdf/a-probabilistic-framework-for-learning-kinematic-models-of-1l9z3e2imk.pdf

A probabilistic framework for learning kinematic models of articulated objects

Industry 4.0 demands the heavy usage of robotic mobile manipulators with high autonomy and intelligence. The goal is to accomplish dexterous manipulation tasks without prior knowledge of the object status in unstructured environments. It is important for the mobile manipulator to recognize and detect the objects, determine manipulation pose, and adjust its pose in the workspace fast and accurately. In this research, we developed a stereo vision algorithm for the object pose estimation using point cloud data from multiple stereo vision systems. An improved iterative closest point algorithm method is developed for the pose estimation. With the pose input, algorithms and several criteria are studied for the robot to select and adjust its pose by maximizing its manipulability on a given manipulation task. The performance of each technical module and the complete robotic system is finally shown by the virtual robot in the simulator and real robot in experiments. This study demonstrates a setup of autonomous mobile manipulator for various flexible manufacturing and logistical scenarios.

Dexterous Grasping by Manipulability Selection for Mobile Manipulator With Visual Guidance

Humanoid robots that have to operate in cluttered and unstructured environments, such as man-made and natural disaster scenarios, require sophisticated sensorimotor capabilities. A crucial prerequisite for the successful execution of whole-body locomotion and manipulation tasks in such environments is the perception of the environment and the extraction of associated environmental affordances, i.e. the action possibilities of the robot in the environment, in order to generate whole-body locomotion and manipulation actions. We believe that such a coupling between perception and action could be a key to substantially increase the flexibility of humanoid robots. In this paper, we present an approach for the generation of whole-body locomotion and manipulation actions based on the affordances associated with environmental elements in the scene which are extracted via multimodal exploration. Based on the properties of detected environmental primitives and the estimated empty space in the scene, we propose methods to generate hypotheses for feasible whole-body actions while taking into account additional task constraints such as manipulability and balance. We combine visual and inertial sensing modalities by means of a novel depth model for generating segmented and categorized geometric primitives. A rule-based system is then incorporated to assign affordance hypotheses to these primitives. Finally, precomputed whole-body manipulability and stability maps are used for filtering affordances that are out of reach and for identifying the most promising locations for the action execution. We tested the developed methods in different scenes, unknown to the robot, demonstrating how reasonable the generated affordance hypotheses are.

/pdf/extracting-whole-body-affordances-from-multimodal-1shaygtkky.pdf

Extracting whole-body affordances from multimodal exploration

In this paper we present a framework for combining force and visual feedback in the task space to deal with humanoid interaction tasks like open doors in a real kitchen environment. We present stereo-vision methods for markerless recognition and estimation of environmetal elements for applying force control strategies for compliant execution using a 6D force-torque sensor mounted in the humnaoids's wrist. The framework consists of components for model-based self-localization, visual planning, door recognition, grasping based on visual servoing, real time handle tracking and task execution considering force feedback during the physical interaction with the environment. Experimental results on the humanoid robot ARMAR-IIIa are presented.

Combining force and visual feedback for physical interaction tasks in humanoid robots

Humanoid robots should be able to grasp and handle objects in the environment, even if the objects are seen for the first time. A plausible solution to this problem is to categorize these objects into existing classes with associated actions and functional knowledge. So far, efforts on visual object categorization using humanoid robots have either been focused on appearance-based methods or have been restricted to object recognition without generalization capabilities. In this work, a shape model-based approach using stereo vision and machine learning for object categorization is introduced. The state-of-the-art features for shape matching and shape retrieval were evaluated and selectively transfered into the visual categorization. Visual sensing from different vantage points allows the reconstruction of 3D mesh models of the objects found in the scene by exploiting knowledge about the environment for model-based segmentation and registration. These reconstructed 3D mesh models were used for shape feature extraction for categorization and provide sufficient information for grasping and manipulation. Finally, the visual categorization was successfully performed with a variety of features and classifiers allowing proper categorization of unknown objects even when object appearance and shape substantially differ from the training set. Experimental evaluation with the humanoid robot ARMAR-IIIa is presented.

/pdf/towards-shape-based-visual-object-categorization-for-23f9zw6uf6.pdf

Towards shape-based visual object categorization for humanoid robots

In this paper, a geometric approach for global self-localization based on a world-model and active stereo vision is introduced. The method uses class specific object recognition algorithms to obtain the location of entities within the surroundings. The perceived entities in recognition trials are simultaneously filtered and fused to provide a robust set of class features. These classified perceptions which simultaneously satisfy geometric and topological constraints are employed for pruning purposes upon the world-model generating the location hypotheses set. Finally, the hypotheses are validated and disambiguated by applying visual recognition algorithms to selected entities of the world-model. The proposed approach has been successfully used with a humanoid robot.

/pdf/model-based-visual-self-localization-using-geometry-and-329k9sxotv.pdf

Model-based visual self-localization using geometry and graphs

Overcoming the perceptual limitations of humanoid robots requires representations exploitable by highly integrable simulation, sensing, planning and acting components. Therefore, a novel active visual localization component for humanoid robots based on particle filtering in CAD environments is introduced. Specifically, two new components are presented: i) A vector-graphics prediction method employing hierarchical CAD environmental representations is presented. ii) A gaze attention method within the prediction-update cycle of the particle filter increases the available amount of visual features for localization while allowing adjustable task coupling. Finally, large and unobstructive ground-truth validation with the humanoid robot ARMAR-IIIb [1] in a made-for-humans environment shows the robustness, accuracy and performance of the proposed methods.

/pdf/robust-real-time-6d-active-visual-localization-for-humanoid-5bxrrruebi.pdf

D. Gonzalez-Aguirre

Papers

Extracting whole-body affordances from multimodal exploration

Combining force and visual feedback for physical interaction tasks in humanoid robots

Towards shape-based visual object categorization for humanoid robots

Model-based visual self-localization using geometry and graphs

Robust real-time 6D active visual localization for humanoid robots