The PASCAL Visual Object Classes Challenge

Towards 3D Point cloud based object maps for household environments

Environment models serve as important resources for an autonomous robot by providing it with the necessary task-relevant information about its habitat. Their use enables robots to perform their tasks more reliably, flexibly, and efficiently. As autonomous robotic platforms get more sophisticated manipulation capabilities, they also need more expressive and comprehensive environment models: for manipulation purposes their models have to include the objects present in the world, together with their position, form, and other aspects, as well as an interpretation of these objects with respect to the robot tasks. The dissertation presented in this article (Rusu, PhD thesis, 2009) proposes Semantic 3D Object Models as a novel representation of the robot’s operating environment that satisfies these requirements and shows how these models can be automatically acquired from dense 3D range data.

/pdf/semantic-3d-object-maps-for-everyday-manipulation-in-human-van9xaq8u7.pdf

Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments

We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar, or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally, for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

/pdf/data-driven-grasp-synthesis-a-survey-3ahemc6ci5.pdf

Data-Driven Grasp Synthesis—A Survey

We present a novel dataset and novel algorithms for the problem of detecting activities of daily living (ADL) in firstperson camera views. We have collected a dataset of 1 million frames of dozens of people performing unscripted, everyday activities. The dataset is annotated with activities, object tracks, hand positions, and interaction events. ADLs differ from typical actions in that they can involve long-scale temporal structure (making tea can take a few minutes) and complex object interactions (a fridge looks different when its door is open). We develop novel representations including (1) temporal pyramids, which generalize the well-known spatial pyramid to approximate temporal correspondence when scoring a model and (2) composite object models that exploit the fact that objects look different when being interacted with. We perform an extensive empirical evaluation and demonstrate that our novel representations produce a two-fold improvement over traditional approaches. Our analysis suggests that real-world ADL recognition is “all about the objects,” and in particular, “all about the objects being interacted with.”

/pdf/detecting-activities-of-daily-living-in-first-person-camera-1smqab7oob.pdf

Detecting activities of daily living in first-person camera views

In this paper we report on a recent public experiment that shows two robots making pancakes using web instructions. In the experiment, the robots retrieve instructions for making pancakes from the World Wide Web and generate robot action plans from the instructions. This task is jointly performed by two autonomous robots: The first robot opens and closes cupboards and drawers, takes a pancake mix from the refrigerator, and hands it to the robot B. The second robot cooks and flips the pancakes, and then delivers them back to the first robot. While the robot plans in the scenario are all percept-guided, they are also limited in different ways and rely on manually implemented sub-plans for parts of the task. We will thus discuss the potential of the underlying technologies as well as the research challenges raised by the experiment.

/pdf/robotic-roommates-making-pancakes-27mtmt6r2n.pdf

Robotic roommates making pancakes

In this article, we describe and discuss the use of information that is available in the World Wide Web and intended for human use as a knowledge resource for autonomous service robots. To this end, we introduce several categories of Web sites that can serve as information sources and explain which kinds of information they provide. We then investigate several information processing methods that can access these Web sites to provide robots with necessary knowledge for performing everyday manipulation tasks. The use of the Web as a knowledge resource is a promising alternative to the hard and tedious task of coding comprehensive specific knowledge bases for robots.

Web-Enabled Robots

Humanoid robotic assistants need capable and comprehensive perception systems that enable them to perform complex manipulation and grasping tasks. This requires the identification and recognition of supporting planes and objects in the world, together with their precise 6D poses. In this paper, we propose a 3D perception system architecture that can robustly fit CAD models in cluttered table setting scenes for the purpose of grasping with a mobile manipulator. Our approach uses a powerful combination of two different camera technologies, Time-Of-Flight (TOF) and RGB, to robustly segment the scene and extract object clusters. Using an a-priori database of object models we then perform a CAD matching in 2D camera images. We validate the proposed system in a number of experiments, and compare the system's performance and reliability with similar initiatives.

Real-time CAD model matching for mobile manipulation and grasping

In this paper we propose a novel approach to detect and reconstruct transparent objects. This approach makes use of the fact that many transparent objects, especially the ones consisting of usual glass, absorb light in certain wavelengths [1]. Given a controlled illumination, this absorption is measurable in the intensity response by comparison to the background. We show the usage of a standard infrared emitter and the intensity sensor of a time of flight (ToF) camera to reconstruct the structure given we have a second view point. The structure can not be measured by the usual 3D measurements of the ToF camera. We take advantage of this fact by deriving this internal sensory contradiction from two ToF images and reconstruct an approximated surface of the original transparent object. Therefor we are using a perspectively invariant matching in the intensity channels from the first to the second view of initially acquired candidates. For each matched pixel in the first view a 3D movement can be predicted given their original 3D measurement and the known distance to the second camera position. If their line of sight did not pass a transparent object or suffered any other major defect, this prediction will highly correspond to the actual measured 3D points of the second view. Otherwise, if a detectable error occurs, we approximate a more exact point to point matching and reconstruct the original shape by triangulating the points in the stereo setup. We tested our approach using a mobile platform with one Swissranger SR4k. As this platform is mobile, we were able to create a stereo setup by moving it. Our results show a detection of transparent objects on tables while simultaneously identifying opaque objects that also existed in the test setup. The viability of our results is demonstrated by a successful automated manipulation of the respective transparent object.

/pdf/transparent-object-detection-and-reconstruction-on-a-mobile-15aeg7r740.pdf

Transparent object detection and reconstruction on a mobile platform

This paper introduces the assistive kitchen as a comprehensive demonstration and challenge scenario for technical cognitive systems. We describe its hardware and software infrastructure. Within the assistive kitchen application, we select particular domain activities as research subjects and identify the cognitive capabilities needed for perceiving, interpreting, analyzing, and executing these activities as research foci. We conclude by outlining open research issues that need to be solved to realize the scenarios successfully.

/pdf/the-assistive-kitchen-a-demonstration-scenario-for-cognitive-3k3fnz90jg.pdf

Ulrich Klank

Papers

Robotic roommates making pancakes

Web-Enabled Robots

Real-time CAD model matching for mobile manipulation and grasping

Transparent object detection and reconstruction on a mobile platform

The Assistive Kitchen — A demonstration scenario for cognitive technical systems