Interactive teaching and experience extraction for learning about objects and robot activities
20 Oct 2014-pp 153-160
TL;DR: The robot's ontology is extended with concepts for representing human-robot interactions as well as the experiences of the robot, and these experiences are extracted and stored in memory and they are used as input for learning methods.
Abstract: Intelligent service robots should be able to improve their knowledge from accumulated experiences through continuous interaction with the environment, and in particular with humans. A human user may guide the process of experience acquisition, teaching new concepts, or correcting insufficient or erroneous concepts through interaction. This paper reports on work towards interactive learning of objects and robot activities in an incremental and open-ended way. In particular, this paper addresses human-robot interaction and experience gathering. The robot's ontology is extended with concepts for representing human-robot interactions as well as the experiences of the robot. The human-robot interaction ontology includes not only instructor teaching activities but also robot activities to support appropriate feedback from the robot. Two simplified interfaces are implemented for the different types of instructions including the teach instruction, which triggers the robot to extract experiences. These experiences, both in the robot activity domain and in the perceptual domain, are extracted and stored in memory, and they are used as input for learning methods. The functionalities described above are completely integrated in a robot architecture, and are demonstrated in a PR2 robot.
Citations
More filters
Posted Content•
TL;DR: In this paper, it was shown that one's acquaintances, one's immediate neighbors in the acquaintance network, are far from being a random sample of the population, and that this biases the numbers of neighbors two and more steps away.
Abstract: Recent work has demonstrated that many social networks, and indeed many networks of other types also, have broad distributions of vertex degree. Here we show that this has a substantial impact on the shape of ego-centered networks, i.e., sets of network vertices that are within a given distance of a specified central vertex, the ego. This in turn affects concepts and methods based on ego-centered networks, such as snowball sampling and the "ripple effect". In particular, we argue that one's acquaintances, one's immediate neighbors in the acquaintance network, are far from being a random sample of the population, and that this biases the numbers of neighbors two and more steps away. We demonstrate this concept using data drawn from academic collaboration networks, for which, as we show, current simple theories for the typical size of ego-centered networks give numbers that differ greatly from those measured in reality. We present an improved theoretical model which gives significantly better results.
239 citations
TL;DR: Experimental results show that the proposed system is able to interact with human users, learn new object categories over time, as well as perform complex tasks.
Abstract: This paper presents an artificial cognitive system tightly integrating object perception and manipulation for assistive robotics. This is necessary for assistive robots, not only to perform manipulation tasks in a reasonable amount of time and in an appropriate manner, but also to robustly adapt to new environments by handling new objects. In particular, this system includes perception capabilities that allow robots to incrementally learn object categories from the set of accumulated experiences and reason about how to perform complex tasks. To achieve these goals, it is critical to detect, track and recognize objects in the environment as well as to conceptualize experiences and learn novel object categories in an open-ended manner, based on human–robot interaction. Interaction capabilities were developed to enable human users to teach new object categories and instruct the robot to perform complex tasks. A naive Bayes learning approach with a Bag-of-Words object representation are used to acquire and refine object category models. Perceptual memory is used to store object experiences, feature dictionary and object category models. Working memory is employed to support communication purposes between the different modules of the architecture. A reactive planning approach is used to carry out complex tasks. To examine the performance of the proposed architecture, a quantitative evaluation and a qualitative analysis are carried out. Experimental results show that the proposed system is able to interact with human users, learn new object categories over time, as well as perform complex tasks.
58 citations
TL;DR: An object perception and perceptual learning system developed for a complex artificial cognitive agent working in a restaurant scenario that integrates detection, tracking, learning and recognition of tabletop objects and the Point Cloud Library is used in nearly all modules.
Abstract: This paper describes a 3D object perception and perceptual learning system developed for a complex artificial cognitive agent working in a restaurant scenario. This system, developed within the scope of the European project RACE, integrates detection, tracking, learning and recognition of tabletop objects. Interaction capabilities were also developed to enable a human user to take the role of instructor and teach new object categories. Thus, the system learns in an incremental and open-ended way from user-mediated experiences. Based on the analysis of memory requirements for storing both semantic and perceptual data, a dual memory approach, comprising a semantic memory and a perceptual memory, was adopted. The perceptual memory is the central data structure of the described perception and learning system. The goal of this paper is twofold: on one hand, we provide a thorough description of the developed system, starting with motivations, cognitive considerations and architecture design, then providing details on the developed modules, and finally presenting a detailed evaluation of the system; on the other hand, we emphasize the crucial importance of the Point Cloud Library (PCL) for developing such system.11This paper is a revised and extended version of Oliveira et?al. (2014). We describe an object perception and perceptual learning system.The system is able to detect, track and recognize tabletop objects.The system learns novel object categories in an open-ended fashion.The Point Cloud Library is used in nearly all modules of the system.The system was developed and used in the European project RACE.
46 citations
TL;DR: An efficient approach capable of learning and recognizing object categories in an interactive and open-ended manner, which is able to interact with human users, learning new object categories continuously over time is presented.
Abstract: 3D object detection and recognition is increasingly used for manipulation and navigation tasks in service robots. It involves segmenting the objects present in a scene, estimating a feature descriptor for the object view and, finally, recognizing the object view by comparing it to the known object categories. This paper presents an efficient approach capable of learning and recognizing object categories in an interactive and open-ended manner. In this paper, “open-ended” implies that the set of object categories to be learned is not known in advance. The training instances are extracted from on-line experiences of a robot, and thus become gradually available over time, rather than at the beginning of the learning process. This paper focuses on two state-of-the-art questions: (1) How to automatically detect, conceptualize and recognize objects in 3D scenes in an open-ended manner? (2) How to acquire and use high-level knowledge obtained from the interaction with human users, namely when they provide category labels, in order to improve the system performance? This approach starts with a pre-processing step to remove irrelevant data and prepare a suitable point cloud for the subsequent processing. Clustering is then applied to detect object candidates, and object views are described based on a 3D shape descriptor called spin-image. Finally, a nearest-neighbor classification rule is used to predict the categories of the detected objects. A leave-one-out cross validation algorithm is used to compute precision and recall, in a classical off-line evaluation setting, for different system parameters. Also, an on-line evaluation protocol is used to assess the performance of the system in an open-ended setting. Results show that the proposed system is able to interact with human users, learning new object categories continuously over time.
44 citations
Cites background from "Interactive teaching and experience..."
...Further details on the supervised object experience gathering developed for the RACE project are available in (Lim et al., 2014a)....
[...]
..., 2013, 2016c), conferences (Hamidreza Kasaei et al., 2014; Oliveira et al., 2014b; Lim et al., 2014a; Dubba et al., 2014; Kasaei et al., 2015b, 2016b) as well as journal articles (Kasaei et al....
[...]
...The interface is used not only for teaching new object categories in situations where the robot encounters with new objects but also for providing corrective feedback in case there is a misclassification (Chauhan et al., 2013; Lim et al., 2014a)....
[...]
...Interaction capabilities are developed to enable human users to teach new object categories and instruct the robot to perform complex tasks (Lim et al., 2014a)....
[...]
..., Ckey_view) whenever the tracking of an object is initialized (Ctrack), or when it becomes static again after being moved and the user’s hands are far away from the object (Lim et al., 2014a)....
[...]
TL;DR: The general system architecture is introduced and some results in detail regarding hybrid reasoning and planning used in RACE are sketches, and instances of learning from the experiences of real robot task execution are sketched.
Abstract: This paper reports on the aims, the approach, and the results of the European project RACE. The project aim was to enhance the behavior of an autonomous robot by having the robot learn from conceptualized experiences of previous performance, based on initial models of the domain and its own actions in it. This paper introduces the general system architecture; it then sketches some results in detail regarding hybrid reasoning and planning used in RACE, and instances of learning from the experiences of real robot task execution. Enhancement of robot competence is operationalized in terms of performance quality and description length of the robot instructions, and such enhancement is shown to result from the RACE system.
29 citations
Cites background from "Interactive teaching and experience..."
...In the case of supervised experience acquisition, experience extraction is triggered by teaching actions from the user [10]....
[...]
References
More filters
TL;DR: A perceptual theory of knowledge can implement a fully functional conceptual system while avoiding problems associated with amodal symbol systems and implications for cognition, neuroscience, evolution, development, and artificial intelligence are explored.
Abstract: Prior to the twentieth century, theories of knowledge were inherently perceptual. Since then, developments in logic, statis- tics, and programming languages have inspired amodal theories that rest on principles fundamentally different from those underlying perception. In addition, perceptual approaches have become widely viewed as untenable because they are assumed to implement record- ing systems, not conceptual systems. A perceptual theory of knowledge is developed here in the context of current cognitive science and neuroscience. During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas. Later, in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. The stor- age and reactivation of perceptual symbols operates at the level of perceptual components - not at the level of holistic perceptual expe- riences. Through the use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory (e.g., individual memories of green, purr, hot). As memories of the same component become organized around a com- mon frame, they implement a simulator that produces limitless simulations of the component (e.g., simulations of purr). Not only do such simulators develop for aspects of sensory experience, they also develop for aspects of proprioception (e.g., lift, run) and introspec- tion (e.g., compare, memory, happy, hungry). Once established, these simulators implement a basic conceptual system that represents types, supports categorization, and produces categorical inferences. These simulators further support productivity, propositions, and ab- stract concepts, thereby implementing a fully functional conceptual system. Productivity results from integrating simulators combinato- rially and recursively to produce complex simulations. Propositions result from binding simulators to perceived individuals to represent type-token relations. Abstract concepts are grounded in complex simulations of combined physical and introspective events. Thus, a per- ceptual theory of knowledge can implement a fully functional conceptual system while avoiding problems associated with amodal sym- bol systems. Implications for cognition, neuroscience, evolution, development, and artificial intelligence are explored.
5,259 citations
"Interactive teaching and experience..." refers background in this paper
...In the perceptual theory of knowledge, schematic representations of perceptual components are extracted from experience and stored in memory [4], [5]....
[...]
TL;DR: In this paper, a 3D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion is presented, which is based on matching surfaces by matching points using the spin image representation.
Abstract: We present a 3D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion. Recognition is based on matching surfaces by matching points using the spin image representation. The spin image is a data level shape descriptor that is used to match surfaces represented as surface meshes. We present a compression scheme for spin images that results in efficient multiple object recognition which we verify with results showing the simultaneous recognition of multiple objects from a library of 20 models. Furthermore, we demonstrate the robust performance of recognition in the presence of clutter and occlusion through analysis of recognition trials on 100 scenes.
2,798 citations
Journal Article•
TL;DR: This algorithm combines a multilevel approach, which effectively overcomes local minimums, with the Barnes and Hut octree technique, which approximates short and long-range force efficiently.
Abstract: We propose a graph drawing algorithm that is both efficient and high quality. This algorithm combines a multilevel approach, which effectively overcomes local minimums, with the Barnes and Hut [1] octree technique, which approximates shortand long-range force efficiently. Our numerical results show that the algorithm is comparable in speed to Walshaw’s [2] highly efficient multilevel graph drawing algorithm, yet gives better results on some of the difficult problems. In addition, an adaptive cooling scheme for the force-directed algorithms and a general repulsive force model are proposed. The proposed graph drawing algorithm and others are included with Mathematica 5.1 and later versions in the package DiscreteMath‘GraphÑ Plot.
563 citations
TL;DR: This paper shows experiments that demonstrate why there has to be a causal role of language on category acquisition and leads effectively to the bootstrapping of communication and shows that other forms of learning do not generate categories usable in communication or make information assumptions which cannot be satisfied.
Abstract: This paper explores the hypothesis that language communication in its very first stage is bootstrapped in a social learning process under the strong influence of culture. A concrete framework for social learning has been developed based on the notion of a language game. Autonomous robots have been programmed to behave according to this framework. We show experiments that demonstrate why there has to be a causal role of language on category acquisition; partly by showing that it leads effectively to the bootstrapping of communication and partly by showing that other forms of learning do not generate categories usable in communication or make information assumptions which cannot be satisfied.
291 citations
"Interactive teaching and experience..." refers methods in this paper
...[10] used the notion of ‘language game’ to develop a social learning framework through which a robot can learn its first words....
[...]
Posted Content•
TL;DR: In this paper, it was shown that one's acquaintances, one's immediate neighbors in the acquaintance network, are far from being a random sample of the population, and that this biases the numbers of neighbors two and more steps away.
Abstract: Recent work has demonstrated that many social networks, and indeed many networks of other types also, have broad distributions of vertex degree. Here we show that this has a substantial impact on the shape of ego-centered networks, i.e., sets of network vertices that are within a given distance of a specified central vertex, the ego. This in turn affects concepts and methods based on ego-centered networks, such as snowball sampling and the "ripple effect". In particular, we argue that one's acquaintances, one's immediate neighbors in the acquaintance network, are far from being a random sample of the population, and that this biases the numbers of neighbors two and more steps away. We demonstrate this concept using data drawn from academic collaboration networks, for which, as we show, current simple theories for the typical size of ego-centered networks give numbers that differ greatly from those measured in reality. We present an improved theoretical model which gives significantly better results.
239 citations