Showing papers presented at "Simulation of Adaptive Behavior in 2012"

Analysing an Evolved Robotic Behaviour Using a Biological Model of Collegial Decision Making

Abstract: Evolutionary robotics can be a powerful tool in studies on the evolutionary origins of self-organising behaviours in biological systems. However, these studies are viable only when the behaviour of the evolved artificial system closely corresponds to the one observed in biology, as described by available models. In this paper, we compare the behaviour evolved in a robotic system with the collegial decision making displayed by cockroaches in selecting a resting shelter. We show that artificial evolution can synthesise a simple self-organising behaviour for a swarm of robots, which presents dynamics that are comparable with the cockroaches behaviour.

Multi-timescale Nexting in a Reinforcement Learning Robot

Abstract: The term “nexting” has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to “next” constitutes a basic kind of awareness and knowledge of one’s environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world’s state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward-like target and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(λ) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with most of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution.

Biologically-Inspired Deceptive Behavior for a Robot

Abstract: A common behavior in animals or human beings is deception. We focus on deceptive behavior in robotics because the appropriate use of deception is beneficial in several domains ranging from the military to a more everyday context. In this research, novel algorithms are developed for the deceptive behavior of a robot, inspired by the observed deceptive behavior of squirrels for cache protection strategies, evaluating the results via simulation studies.

Using Sensorimotor Contingencies for Terrain Discrimination and Adaptive Walking Behavior in the Quadruped Robot Puppy

Abstract: In conventional “sense-think-act” control architectures, perception is reduced to a passive collection of sensory information, followed by a mapping onto a prestructured internal world model. For biological agents, Sensorimotor Contingency Theory (SMCT) posits that perception is not an isolated processing step, but is constituted by knowing and exercising the law-like relations between actions and resulting changes in sensory stimulation. We present a computational model of SMCT for controlling the behavior of a quadruped robot running on different terrains. Our experimental study demonstrates that: (i) Sensory-Motor Contingencies (SMC) provide better discrimination capabilities of environmental properties than conventional recognition from the sensory signals alone; (ii) discrimination is further improved by considering the action context on a longer time scale; (iii) the robot can utilize this knowledge to adapt its behavior for maximizing its stability.

Learning Adjectives and Nouns from Affordances on the iCub Humanoid Robot

Abstract: This article studies how a robot can learn nouns and adjectives in language. Towards this end, we extended a framework that enabled robots to learn affordances from its sensorimotor interactions, to learn nouns and adjectives using labeling from humans. Specifically, an iCub humanoid robot interacted with a set of objects (each labeled with a set of adjectives and a noun) and learned to predict the effects (as labeled with a set of verbs) it can generate on them with its behaviors. Different from appearance-based studies that directly link the appearances of objects to nouns and adjectives, we first predict the affordances of an object through a set of Support Vector Machine classifiers which provided a functional view of the object. Then, we learned the mapping between these predicted affordance values and nouns and adjectives. We evaluated and compared a number of different approaches towards the learning of nouns and adjectives on a small set of novel objects.

Multimodal Integration of Visual Place Cells and Grid Cells for Navigation Tasks of a Real Robot

Abstract: In the present study, we propose a model of multimodal place cells merging visual and proprioceptive primitives. First we will briefly present our previous sensory-motor architecture, highlighting limitations of a visual-only based system. Then we will introduce a new model of proprioceptive localization, giving rise to the so-called grid cells, wich are congruent with neurobiological studies made on rodent.

Evolving Reactive Controller for a Modular Robot: Benefits of the Property of State-Switching in Fractal Gene Regulatory Networks

Abstract: In this paper, we study Fractal Gene Regulatory Networks (FGRNs) evolved as local controllers for a modular robot in snake topology that reacts adaptively to environment. The task is to have the robot moving in a specific direction until it reaches a randomly placed target-zone and stays there. We point to a characteristic of FGRN model, namely “state-switching property” and demonstrate it as a beneficial property in evolving reactive controllers.

Using Sensorimotor Contingencies for Prediction and Action Planning

Abstract: Sensorimotor contingency theory holds that the law-like relations between actions and contingent changes in the sensory signals constitute the basis for sensory experience and awareness in humans. These Sensory-Motor Contingencies (SMCs) are not only passively observed and recorded by the agent, but are actively exercised and used to control behavior. We have previously introduced a computational model of SMCs for robot control that employs a set of Markov models for the conditional probabilities of making sensory observations given an action. In this article we extend this model by showing how prediction and evaluation of future sensorimotor events can be achieved. We investigate this prediction and planning method in a scenario where the robot’s actions do not take immediately effect, so that it has to plan ahead. Exploiting an action selection method that takes into account previous experiences, the robot learns to move in an energy-efficient, naturalistic manner and to avoid known obstacles. We also make a first step towards analyzing the robot’s behavior in a dynamically changing environment.

Automatic Synthesis of Controllers for Real Robots Based on Preprogrammed Behaviors

Abstract: We present a novel methodology for the synthesis of behavioral control for real robotic hardware. In our approach, neural controllers decide when different preprogrammed behaviors should be active during task execution. We evaluate our approach in a double T-maze task carried out by an e-puck robot. We compare results obtained in our setup with results obtained in a traditional evolutionary robotics setup where the neural controller has direct control over the robot’s actuators. The results show that the combination of preprogrammed and evolved control offers two key benefits over a traditional evolutionary robotics approach: (i) solutions are synthesized faster and achieve a higher performance, and (ii) solutions synthesized in simulation maintain their performance when transferred to real robotic hardware.

Degeneration of a von Neumann Self-reproducer into a Self-copier within the Avida World

Abstract: The theory of machine self-reproduction formalised by John von Neumann illustrates the real living organisms’ self-reproduction equipped with genotype and phenotype. However, within such a simulated world as Avida, this particular style of self-reproduction has not been previously studied. In an attempt to characterise the von Neumann style self-reproducer in a computational system, we have implemented a novel seed program that self-reproduces using von Neumann’s architecture. We expected that distinctly different evolutionary dynamics of organisms in the system would be observed, specifically including the possibility of mutationally altered genotype-phenotype mapping. However, what we have observed is degenerative displacement by self-copiers, which are conventional self-reproducers in the system. The mutational easiness of this degeneration was not anticipated, although we knew the selective advantage that such self-copiers intrinsically would have in the system.

Attentional Action Selection Using Reinforcement Learning

Abstract: Reinforcement learning is typically used to model and optimize action selection strategies, in this work we deploy it to optimize attentional allocation strategies while action selection is obtained as a side effect. We present a reinforcement learning approach to attentional allocation and action selection in a behavior-based robotic systems. We detail our attentional allocation mechanisms describing the reinforcement learning problem and analysing its performance in a survival domain.

Natural and Artificial Systems: Compare, Model or Engineer?

Abstract: Some areas of biological research use artificial means to explore the natural world But how the natural and artificial are related across wide-ranging research areas is not always clear Relations differ further for bioengineering fields We propose a taxonomy which would serve to elucidate distinct relations; there are three ways in which the natural is linked to the artificial, corresponding with distinct methods of investigation: i) a comparative approach (natural vs artificial) in which artificial systems are treated in the same way as natural systems, ii) a modeling approach (natural via artificial) in which we use artificial systems to learn about features of natural ones, and iii) an engineering approach (natural pro artificial) in which natural systems are used to draw inspiration for artefacts Ambiguities about and between these approaches limit the development of fields and impact negatively on interdisciplinary communication

Plastic Representation of the Reachable Space for a Humanoid Robot

Abstract: Reaching a target object requires accurate estimation of the object spatial position and its further transformation into a suitable arm-motor command. In this paper, we propose a framework that provides a robot with a capacity to represent its reachable space in an adaptive way. The location of the target is represented implicitly by both the gaze direction and the angles of arm joints. Two paired neural networks are used to compute the direct and inverse transformations between the arm position and the head position. These networks allow reaching the target either through a ballistic movement or through visually-guided actions. Thanks to the latter skill, the robot can adapt its sensorimotor transformations so as to reflect changes in its body configuration. The proposed framework was implemented on the NAO humanoid robot, and our experimental results provide evidences for its adaptative capabilities.

The Search for Beauty: Evolution of Minimal Cognition in an Animat Controlled by a Gene Regulatory Network and Powered by a Metabolic System

Abstract: We have created a model of a hybrid system in which a gene regulatory network (GRN) controls the search for resources (fuel/food and water) necessary to allow an artificial metabolic system (simulated microbial fuel cell) to produce energy. We explore the behaviour of simple animats in a two-dimensional simulated environment requiring minimal cognition. In our system control evolves in a biologically-realistic manner under tight energy constraints. We use a model of GRN in which there is no limit on the size of the network, and the concentration of regulatory substances (transcriptional factors, TFs) change in a continuous fashion. Externally driven concentrations of selected TFs provide the sensory information to the animat, while the concentration of selected internally produced TFs is interpreted as the signal for actuators. We use a genetic algorithm to obtain diverse evolved strategies in ecologically grounded animats with motivational autonomy, even though they lack a dedicated motivational circuit. There are three motivations (or drives) in the system: thirst, hunger, and reproduction. The animats need to search for food and water, but also to perform work. Because the value of such work is arbitrary (in the eye of the beholder), but affects the chances of reproduction, we suggest that the term beauty is more appropriate, and we name the task the Search for Beauty. The results obtained provide a step towards realizing a biologically realistic system with respect to: the way the control is exercised, the way it evolves, and the way the metabolism provides energy.

Synthesis and Adaptation of Effective Motor Synergies for the Solution of Reaching Tasks

Abstract: Taking inspiration from the hypothesis of muscle synergies, we propose a method to generate open loop controllers for an agent solving point-to-point reaching tasks. The controller output is defined as a linear combination of a small set of predefined actuations, termed synergies. The method can be interpreted from a developmental perspective, since it allows the agent to autonomously synthesize and adapt an effective set of synergies to new behavioral needs. This scheme greatly reduces the dimensionality of the control problem, while keeping a good performance level. The framework is evaluated in a planar kinematic chain, and the quality of the solutions is quantified in several scenarios.

The Emergence of Pathological Constructors When Implementing the von Neumann Architecture for Self-reproduction in Tierra

Abstract: John von Neumann’s architecture for genetic reproduction provides an explanation in principle for how arbitrarily complex machines can construct other (“offspring”) machines of equal or even greater complexity. We designed a von Neumann style self-reproducing ancestor, within the framework of the Tierra platform, which implements a (mutable) genotype-phenotype mapping during reproduction. However, we have consistently observed a particular phenomenon where what we call pathological constructors quickly emerge, which ultimately lead to catastrophic ecosystem collapse. Pathological constructors are creatures which rapidly construct multiple short malfunctioning offspring within their lifetime. Pathological constructors are a hindrance to an ecosystem because their offspring, although sterile, still occupy both memory space and CPU time. When several pathological constructors coincide in time, their production rate can be so high that their non-functional offspring displace the entire population of functional self-reproducing creatures, resulting in ecosystem collapse. We investigate the origin of pathological constructors, and consider how a more mutational robust architecture which is less susceptible to the emergence of these creatures can be created.

Adaptation and Genomic Evolution in EcoSim

Abstract: Artificial life evolutionary systems facilitate addressing lots of fundamental questions in evolutionary genetics. Behavioral adaptation requires long term evolution with continuous emergence of new traits, governed by natural selection. We model organism’s genomes coding for their behavioral model and represented by fuzzy cognitive maps (FCM), in an individual-based evolutionary ecosystem simulation (EcoSim). Our system allows the emergence of new traits and disappearing of others, throughout a course of evolution. In this paper we show how continuous adaptation to a changing environment affects genomic structure and genetic diversity. We adopted the notion of Shannon entropy as a measure of genetic diversity. We emphasized the difference in genetic diversity between EcoSim and its neutral model (a partially randomized version of EcoSim). In addition, we studied the effect that genetic diversity has on species fitness and we showed how they correlate with each other. We used Random Forest to build a classifier to further validate our findings, along with some meaningful rule extraction.

Adaptive Quadruped Locomotion: Learning to Detect and Avoid an Obstacle

Abstract: Autonomy and adaptability are key features in the design and construction of a robotic system capable of carrying out tasks in an unstructured and not predefined environment. Such adaptability is generally observed in animals, biological systems that often serve as inspiration models to the design of robots. The autonomy and adaptability of these systems partially arises from their ability to learn.

Examining the Information Requirements for Flocking Motion

Abstract: Flocking is an archetype emergent behavior that is displayed by a wide variety of groups and has been extensively studied in both biological and robotic communities. Still today, the exact requirements on the detail and type of information required for the production of flocking motion is unclear; moreover, these requirements have large potential impacts on biological plausibility and robotic implementations. This work implements a previously published flocking algorithm (Local Crowed Horizon) on a robotic platform and in computer simulations to explore the effects that the type and detail of information have on the produced motions. Specifically, we investigate the level of detail needed for the observation of flock members and study the differences between the use of pose and bearing information. Surprisingly, the results show that there is no significant difference in the motions produce by any observation detail or type of information. From the results, we introduce and define information-abstracted flocking algorithms, which are structured in such a way that the rule is agnostic to the observation detail and/or type of information given as input. Moreover, we believe our implementation of the Local Crowded Horizon flocking algorithm produces motions that require the least and most simplistic type of information (bearing only) which has been validated on robotic hardware to date.

Towards Detecting Group Identities in Complex Artificial Societies

Abstract: This paper presents a framework for modelling group structures and dynamics in both artificial societies and human-populated virtual environments such as computer games. The group modelling (GM) framework proposed focuses on the detection of existing, pre-defined group structures and is composed of a reinforcement learning method that infers collaboration values from the society’s local interactions and a clustering algorithm that detects group identities based on the learned collaboration values. An empirical evaluation of the framework in the social ultimatum bargain game shows that the GM method proposed is robust independently of the size of the society and the locality of the interactions.

Modelling Walking Behaviors Based on CPGs: A Simplified Bio-inspired Architecture

Abstract: In this article, we use a recurrent neural network including four-cell core architecture to model the walking gait and implement it with the simulated and physical NAO robot. Meanwhile, inspired by the biological CPG models, we propose a simplified CPG model which comprises motorneurons, interneurons, sensor neurons and the simplified spinal cord. Within this model, the CPGs do not directly output trajectories to the servo motors. Instead, they only work to maintain the phase relation among ipsilateral and contralateral limbs. The final output is dependent on the integration of CPG signals, outputs of interneurons, motor neurons and sensor neurons (sensory feedback).

Towards Behavioral Consistency in Neuroevolution

Abstract: To survive in its environment, an animat must have a behavior that is not too disturbed by noise or any other distractor. Its behavior is supposed to be relatively unchanged when tested on similar situations. Evolving controllers that are robust and generalize well over similar contexts remains a challenge for several reasons. One of them comes from the evaluation: how to check a controller for such properties? The fitness may evaluate a distance towards a behavior known to be robust, but such an example is not always available. An alternative is to test the behavior in multiple conditions, actually as many as possible, to avoid overfitting, but this significantly slows down the search process. This issue is expected to become even more critical when evolving behaviors of increasing complexity. To tackle this issue, we propose to formulate it as a problem of behavioral consistency in different contexts. We then propose a fitness objective aimed at explicitly rewarding behavioral consistency. Its principle is to define different sets of contexts and compare the evolved system behavior on each of them. The fitness function thus defined aims at rewarding individuals that exhibit the expected consistency. We apply it to the evolution of two simple computational neuroscience models.

Unsupervised Learning of a Reduced Dimensional Controller for a Tendon Driven Robot Platform

Abstract: In this paper we present a developmental framework to carry out goal-oriented learning in a low-dimensional space. The framework uses two stages of learning: one to synthesise a set of motor synergies and reduce the dimensionality of the control space in an unsupervised manner, and another to carry out supervised learning in the reduced control space. We test our framework in a reaching task carried out on a (real) tendon-driven robot actuated by four artificial muscles. Our results show that the robot is capable of learning to reach using a reduced control space using no prior information about its body apart from that inherent to the unsupervised and supervised learning rules.

Which Temporal Difference Learning Algorithm Best Reproduces Dopamine Activity in a Multi-choice Task?

Abstract: The activity of dopaminergic (DA) neurons has been hypothesized to encode a reward prediction error (RPE) which corresponds to the error signal in Temporal Difference (TD) learning algorithms. This hypothesis has been reinforced by numerous studies showing the relevance of TD learning algorithms to describe the role of basal ganglia in classical conditioning. However, recent recordings of DA neurons during multi-choice tasks raised contradictory interpretations on whether DA’s RPE signal is action dependent or not. Thus the precise TD algorithm (i.e. Actor-Critic, Q-learning or SARSA) that best describes DA signals remains unknown. Here we simulate and precisely analyze these TD algorithms on a multi-choice task performed by rats. We find that DA activity previously reported in this task is best fitted by a TD error which has not fully converged, and which converged faster than observed behavioral adaptation.

Synthesising a motor-primitive inspired control architecture for redundant compliant robots

Abstract: This paper presents a control architecture for redundant and compliant robots inspired by the theory of biological motor primitives which are theorised to be the mechanism employed by the central nervous system in tackling the problem of redundancy in motor control. In our framework, inspired by self-organisational principles, the simulated robot is first perturbed by a form of spontaneous motor activity and the resulting state trajectory is utilised to reduce the control dimensionality using proper orthogonal decomposition. Motor primitives are then computed using a method based on singular value decomposition. Controllers for generating reduced dimensional commands to reach desired equilibrium positions in Cartesian space are then presented. The proposed architecture is successfully tested on a simulation of a compliant redundant robotic pendulum platform that uses antagonistically arranged series-elastic actuation.

Gaze Allocation Analysis for a Visually Guided Manipulation Task

Abstract: Findings from eye movement research in humans have demonstrated that the task determines where to look. One hypothesis is that the purpose of looking is to reduce uncertainty about properties relevant to the task. Following this hypothesis, we define a model that poses the problem of where to look as one of maximising task performance by reducing task relevant uncertainty. We implement and test our model on a simulated humanoid robot which has to move objects from a table into containers. Our model outperforms and is more robust than two other baseline schemes in terms of task performance whilst varying three environmental conditions, reach/grasp sensitivity, observation noise and the camera’s field of view.

LocoKit: A Robot Construction Kit for Studying and Developing Functional Morphologies

Abstract: We describe a robot construction kit named LocoKit and a method for studying functional morphologies. LocoKit consists of simple mechanical parts that allow for construction of a wide range of morphologies and modular electronics for instrumentation and control. The method relies on LocoKit for constructing functional morphologies and an experimental setup borrowed from the study of functional morphology in animals. We demonstrate the use of LocoKit and the method in a case study on quadruped locomotion and conclude that the methodology represents a systematic and efficient approach to the study and development of functional robot morphologies.

On the Evolution of Homogeneous Multi-robot Teams: Clonal versus Aclonal Approach

Abstract: This study compares two different evolutionary approaches to the design of homogeneous multi-robot teams in a task that requires the agents to specialise in different roles. Our results diverge from what illustrated in a previous similar comparative study, which advocates for the superiority of the aclonal versus the clonal approach. We question this argument in view of new empirical evidence showing that the two approaches perform equally well in generating homogeneous teams.

A Computational Model of the Role of Serotonin in Reversal Learning

Abstract: It has been shown that the action of serotonin on the orbito-frontal cortex (OFC) is crucial for the inhibition phase of reversal learning. Serotonin has also been shown to facilitate the induction of LTD throughout the prefrontal cortex. We present a biologically realistic, systems level model which proposes a mechanism for the release of serotonin in response to the omission of an expected reward. Serotonin release, as a result of the combination of excitation of the dorsal raphe nucleus (DRN) pathway and the lack of inhibition of the DRN from the lateral habenula, leads to LTD in the OFC and suppression of excitation of the nucleus accumbens shell due to reward predicting sensory stimuli. Behavioural inhibition is controlled via the shell-ventral pallido-mediodorsal pathway, which serves as a feed forward switching mechanism and enables the behavioural inhibition required to achieve reversal learning.

Self-organization of Spinal Reflexes Involving Homonymous, Antagonist and Synergistic Interactions

Abstract: Recent results in spinal research are challenging the historical view that the spinal reflexes are mostly hardwired and fixed behaviours. In previous work we have shown that three of the simplest spinal reflexes could be self-organised in an agonist-antagonist pair of muscles. The simplicity of these reflexes is given from the fact that they entail at most one interneuron mediating the connectivity between afferent inputs and efferent outputs. These reflexes are: the Myotatic, the Reciprocal Inibition and the Reverse Myotatic reflexes. In this paper we apply our framework to a simulated 2D leg model actuated by six muscles (mono- and bi-articular). Our results show that the framework is successful in learning most of the spinal reflex circuitry as well as the corresponding behaviour in the more complicated muscle arrangement.