Showing papers on "Humanoid robot published in 2016"

Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot

Abstract: This paper describes a collection of optimization algorithms for achieving dynamic planning, control, and state estimation for a bipedal robot designed to operate reliably in complex environments. To make challenging locomotion tasks tractable, we describe several novel applications of convex, mixed-integer, and sparse nonlinear optimization to problems ranging from footstep placement to whole-body planning and control. We also present a state estimator formulation that, when combined with our walking controller, permits highly precise execution of extended walking plans over non-flat terrain. We describe our complete system integration and experiments carried out on Atlas, a full-size hydraulic humanoid robot built by Boston Dynamics, Inc.

Safety in Human-Robot Collaborative Manufacturing Environments: Metrics and Control

Abstract: New paradigms in industrial robotics no longer require physical separation between robotic manipulators and humans. Moreover, in order to optimize production, humans and robots are expected to collaborate to some extent. In this scenario, involving a shared environment between humans and robots, common motion generation algorithms might turn out to be inadequate for this purpose.

Design of a Momentum-Based Control Framework and Application to the Humanoid Robot Atlas

Abstract: This paper presents a momentum-based control framework for floating-base robots and its application to the humanoid robot “Atlas”. At the heart of the control framework lies a quadratic program that reconciles motion tasks expressed as constraints on the joint acceleration vector with the limitations due to unilateral ground contact and force-limited grasping. We elaborate on necessary adaptations required to move from simulation to real hardware and present results for walking across rough terrain, basic manipulation, and multi-contact balancing on sloped surfaces (the latter in simulation only). The presented control framework was used to secure second place in both the DARPA Robotics Challenge Trials in December 2013 and the Finals in June 2015.

3D dynamic walking with underactuated humanoid robots: A direct collocation framework for optimizing hybrid zero dynamics

Abstract: Hybrid zero dynamics (HZD) has emerged as a popular framework for dynamic and underactuated bipedal walking, but has significant implementation difficulties when applied to the high degrees of freedom present in humanoid robots. The primary impediment is the process of gait design-it is difficult for optimizers to converge on a viable set of virtual constraints defining a gait. This paper presents a methodology that allows for the fast and reliable generation of efficient multi-contact robotic walking gaits through the framework of HZD, even in the presence of underactuation. To achieve this goal, we unify methods from trajectory optimization with the control framework of multi-domain hybrid zero dynamics. By formulating a novel optimization problem in the context of direct collocation and generating analytic Jacobians for the constraints, solving the resulting nonlinear program becomes tractable for large-scale nonlinear programming solvers, even for systems as high-dimensional as humanoid robots. We experimentally validated our methodology on the spring-legged prototype humanoid, DURUS, showing that the optimization approach yields dynamic and stable 3D walking gaits.

Modeling and Control of Legged Robots

Abstract: The promise of legged robots over wheeled robots is to provide improved mobility over rough terrain. Unfortunately, this promise comes at the cost of a significant increase in complexity. We now have a good understanding of how to make legged robots walk and run dynamically, but further research is still necessary to make them walk and run efficiently in terms of energy, speed, reactivity, versatility, and robustness. In this chapter, we will discuss how legged robots are usually modeled, how their stability analysis is approached, how dynamic motions are generated and controlled, and finally summarize the current trends in trying to improve their performance. The main problem is avoiding to fall. This can prove difficult since legged robots have to rely entirely on available contact forces to do so. The temporality of leg motions appears to be a key aspect in this respect, as current control solutions include continuous anticipation of future motion (using some form of model predictive control), or focusing more specifically on limit cycles and orbital stability.

Detecting object affordances with Convolutional Neural Networks

Abstract: We present a novel and real-time method to detect object affordances from RGB-D images. Our method trains a deep Convolutional Neural Network (CNN) to learn deep features from the input data in an end-to-end manner. The CNN has an encoder-decoder architecture in order to obtain smooth label predictions. The input data are represented as multiple modalities to let the network learn the features more effectively. Our method sets a new benchmark on detecting object affordances, improving the accuracy by 20% in comparison with the state-of-the-art methods that use hand-designed geometric features. Furthermore, we apply our detection method on a full-size humanoid robot (WALK-MAN) to demonstrate that the robot is able to perform grasps after efficiently detecting the object affordances.

Passivity-based whole-body balancing for torque-controlled humanoid robots in multi-contact scenarios

Abstract: This work presents a new control approach to multi-contact balancing for torque-controlled humanoid robots. The controller includes a non-strict task hierarchy, which allows the robot to use a subset of its end effectors for balancing while the remaining ones can be used for interacting with the environment. The controller creates a passive and compliant behavior for regulating the center of mass (CoM) location, hip orientation and the poses of each end effector assigned to the interaction task. This is achieved by applying a suitable wrench (force and torque) at each one of the end effectors used for interaction. The contact wrenches at the balancing end effectors are chosen such that the sum of the balancing and interaction wrenches produce the desired wrench at the CoM. The algorithm requires the solution of an optimization problem, which distributes the CoM wrench to the end effectors taking into account constraints for unilaterality, friction and position of the center of pressure. Furthermore, the f...

A versatile and efficient pattern generator for generalized legged locomotion

Abstract: This paper presents a generic and efficient approach to generate dynamically consistent motions for under-actuated systems like humanoid or quadruped robots. The main contribution is a walking pattern generator, able to compute a stable trajectory of the center of mass of the robot along with the angular momentum, for any given configuration of contacts (e.g. on uneven, sloppy or slippery terrain, or with closed-gripper). Unlike existing methods, our solver is fast enough to be applied as a model-predictive controller. We then integrate this pattern generator in a complete framework: an acyclic contact planner is first used to automatically compute the contact sequence from a 3D model of the environment and a desired final posture; a stable walking pattern is then computed by the proposed solver; a dynamically-stable whole-body trajectory is finally obtained using a second-order hierarchical inverse kinematics. The implementation of the whole pipeline is fast enough to plan a step while the previous one is executed. The interest of the method is demonstrated by real experiments on the HRP-2 robot, by performing long-step walking and climbing a staircase with handrail support.

Perception-less terrain adaptation through whole body control and hierarchical optimization

Abstract: This paper presents a control approach based on a whole body control framework combined with hierarchical optimization. Locomotion is formulated as multiple tasks (e.g. maintaining balance or tracking a desired motion of one of the limbs) which are solved in a prioritized way using QP solvers. It is shown how complex locomotion behaviors can purely emerge from robot-specific inequality tasks (i.e. torque or reaching limits) together with the optimization of balance and system manipulability. Without any specific motion planning, this prioritized task optimization leads to a natural adaption of the robot to the terrain while walking and hence enables blind locomotion over rough grounds. The presented framework is implemented and successfully tested on ANYmal, a torque controllable quadrupedal robot. It enables the machine to walk while accounting for slippage and torque limitation constraints, and even step down from an unperceived 14 cm obstacle. Thereby, ANYmal exploits the maximum reach of the limbs and automatically adapts the body posture and height.

3D dynamic walking on stepping stones with control barrier functions

Abstract: 3D dynamical walking subject to precise footstep placements is crucial for navigating real world terrain with discrete footholds. We present a novel methodology that combines control Lyapunov functions—to achieve periodic walking—and control Barrier functions—to enforce strict constraints on step length and step width—unified in a single optimization-based controller. We numerically validate our proposed method by demonstrating dynamic 3D walking at 0.6 m/s on DURUS, a 23 degree-of-freedom underactuated humanoid robot.

Realizing dynamic and efficient bipedal locomotion on the humanoid robot DURUS

Abstract: This paper presents the methodology used to achieve efficient and dynamic walking behaviors on the prototype humanoid robotics platform, DURUS. As a means of providing a hardware platform capable of these behaviors, the design of DURUS combines highly efficient electromechanical components with “control in the loop” design of the leg morphology. Utilizing the final design of DURUS, a formal framework for the generation of dynamic walking gaits which maximizes efficiency by exploiting the full body dynamics of the robot, including the interplay between the passive and active elements, is developed. The gaits generated through this methodology form the basis of the control implementation experimentally realized on DURUS; in particular, the trajectories generated through the formal framework yield a feedforward control input which is modulated by feedback in the form of regulators that compensate for discrepancies between the model and physical system. The end result of the unified approach to control-informed mechanical design, formal gait design and regulator-based feedback control implementation is efficient and dynamic locomotion on the humanoid robot DURUS. In particular, DURUS was able to demonstrate dynamic locomotion at the DRC Finals Endurance Test, walking for just under five hours in a single day, traveling 3.9 km with a mean cost of transport of 1.61—the lowest reported cost of transport achieved on a bipedal humanoid robot.

Human-like compliant locomotion: state of the art of robotic implementations.

Abstract: This review paper provides a synthetic yet critical overview of the key biomechanical principles of human bipedal walking and their current implementation in robotic platforms. We describe the functional role of human joints, addressing in particular the relevance of the compliant properties of the different degrees of freedom throughout the gait cycle. We focused on three basic functional units involved in locomotion, i.e. the ankle-foot complex, the knee, and the hip-pelvis complex, and their relevance to whole-body performance. We present an extensive review of the current implementations of these mechanisms into robotic platforms, discussing their potentialities and limitations from the functional and energetic perspectives. We specifically targeted humanoid robots, but also revised evidence from the field of lower-limb prosthetics, which presents innovative solutions still unexploited in the current humanoids. Finally, we identified the main critical aspects of the process of translating human principles into actual machines, providing a number of relevant challenges that should be addressed in future research.

Clinical Application of a Humanoid Robot in Pediatric Cancer Interventions

Abstract: This paper propounds a novel approach by exploring the effect of utilizing a social humanoid robot as a therapy-assistive tool in dealing with pediatric distress. The study aims to create a friendship bond between a humanoid robot and young oncology patients to alleviate their pain and distress. Eleven children, ages 7–12, diagnosed with cancer were randomly assigned into two groups: a social robot-assisted therapy (SRAT) group with 6 kids and a psychotherapy group with five kids at two specialized hospitals in Tehran. A NAO robot was programmed and employed as a robotic assistant to a psychologist in the SRAT group to perform various scenarios in eight intervention sessions. These sessions were aimed at instructing the children about their affliction and its symptoms, sympathizing with them, and providing a space for them to express their fears and worries. The same treatment was conducted by the psychologist alone on the control group. The children’s anxiety, anger, and depression were measured with three standard questionnaires obtained from the literature before and after the treatment (March et al., in J Am Acad Child Adolesc Psychiatry 36:554–565, 1997; Nelson and Finch, in Children’s inventory of anger, 2000; Kovacs, in Psychopharmacol Bull 21:995–1124, 1985). The results of descriptive statistics and MANOVA indicated that the children’s stress, depression, and anger were considerably alleviated during SRAT treatment and significant differences were observed between the two groups. Considering the positive reactions from the children to the robot assistant’s presence at the intervention sessions, and observing the numerical results, one can anticipate that utilizing a humanoid robot with different communication abilities can be beneficial, both in elevation of efficacy in interventions, and fomenting kids to be more interactive and cooperative in their treatment sessions. In addition, employing the humanoid robot was significantly useful in teaching children about their affliction and instructing them in techniques such as: relaxation or desensitization in order to help them confront and manage their distress themselves and take control of their situation.

A convex model of humanoid momentum dynamics for multi-contact motion generation

Abstract: Linear models for control and motion generation of humanoid robots have received significant attention in the past years, not only due to their well known theoretical guarantees, but also because of practical computational advantages. However, to tackle more challenging tasks and scenarios such as locomotion on uneven terrain, a more expressive model is required. In this paper, we are interested in contact interaction-centered motion optimization based on the momentum dynamics model. This model is non-linear and non-convex; however, we find a relaxation of the problem that allows us to formulate it as a single convex quadratically-constrained quadratic program (QCQP) that can be very efficiently optimized and is useful for multi-contact planning. This convex model is then coupled to the optimization of end-effector contact locations using a mixed integer program, which can also be efficiently solved. This becomes relevant e.g. to recover from external pushes, where a predefined stepping plan is likely to fail and an online adaptation of the contact location is needed. The performance of our algorithm is demonstrated in several multi-contact scenarios for a humanoid robot.

Stability analysis and design of momentum-based controllers for humanoid robots

Abstract: Envisioned applications for humanoid robots call for the design of balancing and walking controllers. While promising results have been recently achieved, robust and reliable controllers are still a challenge for the control community dealing with humanoid robotics. Momentum-based strategies have proven their effectiveness for controlling humanoids balancing, but the stability analysis of these controllers is still missing. The contribution of this paper is twofold. First, we numerically show that the application of state-of-the-art momentum-based control strategies may lead to unstable zero dynamics. Secondly, we propose simple modifications to the control architecture that avoid instabilities at the zero-dynamics level. Asymptotic stability of the closed loop system is shown by means of a Lyapunov analysis on the linearized system's joint space. The theoretical results are validated with both simulations and experiments on the iCub humanoid robot.

Multi-contact vertical ladder climbing with an HRP-2 humanoid

Abstract: We describe the research and the integration methods we developed to make the HRP-2 humanoid robot climb vertical industrial-norm ladders. We use our multi-contact planner and multi-objective closed-loop control formulated as a QP (quadratic program). First, a set of contacts to climb the ladder is planned off-line (automatically or by the user). These contacts are provided as an input for a finite state machine. The latter builds supplementary tasks that account for geometric uncertainties and specific grasps procedures to be added to the QP controller. The latter provides instant desired states in terms of joint accelerations and contact forces to be tracked by the embedded low-level motor controllers. Our trials revealed that hardware changes are necessary, and parts of software must be made more robust. Yet, we confirmed that HRP-2 has the kinematic and power capabilities to climb real industrial ladders, such as those found in nuclear power plants and large scale manufacturing factories (e.g. aircraft, shipyard) and construction sites.

Believing in BERT: Using expressive communication to enhance trust and counteract operational error in physical Human-robot interaction

Abstract: Strategies are necessary to mitigate the impact of unexpected behavior in collaborative robotics, and research to develop solutions is lacking. Our aim here was to explore the benefits of an affective interaction, as opposed to a more efficient, less error prone but non-communicative one. The experiment took the form of an omelet-making task, with a wide range of participants interacting directly with BERT2, a humanoid robot assistant. Having significant implications for design, results suggest that efficiency is not the most important aspect of performance for users; a personable, expressive robot was found to be preferable over a more efficient one, despite a considerable trade off in time taken to perform the task. Our findings also suggest that a robot exhibiting human-like characteristics may make users reluctant to ‘hurt its feelings’; they may even lie in order to avoid this.

Unifying Representations and Large-Scale Whole-Body Motion Databases for Studying Human Motion

Abstract: Large-scale human motion databases are key for research questions ranging from human motion analysis and synthesis, biomechanics of human motion, data-driven learning of motion primitives, and rehabilitation robotics to the design of humanoid robots and wearable robots such as exoskeletons. In this paper we present a large-scale database of whole-body human motion with methods and tools, which allows a unifying representation of captured human motion, and efficient search in the database, as well as the transfer of subject-specific motions to robots with different embodiments. To this end, captured subject-specific motion is normalized regarding the subject's height and weight by using a reference kinematics and dynamics model of the human body, the master motor map (MMM). In contrast with previous approaches and human motion databases, the motion data in our database consider not only the motions of the human subject but the position and motion of objects with which the subject is interacting as well. In addition to the description of the MMM reference model, we present procedures and techniques for the systematic recording, labeling, and organization of human motion capture data, object motions as well as the subject–object relations. To allow efficient search for certain motion types in the database, motion recordings are manually annotated with motion description tags organized in a tree structure. We demonstrate the transfer of human motion to humanoid robots and provide several examples of motion analysis using the database.

Iterative path optimisation for personalised dressing assistance using vision and force information

Abstract: We propose an online iterative path optimisation method to enable a Baxter humanoid robot to assist human users to dress. The robot searches for the optimal personalised dressing path using vision and force sensor information: vision information is used to recognise the human pose and model the movement space of upper-body joints; force sensor information is used for the robot to detect external force resistance and to locally adjust its motion. We propose a new stochastic path optimisation method based on adaptive moment estimation. We first compare the proposed method with other path optimisation algorithms on synthetic data. Experimental results show that the performance of the method achieves the smallest error with fewer iterations and less computation time. We also evaluate real-world data by enabling the Baxter robot to assist real human users with their dressing.

The interactive effects of robot anthropomorphism and robot ability on perceived threat and support for robotics research

Abstract: The present research examines how a robot's physical anthropomorphism interacts with perceived ability of robots to impact the level of realistic and identity threat that people perceive from robots and how it affects their support for robotics research. Experimental data revealed that participants perceived robots to be significantly more threatening to humans after watching a video of an android that could allegedly outperform humans on various physical and mental tasks relative to a humanoid robot that could do the same. However, when participants were not provided with information about a new generation of robots' ability relative to humans, then no significant differences were found in perceived threat following exposure to either the android or humanoid robots. Similarly, participants also expressed less support for robotics research after seeing an android relative to a humanoid robot outperform humans. However, when provided with no information about robots' ability relative to humans, then participants showed marginally decreased support for robotics research following exposure to the humanoid relative to the android robot. Taken together, these findings suggest that very humanlike robots can not only be perceived as a realistic threat to human jobs, safety, and resources, but can also be seen as a threat to human identity and uniqueness, especially if such robots also outperform humans. We also demonstrate the potential downside of such robots to the public's willingness to support and fund robotics research.

Construction Robots: Elementary Technologies and Single-Task Construction Robots

Abstract: Learn how Single-Task Construction Robots (STCRs) can improve productivity in the construction industry with this cross-disciplinary text. This third volume in The Cambridge Handbooks in Construction Robotics series discusses the STCRs employed on construction sites since the development of the approach in the 1980s, presents current applications, and highlights upcoming trends in the construction automation and robotics field. Two hundred different types of STCR are presented, from the simplest models comprising simple manipulators and mobile platforms, to those utilizing more sophisticated technologies such as aerial robotics, swarm robotics, exoskeletons, additive manufacturing technologies, self-assembling building structures, and humanoid robotics. Real-world case studies demonstrate the different application scenarios for each approach, and highlight the key implementation and management issues. With an easy-to-follow structure, and including hundreds of color illustrations, it provides an excellent toolkit for professional engineers, researchers, and students.

Structured contact force optimization for kino-dynamic motion generation

Abstract: Optimal control approaches in combination with trajectory optimization have recently proven to be a promising control strategy for legged robots. Computationally efficient and robust algorithms were derived using simplified models of the contact interaction between robot and environment such as the linear inverted pendulum model (LIPM). However, as humanoid robots enter more complex environments, less restrictive models become increasingly important. As we leave the regime of linear models, we need to build dedicated solvers that can compute interaction forces together with consistent kinematic plans for the whole-body. In this paper, we address the problem of planning robot motion and interaction forces for legged robots given predefined contact surfaces. The motion generation process is decomposed into two alternating parts computing force and motion plans in coherence. We focus on the properties of the momentum computation leading to sparse optimal control formulations to be exploited by a dedicated solver. In our experiments, we demonstrate that our motion generation algorithm computes consistent contact forces and joint trajectories for our humanoid robot. We also demonstrate the favorable time complexity due to our formulation and composition of the momentum equations.

Why That Nao?: How Humans Adapt to a Conventional Humanoid Robot in Taking Turns-at-Talk

Abstract: This paper explores how humans adapt to a conventional humanoid robot. Video data of participants playing a charade game with a Nao robot were analyzed from a multimodal conversation analysis perspective. Participants soon adjust aspects of turn-design such as word selection, turn length and prosody, thereby adapting to the robot's limited perceptive abilities as they become apparent in the interaction. However, coordination of turns-at-talk remains troublesome throughout the encounter, as evidenced by overlapping turns and lengthy silences around possible turn endings. The study discusses how the robot design can be improved to support the problematic taking of turns-at-talk with humans. Two programming strategies to address the identified problems are presented: 1. to program the robot so that it will be systematically receptive at the equivalence to transition relevance places in human-human interaction, and 2. to make the robot preferably produce verbal actions that require a response in a conditional way, rather than making a response only possible.

Robustness to Joint-Torque-Tracking Errors in Task-Space Inverse Dynamics

Abstract: Task-space inverse dynamics (TSID) is a well-known optimization-based technique for the control of highly redundant mechanical systems, such as humanoid robots. One of its main flaws is that it does not take into account any of the uncertainties affecting these systems: poor torque tracking, sensor noises, delays, and model uncertainties. As a consequence, the resulting control-state trajectories may be feasible for the ideal system, but not for the real one. We propose to improve the robustness of TSID by modeling uncertainties in the joint torques, either as Gaussian random variables or as bounded deterministic variables. Then we try to immunize the constraints of the system to any—or at least most—of the realizations of these uncertainties. When the resulting optimization problem is computationally too expensive for online control, we propose ways to approximate it that lead to computation times below 1 ms. Extensive simulations in a realistic environment show that the proposed robust controllers greatly outperform the classic one, even when other unmodeled uncertainties affect the system (e.g., errors in the inertial parameters, delays in the velocity estimates).

Walking on partial footholds including line contacts with the humanoid robot atlas

Abstract: We present a method for humanoid robot walking on partial footholds such as small stepping stones and rocks with sharp surfaces. Our algorithm does not rely on prior knowledge of the foothold, but information about an expected foothold can be used to improve the stepping performance. After a step is taken, the robot explores the new contact surface by attempting to shift the center of pressure around the foot. The available foothold is inferred by the way in which the foot rotates about contact edges and/or by the achieved center of pressure locations on the foot during exploration. This estimated contact area is then used by a whole body momentum-based control algorithm. To walk and balance on partial footholds, we combine fast, dynamic stepping with the use of upper body angular momentum to regain balance. We applied this method to the Atlas humanoid designed by Boston Dynamics to walk over small contact surfaces, such as line and point contacts. We present experimental results and discuss performance limitations.

Whole-body impedance control of wheeled mobile manipulators

Abstract: Humanoid service robots in domestic environments have to interact with humans and their surroundings in a safe and reliable way One way to manage that is to equip the robotic systems with force-torque sensors to realize a physically compliant whole-body behavior via impedance control To provide mobility, such robots often have wheeled platforms The main advantage is that no balancing effort has to be made compared to legged humanoids However, the nonholonomy of most wheeled systems prohibits the direct implementation of impedance control due to kinematic rolling constraints that must be taken into account in modeling and control In this paper we design a whole-body impedance controller for such a robot, which employs an admittance interface to the kinematically controlled mobile platform The upper body impedance control law, the platform admittance interface, and the compensation of dynamic couplings between both subsystems yield a passive closed loop The convergence of the state to an invariant set is shown To prove asymptotic stability in the case of redundancy, priority-based approaches can be employed In principle, the presented approach is the extension of the well-known and established impedance controller to mobile robots Experimental validations are performed on the humanoid robot Rollin' Justin The method is suitable for compliant manipulation tasks with low-dimensional planning in the task space

Human Expectations of Social Robots

Abstract: A key assumption that drives much of HRI research is that human robot collaboration can be improved by advancing a robot's capabilities. We argue that this assumption has potentially negative implications, as increasing social capabilities in robots can produce an expectations gap where humans develop unrealistically high expectations of social robots due to generalization from human mental models. By conducting two studies with 674 participants, we examine how people develop and adjust mental models of robots. We find that both a robot's physical appearance and its behavior influence how we form these models. This suggests it is possible for a robot to unintentionally manipulate a human into building an inaccurate mental model of its overall abilities simply by displaying a few capabilities that humans possess, such as speaking and turn-taking. We conclude that this expectations gap, if not corrected for, could ironically result in less effective collaborations as robot capabilities improve.

Humanoid and Human Inertia Parameter Identification Using Hierarchical Optimization

Abstract: We propose a method for estimation of humanoid and human links’ inertial parameters. Our approach formulates the problem as a hierarchical quadratic program by exploiting the linear properties of rigid body dynamics with respect to the inertia parameters. In order to assess our algorithm, we conducted experiments with a humanoid robot and a human subject. We compared ground reaction forces and moments estimated from force measurements with those computed using identified inertia parameters and movement information. Our method is able to accurately reconstruct ground reaction forces and force moments. Moreover, our method is able to estimate correctly masses of the robots links and to accurately detect additional masses placed on the human subject during the experiments.

The MuMMER project: Engaging human-robot interaction in real-world public spaces

Abstract: MuMMER (MultiModal Mall Entertainment Robot) is a four-year, EU-funded project with the overall goal of developing a humanoid robot (SoftBank Robotics’ Pepper robot being the primary robot platform) with the social intelligence to interact autonomously and naturally in the dynamic environments of a public shopping mall, providing an engaging and entertaining experience to the general public. Using co-design methods, we will work together with stakeholders including customers, retailers, and business managers to develop truly engaging robot behaviours. Crucially, our robot will exhibit behaviour that is socially appropriate and engaging by combining speech-based interaction with non-verbal communication and human-aware navigation. To support this behaviour, we will develop and integrate new methods from audiovisual scene processing, social-signal processing, high-level action selection, and human-aware robot navigation. Throughout the project, the robot will be regularly deployed in Ideapark, a large public shopping mall in Finland. This position paper describes the MuMMER project: its needs, the objectives, R&D challenges and our approach. It will serve as reference for the robotics community and stakeholders about this ambitious project, demonstrating how a co-design approach can address some of the barriers and help in building follow-up projects.