![Fig. 5: An example of providing haptic and visual feedback (via AR) to the human counterpart in collaborative settings [Glassmire et al., 2004] (image courtesy of M. O’Malley).](/figures/fig-5-an-example-of-providing-haptic-and-visual-feedback-via-18cp02xt.webp)
![Fig. 4: In [Ivaldi et al., 2016] naive participants (not expert in robotics) interacted with the humanoid iCub to build an object: the physical interaction was at the level of the arms, covered by a soft tactile skin (image courtesy of S. Ivaldi).](/figures/fig-4-in-ivaldi-et-al-2016-naive-participants-not-expert-in-246d32ea.webp)
![Fig. 3: Authors in [Peternel et al., 2016b] proposed a human-robot comanipulation framework for robot adaptation to human fatigue. The myoelectric interface provides the robot controller with a feedback about human motor behaviour to achieve an appropriate impedance profile in different phases of the task. The human fatigue estimation system provides the robot with the state of the human physical endurance (image courtesy of L. Peternel).](/figures/fig-3-authors-in-peternel-et-al-2016b-proposed-a-human-robot-1owgurxh.webp)
HAL Id: hal-01643655
https://hal.archives-ouvertes.fr/hal-01643655
Submitted on 21 Nov 2017
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Progress and Prospects of the Human-Robot
Collaboration
Arash Ajoudani, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäer,
Kazuhiro Kosuge, Oussama Khatib
To cite this version:
Arash Ajoudani, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäer, Kazuhiro Kosuge, et al..
Progress and Prospects of the Human-Robot Collaboration. Autonomous Robots, Springer Verlag,
2017, pp.1-17. �10.1007/s10514-017-9677-2�. �hal-01643655�
Noname manuscript No.
(will be inserted by the editor)
Progress and Prospects of the
Human-Robot Collaboration
Arash Ajoudani, Andrea
Maria Zanchettin, Serena
Ivaldi, Alin Albu-Sch
¨
affer,
Kazuhiro Kosuge, and
Oussama Khatib
Received: date / Accepted: date
Abstract Recent technological advances in hardware de-
sign of the robotic platforms enabled the implementation
of various control modalities for improved interactions with
humans and unstructured environments. An important appli-
cation area for the integration of robots with such advanced
interaction capabilities is human-robot collaboration. This
aspect represents high socio-economic impacts and main-
tains the sense of purpose of the involved people, as the ro-
bots do not completely replace the humans from the work
process. The research community’s recent surge of interest
in this area has been devoted to the implementation of vari-
ous methodologies to achieve intuitive and seamless human-
robot-environment interactions by incorporating the collab-
orative partners’ superior capabilities, e.g. human’s cogni-
tive and robot’s physical power generation capacity. In fact,
the main purpose of this paper is to review the state-of-the-
art on intermediate human-robot interfaces (bi-directional),
robot control modalities, system stability, benchmarking and
relevant use cases, and to extend views on the required future
developments in the realm of human-robot collaboration.
Arash Ajoudani is with HRI
2
Lab of the Istituto Italiano di Tecnologia,
Genoa, Italy, Email: arash.ajoudani@iit.it
Andrea Maria Zanchettin is with Politecnico di Milano, Dipartimento
di Elettronica, Informazione e Bioingegneria, Milano, Italy, Email: an-
dreamaria.zanchettin@polimi.it
Serena Ivaldi is with INRIA Nancy Grand-Est, France, and the Intel-
ligent Autonomous Systems Lab of TU Darmstadt, Germany. Email:
serena.ivaldi@inria.fr
Alin Albu-Sch
¨
affer is with the Institute of Robotics and Mechatron-
ics, German Aerospace Center (DLR), Germany, Email: Alin.Albu-
Schaeffer@dlr.de
Kazuhiro Kosuge is with the System Robotics Laboratory, Tohoku Uni-
versity, Japan, Email: kosuge@m.tohoku.ac.jp
Oussama Khatib is with the Stanford Robotics Laboratory, Stanford
University, USA, Email: ok@robo.stanford.edu
0
100
200
300
400
500
600
700
800
Time [Year]
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
Number of published papers
Human Robot "Collaboration OR Cooperation"
"User OR Human" Intention
Fig. 1: Number of publications on the topic of human-robot collabo-
ration from 1996 to 2015. The contribution of the research keyword
“human intention” (red) to the numbers is illustrated in this plot. The
data is extracted from Google Scholar.
1 Introduction
The fast growing demands for service robot applications in
home or industrial workspaces have led to the development
of several well-performing robots equipped with rich pro-
prioception sensing and actuation control. Such systems that
range from robotic manipulators [Albu-Sch
¨
affer et al., 2007]
to full humanoids [Tsagarakis et al., 2016, Ott et al., 2006,
Kaneko et al., 2008, Radford et al., 2015] are expected to
help the human user in various tasks, some of which require
collaborative effort for a safe
1
, successful, and time and en-
ergy efficient execution. In fact, the integration of robotic
systems in collaborative scenarios has seen an extensive and
fast-growing research effort, a tentative estimation of which
is provided in Fig.1 by referring to the number of publica-
tions on this topic over the last two decades.
Physical human-robot collaboration (PHRC), which falls
within the general scope of physical human-robot interac-
tion (see [De Santis et al., 2008, Murphy, 2004,Alami et al.,
2006]), is defined when human(s), robot(s) and the environ-
ment come to contact with each other and form a tightly cou-
pled dynamical system to accomplish a task [Bauer et al.,
2008, Kr
¨
uger et al., 2009]. Ideally, each active component
of such a system must be capable of observing and estimat-
ing the counterparts’ contributions to the overall system’s
response through the fusion and processing of the sensory
information [Argall and Billard, 2010, Ebert and Henrich,
2002, Lall
´
ee et al., 2012]. As a consequence, an appropriate
reactive behaviour can be replicated (e.g. by the human from
a set of obtained skills in previous attempts of performing a
similar task) or developed to complement and improve the
performance of the collaborative partners.
1
The problem of safety in human-robot interaction (HRI) and the
related open issues have been extensively discussed in literature [Had-
dadin et al., 2009, De Santis et al., 2008, Alami et al., 2006]. Hence,
our focus in this review paper will be on other important aspects of
physical human-robot collaboration (PHRC).
2 Ajoudani et al.,
Similar to the humans’ anticipatory (feed-forward [Shad-
mehr and Mussa-Ivaldi, 1994b]) or feed-back [Todorov and
Jordan, 2002] mechanisms to develop a suitable motor be-
haviour, the collaborative robots’ response to sensory in-
put can be achieved through model/knowledge based tech-
niques [Tamei and Shibata, 2011,Ogata et al., 2003,Kimura
et al., 1999,Magnanimo et al., 2014], the implementation of
feedback controllers with pre-set interaction modalities [Pe-
ternel et al., 2016c, Donner and Buss, 2016a] or a combined
approach [Rozo et al., 2013,Peternel et al., 2016b,Lawitzky
et al., 2012b, Palunko et al., 2014]. A key strategy in this
direction is the establishment of a shared authority frame-
work in which the significant capabilities of both humans
and robots can be exploited. For instance, humans’ signifi-
cant cognitive abilities in learning and adaptation to various
tasks demands and disturbances can be used to supervise the
collaborative robots’ superior physical capabilities. The in-
creasing slope in the research community’s interest towards
the integration of human intention in real-time adaptation of
the robot behaviour is illustrated in Fig.1.
With this in mind, the main purpose of this paper is to re-
view the state-of-the-art on the key enabling technologies to
achieve a seamless and intuitive human-robot collaboration.
Although hardware-related components are among the most
critical to achieve this goal, a remarkable effort has been re-
cently devoted to overview the underlying progress in terms
of communication [Wang et al., 2005], sensing [Tegin and
Wikander, 2005], and actuation [Vanderborght et al., 2013a]
developments. Hence, our focus in this review will be on
other relevant key elements of the HRC, i.e. human-robot
interfaces (bi-directional), robot control modalities, system
stability, benchmarking and relevant use cases. In addition,
with the aim to achieve a reasonable convergence of views
for the required future developments, our focus will be mostly
on the physical aspects
2
of the human-robot collaboration.
2 Interfaces for Improved Robot Perception
Humans embrace a diversity of experiences from working
together in pairs or groups. This has contributed to the devel-
opment of implicit and explicit communication standards so
that task-related information can be perceived and commu-
nicated intuitively [Sebanz et al., 2006]. In fact, one of the
main objectives in the realm of physical human-robot col-
laboration is to design and establish similar communication
standards so that the robot is aware of human intentions and
needs in various phases of the collaborative task [Klingspor
et al., 1997, Bauer et al., 2008]. Despite the fact that the
robotic replication of the human sensory system is hardly
2
A good overview of the cognitive aspects in HRC can be found
in the literature, e.g., see [Fong et al., 2003, Freedy et al., 2007, Rani
et al., 2004]
possible with the current technology, the understanding and
implementation of the underlying communication principles
can potentially lead to an enhanced physical human-robot
interaction performance [Reed and Peshkin, 2008].
A widely known example of such a communication in-
terface is built on the use of visual [Perzanowski et al., 1998,
Agravante et al., 2014a,Morel et al., 1998] or language com-
mands [Medina et al., 2012a,Miyake and Shimizu, 1994,Pe-
tit et al., 2013], as user-friendly means of communicating
with a robot, from the human standpoint, given that the hu-
man is not required to learn additional tools. The use of
head, body or arm gestures are common examples in the ar-
eas of human-robot interaction and collaboration [Li et al.,
2005,Carlson and Demiris, 2012]. In this direction a method
to interpret the human intention from the latest history of
the gaze movements and to generate an appropriate reactive
response in a collaborative setup was proposed in [Sakita
et al., 2004]. Authors in [Hawkins et al., 2013] developed
a vision-based interface to predict in a probabilistic manner
when the human will perform different subtasks that may re-
quire robot assistance. The developed technique allows for
the tracking of the human variability, environmental con-
straints, and task structure to accurately analyse the timings
of the human partner’s actions.
Although such interfaces appear natural to the humans,
their usage is mostly limited to activating high-level robot
operations and the task complexity can potentially prevent
the robot from deriving the desired sensorimotor behaviour
from these higher-level modalities. In fact, a large degree
of robot autonomy, which is far beyond current capabilities
of the autonomous robots, is required for vision or auditory
based interfaces to function on a wide range of applications.
An alternative approach to the design of human-robot in-
terfaces recognises the use of force/pressure sensors in con-
tact to anticipate the objective of the human partner and/or
to control the cooperation effort. Due to the simplicity of the
underlying mechanism, it has been explored in several ap-
plications, examples include collaborative object transporta-
tion [Ikeura and Inooka, 1995a,Kosuge and Kazamura, 1997a,
Al-Jarrah and Zheng, 1997a,Tsumugiwa et al., 2002a,Duchaine
and Gosselin, 2007, Agravante et al., 2014a, Gribovskaya
et al., 2011a, Rozo et al., 2014, Rozo et al., 2015, Adams
et al., 1996], object lifting [Evrard and Kheddar, 2009,Evrard
et al., 2009], object placing [Tsumugiwa et al., 2002a,Gams
et al., 2014], object swinging [Donner and Buss, 2016a,Palunko
et al., 2014], posture assistance [Ikemoto et al., 2012, Pe-
ternel and Babi
ˇ
c, 2013], and industrial complex assembly
processes [Kr
¨
uger et al., 2009] (see also Fig. 2).
In most of the above techniques, the interaction forces/torques
are used to regulate the robot control parameters and tra-
jectories following the admittance [Duchaine and Gosselin,
2009,Lecours et al., 2012] or impedance [Tsumugiwa et al.,
2002a,Agravante et al., 2014a] causality [Hogan, 1985]. Notwith-
Progress and Prospects of the HRC 3
Fig. 2: An example of human-robot collaborative manipulation in
a productive environment (image courtesy of ABB received from
www.abb.cz).
standing the wide margin of applications, collaborative tasks
that involve simultaneous interaction with rough or uncer-
tain environments (e.g. co-manipulative tool-use) can induce
various unpredictable force components to the sensor read-
ings [Peternel et al., 2014]. This can significantly reduce the
suitability of such an interface in more complex interaction
scenarios since it can be difficult to distinguish the compo-
nents related to the active counterpart(s) behaviour from the
ones generated from the interaction with the environment.
Bio-signals such as electromyography (EMG) and elec-
troencephalography (EEG), or other physiological indices
such as electrodermal activity [Pecchinenda, 1996,Rani et al.,
2006] can be used to anticipate the human intention in PHRC.
In particular, due to the adaptability and ease-of-use of EMG
measurements, they have found a wide range of applica-
tions in human-in-the-loop robot control such as: prosthe-
sis [Farry et al., 1996, Jiang et al., 2009, Farina et al., 2014,
Castellini et al., 2014, Strazzulla et al., 2017], exoskeletons
[Rosen et al., 2001,Fleischer and Hommel, 2008] and indus-
trial manipulator control [Vogel et al., 2011, Peternel et al.,
2014, Ajoudani, 2016, Gijsberts et al., 2014]. Peternel et al.,
used EMG signals to anticipate the stiffening/complying be-
haviour of a torque controlled robotic arm in a co-manipulation
task [Peternel et al., 2016c]. Through this interface, the lead-
ing/following roles of human and the robot counterpart were
estimated online. In another study, Bell et al., used EEG sig-
nals to command a partially autonomous humanoid robot
through high-level descriptions of the task [Bell et al., 2008].
A remarkable use of bio-signals in the development of
HR interfaces is to estimate the human physical (e.g. fa-
tigue) or cognitive (anxiety, in-attention, etc.) state varia-
tions that might deteriorate the collaborative robot’(s) per-
formance. The authors in [Rani et al., 2004] developed a
method to detect human anxiety in a collaborative setup by
extracting features from EMG, electrocardiography (ECG)
and Electrodermal responses. In a similar work, the human
physical fatigue was detected and used to increase the ro-
bot’s contribution to the task execution [Peternel et al., 2016b].
Fig. 3: Authors in [Peternel et al., 2016b] proposed a human-robot co-
manipulation framework for robot adaptation to human fatigue. The
myoelectric interface provides the robot controller with a feedback
about human motor behaviour to achieve an appropriate impedance
profile in different phases of the task. The human fatigue estimation
system provides the robot with the state of the human physical en-
durance (image courtesy of L. Peternel).
Although an interface that is build on a unique source
of sensory data can configure a pre-defined robot behav-
iour in collaborative settings, the underlying functionality
is limited and cannot be easily generalised to cross domain
scenarios. For instance, the use of visual feedback for the
estimation of the exchanged amount of energy between the
counterparts is less effective than the use of force or pressure
sensors. Similarly, the use of bio-signals such as EMGs for
tracking of the human limb movements may result in less
accurate performances in comparison to the external opti-
cal or IMU-based (inertial measurement unit) tracking sys-
tems [Corrales et al., 2008]. To address this, a combined ap-
proach, associating multi-modal sensory information to dif-
ferent robot control modalities (commonly known as multi-
modal interfaces [Mittendorfer et al., 2015, Peternel et al.,
2016c]), can be exploited. In this direction, the authors in
[Agravante et al., 2014a] proposed a hybrid approach by
merging vision and force sensing, to decouple high- and
low-level interaction components in a joint transportation
task where a human and humanoid robot carry a table with
a freely moving ball on top (see also [Rozo et al., 2016]). A
similar work proposed a multi-modal scheme for intelligent
and natural human-robot interaction [B
¨
ohme et al., 2003] by
merging vision-based techniques for user localisation, per-
son localisation and person tracking and their embodiment
into a multi-modal overall interaction schema.
In a similar fashion, voice commands were used to pause,
stop or resume the execution of a dynamic co-manipulation
task, the control parameters of which regulated by an EMG
based interface (See Fig. 3). In this work, an external track-
ing system detected the human arm configuration to regulate
the robot task frame in real-time. By the same token, au-
thors in [Yang et al., 2016] developed a multi-modal teach-
ing interface on a dual-arm robotic platform. The interface
was built on the use of EMG on the user arm and force sen-
sors on robot end-effector. In this setup, one robotic arm is
4 Ajoudani et al.,
Fig. 4: In [Ivaldi et al., 2016] naive participants (not expert in robot-
ics) interacted with the humanoid iCub to build an object: the physical
interaction was at the level of the arms, covered by a soft tactile skin
(image courtesy of S. Ivaldi).
connected to the tutee’s arm providing guidance through a
variable stiffness control approach, and the other to the tu-
tor to capture the motion and to feedback the tutees perfor-
mance in a haptic manner. The reference stiffness for the
tutors arm stiffness was estimated in real-time and repli-
cated by the tutee’s robotic arm. Ivaldi et al., [Ivaldi et al.,
2016] studied multi-modal communication of people inter-
acting physically with the humanoid iCub to build objects
(see Fig. 4). Participants would naturally gaze at the ro-
bot’s hands or face to communicate the focus of attention
of the collaborative action, while speaking to the robot to
describe each action. The authors found that individual fac-
tors of the participants influence the production of refer-
ential cues, both in speech and gaze: particularly, people
with negative attitude towards robots avoid gazing at the ro-
bot, while extroverted people speak more to the robot dur-
ing the collaboration. The robot, controlled in impedance
with a low stiffness at joints, switched to zero-torque control
when the humans were grasping the robot arms covered by a
tactile skin (enabling precise contact estimation [Fumagalli
et al., 2012]) and giving the voice command to “be compli-
ant”. This multi-modal command strategy allowed the par-
ticipants, not experts in robotics and mostly interacting with
a robot for the first time, to physically move the robot arms
in an intuitive way, without generating anxiety for their safety
as reported by the participants in their interviews. Despite
the lack of experience, all the human participants were able
to interact physically with the robot to teach the task. Fa-
cilitated probably by the child-like appearance of the iCub,
the participants naturally acted as teachers/caregivers, in line
with the observations of [Nagai and Rohlfing, 2009]; how-
ever, it has to be remarked that in that situation of physical
interaction the authors did not observe exaggerated move-
ments typical of parental motionese/scaffolding situations
in HRI, where often there is little physical interaction and
more social cues as gaze or gestures. These results are in
accord with previous work presented, see e.g. [Kilner et al.,
2003, Ugur et al., 2015].
The improved performance of the multi-modal interfaces
in the generation of compound robot behaviours that are
required to execute more complex collaborative tasks has
shifted the attention towards the usage and fusion of multi-
source sensory information. Results of Google Scholar sug-
gest that over 76% of the publications in the area of human-
robot collaboration used multi-modal interfaces in year 2015.
Nevertheless, the inclusion of more communication chan-
nels in the development of the intermediate interfaces will
potentially contribute to an increase in the human cogni-
tive burden and the low level robot control complexity. This
may affect the intuitiveness of the interface and result in an
excessive human effort to operate a specific robot modal-
ity. A solution to this issue can be obtained by the intro-
duction of shared communication modalities [Green et al.,
2008,Lackey et al., 2011,Lackey et al., 2011]. Alternatively,
robotic learning techniques such as: gradual mutual adap-
tation [Ikemoto et al., 2012, Peternel et al., 2016a], rein-
forcement learning [Palunko et al., 2014] or learning from
demonstration [Evrard et al., 2009, Lawitzky et al., 2012a,
Rozo et al., 2015] can be exploited to weaken the communi-
cation loops’ demands (e.g. bandwidth, number of feedback
modalities) due to an increased level of robot autonomy.
3 Interfaces for Improved Human Perception
The visual and auditory systems of the humans provide pow-
erful sensory inputs that contribute to a fast and accurate per-
ception of the movement kinematics and the environment,
and a constant update of the internal models. The role of
such sensory inputs in dynamic perception of the environ-
ment, e.g. anticipating the weight of an object through vi-
sion [Gordon et al., 1993], and estimating a required amount
of force to move the object along a pre-defined path [Johans-
son, 1998], has also been investigated.
During collaboration and dyadic interaction, mutual gaze
and joint attention are common ways of conveying informa-
tion [Tomasello, 2009]. Such mechanisms are often imple-
mented in robots to make the interaction more effective by
providing additional back-channels. For example in [Ivaldi
et al., 2014] the robot was equipped with anticipatory gaze
mechanisms and proactive behaviours, increasing the pace
of the interaction and reducing the reaction time of the hu-
man to the robot’s cues.
In a similar study, Dumora et al. [Dumora et al., 2012],
studying haptic communication between an operator and a
robot for a bar transportation task, observed that wrench
measurements provide incomplete information to detect the
operator’s intent of motion. This has been also observed
in [Reed, 2012] for cooperating dyads able to communicate