Benchmarking Bipedal Locomotion: A Unified Scheme for Humanoids, Wearable Robots, and Humans

TLDR: The objective of this article is to define the basis of a benchmarking scheme for the assessment of bipedal locomotion that could be applied and shared across different research communities.

Abstract: In the field of robotics, there is a growing awareness of the importance of benchmarking [1], [2]. Benchmarking not only allows the assessment and comparison of the performance of different technologies but also defines and supports the standardization and regulation processes during their introduction to the market. Its importance has been recently emphasized by the adoption of the technology readiness levels (TRLs) in the Horizon 2020 information and communication technologies by the European Union as an important guideline to assess when a technology can shift from one TRL to the other. The objective of this article is to define the basis of a benchmarking scheme for the assessment of bipedal locomotion that could be applied and shared across different research communities.

Figures

Figure 2. The basic components of our benchmarking taxonomy: motor skills, motor abilities and motor performance. These components have important interdependencies. In order to achieve a desired motor performance (e.g. moving from A to B), different motor abilities (e.g. coordination, equilibrium, reaction time) should be combined together, resulting in a functional motor skill (e.g. walking movements). These three motor aspects, whose quantitative measurement is the main objective of the proposed scheme, are associated to three corresponding internal processes (indicated in blue): i) the perception of the sensory feedback resulting from the actual performance, ii) the learning of new control strategies, and iii) the adaptation of motor abilities necessary to generate an improved motor skill. The analysis of these internal processes, very specific to each community, goes behind the scope of the proposed scheme.

Figure 5. Motor abilities and related benchmarks classified in two categories: performance and human likeness. The performance category includes all those abilities related to stability (ability of maintaining equilibrium) and efficiency. The human likeness category includes all those abilities related to typical human behavior, under the perspective of kinematics

Figure 3. The Gentile’s taxonomy classifies motor skills according to two main dimensions, environment and function, and four intrinsic characteristics, i.e. environment motion, intertrial variability, body motion, and manipulation. The table allows for 16 possible categories, ordered in a simple-to-complex progression, from top-left to bottom-right. Adapted from [8].

Figure 6. Template of work sheet. This work sheet should be used each time someone wants to propose a new benchmarking protocol to the community, in order others to replicate the experiment on different platforms and hardware configurations. Sections 1 and 2 should be used to contextualize the motor skill and the type of disturbances. Section 3 should contain a step-by-step description of the experimental protocol, to allow for its replication. Section 4 should include the variables to be measured to allow for computing the benchmarks specified in Section 5. Finally, in section 6, the researcher should specify how results will be presented. Practical examples of how to use the work sheet in practical systems and experimental scenarios will be gathered in the webpage www.benchmarkinglocomotion.org, to promote their iterative use and improvement.

Figure 4. Motor skills considered in the benchmarking scheme. Since this scheme is limited to bipedal locomotion skills, the “manipulation” category originally included in the Gentile’s taxonomy (see Figure 3) has been omitted. The concept of intertrial variability is analogous to the concept of unexpected disturbances.

Figure 1. Results of the web-based survey. The 161 respondents to the questionnaire are international experts that were contacted through communication means of the following workshops, networks or forums: the involved research projects listed on page 1, euRobotics or Euron mailing list, Biomch-L mailing list, robotics_worldwide mailing list, WeRob2014 - The 2014 International Workshop on Wearable Robotics (werob2014.org), European Network on Robotics for NeuroRehabilitation (EC COST Action TD1006) (www.rehabilitationrobotics.eu), Dynamic Walking (dynamicwalking.org/), RehabRobotics mailing list (associated with ICORR - http://www.rehabrobotics.org/).

Full text

Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/281823589
BenchmarkingBipedalLocomotion:AUnified
SchemeforHumanoids,WearableRobots,and
Humans
ARTICLEinIEEEROBOTICS&AMPAMPAMPAUTOMATIONMAGAZINE·SEPTEMBER2015
ImpactFactor:2.41·DOI:10.1109/MRA.2015.2448278
READS
73
9AUTHORS,INCLUDING:
DiegoTorricelli
SpanishNationalResearchCouncil
51PUBLICATIONS77CITATIONS
SEEPROFILE
JanFVeneman
Tecnalia
27PUBLICATIONS856CITATIONS
SEEPROFILE
AntonioJ.del-Ama
ServiciodeSaluddeCastilla-LaMancha
41PUBLICATIONS159CITATIONS
SEEPROFILE
JuanC.Moreno
CajalInstitute
80PUBLICATIONS468CITATIONS
SEEPROFILE
Availablefrom:DiegoTorricelli
Retrievedon:24September2015

1
The current document represents the accepted version of the following article:
Torricelli, D.; Gonzalez-Vargas, J.; Veneman, J.; Mombaur, K.; Tsagarakis, N.; del-Ama, A.; Gil-
Agudo, A.; Moreno, J.; Pons, J., "Benchmarking Bipedal Locomotion: A Unified Scheme for
Humanoids, Wearable Robots, and Humans," in Robotics & Automation Magazine, IEEE, vol.22,
no.3, pp.103-115, Sept. 2015. DOI: 10.1109/MRA.2015.244827
The published version of the article can be downloaded at:
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7254255&isnumber=7254280
IEEE Copyright notice
“©2015 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or lists, or
reuse of any copyrighted component of this work in other works.”

2
Benchmarking bipedal locomotion: A unified scheme
for humanoids, wearable robots and humans
Diego Torricelli
1
, Jose Gonzalez
1
, Jan Veneman
2
, Katja Mombaur
3
, Nikos Tsagarakis
4
, Antonio J.
del-Ama
5
, Ángel Gil-Agudo
5
, Juan C. Moreno
1
, and Jose L. Pons
1
1
Spanish National Research Council (CSIC), Cajal Institute, Neural Rehabilitation Group, Madrid, Spain
2
Tecnalia Research and Innovation, Donostia-San Sebastian, Spain
3
University of Heidelberg, Germany
4
IIT, Istituto Italiano di Tecnologia, Department of Advanced Robotics, Italy
5
National Hospital for Spinal Cord Injury (SESCAM), Biomechanics Unit, Spain
Introduction
In the field of robotics, there is a growing awareness of the importance of benchmarking [1], [2].
Benchmarking not only allows assessing and comparing the performance of different technologies, but also
defines and supports the standardization and regulation processes during their introduction to the market.
Its importance has been recently emphasized by the adoption of the technology readiness levels (TRLs) in
the Horizon 2020 ICT by the EU as an important guideline to assess when a technology can shift from one
TRL to the other.
In the field of humanoid robots the main obstacle in identifying common benchmarks is that different
methods and metrics are typically employed for specific robotic systems and functional scenarios.
Benchmarking of humanoid locomotion is typically approached on a competition-based level, and is mostly
focused on global functional goals (e.g. playing soccer, obstacle avoiding, stair climbing [1], [3]). In the
field of wearable robots, performance is usually reported in terms of the effects on the user’s motor
function. New standards are highly expected, especially now that these products are appearing on the
market. Yet, there are no accepted schemes for comparing wearable robots performance on a vast scale.
The only initiative in this direction is represented by the upcoming CYBATHLON competition [4]. In the
clinical and biomechanics field, many metrics and clinical scales have been defined and are regularly used
to assess locomotion functions [5]. Most of these scales are based on observation by skilled personnel or
defined on very general level, measuring variables like average speed or timed up-and-go. With the
increasing application of sensorized and robotic technology in clinics, the expectation for new quantitative
and reliable metrics is rapidly growing.
The objective of this article is to define the basis of a benchmarking scheme for the assessment of bipedal
locomotion, which could be applied and shared across different research communities. Our approach does
not aim to compare systems on a global level to see which one is better, but to assess the several aspects of
multi-facetted performance, allowing a truthful comparison of each feature independently. We envision a
scenario in which using this scheme will encourage the collaboration between different research groups
towards the consolidation of standardized benchmarks and experimental procedures, and promote its use as
a complementary tool to competition-based approaches. The scheme presented in this paper is the result of
the joint efforts of five European projects, i.e. H2R
1
, BALANCE
2
, Koroibot
3
, Walkman
4
and Biomot
5
. We
think that the proposed scheme can be taken as a starting point for a global iterative process that could lead
to an international consensus, based on its practical use across different laboratories.
1
www.h2rproject.eu
2
www.balance-fp7.eu
3
www.koroibot.eu
4
www.walk-man.eu
5
www.biomotproject.eu

3
1 Design approach
1.1 Analysis of the needs: the web-based survey
A benchmark can be considered successful if and only if it is widely accepted by the community at which it
is targeted. In order to reach this goal, a number of key principles for a successful benchmarking scheme
have been identified [6]:
1. The benchmarks must be well defined, i.e. they really must serve their purpose. As a consequence,
the purpose should be clear.
2. Benchmarks should be rigorously focused on limited, particular sub-domains.
3. It is more likely that a benchmark is successful within a scientific (sub) community if it arises from
that community itself.
Our design process started with a web-based survey, in order to identify the needs of the different users to
which the scheme is addressed. The research communities considered were humanoid robotics, wearable
robotics, and human biomechanics. The last has been included because of the increasing need to merge
insights from biomechanics and human motor control in robotic research. The survey (see Figure 1)
comprised 9 questions, which overall address the first two aforementioned design principles. The first three
questions aimed to collect general information about the respondents, such as their background and their
overall interest in using a benchmarking scheme and in sharing the data obtained by its use. The last six
questions focused on the contents of the ideal benchmarking scheme, in terms of: general purpose, motor
function addressed, performance variables to be measured, conditions to be included, technical properties
of the scheme, and information needed to contextualize results. In these questions, the user was asked to
give a score from 1 to 5 to each of the predefined options. Results are represented in Figure 1 in terms of
mean values and standard deviations, and divided by the background of the respondents. Statistical analysis
of similarity across communities has been performed by one-way ANOVA (level of significance p=0.05).
Figure 1 also provides a rough classification of results in three classes, according to mean scores across all
communities: items of high relevance (mean score over 4.00, highlighted in green), items of medium
relevance (mean score over 3.00, in orange) and items of low relevance (mean score lower than 3.00, in
red). An asterisk indicates which items presented a significant different response among the communities
(excluding the “other” group).

4
Figure 1. Results of the web-based survey. The 161 respondents to the questionnaire are international experts that were
contacted through communication means of the following workshops, networks or forums: the involved research projects
listed on page 1, euRobotics or Euron mailing list, Biomch-L mailing list, robotics_worldwide mailing list, WeRob2014 -
The 2014 International Workshop on Wearable Robotics (werob2014.org), European Network on Robotics for
NeuroRehabilitation (EC COST Action TD1006) (www.rehabilitationrobotics.eu), Dynamic Walking
(dynamicwalking.org/), RehabRobotics mailing list (associated with ICORR - http://www.rehabrobotics.org/). !

Frequently asked questions

Q1. What are the contributions in "Benchmarking bipedal locomotion: a unified scheme for humanoids, wearable robots, and humans" ?

In this paper, the authors proposed a taxonomic classification of bipedal motor functions based on motor skills, abilities and performance. 

Q2. What are the major motor abilities of the Fleishman’s list related to lower ?

They are inter-limb coordination, static and dynamic strength, limb flexibility, gross body equilibrium, reaction time, speed of limbs, and control precision. 

Q3. What is the key prerequisite for the study of the mechanisms behind human motor control?

The correct classification and assessment of abilities, skills, and performance is a key prerequisite for the study of the mechanisms behind human motor control. 

Q4. What is the typical rationale for a motor learning procedure?

A typical rationale during a motor learning procedure is to begin with stationary environment and no intertrial variability (e.g. repetitive trials of single movement), towards a complete moving environment with intertrial variability (e.g. real-life and out-of-the-lab conditions). 

Q5. What are the main problems when defining similarity between different systems?

One problem when defining similarity between different systems is that the dynamic and kinematic properties, including elementary properties such as weight, size, mass distribution or number of degrees of freedoms (DOFs), but also the corresponding kinematic and dynamic constraints, have to be taken into account. 

Q6. What are the common performance measures in clinical and robotic scenarios?

Performance measures usually consist of discrete scales based on time, distance, or percentage of goal achievement, and can be obtained experimentally with no particular difficulty. 

Q7. What is the typical rationale for a moving environment?

A typical rationale is to begin with stationary environment and no intertrial variability (e.g. repetitive trials of single movement), towards a complete moving environment with intertrial variability (e.g. real-life and outof-the-lab conditions). 

Q8. What kind of information should be included in the survey?

Four kinds of information should be included: the experimental procedure, the applicable benchmarks, the variables to be measured, and way of representing the results, namely numerical, graphical, or single-scale. 

Q9. What is the main purpose of the work sheet?

the scheme is collaborative, i.e. it requires the participation of the community in proposing and refining new protocols and benchmarks, e.g. by means of the work sheet tool provided in this paper. 

Q10. What is the way to classify the environment?

The environment, represented by the elements in contact with the person during the execution of the skill, can be classified according to two intrinsic characteristics: i) its absolute motion and ii) the presence of intertrial variability, i.e. if the environment changes from two consecutive trials. 

Q11. How can the authors measure energy efficiency of robots and humans?

Measuring energy efficiency of robots and humans can be done by the specific cost of transport (ct) [12], [13], defined as the ratio of the energy consumed and the weight times the distance travelled. 

Q12. What is the classification of motor skills?

Gentile’s taxonomy (see Figure 3) classifies motor skills according to two general dimensions:1) The environment, represented by the elements in contact with the person during the execution of the skill, which can be classified according to two intrinsic characteristics: i) its absolute motionand ii) the presence of intertrial variability, which indicates whether the environmental condition changes from two consecutive trials. 

Q13. What is the way to classify skills and abilities?

Measures for skills and abilities are more difficult to obtain, because they rely on generic concepts (e.g. “walking”, “standing”) and depend on continuous variables such as kinematics, kinetics, muscular activity, which can hardly be translated into absolute metrics.