On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness

TLDR: In this article, the authors put composite indicators under the spotlight, examining the wide variety of methodological approaches in existence and offered a more recent outlook on the advances made in this field over the past years.

Abstract: In recent times, composite indicators have gained astounding popularity in a wide variety of research areas. Their adoption by global institutions has further captured the attention of the media and policymakers around the globe, and their number of applications has surged ever since. This increase in their popularity has solicited a plethora of methodological contributions in response to the substantial criticism surrounding their underlying framework. In this paper, we put composite indicators under the spotlight, examining the wide variety of methodological approaches in existence. In this way, we offer a more recent outlook on the advances made in this field over the past years. Despite the large sequence of steps required in the construction of composite indicators, we focus particularly on two of them, namely weighting and aggregation. We find that these are where the paramount criticism appears and where a promising future lies. Finally, we review the last step of the robustness analysis that follows their construction, to which less attention has been paid despite its importance. Overall, this study aims to provide both academics and practitioners in the field of composite indices with a synopsis of the choices available alongside their recent advances.

Figures

Fig. 1 Results for ‘composite indicators’ on SCOPUS for the period 1997–2016

Full text

Soc Indic Res (2019) 141:61–94
https://doi.org/10.1007/s11205-017-1832-9
1 3
On theMethodological Framework ofComposite
Indices: AReview oftheIssues ofWeighting,
Aggregation, andRobustness
SalvatoreGreco
1,2
· AlessioIshizaka
2
· MenelaosTasiou
3
· GianpieroTorrisi
1,3
Accepted: 28 December 2017 / Published online: 17 January 2018
© The Author(s) 2018. This article is an open access publication
Abstract In recent times, composite indicators have gained astounding popularity in a
wide variety of research areas. Their adoption by global institutions has further captured the
attention of the media and policymakers around the globe, and their number of applications
has surged ever since. This increase in their popularity has solicited a plethora of methodo-
logical contributions in response to the substantial criticism surrounding their underlying
framework. In this paper, we put composite indicators under the spotlight, examining the
wide variety of methodological approaches in existence. In this way, we offer a more recent
outlook on the advances made in this field over the past years. Despite the large sequence
of steps required in the construction of composite indicators, we focus particularly on two
of them, namely weighting and aggregation. We find that these are where the paramount
criticism appears and where a promising future lies. Finally, we review the last step of the
robustness analysis that follows their construction, to which less attention has been paid
despite its importance. Overall, this study aims to provide both academics and practitioners
in the field of composite indices with a synopsis of the choices available alongside their
recent advances.
* Menelaos Tasiou
Menelaos.Tasiou@myport.ac.uk
Salvatore Greco
salgreco@unict.it
Alessio Ishizaka
alessio.ishizaka@port.ac.uk
Gianpiero Torrisi
gianpiero.torrisi@port.ac.uk
1
Department ofEconomics andBusiness, University ofCatania, Catania, Italy
2
Centre ofOperations Research andLogistics, Portsmouth Business School, University
ofPortsmouth, Portsmouth, UK
3
Portsmouth Business School, University ofPortsmouth, Portland Building 3.09, Portland Street,
Portsmouth, HampshirePO13AH, UK

62
S.Greco et al.
1 3
Keywords Composite indicators· Weighting· Aggregation· Robustness
1 Introduction
In the past decades, we have witnessed an enormous upsurge in available information, the
extent and use of which are characterised by the founder of the World Economic Forum as
the ‘Fourth Industrial Revolution’ (Schwab 2016, para. 2). While Schwab focuses on the
use and future impact of these data—ranging from policy and business analysis to artifi-
cial intelligence—one of the key underlying points is that this enormous and exponential
increase in available information hides another issue: the need for its interpretation and
consolidation. Indeed, an ever-increasing variety of information, broadly speaking in the
form of indicators, increases the difficulty involved in interpreting a complex system. To
illustrate this, consider for example a phenomenon like well-being. In principle, it is a
very complex concept that is particularly difficult to capture with only a single indicator
(Decancq and Lugo 2013; Decancq and Schokkaert 2016; Patrizii etal. 2017). Hence, one
should enlarge the range of indicators to encompass all the necessary information on a
matter that is generally multidimensional in nature (Greco etal. 2016). However, in such a
case, it would be very difficult for the public to understand ‘well-being’ by, say, identifying
common trends among several individual indices. They would understand a complex con-
cept more easily in the form of a sole number that encompasses this plethora of indicators
(Saltelli 2007). Reasonably, this argument may raise more questions than it might answer.
For instance, how would this number be produced? Which aspects of a concept would it
encompass? How would they be aggregated into the form of a simple interpretation for
the public and so on? This issue, and the questions that it raises, introduce the concept of
‘composite indicators’.
Defining ‘composite’ (sometimes also encountered as ‘synthetic’) indicators should be
a straightforward task given their widespread use nowadays. Even though it appears that
there is no single official definition to explain this concept, the literature provides a wide
variety of definitions. According to the European Commission’s first state-of-the-art report
(Saisana and Tarantola 2002, p. 5), composite indicators are ‘[…] based on sub-indicators
that have no common meaningful unit of measurement and there is no obvious way of
weighting these sub-indicators’. Freudenberg (2003, p. 5) identifies composite indicators as
‘synthetic indices of multiple individual indicators’. Another potential definition provided
by the OECD’s first handbook for constructing composite indicators (Nardo etal. 2005, p.
8) is that a composite indicator ‘[…] is formed when individual indicators are compiled
into a single index, on the basis of an underlying model of the multi-dimensional con-
cept that is being measured’. This list of definitions could continue indefinitely. By pooling
them together, a common pattern emerges and relates to the central idea of the landmark
work of Rosen (1991). Essentially, a composite indicator might reflect a ‘complex system’
that consists of numerous ‘components’, making it easier to understand in full rather than
reducing it back to its ‘spare parts’. Although this ‘complexity’, from a biologist’s view-
point, refers to the causal impact that organisations exert on the system as a whole, the
intended meaning here is astonishingly appropriate for the aim of composite indicators.
After all, Rosen asserts that this ‘complexity’ is a universal and interdisciplinary feature.
Despite their vague definition, composite indicators have gained astounding popularity
in all areas of research. From social aspects to governance and the environment, the num-
ber of their applications is constantly growing at a rapid pace (Bandura 2005, 2008, 2011).

63On theMethodological Framework ofComposite Indices: AReview…
1 3
For instance, Bandura (2011) identifies over 400 official composite indices that rank or
assess a country according to some economic, political, social, or environmental measures.
In a complementary report by the United Nations’ Development Programme, Yang (2014)
documents over 100 composite measures of human progress. While these inventories are
far from being exhaustive—compared with the actual number of applications in exist-
ence—they give us a good understanding of the popularity of composite indicators. Moreo-
ver, a search for ‘composite indicators’ in SCOPUS, conducted in January 2017, shows
this trend (see Fig.1). The increase over the past 20years is exponential, and the number
of yearly publications shows no sign of a decline. Moreover, their widespread adoption by
global institutions (e.g. the OECD, World Bank, EU, etc.) has gradually captured the atten-
tion of the media and policymakers around the globe (Saltelli 2007), while their simplicity
has further strengthened the case for their adoption in several practices.
Nevertheless, composite indicators have not always been so popular, and there was a
time when considerable criticism surrounded their use (Sharpe 2004). In fact, according
to the author, their very existence was responsible for the creation of two camps in the
literature: aggregators versus non-aggregators. In brief,
1
the first group supports the con-
struction of synthetic indices to describe an overall complex phenomenon, while the lat-
ter opposes it, claiming that the final product is statistically meaningless. While it seems
idealistic to assume that this debate will ever be resolved (Saisana etal. 2005), it quickly
drew the attention of policymakers and the public. Sharpe (2004) describes the example of
the Human Development Index (HDI), which has received a vast amount of criticism since
its creation due to the arbitrariness of its methodological framework (Ray 2008). However,
it is the most well-known composite index to date. Moreover, it led the 1998 Nobel Prize-
winning economist A. K. Sen, once one of the main critics of aggregators, to change his
position due to the attention that the HDI attracted and the debate that it fostered after-
wards (Saltelli 2007). He characterised it as a ‘success’ that would not have happened in
the case of non-aggregation (Sharpe 2004, p. 11). Seemingly, this might be considered as
the first win for the camp of aggregators. Nevertheless, the truth is that we are still far from
settling the disputes and the criticism concerning the stages of the construction process
(Saltelli 2007).
This is natural, as there are many stages in the construction process of a composite
index and criticism could grow simultaneously regarding each of them (Booysen 2002).
Moreover, if the procedure followed is not clear and reasonably justified to everyone, there
is considerable room for manipulation of the outcome (Grupp and Mogee 2004; Grupp
and Schubert 2010). Working towards a solution to this problem, the OECD (2008, p. 15)
identifies a ten-step process, namely a ‘checklist’. Its aim is to establish a common guide-
line as a basis for the development of composite indices and to enhance the transparency
and the soundness of the process. Undeniably, this checklist aids the developer in gaining a
better understanding of the benefits and drawbacks of each choice and overall in achieving
the kind of coherency required in the steps of constructing a composite index. In practice,
though, this hardly reduces the criticism that an index might receive. This is because, even
if one does indeed achieve perfect coherency (from choosing the theoretical framework to
developing the final composite index), there might still be certain drawbacks in the meth-
odological framework itself.
1
For a more detailed analysis of the debate between the two groups, see Sharpe (2004, pp. 9–11).

64
S.Greco et al.
1 3
The purpose of this study is to review the literature with respect to the methodologi-
cal framework used to construct a composite index. While the existing literature con-
tains a number of reviews of composite indicators, the vast majority particularly focuses
on covering the applications for a specific discipline. To be more precise, several reviews
of composite indicators’ applications exist in the fields of sustainability (Bohringer and
Jochem 2007; Singh etal. 2009, 2012; Pissourios 2013; Huang etal. 2015), the environ-
ment (Juwana etal. 2012; Wirehn etal. 2015), innovation (Grupp and Mogee 2004; Grupp
and Schubert 2010), and tourism (Mendola and Volo 2017). However, the concept of com-
posite indicators is interdisciplinary in nature, and it is applied to practically every area
of research (Saisana and Tarantola 2002). Since the latest reviews on the methodological
framework of composite indices were published a decade ago (Booysen 2002; Saisana and
Tarantola 2002; Freudenberg 2003; Sharpe 2004; Nardo etal. 2005; OECD 2008) and a
great number of new publications have appeared since then (see Fig.1), we re-examine
the literature focusing on the methodological framework of composite indicators and more
specifically on the weighting, aggregation, and robustness steps. These steps are the focus
of the paramount criticism as well as the recent development. In the following, Sect. 2
describes the weighting schemes found in the literature and Sect. 3 covers the step of
aggregation. Section4 provides an overview of the methods used for robustness checks
following the construction of an index, and Sect.5 contains a discussion and concluding
remarks.
2 On theWeighting ofComposite Indicators
The meaning of weighting in the construction of composite indicators is twofold (OECD
2008, pp. 31–33). First, it refers to the ‘explicit importance’ that is attributed to every cri-
terion in a composite index. More specifically, a weight may be considered as a kind of
coefficient that is attached to a criterion, exhibiting its importance relative to the rest of
the criteria. Second, it relates to the implicit importance of the attributes, as this is shown
by the ‘trade-off’ between the pairs of criteria in an aggregation process. A more detailed
description of the latter and the difference between these two meanings is presented in
Sect.3, in which we describe the stage of aggregation and explain the distinction between
‘compensatory’ and ‘non-compensatory’ approaches.
Undeniably, the selection of weights might have a significant effect on the units ranked.
For instance, Saisana etal. (2005) show that, in the case of the Technology Achievement
Index, changing the weights of certain indicators seems to affect several of the units evalu-
ated, especially those that are ranked in middle positions.
2
Grupp and Mogee (2004) and
Grupp and Schubert (2010, p. 69) present two further cases of science and technology indi-
cators, for which the country rankings could significantly change or otherwise be ‘manip-
ulated’ in the case of different weighting schemes. This is a huge challenge in the con-
struction of a composite indicator, often referred to as the ‘index problem’ (Rawls 1971).
Basically, even if we reach an agreement about the indicators that are to be used, the ques-
tion that follows—and the most ‘pernicious’ one (Freudenberg 2003)—is how a weighting
scheme might be achieved. Although far from reaching a consensus (Cox etal. 1992), the
literature tries to solve this puzzle in several ways. Before we venture further to analyse the
2
Freudenberg (2003) presents a similar case during the construction of an index of innovation perfor-
mance.

65On theMethodological Framework ofComposite Indices: AReview…
1 3
weighting approaches in existence, we should first note that no weighting system is above
criticism. Each approach has its benefits and drawbacks, and there is no ultimate case of a
clear winner or a kind of ‘one-size-fits-all’ solution. On the contrary, it is up to the index
developer to choose a weighting system that is best fitted to the purpose of the construc-
tion, as disclosed in the theoretical framework (see OECD 2008, p. 22).
2.1 No orEqual Weights
As simple as it sounds, the first option is not to distribute any weights to the indicators, oth-
erwise called an ‘attributes-based weighting system’ (see e.g. Slottje 1991, pp. 686–688).
This system may have two consequences. First, the overall score (index) could simply be
the non-weighted arithmetic average of the normalised indicators (Booysen 2002; Singh
etal. 2009; Karagiannis 2017). A common problem that appears here, though, is that of
‘double counting’
3
(Freudenberg 2003; OECD 2008). Of course, this issue might partially
be moderated by averaging the collinear indicators as well prior to their aggregation into
a composite (Kao etal. 2008). The second alternative in the absence of weights is that
the composite index is equal to the sum of the individual rankings that each unit obtains
in each of the sub-indicators (e.g. see the Information and Communication Technologies
Index in Saisana and Tarantola 2002, p. 9). By relying solely on aggregating rankings, this
approach fails to achieve the purpose of vastly improving the statistical information, as
Fig. 1 Results for ‘composite indicators’ on SCOPUS for the period 1997–2016
3
In brief, ‘double counting’ refers to the issue of implicitly weighting an indicator higher than the desired
level. This happens when two collinear indicators are included in the aggregation process without moderat-
ing their weighting for this effect.

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "On the methodological framework of composite indices: a review of the issues of weighting, aggregation, and robustness" ?

This increase in their popularity has solicited a plethora of methodological contributions in response to the substantial criticism surrounding their underlying framework. In this paper, the authors put composite indicators under the spotlight, examining the wide variety of methodological approaches in existence. Finally, the authors review the last step of the robustness analysis that follows their construction, to which less attention has been paid despite its importance. Overall, this study aims to provide both academics and practitioners in the field of composite indices with a synopsis of the choices available alongside their recent advances. The authors find that these are where the paramount criticism appears and where a promising future lies. 

Q2. What is the standard procedure for using PCA as a weight elicitation technique?

The standard procedure in using PCA as a weight elicitation technique is to use the factor loadings of the first component to serve as weights for the indicators (Greyling and Tregenna 2016). 

Q3. What is the interesting development in the literature?

but not least, another interesting development in the literature is the introduction of SMAA, a tool that extends above and beyond the concept of the representative agent by considering the viewpoints of the whole population associated with the evaluation process. 

Q4. What other methods deal with the issue of adjusting the discrimination in BoD models?

There are several other methods in the literature17 that deal with the issue of adjusting the discrimination in BoD models, the most popular being the super-efficiency (Andersen and Petersen 1993), cross-efficiency (Sexton et al. 

Q5. Why do some methodologies appear in the literature?

Owing to the unresolved issues of choosing a weighting and an aggregation approach, several methodologies appear in the literature, dealing with these steps in different manners. 

Q6. What is the difference between the composite index and the compensatory aggregation approach?

The composite index is formed through an additive utility function, in which the composite equals the sum of the products of weights and indicators.