Ratings and rankings: voodoo or science?
TLDR
In this article, the authors measure the importance of a given variable within existing composite indicators via Karl Pearson's "correlation ratio"; they call this measure the main effect, and they discuss to what extent the mapping from nominal weights to main effects can be inverted.Abstract:
Summary. Composite indicators aggregate a set of variables by using weights which are understood to reflect the variables’ importance in the index. We propose to measure the importance of a given variable within existing composite indicators via Karl Pearson's ‘correlation ratio’; we call this measure the ‘main effect’. Because socio-economic variables are heteroscedastic and correlated, relative nominal weights are hardly ever found to match relative main effects; we propose to summarize their discrepancy with a divergence measure. We discuss to what extent the mapping from nominal weights to main effects can be inverted. This analysis is applied to six composite indicators, including the human development index and two popular league tables of university performance. It is found that in many cases the declared importance of single indicators and their main effect are very different, and that the data correlation structure often prevents developers from obtaining the stated importance, even when modifying the nominal weights in the set of non-negative numbers with unit sum.read more
Citations
More filters
Journal ArticleDOI
On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness
Salvatore Greco,Salvatore Greco,Alessio Ishizaka,Menelaos Tasiou,Gianpiero Torrisi,Gianpiero Torrisi +5 more
TL;DR: In this article, the authors put composite indicators under the spotlight, examining the wide variety of methodological approaches in existence and offered a more recent outlook on the advances made in this field over the past years.
Journal ArticleDOI
Weights and importance in composite indicators: Closing the gap.
TL;DR: Three tools are presented which help developers and users to investigate effects of weights in composite indicators and case studies related to sustainable development demonstrate the benefits.
Journal ArticleDOI
Building composite indicators using multicriteria methods: a review
TL;DR: A literature review of papers published after 2002 in leading international journals indexed in a recognised database (JCR) is conducted in order to identify the different MCDM methods used for aggregating single indicators into composite ones.
Journal ArticleDOI
Stochastic predictions of interfacial characteristic of polymeric nanocomposites (PNCs)
TL;DR: In this article, the effect of the single-walled carbon nanotube (SWCNT) radius, the temperature and the pulling velocity on interfacial shear stress (ISS) was studied by using the molecular dynamics (MD) simulations.
Journal ArticleDOI
Building Composite Indicators in Tourism Studies: Measurements and Applications in Destination Competitiveness
Daria Mendola,Serena Volo +1 more
TL;DR: This paper provides tourism scholars and practitioners with a set of statistical guidelines to build composite indicators and with an operative scheme to assess indicators' effectiveness in empirical evaluations.
References
More filters
Journal ArticleDOI
Performance indicators: good, bad, and ugly
TL;DR: In the UK public services, performance monitoring (PM) has been used to assess the impact of Government policies on those services or to identify well performing or underperforming institutions and public servants.
Journal ArticleDOI
Composite Indicators between Analysis and Advocacy.
TL;DR: In this paper, the authors explore to what extent composite indicators, capable of aggregating multi-dimensional processes into simplified, stylised concepts, are up to the task of underpinning the development of data-based narratives for political advocacy.
Book
Evaluation and Decision Models with Multiple Criteria: Stepping Stones for the Analyst
TL;DR: This work presents a meta-modelling architecture suitable for multi-Dimensional preference models and shows clear trends in preference-based decision-making that have been identified in recent years.
Journal ArticleDOI
Rickety numbers: Volatility of university rankings and policy implications
TL;DR: In this article, a robustness analysis based on a multi-modeling approach is proposed to test the validity of the inference about the rankings produced in the Academic Ranking of World Universities of Shanghai Jiao Tong University and those produced by the UK's Times Higher Education Supplement (THES).
Journal ArticleDOI
Random balance designs for the estimation of first order global sensitivity indices
TL;DR: The methods adopt Satterthwaite's application of random balance designs in regression problems, and extend it to sensitivity analysis of model output for non-linear, non-additive models to reduce significantly the computational cost of the analysis.