Pairwise comparison

About: Pairwise comparison is a research topic. Over the lifetime, 6804 publications have been published within this topic receiving 174081 citations.

New results on inconsistency indices and their relationship with the quality of priority vector estimation

Abstract: New results on inconsistency indices (ICIs) of pairwise comparison matrices (PCM).A first study on the relationship between ICIs and priorities estimation (PE) errors.Results allowing for a more profound interpretation of the well-known ICIs.An introduction of a new ICI that manifests high correlation with PE errors.A proposal of a PCM acceptance approach based on classical statistical methodology. The article is devoted to the problem of inconsistency in the pairwise comparisons based prioritization methodology. The issue of "inconsistency" in this context has gained much attention in recent years. The literature provides us with a number of different "inconsistency" indices suggested for measuring the inconsistency of the pairwise comparison matrix (PCM). The latter is understood as a deviation of the PCM from the consistent case - a notion that is formally defined in this theory. However the usage of the indices is justified only by some heuristics. It is still unclear what they really "measure". What is even more important and still not known is the relationship between their values and the "consistency" of the decision maker's judgments on the one hand, and the prioritization results upon the other.In this paper we argue that it is necessary to distinguish between the three following tasks: the "measuring" of the "PCM inconsistency", the PCM-based "measuring" of the consistency of the decision maker's judgments and, finally, the "measuring" of the usefulness of the PCM as a source of information for estimation of the priority vector (PV). We present examples showing that improving the consistency of PCM may lead to poorer PV estimation results, and that such a situation may occur quite naturally. Next we focus on the third of the above tasks, which is very important one in multi-criteria decision making. For the first time in literature, with the help of Monte Carlo simulations, we analyze the performance of the most common inconsistency indices as indicators of the final PV estimates quality. We consider two types of PV estimation errors and examine their distributions as well as their relationship with the indices values. The new results presented here allow for a more profound interpretation of the well-known inconsistency characteristics. Moreover, based on the analysis, we also introduce a new inconsistency index. In comparison with the other ones, the new index manifests significantly higher correlation with PV estimation errors. This fact also enables us to propose a novel PCM acceptance approach that is supported by the classical statistical methodology.

G2Way A Backtracking Strategy for Pairwise Test Data Generation

Abstract: Our continuous dependencies on software (i.e. to assist as well as facilitate our daily chores) often raise dependability issue particularly when software is being employed harsh and life threatening or (safety) critical applications. Here, rigorous software testing becomes immensely important. Many combinations of possible input parameters, hardware/software environments, and system conditions need to be tested and verified against for conformance. Due to resource constraints as well as time and costing factors, considering all exhaustive test possibilities would be impossible (i.e. due to combinatorial explosion problem). Earlier work suggests that pairwise sampling strategy (i.e. based on two-way parameter interaction) can be effective. Building and complementing earlier work, this paper discusses an efficient pairwise test data generation strategy, called G2Way. In doing so, this paper demonstrates the correctness of G2Way as well as compares its effectiveness against existing strategies including AETG and its variations, IPO, SA, GA, ACA, and All Pairs. Empirical evidences demonstrate that G2Way, in some cases, outperformed other strategies in terms of the number of generated test data within reasonable execution time.

Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Abstract: We address the problem of calibration of workers whose task is to label patterns with continuous variables, which arises for instance in labeling images of videos of humans with continuous traits. Worker bias is particularly difficult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of videos at a time and must order by preference. The variable levels are reconstructed by fitting a Bradley-Terry-Luce model with maximum likelihood. This method may at first sight, seem prohibitively expensive because for N videos, \(p=N(N-1)/2\) pairs must be potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is affordable. We apply the method to the labeling of a large scale dataset of 10,000 videos used in the ChaLearn Apparent Personality Trait challenge.

Techniques for inferring an organizational hierarchy from a social graph

Abstract: A method, system and article of manufacture for determining a global hierarchy of an entity by computing subjective coherence strength between nodes of first members of a social network service, computing objective entanglement strength between nodes of second members of the social network service, using subjective coherence strength and objective entanglement strength, compute the pairwise latent relationship probabilities between different ones of the nodes of the social network service, and using the pairwise relationship probabilities, compute the global hierarchy relationship.