UC Berkeley
UC Berkeley Previously Published Works
Title
Measurement Validity: A Shared Standard for Qualitative and Quantitative Research
Permalink
https://escholarship.org/uc/item/945280s6
Journal
American Political Science Review, 95(3)
Authors
Collier, David
Adcock, Robert
Publication Date
2001-09-01
Peer reviewed
eScholarship.org Powered by the California Digital Library
University of California
Electronic copy available at: http://ssrn.com/abstract=1540880Electronic copy available at: http://ssrn.com/abstract=1540880
1094-2939/99/0616-0537$08.00
537
Annu. Rev. Polit. Sci. 1999. 2:537–65
Copyright © 1999 by Annual Reviews. All rights reserved
DEMOCRACY AND DICHOTOMIES:
A Pragmatic Approach to Choices
about Concepts
David Collier and Robert Adcock
Department of Political Science, University of California, Berkeley, California 94720-
1950; e-mail: dcollier@socrates.berkeley.edu
KEY WORDS: methodology, concept formation, validity, levels of measurement, regimes
ABSTRACT
Prominent scholars engaged in comparative research on democratic regimes
are in sharp disagreement over the choice between a dichotomous or graded
approach to the distinction between democracy and nondemocracy. This
choice is substantively important because it affects the findings of empirical
research. It is methodologically important because it raises basic issues,
faced by both qualitative and quantitative analysts, concerning appropriate
standards for justifying choices about concepts. In our view, generic claims
that the concept of democracy should inherently be treated as dichotomous
or graded are incomplete. The burden of demonstration should instead rest
on more specific arguments linked to the goals of research. We thus take
the pragmatic position that how scholars understand and operationalize a
concept can and should depend in part on what they are going to do with it.
We consider justifications focused on the conceptualization of democratiza-
tion as an event, the conceptual requirements for analyzing subtypes of de-
mocracy, the empirical distribution of cases, normative evaluation, the idea
of regimes as bounded wholes, and the goal of achieving sharper analytic
differentiation.
INTRODUCTION
Should scholars engaged in comparative research on democracy treat the dis-
tinction between democracy and nondemocracy as a dichotomy, or in terms of
gradations? This recurring and much debated question has important implica-
Electronic copy available at: http://ssrn.com/abstract=1540880Electronic copy available at: http://ssrn.com/abstract=1540880
tions for how research is organized, for how data are collected and analyzed,
and for inferences about the causes and consequences of democracy. It also
serves as a reminder that discussions of research design in political science
must pay central attention to conceptual issues.
Among the authors who have advocated an approach based on grading and
ranking, Bollen & Jackman argue that “democracy is always a matter of de-
gree” and that treating it as dichotomous is a “flawed” practice (1989:618,
612). A graded perspective is likewise adopted by Dahl, using the term poly-
archy (1971:2, 8, 231–35; 1989:241, 316–17), and later by Coppedge & Rein-
icke (1990). By contrast, Sartori finds that treating the distinction between
democracy and nondemocracy in graded terms is an analytically “stultifying”
exercise in “degreeism,” which misses the basic fact that political systems
are “bounded wholes” (1987:184; also 1991:248). Other scholars who have
adopted a dichotomous approach include Linz (1975:184–85), Huntington
(1991:11–12), and Geddes (1999). Przeworski and collaborators have speci-
fically rejected Bollen & Jackman’s argument as “confused” because it does
not recognize that regimes “cannot be half-democratic: there is a natural zero
point” (Alvarez et al 1996:21). Their position is especially striking because
their larger project (Przeworski et al 1996, Przeworski & Limongi 1997) is
based on quantitative data and sophisticated forms of statistical analysis. Yet
when it comes to measuring democracy versus nondemocracy, they select a
dichotomy.
We see an interesting puzzle here. The choice of a dichotomy in effect
places this distinction at what is traditionally viewed as the lowest level of
measurement (Stevens 1946, Roberts 1976:492–93). This choice thereby ap-
pears to underutilize more fine-grained information that may routinely be
available about differences among regimes. Yet both Sartori and Przeworski
and collaborators are convinced that this lowest level of measurement is more
valid in conceptual terms. This puzzle points to a question: What, indeed, are
the grounds for viewing this as a valid dichotomy, and not a “false dichot-
omy”?
This is an important question. First, quite apart from the scholars who are
explicitly debating this choice, large numbers of qualitative researchers (on
the one side) and quantitative researchers (on the other side) in effect take a po-
sition on this issue without ever directly addressing it. Second, this choice is
important because it affects substantive findings of research on democracy.
Although alternative dichotomous and graded measures are strongly corre-
lated with one another (Alvarez et al 1996:21), Elkins (1999) has shown that,
in assessing the impact of regime type on the initiation of war, a graded meas-
ure reveals interesting incremental effects that would not be detected with a
dichotomy. He likewise shows that, in studies of the effect of regime type on
political stability, the use of a single cut-point can mask a relationship that
538 COLLIER & ADCOCK
emerges only if one looks at a different cut-point. Similarly, Coppedge (1997:
181, 189–97) finds that in cross-national tests evaluating explanations for
levels of polyarchy, different results emerge depending on which cut-point is
employed in creating a dichotomous measure of polyarchy. Restricting the
analysis to any single dichotomous cut-point can thus obscure potential find-
ings.
Given that the findings of research can be influenced by these choices, it is
important to examine the conceptual reasoning that justifies alternative ap-
proaches. However, somewhat surprisingly, recent methodological writing in
political science provides insufficient guidance for dealing with such ques-
tions, and various contributors to recent debates on research design have called
for greater attention to conceptual issues (Laitin 1995:455–56, Collier 1995:
463, Brady 1995:16–18, Munck 1999).
This paper examines the conceptual justifications that lead scholars to
choose a dichotomous or graded approach. Part 1 focuses on general methodo-
logical arguments about concept formation, dichotomies, and gradations. Part
2 reviews examples of the generic justifications employed by prominent
authors in the literature on democracy. Part 3 considers more specific justifica-
tions, which we believe provide a better rationale for the choice between di-
chotomies and gradations.
In favoring these more specific justifications, we adopt a pragmatic posi-
tion.
1
While recognizing that usage is shaped and constrained by the broader
scholarly understanding of a concept’s meaning, we hold that specific meth-
odological choices are often best understood and justified in light of the
theoretical framework, analytic goals, and context of research involved in any
particular study. As theory, goals, and context evolve, choices about concepts
likewise may evolve.
To explicate our pragmatic approach, it is useful to identify two interrelated
priorities that underlie this perspective. First, it rejects the idea that there is a
single correct, or “best,” meaning for all concepts and views the search for a
single best meaning as frequently being an unproductive enterprise. Second,
this approach focuses on understanding how alternative meanings are con-
nected with the specific goals and context of research. Thus, how scholars un-
derstand and operationalize a concept can and should depend in part on what
they are going to do with it.
Our approach thus shares important concerns with the tradition of concept
analysis that, in conjunction with a broad focus on the structure of meaning,
A PRAGMATIC APPROACH TO CONCEPTS 539
1
1
In developing our position, we have drawn on discussions of concepts and measurement in
Popper (1976:17–31), Kaplan (1964:34–81), Adams & Adams (1987), Jones (1974), Shapiro
(1989), and Collier (1998). In relation to recent debates on the philosophy of science, we see our
position on concepts as compatible with the general position of “pragmatic realism” advocated by
Shapiro (1990:231–38).
explores variations in concept usage among different authors and different
schools of thought. This tradition is identified both with the field of political
theory (Gallie 1956, Pitkin 1967, Freeden 1996), and also with studies that
draw directly or indirectly on Sartori’s (1984b:40–42) methodology for the
reconstruction of concepts. Examples of this latter approach include studies of
consensus (Graham 1984), elite (Zannoni 1978), ideology (Gerring 1997),
political culture (Patrick 1984), revolution (Kotowski 1984), and social move-
ment (Diani 1992). This tradition generally avoids preemptively ruling out
particular meanings or usages, and instead focuses on understanding each us-
age in its own terms.
Our pragmatic approach also shares important concerns with the focus on
“construct validity,” which is one central consideration in efforts by quantita-
tive researchers in political science to evaluate choices about concepts and
operationalization (Zeller & Carmines 1980:79–81). A central goal in assess-
ments of construct validity is to evaluate whether a given operationalization of
a concept, when used in testing a well-established hypothesis, yields results
that are plausible and interesting in light of theoretical expectations regarding
that hypothesis. With regard to the concept of democracy, Elkins (1999) is an
excellent example of this approach.
We share with the construct validity approach a concern with the details of
how particular concepts are actually used in exploring specific research
questions. However, in contrast to this approach, we address this concern in
relation to a broader range of issues about how concepts are applied and under-
stood in empirical research. For example, we consider the implications of
treating democratization as a well-bounded “event” and the conceptual re-
quirements for analyzing subtypes of democracy, as well as issues of the em-
pirical distribution of cases and normative assessment that arise in specific
contexts of research.
These questions of justifying choices about concepts are complex, and we
wish to underscore two issues not addressed here. First, we take as given the
procedural definition of democracy that has predominated in the recent com-
parative literature on democratization. The interesting question of dichotomies
in relation to other definitions of democracy is not addressed. Second, if a
scholar adopts gradations, a further choice concerns the choice of procedures
for aggregating observations in scales.
2
This choice is crucial but is likewise
beyond our focus.
540 COLLIER & ADCOCK
2
2
For example, among advocates of a graded approach, Bollen (1993) and Bollen & Paxton
(2000) begin with eight ordered scales (each involving between two and seven categories) and
employ structural equation models with latent variables to produce an aggregated scale that ranges
from 0 to 100. Coppedge & Reinicke (1990) begin with five graded measures (each involving either
three or four categories) and employ Guttman scale analysis to produce an aggregated scale that
ranges from 0 to 10.