Research Design: Toward a Realistic Role for Causal Analysis

TLDR: In this article, the authors adopt the design typology of Leslie Kish (1987), which advocates an appropriate balance of randomization, representation, and realism, and illustrate how all three elements are integrated aspects of meaningful causal analysis.

Abstract: For a half-century, sociology and allied social sciences have worked with a model of research design founded on a distinction between internal validity, the capacity of designs to support statements about cause and effect, and external validity, the extent to which the results from specific studies can be generalized beyond the batch of data on which they are founded. The distinction is conceptually useful and has great pedagogic value, that is, the association of the experimental model with internal validity, and random sampling with external validity. The advent of the potential outcomes model of causation, by emphasizing the definition of a causal effect at the unit level and the heterogeneity of causal effects, has made it clear how indistinct (and interpenetrated) are these “twin pillars” of research design. This is the theme of this chapter, which inveighs against the idea of a hierarchy of research design desiderata, with causal inference at the peak. Rather, I adopt the design typology of Leslie Kish (1987), which advocates an appropriate balance of randomization, representation, and realism, and illustrate how all three elements (and not just randomization, the internal validity design mechanism) are integrated aspects of meaningful causal analysis. What is meaningful causal analysis? It depends first and foremost on getting straight why we are doing what we are doing. Understanding why something has happened may tell us a lot about what will happen if we were actually to do something, but this is not necessarily so.