We examine the trade-offs associated with using Amazon.com’s Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.

/pdf/evaluating-online-labor-markets-for-experimental-research-2g48xhj8u4.pdf

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g., students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon’s Mechanical Turk. The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

/pdf/the-generalizability-of-survey-experiments-5eofcolm20.pdf

The Generalizability of Survey Experiments

As Amazon’s Mechanical Turk (MTurk) has surged in popularity throughout political science, scholars have increasingly challenged the external validity of inferences made drawing upon MTurk samples....

https://journals.sagepub.com/doi/pdf/10.1177/2053168015604648

“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents:

We investigate the extent to which using students as experimental participants creates problems for causal inference. First, we discuss the impact of student subjects on a study’s internal and external validity. In contrast to common claims, we argue that student subjects do not intrinsically pose a problem for a study’s external validity. Second, we use simulations to identify situations when student subjects are likely to constrain experimental inferences. We show that such situations are relatively limited; any convenience sample poses a problem only when the size of an experimental treatment effect depends upon a characteristic on which the convenience sample has virtually no variance. Third, we briefly survey empirical evidence that provides guidance on when researchers should be particularly attuned to taking steps to ensure appropriate generalizability from student subjects. We conclude with a discussion of the practical implications of our findings. In short, we argue that student subjects are not an inherent problem to experimental research; moreover, the burden of proof - of student subjects being a problem - should lie with critics rather than experimenters.

https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID1498843_code844841.pdf?abstractid=1498843&mirid=3

Students as Experimental Participants: A Defense of the ‘Narrow Data Base

Social science researchers seek to establish causal relationships that are generalizable—that is, they try to maximize internal and external validity. Survey experiments are becoming more popular among scholars because they seem to possess both properties. The random assignment of respondents to treatment and control conditions reveals whether one factor causes another, whereas the use of a representative sample allows generalization to the larger population. However, even in nationally representative survey experiments, external validity may still be a concern if the treatments do not resemble the relevant phenomena in question or if the experimental setting exaggerates the effect of the stimulus. This article investigates a question that recent studies have raised (e.g., Gaines, Kuklinski, and Quirk 2007; Kinder 2007) but that has not been examined empirically: are the causal findings of survey experiments reliable predictors of how opinion changes in the wake of actual political events? For researchers using survey experiments, the implicit assumption is that a significant treatment effect says something about the direction, if not the rough magnitude, of effects that might be expected to occur in the real world (Gaines, Kuklinski, and Quirk 2007, 5). Our study investigates this assumption. We do so by

/pdf/are-survey-experiments-externally-valid-18kpatat67.pdf

Are Survey Experiments Externally Valid

The experimental approach has begun to permeate political science research, increasingly so in the last decade. Laboratory researchers face at least two challenges: determining who to study and how to lure them into the lab. Most experimental studies rely on student samples, yet skeptics often dismiss student samples for lack of external validity. In this article, we propose another convenience sample for laboratory research: campus staff. We report on a randomized experiment to investigate the characteristics of samples drawn from a general local population and from campus staff. We report that campus staff evidence significantly higher response rates, and we find few discernible differences between the two samples. We also investigate the second challenge facing researchers: how to lure subjects into the lab. We use evidence from three focus groups to identify ways of luring this alternative convenience sample into the lab. We analyze the impact of self-interest, social-utility, and neutral appeals on encouraging study participation, and we find that campus staff respond better to a no-nonsense approach compared to a hard-sell that promises potential policy benefits to the community or, and especially, to the self. We conclude that researchers should craft appeals with caution as they capitalize on this heretofore largely untapped reservoir for experimental research: campus employees.

Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research

This article focuses on gender and ethnic inequalities in political participation across non-Hispanic whites and Mexican Americans. Using a mainstream model of participation, the authors find that differences in the levels of resources, motivations, and opportunities effectively account for gender gaps within the two populations. However, this mainstream model leaves largely unexplained the chasm in participation across non-Hispanic whites and Mexican Americans. The authors incorporate socialization experiences specific to Mexican Americans to identify the roots of participatory inequality across these groups. Differences in linguistic, educational, and general assimilation account for participatory differences across Mexican Americans and non-Hispanic whites. Equalizing these factors closes the chasm in participation.

From the Gap to the Chasm: Gender and Participation among Non-Hispanic Whites and Mexican Americans

Elections are now a common feature of countries across regime types, yet we know little about what leads people to perceive an election as fair, or how the democratic context shapes the ingredients of fairness judgments. While the conduct of a process is most important for perceptions of fairness in established democracies, “procedural fairness” may not travel to non-democracies, where economic outcomes occasionally take precedence over procedure. Additionally, individual level characteristics, such as political engagement, may also shape how people view the fairness of elections. Using original experiments conducted in the United States and China, I find procedural considerations are most important for fairness judgments, across democratic contexts and largely independent of political engagement.

https://link.springer.com/content/pdf/10.1007%2Fs11109-010-9119-8.pdf

The Portability of Electoral Procedural Fairness: Evidence from Experimental Studies in China and the United States

“Well, that’s like night and day, being homeless, having nothing.” A qualitative exploration of the experiences of residents and service providers of permanent supportive housing

This mixed methods study explores predictive factors for housing insecurity (HI) and homelessness. Using multi-variable analyses with data from an original survey (48% response rate, n = 1,416 students), we found that income, race/ethnicity, awareness of services and being impacted by a natural disaster, all significantly impact HI and homelessness. Focus groups and interviews corroborate the quantitative results. Implications for practice are discussed. 

Jennifer R. Wilking

Papers

Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research

From the Gap to the Chasm: Gender and Participation among Non-Hispanic Whites and Mexican Americans

The Portability of Electoral Procedural Fairness: Evidence from Experimental Studies in China and the United States

“Well, that’s like night and day, being homeless, having nothing.” A qualitative exploration of the experiences of residents and service providers of permanent supportive housing

Understanding Student Housing Insecurity and Homelessness: A Mixed Methods and Multi-variable Analysis