Validity and Mechanical Turk

High and persistent unemployment has been a major blot on the economic and social record of most OECD countries since the early 1970s: the OECD average standardised unemployment rate rose from just over 3 per cent in 1973 to 7.3 per cent in 1997. In response to growing political concerns about the seemingly inexorable rise in unemployment, OECD Ministers gave the Organisation a mandate in 1992 to analyse the causes and consequences of high and persistent unemployment and propose effective remedies to deal with the problem.

What Works Among Active Labour Market Policies: Evidence from OECD Countries' Experiences | Conference – 1998

Researchers are concerned about whether manipulations have the intended effects. Many journals and reviewers view manipulation checks favorably, and they are widely reported in prestigious journals. However, the prototypical manipulation check is a verbal (rather than behavioral) measure that always appears at the same point in the procedure (rather than its order being varied to assess order effects). Embedding such manipulation checks within an experiment comes with problems. While we conceptualize manipulation checks as measures, they can also act as interventions which initiate new processes that would otherwise not occur. The default assumption that manipulation checks do not affect experimental conclusions is unwarranted. They may amplify, undo, or interact with the effects of a manipulation. Further, the use of manipulation checks in mediational analyses does not rule out confounding variables, as any unmeasured variables that correlate with the manipulation check may still drive the relationship. Alternatives such as non-verbal and behavioral measures as manipulation checks and pilot testing are less problematic. Reviewers should view manipulation checks more critically, and authors should explore alternative methods to ensure the effectiveness of manipulations.

/pdf/are-manipulation-checks-necessary-4xmfvkgxqx.pdf

Are Manipulation Checks Necessary

Attention checks have become increasingly popular in survey research as a means to filter out careless respondents. Despite their widespread use, little research has empirically tested the impact of attention checks on scale validity. In fact, because attention checks can induce a more deliberative mindset in survey respondents, they may change the way respondents answer survey questions, posing a threat to scale validity. In two studies, we tested this hypothesis (N 5 816). We examined whether common attention checks—instructedresponse items (Study 1) and an instructional manipulation check (Study 2)— impact responses to a well-validated management scale. Results showed no evidence that they affect scale validity, both in reported scale means and tests of measurement invariance. These findings allow researchers to justify the use of attention checks without compromising scale validity and encourage future research to examine other survey characteristic-respondent dynamics to advance our use of survey methods.

Are Attention Check Questions a Threat to Scale Validity

Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)—that is, objective questions about key elements of the experiment—can identify individual-level attentiveness to experimental information and, as a consequence, better enable researchers to diagnose experimental findings. We then find, through four replication studies, little evidence that FMC placement affects treatment effects, and that placing FMCs immediately post-outcome does not attenuate FMC passage rates. Additionally, FMC and IMC passage rates are only weakly related, suggesting that each technique identifies different sets of attentive subjects. Thus, unlike other methods, FMCs can confirm attentiveness to experimental protocols. Replication Materials: The data, code, and any additional materials required to replicate all analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at: https://dataverse.harvard.edu/dataset.xhtml?persistentId=https://doi.org/10.7910/DVN/GSQG1K. Experiments are a popular method for testing hypotheses and advancing theory within the social sciences. Essential for the validity of an experiment is the extent to which a study’s participants are actually “treated” (Oppenheimer, Meyvis, and Davidenko 2009). As a precondition for being treated, respondents must (in most instances) first be attentive to treatments. Ensuring that respondents attend to stimuli, however, presents a challenge, particularly given the growth in online surveys and reliance upon subjects from opt-in panels who may be distracted (e.g., Chandler and Shapiro 2016; Clifford and Jerit 2015). Within a study, therefore, being able to accurately gauge receipt of treatment is paramount. One technique for assessing treatment receipt is that of a manipulation check (MC). Broadly defined, MCs are “used to check whether the manipulation conducted in an experiment is perceived by the subjects as the experimenter wishes it to be perceived” (Morton and John V. Kane is Assistant Professor, Center for Global Affairs, New York University, 15 Barclay Street, New York, NY 10007 (john.kane@nyu.edu).Jason Barabas is Professor, Department of Political Science, Stony Brook University, Social & Behavioral Sciences, 7th Floor, Stony Brook, NY 11794 (jason.barabas@stonybrook.edu). We would like to thank Scott Clifford, Jennifer Jerit, Yanna Krupnikov, David Nickerson, and Christine Peterson for their helpful comments on earlier drafts of this article. Additionally, Ben Carter, Mike Kriner, Michelle lo-Low, and Alecia Nepaul provided outstanding research assistance. Matthew Hitt and Vittorio Mérola graciously provided data and code. Finally, we also appreciate the suggestions that we received from the anonymous reviewers, faculty and graduate students at Stony Brook University, and panelists at the annual conferences of the American Political Science Association and the International Society for Political Psychology. Williams 2010, 108). However, there is much variation in why researchers implement MCs. In practice, scholars sometimes report using manipulation checks to determine whether the latent independent variable of interest has been affected by experimental stimuli (Mutz and Pemantle 2015, 196). More generally, researchers also use MCs to assess respondent attentiveness during a study, such as via specific questions given to participants to judge whether they are reading carefully (e.g., Anduiza and Galais 2016; Crump, McDonnell, and Gureckis 2013; Maniaci and Rogge 2014). MCs therefore provide researchers with leverage on the nature of their experimental findings. As Mutz (2011, 84–85) contends, “Experiments for which manipulation checks can be considered ‘optional’ are relatively few and far between,” primarily because failing to include an MC “undermines what can be learned from any given study.” Perhaps because of the varying functions of MCs, there is widespread heterogeneity in the forms that MCs American Journal of Political Science, Vol. 63, No. 1, January 2019, Pp. 234–249 C ©2018, Midwest Political Science Association DOI: 10.1111/ajps.12396

No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments

Current levels of youth unemployment need to be understood in the context of increased labor market flexibility, an expansion of higher education, youth migration, and family legacies of long-term unemployment. Compared with previous recessions, European-wide policies and investments have significantly increased with attempts to support national policies. By mapping these developments and debates, we illustrate the different factors shaping the future of European labor markets. We argue that understanding youth unemployment requires a holistic approach that combines an analysis of changes in the economic sphere around labor market flexibility, skills attainment, and employer demand, as well as understanding the impact of family legacies affecting increasingly polarized trajectories for young people today. The success of EU policy initiatives and investments will be shaped by the ability of national actors to implement these effectively.

/pdf/five-characteristics-of-youth-unemployment-in-europe-4g46fv115e.pdf

Lucia Mýtna Kureková

Papers

Five Characteristics of Youth Unemployment in Europe: Flexibility, Education, Migration, Family Legacies, and EU Policy