Open AccessPosted Content
Running experiments on Amazon Mechanical Turk
Reads0
Chats0
TLDR
The authors presented new demographic data about the Mechanical Turk subject population, reviewed the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and compared the magnitude of effects obtained using Mechanical Turk and traditional subject pools.Abstract:
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.read more
Citations
More filters
Journal ArticleDOI
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
TL;DR: It is shown that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples but less representative than subjects in Internet-based panels or national probability samples.
Journal ArticleDOI
Conducting behavioral research on Amazon's Mechanical Turk.
Winter Mason,Siddharth Suri +1 more
TL;DR: It is shown that when taken as a whole Mechanical Turk can be a useful tool for many researchers, and how the behavior of workers compares with that of experts and laboratory subjects is discussed.
Journal ArticleDOI
Inside the Turk Understanding Mechanical Turk as a Participant Pool
Gabriele Paolacci,Jesse Chandler +1 more
TL;DR: The characteristics of Mechanical Turk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become Mechanical Turk workers and research participants, and how data quality on Mechanical Turk compares to that from other pools and depends on controllable and uncontrollable factors as mentioned in this paper.
Journal ArticleDOI
Data collection in a flat world: the strengths and weaknesses of mechanical turk samples
TL;DR: The authors compared Mechanical Turk participants with community and student samples on a set of personality dimensions and classic decision-making biases and found that MTurk participants are less extraverted and have lower self-esteem than other participants, presenting challenges for some research domains.
Journal ArticleDOI
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
TL;DR: This article found that participants on both platforms were more naive and less dishonest compared to MTurk participants, and ProA and CrowdFlower participants produced data quality that was higher than CF's and comparable to M-Turk's.
References
More filters
Journal ArticleDOI
Instructional manipulation checks: Detecting satisficing to increase statistical power
TL;DR: This paper proposed Instructional manipulation check (IMC), a new tool for detecting participants who are not following instructions and demonstrated how the inclusion of an IMC can increase statistical power and reliability of a dataset.
OtherDOI
Current Population Survey
TL;DR: The survey has been conducted for more than 50 years and has been used by the Census Bureau for the Bureau of Labor Statistics (BLS) as discussed by the authors to estimate employment, unemployment, earnings, hours of work, and other indicators.
Journal ArticleDOI
General performance on a numeracy scale among highly educated samples.
TL;DR: The results suggest that even highly educated participants have difficulty with relatively simple numeracy questions, thus replicating in part earlier studies that usual strategies for communicating numerical risk may be flawed.
Proceedings ArticleDOI
Who are the crowdworkers?: shifting demographics in mechanical turk
TL;DR: How the worker population has changed over time is described, shifting from a primarily moderate-income, U.S. based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers.
Journal ArticleDOI
Interpersonal expectancy effects: the first 345 studies
Robert Rosenthal,Donald B. Rubin +1 more
TL;DR: For instance, the authors summarizes the results of 345 experiments investigating interpersonal self-fulfilling prophecies, including those conducted by experimenters, teachers, employers, and therapists, with the goal of evaluating their substantive and methodological importance.