Beyond the Turk: Alternative platforms for crowdsourcing behavioral research

TLDR: This article found that participants on both platforms were more naive and less dishonest compared to MTurk participants, and ProA and CrowdFlower participants produced data quality that was higher than CF's and comparable to M-Turk's.

About: This article is published in Journal of Experimental Social Psychology.The article was published on 2017-05-01 and is currently open access. It has received 1537 citations till now. The article focuses on the topics: Population.

Figures

Figure 2. Percent of participants included in each sample by exclusion policy (lenient = excluding participants that failed more than one ACQ; strict = excluding participants that failed any ACQ).

Figure 1. Response rates across samples.

Table 1. Comparison of platforms’ properties and features (extracted from the platforms’ websites)

Table 4. Percentage of participants reporting using platforms more than “a few times.” Uses MTurk Uses CF Uses ProA

Table 5. Effect sizes across samples (p-values are in parantheses). Asian disease

Figure 10. Response rates between the two platforms.

Frequently asked questions

Q1. What have the authors contributed in "Beyond the turk: alternative platforms for crowdsourcing behavioral research" ?

After surveying several options, the authors empirically examined two such platforms, CrowdFlower ( CF ) and Prolific Academic ( ProA ). Across the three platforms, CF provided the best response rate, but CF participants failed more attention-check questions and did not reproduce known effects replicated on ProA and MTurk.