Methods to detect low quality data and its implication for psychological research
Reads0
Chats0
TLDR
This algorithm can be a promising tool to identify low quality or automated data via AMT or other online data collection platforms and be used as part of sensitivity analyses to warrant exclusion from further analyses.Abstract:
Web-based data collection methods such as Amazon's Mechanical Turk (AMT) are an appealing option to recruit participants quickly and cheaply for psychological research. While concerns regarding data quality have emerged with AMT, several studies have exhibited that data collected via AMT are as reliable as traditional college samples and are often more diverse and representative of noncollege populations. The development of methods to screen for low quality data, however, has been less explored. Omitting participants based on simple screening methods in isolation, such as response time or attention checks may not be adequate identification methods, with an inability to delineate between high or low effort participants. Additionally, problematic survey responses may arise from survey automation techniques such as survey bots or automated form fillers. The current project developed low quality data detection methods while overcoming previous screening limitations. Multiple checks were employed, such as page response times, distribution of survey responses, the number of utilized choices from a given range of scale options, click counts, and manipulation checks. This method was tested on a survey taken with an easily available plug-in survey bot, as well as compared to data collected by human participants providing both high effort and randomized, or low effort, answers. Identified cases can then be used as part of sensitivity analyses to warrant exclusion from further analyses. This algorithm can be a promising tool to identify low quality or automated data via AMT or other online data collection platforms.read more
Citations
More filters
Journal ArticleDOI
MTurk Research: Review and Recommendations:
TL;DR: The use of Amazon's Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019.
Journal ArticleDOI
Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks
TL;DR: The aim of this paper is to warn researchers of the threat posed by bots and to highlight practical strategies that can be used to detect and prevent these bots.
Journal ArticleDOI
Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices.
TL;DR: Three of the seven indices in this study appear to be the best estimators for detecting nonhuman response sets and every researcher working with online questionnaires could use them to screen for the presence of such invalid data.
Journal ArticleDOI
How Passion for Playing World of Warcraft Predicts In-Game Social Capital, Loneliness, and Wellbeing.
TL;DR: This paper sampled 300 frequent World of Warcraft players, recruited from online forums, and used structural equation modeling (SEM) to investigate the effects of their passion for playing WoW on in-game social capital, loneliness, and wellbeing.
Journal ArticleDOI
Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry
TL;DR: This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification, including demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods.
References
More filters
Journal ArticleDOI
Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?
TL;DR: Findings indicate that MTurk can be used to obtain high-quality data inexpensively and rapidly and the data obtained are at least as reliable as those obtained via traditional methods.
Book
The WEIRDest People in the World
TL;DR: A review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers.
Book
Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs
TL;DR: A practical primer on how to calculate and report effect sizes for t-tests and ANOVA's such that effect sizes can be used in a-priori power analyses and meta-analyses and a detailed overview of the similarities and differences between within- and between-subjects designs is provided.