Proceedings ArticleDOI
Worker types and personality traits in crowdsourcing relevance labels
Gabriella Kazai,Jaap Kamps,Natasa Milic-Frayling +2 more
- pp 1941-1944
Reads0
Chats0
TLDR
This paper uses behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent, and the `Big Five' personality dimensions.Abstract:
Crowdsourcing platforms offer unprecedented opportunities for creating evaluation benchmarks, but suffer from varied output quality from crowd workers who possess different levels of competence and aspiration. This raises new challenges for quality control and requires an in-depth understanding of how workers' characteristics relate to the quality of their work.In this paper, we use behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent. Using data collected from workers engaged in the crowdsourced evaluation of the INEX 2010 Book Track Prove It task, we relate the worker types to label accuracy and personality trait information along the `Big Five' personality dimensions.We expect that these new insights about the types of crowd workers and the quality of their work will inform how to design HITs to attract the best workers to a task and explain why certain HIT designs are more effective than others.read more
Citations
More filters
Journal ArticleDOI
Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample
TL;DR: The results support the accumulating evidence for the promise of crowdsource recruitment for online experiments, including factorial surveys, according to demographic differences and indicators of data quality.
Proceedings ArticleDOI
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
TL;DR: The prevalent malicious activity on crowdsourcing platforms is analyzed and different types of workers in the crowd are defined, a method to measure malicious activity is proposed, and guidelines for the efficient design of crowdsourced surveys are presented.
Journal ArticleDOI
Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions
TL;DR: In this paper, a survey of quality in the context of crowdsourcing along several dimensions is presented to define and characterize it and to understand the current state-of-the-art.
Proceedings ArticleDOI
Quality through flow and immersion: gamifying crowdsourced relevance assessments
TL;DR: In this paper, the authors proposed the use of a game in order to attract and retain a larger share of reliable workers to frequently-requested crowdsourcing tasks such as relevance assessments and clustering.
Journal ArticleDOI
Increasing cheat robustness of crowdsourcing tasks
TL;DR: This work takes a different approach by investigating means of a priori making crowdsourced tasks more resistant against cheaters by investigating methods of identifying workers primarily interested in producing quick generic answers.
References
More filters
Journal ArticleDOI
Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German
TL;DR: In this paper, the Big Five Inventory (BFI-44) was abbreviated to a 10-item version, the BFI-10, which was developed simultaneously in several samples in both English and German.
Proceedings ArticleDOI
Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
TL;DR: This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
Proceedings ArticleDOI
Crowdsourcing user studies with Mechanical Turk
TL;DR: Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
Book
Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business
TL;DR: The idea of crowdsourcing was first identified by journalist Jeff Howe in a June 2006 Wired article as mentioned in this paper, which describes the process by which the power of the many can be leveraged to accomplish feats that were once the province of the specialized few.