Proceedings ArticleDOI
Crowd IQ: measuring the intelligence of crowdsourcing platforms
Michal Kosinski,Yoram Bachrach,Gjergji Kasneci,Jurgen Van-Gael,Thore Graepel +4 more
- pp 151-160
Reads0
Chats0
TLDR
It is shown that crowds composed of workers of high reputation achieve higher performance than low reputation crowds, and the effect of the amount of payment is non-monotone---both paying too much and too little affects performance.Abstract:
We measure crowdsourcing performance based on a standard IQ questionnaire, and examine Amazon's Mechanical Turk (AMT) performance under different conditions. These include variations of the payment amount offered, the way incorrect responses affect workers' reputations, threshold reputation scores of participating AMT workers, and the number of workers per task. We show that crowds composed of workers of high reputation achieve higher performance than low reputation crowds, and the effect of the amount of payment is non-monotone---both paying too much and too little affects performance. Furthermore, higher performance is achieved when the task is designed such that incorrect responses can decrease workers' reputation scores. Using majority vote to aggregate multiple responses to the same task can significantly improve performance, which can be further boosted by dynamically allocating workers to tasks in order to break ties.read more
Citations
More filters
Posted Content
How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing
TL;DR: An active learning/adaptive testing scheme based on a greedy minimization of expected model entropy is devised, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on the previous responses.
Proceedings Article
How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing
TL;DR: This article proposed a probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings, and devised an active learning/adaptive testing scheme based on a greedy minimization of expected model entropy, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on previous responses.
Book
The Measure of All Minds: Evaluating Natural and Artificial Intelligence
TL;DR: Using algorithmic information theory as a foundation, the book elaborates on the evaluation of perceptual, developmental, social, verbal and collective features and critically analyzes what the future of intelligence might look like.
Proceedings ArticleDOI
Reactive crowdsourcing
TL;DR: This work presents an approach to crowdsourcing which provides fine-level, powerful and flexible controls and progressively transforms these high level specifications into the features of a reactive execution environment that supports task planning, assignment and completion as well as performer monitoring and exclusion.
Proceedings Article
Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels
TL;DR: This work proposes three new annotation methodologies for gathering word senses where untrained annotators are allowed to use multiple labels and weight the senses, showing that given the appropriate annotation task, untrained workers can obtain at least as high agreement as annotators in a controlled setting and in aggregate generate equally as good of a sense labeling.
References
More filters
Proceedings ArticleDOI
Item-based collaborative filtering recommendation algorithms
TL;DR: This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms.
Book ChapterDOI
A Value for n-person Games
TL;DR: In this paper, an examination of elementary properties of a value for the essential case is presented, which is deduced from a set of three axioms, having simple intuitive interpretations.
Journal Article
Industry Report: Amazon.com Recommendations: Item-to-Item Collaborative Filtering.
TL;DR: This work compares three common approaches to solving the recommendation problem: traditional collaborative filtering, cluster models, and search-based methods, and their algorithm, which is called item-to-item collaborative filtering.
Journal ArticleDOI
Amazon.com recommendations: item-to-item collaborative filtering
TL;DR: Item-to-item collaborative filtering (ITF) as mentioned in this paper is a popular recommendation algorithm for e-commerce Web sites that scales independently of the number of customers and number of items in the product catalog.