scispace - formally typeset
Search or ask a question
Topic

Crowdsourcing

About: Crowdsourcing is a research topic. Over the lifetime, 12889 publications have been published within this topic receiving 230638 citations.


Papers
More filters
Journal ArticleDOI
TL;DR: In this article, the authors used multilevel regression analysis to estimate players' market value in the top five European leagues and a period of six playing seasons, and the regression results suggest that data-driven estimates of market value can overcome several of the crowd's practical limitations while producing comparably accurate numbers.

106 citations

Journal ArticleDOI
TL;DR: This work investigates the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study and collecting over 50,000 magnitude estimation judgments using crowdsourcing, and calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance.
Abstract: Magnitude estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting over 50,000 magnitude estimation judgments using crowdsourcing. Our analysis shows that magnitude estimation judgments can be reliably collected using crowdsourcing, are competitive in terms of assessor cost, and are, on average, rank-aligned with ordinal judgments made by expert relevance assessors.We explore the application of magnitude estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and magnitude estimation relevance shows substantial variation; in particular, the top systems ranked using magnitude estimation and ordinal judgments differ substantially. Analysis of the magnitude estimation scores shows that this effect is due in part to varying perceptions of relevance: different users have different perceptions of the impact of relative differences in document relevance. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold.

106 citations

Proceedings ArticleDOI
26 Apr 2014
TL;DR: This work introduces Twitch, a mobile phone application that asks users to make a micro-contribution each time they unlock their phone, and presents twitch crowdsourcing: crowdsourcing via quick contributions that can be completed in one or two seconds.
Abstract: To lower the threshold to participation in crowdsourcing, we present twitch crowdsourcing: crowdsourcing via quick contributions that can be completed in one or two seconds. We introduce Twitch, a mobile phone application that asks users to make a micro-contribution each time they unlock their phone. Twitch takes advantage of the common habit of turning to the mobile phone in spare moments. Twitch crowdsourcing activities span goals such as authoring a census of local human activity, rating stock photos, and extracting structured data from Wikipedia pages. We report a field deployment of Twitch where 82 users made 11,240 crowdsourcing contributions as they used their phone in the course of everyday life. The median Twitch activity took just 1.6 seconds, incurring no statistically distinguishable costs to unlock speed or cognitive load compared to a standard slide-to-unlock interface.

105 citations

Journal ArticleDOI
TL;DR: A methodology for determining the language skills of anonymous crowd workers is established that is more robust than simple surveying and is useful both to create bilingual dictionaries and to act as census of the bilingual speakers on MTurk.
Abstract: We present a large scale study of the languages spoken by bilingual workers on Mechanical Turk (MTurk). We establish a methodology for determining the language skills of anonymous crowd workers that is more robust than simple surveying. We validate workers’ self-reported language skill claims by measuring their ability to correctly translate words, and by geolocating workers to see if they reside in countries where the languages are likely to be spoken. Rather than posting a one-off survey, we posted paid tasks consisting of 1,000 assignments to translate a total of 10,000 words in each of 100 languages. Our study ran for several months, and was highly visible on the MTurk crowdsourcing platform, increasing the chances that bilingual workers would complete it. Our study was useful both to create bilingual dictionaries and to act as census of the bilingual speakers on MTurk. We use this data to recommend languages with the largest speaker populations as good candidates for other researchers who want to develop crowdsourced, multilingual technologies. To further demonstrate the value of creating data via crowdsourcing, we hire workers to create bilingual parallel corpora in six Indian languages, and use them to train statistical machine translation systems.

105 citations

Journal ArticleDOI
TL;DR: This paper examines the phenomena of online crowdsourcing from the perspectives of both volunteers and the campaign coordinator of Tomnod – an online mapping project that uses crowdsourcing to identify objects and places in satellite images.

105 citations


Network Information
Related Topics (5)
Social network
42.9K papers, 1.5M citations
87% related
User interface
85.4K papers, 1.7M citations
86% related
Deep learning
79.8K papers, 2.1M citations
85% related
Cluster analysis
146.5K papers, 2.9M citations
85% related
The Internet
213.2K papers, 3.8M citations
85% related
Performance
Metrics
No. of papers in the topic in previous years
YearPapers
2023637
20221,420
2021996
20201,250
20191,341
20181,396