Crowdsourcing

About: Crowdsourcing is a research topic. Over the lifetime, 12889 publications have been published within this topic receiving 230638 citations.

And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions

Abstract: Crowdsourcing has become a popular and indispensable component of many problem-solving pipelines in the research literature, with crowd workers often treated as computational resources that can reliably solve problems that computers have trouble with, such as image labeling/classification, natural language processing, or document writing. Yet, obviously crowd workers are human, and long sequences of the same monotonous tasks might intuitively reduce the amount of good quality work done by the workers. Here we propose an investigation into how we can use diversions containing small amounts of entertainment to improve crowd workers' experiences. We call these small period of entertainment ``micro-diversions", which we hypothesize to provide timely relief to workers during long sequences of micro-tasks. We hope to improve productivity by retaining workers to work on our tasks longer and to either improve or retain the quality of work. We experimentally test micro-diversions on Amazon's Mechanical Turk, a large paid-crowdsourcing platform. We find that micro-diversions can significantly improve worker retention rate while retaining the same work quality.

Minimizing Efforts in Validating Crowd Answers

Abstract: In recent years, crowdsourcing has become essential in a wide range of Web applications. One of the biggest challenges of crowdsourcing is the quality of crowd answers as workers have wide-ranging levels of expertise and the worker community may contain faulty workers. Although various techniques for quality control have been proposed, a post-processing phase in which crowd answers are validated is still required. Validation is typically conducted by experts, whose availability is limited and who incur high costs. Therefore, we develop a probabilistic model that helps to identify the most beneficial validation questions in terms of both, improvement of result correctness and detection of faulty workers. Our approach allows us to guide the expert's work by collecting input on the most problematic cases, thereby achieving a set of high quality answers even if the expert does not validate the complete answer set. Our comprehensive evaluation using both real-world and synthetic datasets demonstrates that our techniques save up to 50% of expert efforts compared to baseline methods when striving for perfect result correctness. In absolute terms, for most cases, we achieve close to perfect correctness after expert input has been sought for only 20\% of the questions.

On Task Assignment for Real-Time Reliable Crowdsourcing

Abstract: With the rapid growth of mobile smartphone users, several commercial mobile companies have exploited crowd sourcing as an effective approach to collect and analyze data, to improve their services. In a crowd sourcing system, "human workers" are enlisted to perform small tasks, that are difficult to be automated, in return for some monetary compensation. This paper presents our crowd sourcing system that seeks to address the challenge of determining the most efficient allocation of tasks to the human crowd. The goal of our algorithm is to efficiently determine the most appropriate set of workers to assign to each incoming task, so that the real-time demands are met and high quality results are returned. We empirically evaluate our approach and show that our system effectively meets the requested demands, has low overhead and can improve the number of tasks processed under the defined constraints over 71% compared to traditional approaches.

Embedding AI and Crowdsourcing in the Big Data Lake

Abstract: Daniel E O'Leary examines the notion of the Big Data Lake and contrasts it with decision support-based data warehouses In addition, some of the risks of the emerging Lake concept that ultimately require data governance are analyzed O'Leary investigates using different AI and crowdsourcing (human intelligence) applications in that lake in order to integrate disparate data sources, facilitate master data management and analyze data quality Although data governance often is not seen as a technology issue, it is seen as a critical component of making the Big Data Lake "work"

Pervasive urban crowdsourcing: Visions and challenges

Abstract: Pervasive computing technologies can enable very flexible situated collaboration patterns among citizens and, via crowdsourcing, can promote a participatory way of contributing to the wealth and quality of life of our urban environments. This position paper firstly sketches a future vision of pervasive computing rich and crowdsourcing-enabled urban environments. Then, it presents several case studies showing how such environments can be of great use and highly impactful from both the individual and societal viewpoint. Finally, it discusses several open research challenges to be faced for these ideas to become reality.