Topic
Crowdsourcing
About: Crowdsourcing is a research topic. Over the lifetime, 12889 publications have been published within this topic receiving 230638 citations.
Papers published on a yearly basis
Papers
More filters
DOI•
07 Sep 2014TL;DR: This paper investigates the dependence of worker metrics for detecting spam on the quality of sentences in the dataset, and thequality of the target semantics, and shows that worker quality metrics can improve significantly when theQuality of these other aspects of semantic interpretation are considered.
Abstract: Crowdsourcing is often used to gather annotated data for training and evaluating computational systems that attempt to solve cognitive problems, such as understanding Natural Language sentences. Crowd workers are asked to perform semantic interpretation of sentences to establish a ground truth. This has always been done under the assumption that each task unit, e.g. each sentence, has a single correct interpretation that is contained in the ground truth. We have countered this assumption with CrowdTruth, and have shown that it can be better suited to tasks for which semantic interpretation is subjective. In this paper we investigate the dependence of worker metrics for detecting spam on the quality of sentences in the dataset, and the quality of the target semantics. We show that worker quality metrics can improve significantly when the quality of these other aspects of semantic interpretation are considered.
62 citations
TL;DR: It is suggested that a well-articulated problem statement reduces uncertainties of potential solvers and increases their willingness to participate, and that the ability of seekers to draft high-quality problem statements depends on the distance between the problem domain and their current knowledge stock.
62 citations
22 Oct 2013
TL;DR: High correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal is found, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.
Abstract: Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.
62 citations
TL;DR: This paper focuses on the use of implicit priming in the context of microtask crowdsourcing environments, where tasks are short and circumscribed, and how this can lead to significant performance gains.
Abstract: Although microtask platforms are desirable for their speed, scalability, and low cost, task performance varies greatly. Many researchers have focused on improving the quality of the work performed on such platforms. Priming uses implicit mechanisms to induce observable changes in behavior. Although priming has been effective in the laboratory, its use hasn't been explored extensively in software design, perhaps because the effects are often short-lived. In the context of microtask crowdsourcing environments, however, where tasks are short and circumscribed, temporary priming effects can lead to significant performance gains.
62 citations
10 Jul 2014
TL;DR: The Robot Management System (RMS) is presented, a novel framework for bringing robotic experiments to the web and validating the use of web-based crowdsourcing techniques for certain types of HRI evaluations.
Abstract: Human-Robot Interaction (HRI) is a rapidly expanding field of study that focuses on allowing non-roboticist users to naturally and effectively interact with robots. The importance of conducting extensive user studies has become a fundamental component of HRI research; however, due to the nature of robotics research, such studies often become expensive, time consuming, and limited to constrained demographics. In this work, we present the Robot Management System (RMS), a novel framework for bringing robotic experiments to the web. We present a detailed description of our open-source system and describe an initial trial of the RMS as a means of conducting user studies. Using a series of navigation and manipulation tasks with a PR2 robot, we compare three user study conditions: users that are co-present with the robot, users that are recruited to the university laboratory but control the robot from a different room, and remote web-based users. Our findings show little statistical differences between usability patterns across these groups, validating the use of web-based crowdsourcing techniques for certain types of HRI evaluations.
62 citations