The usage and adoption of cloud computing by small and medium businesses

Crowdsourcing relevance judgments for the evaluation of search engines is used increasingly to overcome the issue of scalability that hinders traditional approaches relying on a fixed group of trusted expert judges. However, the benefits of crowdsourcing come with risks due to the engagement of a self-forming group of individuals--the crowd, motivated by different incentives, who complete the tasks with varying levels of attention and success. This increases the need for a careful design of crowdsourcing tasks that attracts the right crowd for the given task and promotes quality work. In this paper, we describe a series of experiments using Amazon's Mechanical Turk, conducted to explore the `human' characteristics of the crowds involved in a relevance assessment task. In the experiments, we vary the level of pay offered, the effort required to complete a task and the qualifications required of the workers. We observe the effects of these variables on the quality of the resulting relevance labels, measured based on agreement with a gold set, and correlate them with self-reported measures of various human factors. We elicit information from the workers about their motivations, interest and familiarity with the topic, perceived task difficulty, and satisfaction with the offered pay. We investigate how these factors combine with aspects of the task design and how they affect the accuracy of the resulting relevance labels. Based on the analysis of 960 HITs and 2,880 HIT assignments resulting in 19,200 relevance labels, we arrive at insights into the complex interaction of the observed factors and provide practical guidelines to crowdsourcing practitioners. In addition, we highlight challenges in the data analysis that stem from the peculiarity of the crowdsourcing environment where the sample of individuals engaged in specific work conditions are inherently influenced by the conditions themselves.

An analysis of human factors and label accuracy in crowdsourcing relevance judgments

Examining the critical success factors of cloud computing adoption in the MSMEs by using ISM model

The Crowdsourcing for Search Evaluation Workshop (CSE 2010) was held on July 23, 2010 in Geneva, Switzerland, in conjunction with the 33rd Annual ACM SIGIR Conference. The workshop addressed the latest advances in theory and empirical methods in crowdsourcing for search evaluation, as well as novel applications of crowdsourcing for evaluating search systems. Three invited talks were presented, along with seven refereed papers. Proceedings from the workshop, along with presentation slides, have been made available online.

Crowdsourcing for search evaluation

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems
The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in
setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively
restricted set of users In this paper, we propose an alternative experimental methodology to laboratory-based user studies Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations

Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems

A professor and several PhD students at MIT examine the challenges and opportunities in human computation.

Heads in the cloud

Conducting focused but large-scale studies and experiments of user search behavior is highly desirable. Crowd-sourcing services such as the Amazon Mechanical Turk allow such studies to be conducted quickly and cheaply. They also have the potential to mitigate the problems associated with traditional experimental methods, in particular the relatively small and homogenous participant samples used in typical experiments. Our current research project addresses the relationship between searcher self-ecacy assessments and their strategies for conducting complex searches. In this work-in-progress paper, we describe our initial tests of using Amazon Mechanical Turk to conduct experiments in this area. We describe a platform for logging the actions taken by Turkers, and a questionnaire we conducted to assess search self-ecacy of average Turkers. Our results indicate Turkers have a similar range of search self-ecacy

/pdf/logging-the-search-self-efficacy-of-amazon-mechanical-4qwd19tjcd.pdf

Rajeev Nayak

Papers

Heads in the cloud

Logging the Search Self-Efficacy of Amazon Mechanical Turkers

h eads in the Cloud