Crowdsourcing systems on the World-Wide Web

Modern machine learning-based approaches to computer vision require very large databases of hand labeled images. Some contemporary vision systems already require on the order of millions of images for training (e.g., Omron face detector [9]). New Internet-based services allow for a large number of labelers to collaborate around the world at very low cost. However, using these services brings interesting theoretical and practical challenges: (1) The labelers may have wide ranging levels of expertise which are unknown a priori, and in some cases may be adversarial; (2) images may vary in their level of difficulty; and (3) multiple labels for the same image must be combined to provide an estimate of the actual label of the image. Probabilistic approaches provide a principled way to approach these problems. In this paper we present a probabilistic model and use it to simultaneously infer the label of each image, the expertise of each labeler, and the difficulty of each image. On both simulated and real data, we demonstrate that the model outperforms the commonly used "Majority Vote" heuristic for inferring image labels, and is robust to both noisy and adversarial labelers.

/pdf/whose-vote-should-count-more-optimal-integration-of-labels-4muru42i5x.pdf

Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

Computational thinking will influence everyone in every field of endeavour. This vision poses a new educational challenge for our society, especially for our children. In thinking about computing, we need to be attuned to the three drivers of our field: science, technology and society. Accelerating technological advances and monumental societal demands force us to revisit the most basic scientific questions of computing.

/pdf/computational-thinking-and-thinking-about-computing-4jgz0pu7z5.pdf

Computational thinking and thinking about computing

Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different "schools of thought" amongst the annotators, and can group together images belonging to separate categories.

/pdf/the-multidimensional-wisdom-of-crowds-35s8gd1f1u.pdf

The Multidimensional Wisdom of Crowds

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

/pdf/human-computation-a-survey-and-taxonomy-of-a-growing-field-ecwxj9twv7.pdf

Human computation: a survey and taxonomy of a growing field

Motivated by the dynamics of the ever-popular online movie rental business, we study a range of assignment problems in rental markets. The assignment problems associated with rental markets possess a rich mathematical structure and are closely related to many well-studied one-sided matching problems. We formalize and characterize the assignment problems in rental markets in terms of one-sided matching problems, and consider several solution concepts for these problems. In order to evaluate and compare these solution concepts (and the corresponding algorithms), we define some “value” functions to capture our objectives, which include fairness, efficiency and social welfare. Then, we bound the value of the output of these algorithms in terms of the chosen value functions.

We also consider models of rental markets corresponding to static, online, and dynamic customer valuations. We provide several constant-factor approximation algorithms for the assignment problem, as well as hardness of approximation results for the different models. Finally, we describe some experiments with a discrete event simulator compare the various algorithms in a practical setting, and present some interesting experimental results.

/pdf/assignment-problems-in-rental-markets-c40nvcpkg1.pdf

Assignment problems in rental markets

We consider a matching market, in which the aim is to maintain a popular matching between a set of applicants and a set of posts, where each applicant has a preference list that ranks some subset of acceptable posts. The setting is dynamic: applicants and posts can enter and leave the market, and applicants can also change their preferences arbitrarily. After any change, the current matching may no longer be popular, in which case, we are required to update it. However, our model demands that we can switch from one matching to another only if there is consensus among the applicants to agree to the switch. Hence, we need to update via a voting path, which is a sequence of matchings, each more popular than its predecessor, that ends in a popular matching. In this paper, we show that, as long as some popular matching exists, there is a 2-step voting path from any given matching to some popular matching. Furthermore, given any popular matching, we show how to find a shortest-length such voting path in linear time.

Dynamic matching markets and voting paths

We consider a matching market, in which the aim is to maintain a popular matching between a set of applicants and a set of posts, where each applicant has a preference list that ranks some subset of acceptable posts. The setting is dynamic: applicants and posts can enter and leave the market, and applicants can also change their preferences arbitrarily. After any change, the current matching may no longer be popular, in which case, we are required to update it. However, our model demands that we can switch from one matching to another only if there is consensus among the applicants to agree to the switch. Hence, we need to update via a voting path, which is a sequence of matchings, each more popular than its predecessor, that ends in a popular matching. In this paper, we show that, as long as some popular matching exists, there is a 2-step voting path from any given matching to some popular matching. Furthermore, given any popular matching, we show how to find a shortest-length such voting path in linear time

A market consists of buyers and sellers of some commodity, say a DVD movie. In this thesis, we assume the role of market operator. Our goal is to ensure that the market has certain desirable properties, including truthfulness, fairness and stability. We explore how to achieve these properties by designing rules for how the participants (buyers and sellers) can interact. We also explore how to efficiently compute the outcome of large numbers of participants interacting at once. 
The main type of market we study is called a matching market. We study several particular matching markets, including keyword auction, kidney exchange and stable roommate markets. In each of these cases, the aim is to match the participants to each other, somehow taking into account their preferences for one another. Our results focus on properties of the matching process and the design of efficient algorithms for finding various types of matchings. In particular, we present new polynomial time algorithms for finding matchings that have one of the following properties: popularity, rank-maximality and fairness. We also give an efficient algorithm for clearing large swap markets such as kidney exchanges. Finally, we present a new decomposition technique for designing keyword auctions.

Matching markets: design and analysis

The Logic-ITA is an Intelligent Teaching Assistant system for the teaching/learning of propositional logic. The system is tailored to two different types of uses: for students, it is an autonomous Intelligent Tutoring System, whilst for teachers, it includes functionality to set up learning levels, adjust parameters for progressing through these levels, monitor the class' progress and collect data. This paper presents some characteristics of the Web-based version of the Logic-ITA. In particular, the web-based version allows a centralisation on the server of the student models, which contain all the users' individual information related to their learning. This means that all the adaptation features of the Logic Tutor are maintained, whilst providing (1) students with a large degree of flexibility in terms of when and where they choose to use the tool, and (2) teachers with the possibility to monitor accurately the results and statistics.

David J. Abraham

Papers

Assignment problems in rental markets

Dynamic matching markets and voting paths

Dynamic matching markets and voting paths

Matching markets: design and analysis

Adaptation in the Web-Based Logic-ITA