Crowdsourcing systems on the World-Wide Web

Modern machine learning-based approaches to computer vision require very large databases of hand labeled images. Some contemporary vision systems already require on the order of millions of images for training (e.g., Omron face detector [9]). New Internet-based services allow for a large number of labelers to collaborate around the world at very low cost. However, using these services brings interesting theoretical and practical challenges: (1) The labelers may have wide ranging levels of expertise which are unknown a priori, and in some cases may be adversarial; (2) images may vary in their level of difficulty; and (3) multiple labels for the same image must be combined to provide an estimate of the actual label of the image. Probabilistic approaches provide a principled way to approach these problems. In this paper we present a probabilistic model and use it to simultaneously infer the label of each image, the expertise of each labeler, and the difficulty of each image. On both simulated and real data, we demonstrate that the model outperforms the commonly used "Majority Vote" heuristic for inferring image labels, and is robust to both noisy and adversarial labelers.

/pdf/whose-vote-should-count-more-optimal-integration-of-labels-4muru42i5x.pdf

Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

Computational thinking will influence everyone in every field of endeavour. This vision poses a new educational challenge for our society, especially for our children. In thinking about computing, we need to be attuned to the three drivers of our field: science, technology and society. Accelerating technological advances and monumental societal demands force us to revisit the most basic scientific questions of computing.

/pdf/computational-thinking-and-thinking-about-computing-4jgz0pu7z5.pdf

Computational thinking and thinking about computing

Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different "schools of thought" amongst the annotators, and can group together images belonging to separate categories.

/pdf/the-multidimensional-wisdom-of-crowds-35s8gd1f1u.pdf

The Multidimensional Wisdom of Crowds

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

/pdf/human-computation-a-survey-and-taxonomy-of-a-growing-field-ecwxj9twv7.pdf

Human computation: a survey and taxonomy of a growing field

The XMLTutor is an authoring tool that generically delivers personalised teaching on an arbitrary domain, which must be specified in Extensible Markup Language (XML). The tool was built to explore the consequences of using XML as the main ontological structure in an intelligent tutoring system (ITS).

XMLTutor-an authoring tool for factual domains

In this paper we introduce a new algorithm for secondary school timetabling, inspired by the classical bipartite graph edge colouring algorithm for basic class-teacher timetabling. We give practical methods for generating large sets of meetings that can be timetabled to run simultaneously, and for building actual timetables based on these sets. We report promising empirical results for one real-world instance of the problem.

Generalizing bipartite edge colouring to solve real instances of the timetabling problem

The logic tutor.

In this paper we introduce a new algorithm for secondary school timetabling, inspired by the classical bipartite graph edge colouring algorithm for basic class–teacher timetabling. We give practical methods for generating large sets of meetings that can be timetabled to run simultaneously, and for building actual timetables based on these sets. We report promising empirical results for one real-world instance of the problem.

/pdf/generalizing-bipartite-edge-colouring-to-solve-real-jxs56bxjzv.pdf

Generalizing Bipartite Edge Colouring to Solve Real Instances of the Timetabling Problem

This paper presents a Logic Tutor, a tool to support computer science students in their learning of logic, more specificall y in their learning of formal proofs. The current tool is equipped with a deduction system for propositional logic. However its modular conception makes it easy to change to another logic. Preliminary evaluation shows that this tool has a high educational value, thanks, among others features, to its simple, attractive interface and its specific error messages. The Logic Tutor will be integrated in our Logic teaching course in 200

David J. Abraham

Papers

XMLTutor-an authoring tool for factual domains

Generalizing bipartite edge colouring to solve real instances of the timetabling problem

The logic tutor.

Generalizing Bipartite Edge Colouring to Solve Real Instances of the Timetabling Problem

A tool to practice formal proofs