The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.

/pdf/imagenet-a-large-scale-hierarchical-image-database-1e8f663s1h.pdf

ImageNet: A large-scale hierarchical image database

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the 5 years of the challenge, and propose future directions and improvements.

/pdf/imagenet-large-scale-visual-recognition-challenge-caky5nmmix.pdf

ImageNet Large Scale Visual Recognition Challenge

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.

This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.

/pdf/the-pascal-visual-object-classes-voc-challenge-ugvl0p2h51.pdf

The Pascal Visual Object Classes (VOC) Challenge

Basics of qualitative research: Grounded theory procedures and techniques

Recent years have seen a rapid proliferation of mass-market consumer software that takes inspiration from video games. Usually summarized as "gamification", this trend connects to a sizeable body of existing concepts and research in human-computer interaction and game studies, such as serious games, pervasive games, alternate reality games, or playful design. However, it is not clear how "gamification" relates to these, whether it denotes a novel phenomenon, and how to define it. Thus, in this paper we investigate "gamification" and the historical origins of the term in relation to precursors and similar concepts. It is suggested that "gamified" applications provide insight into novel, gameful phenomena complementary to playful phenomena. Based on our research, we propose a definition of "gamification" as the use of game design elements in non-game contexts.

/pdf/from-game-design-elements-to-gamefulness-defining-1wnas1kptm.pdf

From game design elements to gamefulness: defining "gamification"

We introduce a new interactive system: a game that is fun and can be used to create valuable output. When people play the game they help determine the contents of images by providing meaningful labels for them. If the game is played as much as popular online games, we estimate that most images on the Web can be labeled in a few months. Having proper labels associated with each image on the Web would allow for more accurate image search, improve the accessibility of sites (by providing descriptions of images to visually impaired individuals), and help users block inappropriate images. Our system makes a significant contribution because of its valuable output and because of the way it addresses the image-labeling problem. Rather than using computer vision techniques, which don't work well enough, we encourage people to do the work by taking advantage of their desire to be entertained.

/pdf/labeling-images-with-a-computer-game-4ame96zv1j.pdf

Labeling images with a computer game

Data generated as a side effect of game play also solves computational problems and trains AI algorithms

https://dl.acm.org/doi/pdf/10.1145/1378704.1378719

Designing games with a purpose

Social applications on the web let users track and follow the activities of a large number of others regardless of location or affiliation. There is a potential for this transparency to radically improve collaboration and learning in complex knowledge-based activities. Based on a series of in-depth interviews with central and peripheral GitHub users, we examined the value of transparency for large-scale distributed collaborations and communities of practice. We find that people make a surprisingly rich set of social inferences from the networked activity information in GitHub, such as inferring someone else's technical goals and vision when they edit code, or guessing which of several similar projects has the best chance of thriving in the long term. Users combine these inferences into effective strategies for coordinating work, advancing technical skills and managing their reputation.

/pdf/social-coding-in-github-transparency-and-collaboration-in-an-1hby48swok.pdf

Social coding in GitHub: transparency and collaboration in an open software repository

Software algorithms are changing how people work in an ever-growing number of fields, managing distributed human workers at a large scale. In these work settings, human jobs are assigned, optimized, and evaluated through algorithms and tracked data. We explore the impact of this algorithmic, data-driven management on human workers and work practices in the context of Uber and Lyft, new ridesharing services. Our findings from a qualitative study describe how drivers responded when algorithms assigned work, provided informational support, and evaluated their performance, and how drivers used online forums to socially make sense of the algorithm features. Implications and future work are discussed.

http://www.cs.cmu.edu/~mklee/materials/Publication/2015-CHI_algorithmic_management.pdf

Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers

Open source software is commonly portrayed as a meritocracy, where decisions are based solely on their technical merit. However, literature on open source suggests a complex social structure underlying the meritocracy. Social work environments such as GitHub make the relationships between users and between users and work artifacts transparent. This transparency enables developers to better use information such as technical value and social connections when making work decisions. We present a study on open source software contribution in GitHub that focuses on the task of evaluating pull requests, which are one of the primary methods for contributing code in GitHub. We analyzed the association of various technical and social measures with the likelihood of contribution acceptance. We found that project managers made use of information signaling both good technical contribution practices for a pull request and the strength of the social connection between the submitter and project manager when evaluating pull requests. Pull requests with many comments were much less likely to be accepted, moderated by the submitter's prior interaction in the project. Well-established projects were more conservative in accepting pull requests. These findings provide evidence that developers use both technical and social information when evaluating potential contributions to open source software projects.

/pdf/influence-of-social-and-technical-factors-for-evaluating-2ksrhmzybz.pdf

Laura Dabbish

Papers

Labeling images with a computer game

Designing games with a purpose

Social coding in GitHub: transparency and collaboration in an open software repository

Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers

Influence of social and technical factors for evaluating contribution in GitHub