Showing papers on "Crowdsourcing published in 2018"

Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

Abstract: We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. We solicit and verify questions and answers for this challenge through a 4-step crowdsourcing experiment. Our challenge dataset contains 6,500+ questions for 1000+ paragraphs across 7 different domains (elementary school science, news, travel guides, fiction stories, etc) bringing in linguistic diversity to the texts and to the questions wordings. On a subset of our dataset, we found human solvers to achieve an F1-score of 88.1%. We analyze a range of baselines, including a recent state-of-art reading comprehension system, and demonstrate the difficulty of this challenge, despite a high human performance. The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills.

Doing Your Research Project: A Guide for First-time Researchers

Abstract: Step-by-step advice on completing an outstanding research project. This is the market-leading book for anyone doing a research project for the first time. Clear, concise and extremely readable, this bestselling resource provides a practical, step-by-step guide from initial concept through to completion of your final written research report. Thoroughly updated but retaining its well-loved style, this 6th edition provides: A brand new chapter describing the benefits of using social media in research, including how to use blogs, sharing ideas on research community sites, crowdsourcing, LinkedIn, Twitter and Facebook Tips on using online tools such as Delicious, Mendeley, Dropbox, EndNote and RefWorks to manage and organize your research Guidance on searching efficiently and effectively online using Google Scholar, Google Books and library databases and on correctly citing online sources Advice on creating online surveys for your research project, plus new material on using Skype and Google Hangouts for online interviewing To support your learning, each chapter contains introductory key points, "Dead End" boxes to warn of pitfalls, "Success" checklists and further reading sections. This practical, no-nonsense guide is vital reading for all those embarking on undergraduate or postgraduate study in any discipline, and for professionals in such fields as social science, education and health.

Large scale crowdsourcing and characterization of twitter abusive behavior

Abstract: In recent years online social networks have suffered an increase in sexism, racism, and other types of aggressive and cyberbullying behavior, often manifesting itself through offensive, abusive, or hateful language. Past scientific work focused on studying these forms of abusive activity in popular online social networks, such as Facebook and Twitter. Building on such work, we present an eight month study of the various forms of abusive behavior on Twitter, in a holistic fashion. Departing from past work, we examine a wide variety of labeling schemes, which cover different forms of abusive behavior. We propose an incremental and iterative methodology that leverages the power of crowdsourcing to annotate a large collection of tweets with a set of abuse-related labels. By applying our methodology and performing statistical analysis for label merging or elimination, we identify a reduced but robust set of labels to characterize abuse-related tweets. Finally, we offer a characterization of our annotated dataset of 80 thousand tweets, which we make publicly available for further scientific exploration.

Exploring product design quality control and assurance under both traditional and crowdsourcing-based design environments

Abstract: Small and medium-sized enterprises face the challenges that they do not have enough employees and related resources to produce high-quality products with limited budget and time. The emergence of c...

Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

Abstract: Researchers in a variety of disciplines use Amazon’s crowdsourcing platform called Mechanical Turk as a way to collect data from a respondent pool that is much more diverse than a typical student s...

Crowdsourcing: A Review and Suggestions for Future Research

Abstract: As academic and practitioner studies on crowdsourcing have been building up since 2006, the subject itself has progressively gained in importance within the broad field of management. No systematic review on the topic has so far appeared in management journals, however; moreover, the field suffers from ambiguity in the topic's definition, which in turn has led to its largely unstructured evolution. The authors therefore investigate the existing body of knowledge on crowdsourcing systematically through a penetrating review in which the strengths and weakness of this literature stream are presented clearly and then future avenues of research are set out. The review is based on 121 scientific articles published between January 2006 and January 2015. The review recognizes that crowdsourcing is ingrained in two mainstream disciplines within the broader subject matter of innovation and management: (1) open innovation; and (2) co-creation. The review, in addition, also touches on several issues covered in other theoretical streams: (3) information systems management; (4) organizational theory and design; (5) marketing; and (6) strategy. The authors adopt a process perspective, applying the ‘Input–Process–Output’ framework to interpret research on crowdsourcing within the broad lines of: (1) Input (Problem/Task); (2) Process (session management; problem management; knowledge management; technology); and (3) Outcome (solution/completed task; seekers’ benefits; solvers’ benefits). This framework provides a detailed description of how the topic has evolved over time, and suggestions concerning the future direction of research are proposed in the form of research questions that are valuable for both academics and managers.

Flexibility in the gig economy: managing time on three online piecework platforms

Abstract: Gig economy platforms seem to provide extreme temporal flexibility to workers, giving them full control over how to spend each hour and minute of the day. What constraints do workers face when attempting to exercise this flexibility? We use 30 worker interviews and other data to compare three online piecework platforms with different histories and worker demographics: Mechanical Turk, MobileWorks, and CloudFactory. We find that structural constraints (availability of work and degree of worker dependence on the work) as well as cultural‐cognitive constraints (procrastination and presenteeism) limit worker control over scheduling in practice. The severity of these constraints varies significantly between platforms, the formally freest platform presenting the greatest structural and cultural‐cognitive constraints. We also find that workers have developed informal practices, tools, and communities to address these constraints. We conclude that focusing on outcomes rather than on worker control is a more fruitful way to assess flexible working arrangements.

Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions

Abstract: Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar—all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives, and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.

Building a Conversational Agent Overnight with Dialogue Self-Play

Abstract: We propose Machines Talking To Machines (M2M), a framework combining automation and crowdsourcing to rapidly bootstrap end-to-end dialogue agents for goal-oriented dialogues in arbitrary domains. M2M scales to new tasks with just a task schema and an API client from the dialogue system developer, but it is also customizable to cater to task-specific interactions. Compared to the Wizard-of-Oz approach for data collection, M2M achieves greater diversity and coverage of salient dialogue flows while maintaining the naturalness of individual utterances. In the first phase, a simulated user bot and a domain-agnostic system bot converse to exhaustively generate dialogue "outlines", i.e. sequences of template utterances and their semantic parses. In the second phase, crowd workers provide contextual rewrites of the dialogues to make the utterances more natural while preserving their meaning. The entire process can finish within a few hours. We propose a new corpus of 3,000 dialogues spanning 2 domains collected with M2M, and present comparisons with popular dialogue datasets on the quality and diversity of the surface forms and dialogue flows.

Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior

Abstract: In recent years, offensive, abusive and hateful language, sexism, racism and other types of aggressive and cyberbullying behavior have been manifesting with increased frequency, and in many online social media platforms. In fact, past scientific work focused on studying these forms in popular media, such as Facebook and Twitter. Building on such work, we present an 8-month study of the various forms of abusive behavior on Twitter, in a holistic fashion. Departing from past work, we examine a wide variety of labeling schemes, which cover different forms of abusive behavior, at the same time. We propose an incremental and iterative methodology, that utilizes the power of crowdsourcing to annotate a large scale collection of tweets with a set of abuse-related labels. In fact, by applying our methodology including statistical analysis for label merging or elimination, we identify a reduced but robust set of labels. Finally, we offer a first overview and findings of our collected and annotated dataset of 100 thousand tweets, which we make publicly available for further scientific exploration.

Chaotic Map-Based Anonymous User Authentication Scheme With User Biometrics and Fuzzy Extractor for Crowdsourcing Internet of Things

Abstract: The recent proliferation of mobile devices, such as smartphones and wearable devices has given rise to crowdsourcing Internet of Things (IoT) applications. E-healthcare service is one of the important services for the crowdsourcing IoT applications that facilitates remote access or storage of medical server data to the authorized users (for example, doctors, patients, and nurses) via wireless communication. As wireless communication is susceptible to various kinds of threats and attacks, remote user authentication is highly essential for a hazard-free use of these services. In this paper, we aim to propose a new secure three-factor user remote user authentication protocol based on the extended chaotic maps. The three factors involved in the proposed scheme are: 1) smart card; 2) password; and 3) personal biometrics. As the proposed scheme avoids computationally expensive elliptic curve point multiplication or modular exponentiation operation, it is lightweight and efficient. The formal security verification using the widely-accepted verification tool, called the ProVerif 1.93, shows that the presented scheme is secure. In addition, we present the formal security analysis using the both widely accepted real-or-random model and Burrows–Abadi–Needham logic. With the combination of high security and appreciably low communication and computational overheads, our scheme is very much practical for battery limited devices for the healthcare applications as compared to other existing related schemes.

Comparing Automatic and Human Evaluation of Local Explanations for Text Classification

Abstract: Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.

Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning

Abstract: End-to-end neural models show great promise towards building conversational agents that are trained from data and on-line experience using supervised and reinforcement learning. However, these models require a large corpus of dialogues to learn effectively. For goal-oriented dialogues, such datasets are expensive to collect and annotate, since each task involves a separate schema and database of entities. Further, the Wizard-of-Oz approach commonly used for dialogue collection does not provide sufficient coverage of salient dialogue flows, which is critical for guaranteeing an acceptable task completion rate in consumer-facing conversational agents. In this paper, we study a recently proposed approach for building an agent for arbitrary tasks by combining dialogue self-play and crowd-sourcing to generate fully-annotated dialogues with diverse and natural utterances. We discuss the advantages of this approach for industry applications of conversational agents, wherein an agent can be rapidly bootstrapped to deploy in front of users and further optimized via interactive learning from actual users of the system.

ZebraLancer: Private and Anonymous Crowdsourcing System atop Open Blockchain

Abstract: We design and implement the first private and anonymous decentralized crowdsourcing system ZebraLancer, and overcome two fundamental challenges of decentralizing crowdsourcing, i.e. data leakage and identity breach. First, our outsource-then-prove methodology resolves the tension between blockchain transparency and data confidentiality, which is critical in crowdsourcing use-case. ZebraLancer ensures: (i) a requester will not pay more than what data deserve, according to a policy announced when her task is published via the blockchain; (ii) each worker indeed gets a payment based on the policy, if he submits data to the blockchain; (iii) the above properties are realized not only without a central arbiter, but also without leaking the data to the open blockchain. Furthermore, the transparency of blockchain allows one to infer private information about workers and requesters through their participation history. On the other hand, allowing anonymity will enable a malicious worker to submit multiple times to reap rewards. ZebraLancer overcomes this problem by allowing anonymous requests/submissions without sacrificing the accountability. The idea behind is a subtle linkability: if a worker submits twice to a task, anyone can link the submissions, or else he stays anonymous and unlinkable across tasks. To realize this delicate linkability, we put forward a novel cryptographic concept, i.e. the common-prefix-linkable anonymous authentication. We remark the new anonymous authentication scheme might be of independent interest. Finally, we implement our protocol for a common image annotation task and deploy it in a test net of Ethereum. The experiment results show the applicability of our protocol with the existing real-world blockchain.

Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research

Abstract: This survey provides a comprehensive overview of the landscape of crowdsourcing research, targeted at the machine learning community. We begin with an overview of the ways in which crowdsourcing can be used to advance machine learning research, focusing on four application areas: 1) data generation, 2) evaluation and debugging of models, 3) hybrid intelligence systems that leverage the complementary strengths of humans and machines to expand the capabilities of AI, and 4) crowdsourced behavioral experiments that improve our understanding of how humans interact with machine learning systems and technology more broadly. We next review the extensive literature on the behavior of crowdworkers themselves. This research, which explores the prevalence of dishonesty among crowdworkers, how workers respond to both monetary incentives and intrinsic forms of motivation, and how crowdworkers interact with each other, has immediate implications that we distill into best practices that researchers should follow when using crowdsourcing in their own research. We conclude with a discussion of additional tips and best practices that are crucial to the success of any project that uses crowdsourcing, but rarely mentioned in the literature.

Cognitive Biases in Crowdsourcing

Abstract: Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases. This paper investigates the prevalence and effect size of a range of common cognitive biases on a standard relevance judgment task. Our experiments are based on three sizable publicly available document collections and note significant detrimental effects on annotation quality, system ranking and the performance of derived rankers when task design does not account for such biases.

Task Allocation in Spatial Crowdsourcing: Current State and Future Directions

Abstract: Spatial crowdsourcing (SC) is an emerging paradigm of crowdsourcing, which commits workers to move to some particular locations to perform spatio-temporal-relevant tasks (e.g., sensing and activity organization). Task allocation or worker selection is a significant problem that may impact the quality of completion of SC tasks. Based on a conceptual model and generic framework of SC task allocation, this paper first gives a review of the current state of research in this field, including single task allocation, multiple task allocation, low-cost task allocation, and quality-enhanced task allocation. We further investigate the future trends and open issues of SC task allocation, including skill-based task allocation, group recommendation and collaboration, task composition and decomposition, and privacy-preserving task allocation. Finally, we discuss the practical issues on real-world deployment as well as the challenges for large-scale user study in SC task allocation.

Multi-Objective Optimization Based Allocation of Heterogeneous Spatial Crowdsourcing Tasks

Abstract: With the rapid development of mobile networks and the proliferation of mobile devices, spatial crowdsourcing, which refers to recruiting mobile workers to perform location-based tasks, has gained emerging interest from both research communities and industries. In this paper, we consider a spatial crowdsourcing scenario: in addition to specific spatial constraints, each task has a valid duration, operation complexity, budget limitation, and the number of required workers. Each volunteer worker completes assigned tasks while conducting his/her routine tasks. The system has a desired task probability coverage and budget constraint. Under this scenario, we investigate an important problem, namely heterogeneous spatial crowdsourcing task allocation (HSC-TA), which strives to search a set of representative Pareto-optimal allocation solutions for the multi-objective optimization problem, such that the assigned task coverage is maximized and incentive cost is minimized simultaneously. To accommodate the multi-constraints in heterogeneous spatial crowdsourcing, we build a worker mobility behavior prediction model to align with allocation process. We prove that the HSC-TA problem is NP-hard. We propose effective heuristic methods, including multi-round linear weight optimization and enhanced multi-objective particle swarm optimization algorithms to achieve adequate Pareto-optimal allocation. Comprehensive experiments on both real-world and synthetic data sets clearly validate the effectiveness and efficiency of our proposed approaches.

A Survey on Security, Privacy, and Trust in Mobile Crowdsourcing

Abstract: With the popularity of sensor-rich mobile devices (e.g., smart phones and wearable devices), mobile crowdsourcing (MCS) has emerged as an effective method for data collection and processing. Compared with traditional wireless sensor networking, MCS holds many advantages such as mobility, scalability, cost-efficiency, and human intelligence. However, MCS still faces many challenges with regard to security, privacy, and trust. This paper provides a survey of these challenges and discusses potential solutions. We analyze the characteristics of MCS, identify its security threats, and outline essential requirements on a secure, privacy-preserving, and trustworthy MCS system. Further, we review existing solutions based on these requirements and compare their pros and cons. Finally, we point out open issues and propose some future research directions.

Practical Attribute-Based Multi-Keyword Search Scheme in Mobile Crowdsourcing

Abstract: Cloud-based mobile crowdsourcing has been an attractive solution to provide data storage and share services for resource-limited mobile devices in a privacy-preserving manner, but how to enable mobile users to issue search queries and achieve fine-grained access control over ciphertexts simultaneously is still a big challenge for various circumstances. Although the ciphertext-policy attribute-based keyword search technology combining attribute-based encryption with searchable encryption has become a hot research topic, it just deals with equivalent attributes rather than more practical attribute comparisons, like “greater than” or “less than.” In this paper, we devise a practical cryptographic primitive called attribute-based multi-keyword search scheme to support comparable attributes through utilizing 0-encoding and 1-encoding. Formal security analysis proves that our scheme is selectively secure against chosen-keyword attack in generic bilinear group model and extensive experiments using real-world dataset demonstrate that our scheme can drastically decrease both computational and storage costs.

Mapping of Crowdsourcing in Health: Systematic Review

Abstract: Crowdsourcing involves obtaining ideas, needed services, or content by soliciting Web-based contributions from a crowd. The 4 types of crowdsourced tasks (problem solving, data processing, surveillance or monitoring, and surveying) can be applied in the 3 categories of health (promotion, research, and care). : This study aimed to map the different applications of crowdsourcing in health to assess the fields of health that are using crowdsourcing and the crowdsourced tasks used. We also describe the logistics of crowdsourcing and the characteristics of crowd workers. : MEDLINE, EMBASE, and ClinicalTrials.gov were searched for available reports from inception to March 30, 2016, with no restriction on language or publication status. : We identified 202 relevant studies that used crowdsourcing, including 9 randomized controlled trials, of which only one had posted results at ClinicalTrials.gov. Crowdsourcing was used in health promotion (91/202, 45.0%), research (73/202, 36.1%), and care (38/202, 18.8%). The 4 most frequent areas of application were public health (67/202, 33.2%), psychiatry (32/202, 15.8%), surgery (22/202, 10.9%), and oncology (14/202, 6.9%). Half of the reports (99/202, 49.0%) referred to data processing, 34.6% (70/202) referred to surveying, 10.4% (21/202) referred to surveillance or monitoring, and 5.9% (12/202) referred to problem-solving. Labor market platforms (eg, Amazon Mechanical Turk) were used in most studies (190/202, 94%). The crowd workers’ characteristics were poorly reported, and crowdsourcing logistics were missing from two-thirds of the reports. When reported, the median size of the crowd was 424 (first and third quartiles: 167-802); crowd workers’ median age was 34 years (32-36). Crowd workers were mainly recruited nationally, particularly in the United States. For many studies (58.9%, 119/202), previous experience in crowdsourcing was required, and passing a qualification test or training was seldom needed (11.9% of studies; 24/202). For half of the studies, monetary incentives were mentioned, with mainly less than US $1 to perform the task. The time needed to perform the task was mostly less than 10 min (58.9% of studies; 119/202). Data quality validation was used in 54/202 studies (26.7%), mainly by attention check questions or by replicating the task with several crowd workers. : The use of crowdsourcing, which allows access to a large pool of participants as well as saving time in data collection, lowering costs, and speeding up innovations, is increasing in health promotion, research, and care. However, the description of crowdsourcing logistics and crowd workers’ characteristics is frequently missing in study reports and needs to be precisely reported to better interpret the study findings and replicate them.

Take-over requests in highly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays

Abstract: An important research question in the domain of highly automated driving is how to aid drivers in transitions between manual and automated control. Until highly automated cars are available, knowledge on this topic has to be obtained via simulators and self-report questionnaires. Using crowdsourcing, we surveyed 1692 people on auditory, visual, and vibrotactile take-over requests (TORs) in highly automated driving. The survey presented recordings of auditory messages and illustrations of visual and vibrational messages in traffic scenarios of various urgency levels. Multimodal TORs were the most preferred option in high-urgency scenarios. Auditory TORs were the most preferred option in low-urgency scenarios and as a confirmation message that the system is ready to switch from manual to automated mode. For low-urgency scenarios, visual-only TORs were more preferred than vibration-only TORs. Beeps with shorter interpulse intervals were perceived as more urgent, with Stevens’ power law yielding an accurate fit to the data. Spoken messages were more accepted than abstract sounds, and the female voice was more preferred than the male voice. Preferences and perceived urgency ratings were similar in middle- and high-income countries. In summary, this international survey showed that people’s preferences for TOR types in highly automated driving depend on the urgency of the situation.