Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

https://www.annualreviews.org/doi/pdf/10.1146/annurev-clinpsy-021815-093623

Conducting Clinical Research Using Crowdsourced Convenience Samples

The so-called “gig-economy” has been growing exponentially in numbers and importance in recent years but its impact on labour rights has been largely overlooked. Forms of work in the “gig-economy” include “crowdwork”, and “work-on-demand via apps”, under which the demand and supply of working activities is matched online or via mobile apps. These forms of work can provide a good match of job opportunities and allow flexible working schedules. However, they can also pave the way to a severe commodification of work. This paper discusses the implications of this commodification and advocates the full recognition of activities in the gig-economy as “work”.

/pdf/the-rise-of-the-just-in-time-workforce-on-demand-work-3z74p8d84v.pdf

The rise of the 'just-in-time workforce': On-demand work, crowdwork and labour protection in the 'gig-economy'

The widespread implementation of algorithmic technologies in organizations prompts questions about how algorithms may reshape organizational control. We use Edwards’ (1979) perspective of “conteste...

Algorithms at Work: The New Contested Terrain of Control

Software algorithms are changing how people work in an ever-growing number of fields, managing distributed human workers at a large scale. In these work settings, human jobs are assigned, optimized, and evaluated through algorithms and tracked data. We explore the impact of this algorithmic, data-driven management on human workers and work practices in the context of Uber and Lyft, new ridesharing services. Our findings from a qualitative study describe how drivers responded when algorithms assigned work, provided informational support, and evaluated their performance, and how drivers used online forums to socially make sense of the algorithm features. Implications and future work are discussed.

http://www.cs.cmu.edu/~mklee/materials/Publication/2015-CHI_algorithmic_management.pdf

Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers

The Information Age: Economy, Society and Culture. Volume I. The Rise of the Network Society

Crowdsourcing is a key current topic in CSCW. We build upon findings of a few qualitative studies of crowdworkers. We conducted an ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users. Using forum data we provide novel depth and detail on how the Turker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them. We show some of the key ways Turker Nation functions as a community and also look further into Turker-Requester relationships from the Turker perspective -- considering practical, emotional and moral aspects. Finally, following Star and Strauss [25] we analyse Turking as a form of invisible work. We do this to illustrate practical and ethical issues relating to working with Turkers and AMT, and to promote design directions to support Turkers and their relationships with Requesters.

Being a turker

In the so called 'post-truth' era, characterized by a loss of public trust in various institutions, and the rise of 'fake news' disseminated via the internet and social media, individuals may face uncertainty about the veracity of information available, whether it be satire or malicious hoax. We investigate attitudes to news delivered by social media, and subsequent verification strategies applied, or not applied, by individuals. A survey reveals that two thirds of respondents regularly consumed news via Facebook, and that one third had at some point come across fake news that they initially believed to be true. An analysis task involving news presented via Facebook reveals a diverse range of judgement forming strategies, with participants relying on personal judgements as to plausibility and scepticism around sources and journalistic style. This reflects a shift away from traditional methods of accessing the news, and highlights the difficulties in combating the spread of fake news.

/pdf/falling-for-fake-news-investigating-the-consumption-of-news-3p3m3mdk5v.pdf

Falling for Fake News: Investigating the Consumption of News via Social Media

Previous studies on Amazon Mechanical Turk (AMT), the most well-known marketplace for microtasks, show that the largest population of workers on AMT is U.S. based, while the second largest is based in India. In this paper, we present insights from an ethnographic study conducted in India to introduce some of these workers or "Turkers" -- who they are, how they work and what turking means to them. We examine the work they do to maintain their reputations and their work-life balance. In doing this, we illustrate how AMT's design practically impacts on turk-work. Understanding the "lived work" of crowdwork is a valuable first step for technology design.

Turk-Life in India

This paper examines how working in the global labour market of Amazon Mechanical Turk (AMT) impacts upon and is understood by two different groups of workers. To do this we report on two qualitative studies; one of US and another of Indian crowdworkers (or `Turkers') that we analysed from an ethnomethodological orientation. Our data is naturalistic and comes from a variety of sources--interviews, observations and forum posts--where Turkers describe their work, and their understandings of that work and of the transnational market they work in. We compare and contrast their situations, their reflections on the marketplace and its global reach, and we take a look at their understandings of one another. Our focus is on `the work to making turking work' (Martin et al. 2014). That is, the work that turkers do to organise and make sense of their work as they operate in the AMT marketplace, such that they can do so effectively. AMT is a technologically mediated marketplace--that is the distribution, completion and payment of work is done online, almost completely through the AMT platform. The design of the platform has important consequences for how Turkers experience and understand the market (including its global or transnational nature). We discuss how our findings relate to a variety of CSCW issues and provide an initial examination of how they relate to globalisation both as a mundanely experienced phenomenon and as a topic of academic interest. We finish the article by drawing on our own experiences in research and design to look at how technology can be used to intervene in a market like this to try to address imbalances in power and agency between employers and workers.

/pdf/turking-in-a-global-labour-market-38rnrsyawi.pdf

Turking in a Global Labour Market

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that of work environments; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. In this paper, we reveal the significant role of work environments in the shaping of crowd work. First, through a pilot study surveying the good and bad experiences workers had with UI elements in crowd work, we revealed the typical issues workers face. Based on these findings, we then deployed over 100 distinct microtasks on CrowdFlower, addressing workers in India and USA in two identical batches. These tasks emulate the good and bad UI element designs that characterize crowdsourcing microtasks. We recorded hardware specifics such as CPU speed and device type, apart from software specifics including the browsers used to complete tasks, operating systems on the device, and other properties that define the work environments of crowd workers. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. To confirm and validate our data-driven findings we then carried out semi-structured interviews with a sample of Indian and American crowd workers from this platform. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others. Based on our overall findings resulting from all the three studies, we introduce ModOp, a tool that helps to design crowdsourcing microtasks that are suitable for diverse crowd work environments. We empirically show that the use of ModOp results in reducing the cognitive load of workers, thereby improving their user experience without affecting the accuracy or task completion time.

/pdf/modus-operandi-of-crowd-workers-the-invisible-role-of-9zpxturigd.pdf

Neha Gupta

Papers

Being a turker

Falling for Fake News: Investigating the Consumption of News via Social Media

Turk-Life in India

Turking in a Global Labour Market

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments