Fast, Free, and Targeted: Reddit as a Source for Recruiting Participants Online
TL;DR: The article discusses current online recruitment sources and their limitations, provides an overview of Reddit, validates its use for research purposes, examines participation data from previous studies which recruited through Reddit, and suggests guidelines that can improve recruitment and retention rates for scientists looking to use Reddit for their research.
Abstract: Recruiting participants is a necessary step in many studies. With the advent of online research techniques, scientists are looking for new places where participants can be recruited online, in orde...
Citations
More filters
TL;DR: Empirical support is provided for the proposed multivariate model and the importance of trust in science is underlined in explaining the different levels of compliance with COVID-19 prevention guidelines.
Abstract: The coronavirus pandemic is one of the biggest health crises of our time. In response to this global problem, various institutions around the world had soon issued evidence-based prevention guidelines. However, these guidelines, which were designed to slow the spread of COVID-19 and contribute to public well-being, are (deliberately) disregarded by some individuals. In the present study, we aimed to develop and test a multivariate model that could help us identify individual characteristics that make a person more/less likely to comply with COVID-19 prevention guidelines. A total of 525 attentive participants completed the online survey. The results of structural equation modeling (SEM) show that COVID-19 risk perception and trust in science both independently predict compliance with COVID-19 prevention guidelines, while the remaining variables in the model (political conservatism, religious orthodoxy, conspiracy ideation and intellectual curiosity) do so via the mediating role of trust in science. The described model exhibited an acceptable fit (χ2(1611) = 2485.84, p < .001, CFI = .91, RMSEA = .032, SRMR = .055). These findings thus provide empirical support for the proposed multivariate model and underline the importance of trust in science in explaining the different levels of compliance with COVID-19 prevention guidelines.
329 citations
TL;DR: Assessment of the independent associations between active versus passive SMU and depressive symptoms in adults 18-49 of age found each one-point increase in active SMU was associated with a 15 percent decrease in depressive symptoms, however, in the same multivariable model, eachone-point decrease in activeSMU was linked to a 15% decrease in depression symptoms.
Abstract: Social media allows users to explore self-identity and express emotions or thoughts. Research looking into the association between social media use (SMU) and mental health outcomes, such as anxiety or depressive symptoms, have produced mixed findings. These contradictions may best be addressed by examining different patterns of SMU as they relate to depressive symptomatology. We sought to assess the independent associations between active versus passive SMU and depressive symptoms. For this, we conducted an online survey of adults 18-49 of age. Depressive symptoms were measured using the Patient-Reported Outcomes Measurement Information System brief depression scale. We measured active and passive SMU with previously developed items. Factor analysis was used to explore the underlying factor structure. Then, we used ordered logistic regression to assess associations between both passive and active SMU and depressive symptoms while controlling for sociodemographic covariates. Complete data were received from 702 participants. Active and passive SMU items loaded on separate factors. In multivariable analyses that controlled for all covariates, each one-point increase in passive SMU was associated with a 33 percent increase in depressive symptoms (adjusted odds ratio [AOR] = 1.33, 95 percent confidence interval [CI] = 1.17-1.51). However, in the same multivariable model, each one-point increase in active SMU was associated with a 15 percent decrease in depressive symptoms (AOR = 0.85, 95 percent CI = 0.75-0.96). To inform interventions, future research should determine directionality of these associations and investigate related factors.
166 citations
TL;DR: The authors examines discussions on the reddit.com forum r/atheism in comparison with rhetoric found in contemporary atheist organizations and among leading figures within the atheist movement, and concludes that "the majority of the participants in r/theism are either agnostic or non-believing".
Abstract: This article examines discussions on the reddit.com forum r/atheism in comparison with rhetoric found in contemporary atheist organizations and among leading figures within the atheist movement. We...
57 citations
Cites background from "Fast, Free, and Targeted: Reddit as..."
...While later studies on the site has found that there is a fairly equal gender distribution on the site, US reddit users are on average likely to lean liberal, be higher education, and be classified as White non-Hispanics, when compared to the general US population (Shatz, 2017)....
[...]
TL;DR: This study investigated whether the psychological impact of COVID19-related circumstantial changes was moderated by introversion, based on outcome measures across psychosocial, cognitive, and affective domains, and highlighted the possibility that living with others may have protective and detrimental effects on different domains of mental health during the CO VID19 pandemic.
Abstract: The coronavirus disease 2019 (COVID19) pandemic has effected the implementation of social distancing and lockdown measures across the globe, and the psychological impact of associated life changes is experienced more severely by some individuals than others. Anecdotal evidence points to a common belief among the general public that introverts are faring better than their extraverted counterparts to this end. However, the claim lacks empirical research, and seems counterintuitive when the broader literature on the association between introversion and mental health is considered. The current study investigated whether the psychological impact of COVID19-related circumstantial changes was moderated by introversion, based on outcome measures across psychosocial, cognitive, and affective domains. The role of several demographic factors in determining COVID19-related mental health symptoms was also examined. One hundred and fourteen individuals (64 USA residents) completed measures of introversion, and reported on the extent to which they experienced loneliness, anxiety, depression and cognitive impairments as a function of COVID19-related circumstantial changes. Results showed that introversion predicted more severe loneliness, anxiety, and depression experienced as a function of COVID19-related circumstantial changes, but not cognitive impairments. Among the range of demographic factors examined (age, gender, living condition, recent unemployment), living with others (vs. living alone) predicted more severe COVID19-related mental health symptoms. However, these effects were only observed on outcome measures pertaining to anxiety and cognitive impairments, but not loneliness and depression. Current findings have implications for both consumers and disseminators of information on popular internet hubs. Current findings also highlight the possibility that living with others (close human affiliation) may have protective and detrimental effects on different domains of mental health during the COVID19 pandemic.
39 citations
Cites methods from "Fast, Free, and Targeted: Reddit as..."
...Based on previous research (Shatz, 2016; Jamnik and Lane, 2017) and the current author’s own experience, recruitment using this platform reliably produces quality data from adult individuals dominantly residing in the United States of America (USA)....
[...]
22 May 2017
TL;DR: It is argued that the value of the practice should be evaluated according to the team needs and why senior developers do not perceive the meetings as valuable and how to apply the practice successfully in large teams is needed.
Abstract: The daily stand-up meeting is a widely used practice. However, what is more uncertain is how valuable the practice is to team members. We invited professional developers of a programming forum to a survey and obtained 221 responses. Results show that the daily stand-up meeting was used by 87% of those who employ agile methods. We found that even though the respondents on average were neutral towards the practice, the majority were either positive or negative. Junior developers were most positive and senior developers and members of large teams most negative. We argue that the value of the practice should be evaluated according to the team needs. Further, more work is needed to understand why senior developers do not perceive the meetings as valuable and how to apply the practice successfully in large teams.
24 citations
Cites background or methods from "Fast, Free, and Targeted: Reddit as..."
...Further, no compensation was offered, which increase the quality of the data because the incentive to cheat is largely reduced [15]....
[...]
...Accordingly, we posted the survey on Reddit, which is a social media website that allows scientists to recruit a targeted population [15]....
[...]
References
More filters
TL;DR: Findings indicate that MTurk can be used to obtain high-quality data inexpensively and rapidly and the data obtained are at least as reliable as those obtained via traditional methods.
Abstract: Amazon's Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.
9,562 citations
"Fast, Free, and Targeted: Reddit as..." refers background in this paper
...The most prominent is Amazon Mechanical Turk (MTurk), a crowdsourcing Internet marketplace, where workers receive wages in return for performing tasks online (Berinsky, Huber, & Lenz, 2012; Buhrmester et al., 2011; Casler, Bickel, & Hackett, 2013; Paolacci, Chandler, & Ipeirotis, 2010)....
[...]
TL;DR: It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Abstract: In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
4,727 citations
01 Aug 2010
TL;DR: The authors presented new demographic data about the Mechanical Turk subject population, reviewed the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and compared the magnitude of effects obtained using Mechanical Turk and traditional subject pools.
Abstract: textAlthough Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.
3,744 citations
TL;DR: It is shown that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples but less representative than subjects in Internet-based panels or national probability samples.
Abstract: We examine the trade-offs associated with using Amazon.com’s Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.
3,517 citations
"Fast, Free, and Targeted: Reddit as..." refers background in this paper
...In addition, even if such a source existed, methodological issues can often arise around repeated overuse of the sample (Berinsky et al., 2012; Chandler et al., 2014; Paolacci & Chandler, 2014; majority of tasks, and experienced workers often become familiar with classic paradigms and tasks used in…...
[...]
TL;DR: A checklist of recommendations for authors is being presented by theJMIR in an effort to ensure complete descriptions of Web-based surveys and it is hoped that author adherence to the checklist will increase the usefulness of such reports.
Abstract: An error in the CHERRIES statement has been corrected (J Med Internet Res 2004;6[3]:e34). In the original paper, in table 1, denominator and numerator were flipped in the recommendations on how response rates (view rate, participation rate, and completion rate) should be calculated. The view rate should be the ratio of unique survey visitors divided by unique site visitors. The participation rate should be the ratio of those who agreed to participate divided by unique first survey page visitors. The completion rate is the ratio of the number of people who finished the survey divided by those who agreed to participate. The corrections have been made in the table in both columns. [J Med Internet Res 2012;14(1):e8]
3,009 citations
"Fast, Free, and Targeted: Reddit as..." refers background in this paper
...…and experimental design, and do not prevent Reddit in particular and online sources overall from being valid for research purposes (Birnbaum, 2004; Couper, 2008; Evans & Mathur, 2005; Eysenbach, 2004; Gosling et al., 2000; Miller & Dickson, 2001; Pedersen & Nielsen, 2016; Wilson & Dewaele, 2010)....
[...]
...However, these issues can be dealt with through proper methodology and experimental design, and do not prevent Reddit in particular and online sources overall from being valid for research purposes (Birnbaum, 2004; Couper, 2008; Evans & Mathur, 2005; Eysenbach, 2004; Gosling et al., 2000; Miller & Dickson, 2001; Pedersen & Nielsen, 2016; Wilson & Dewaele, 2010)....
[...]