scispace - formally typeset
Search or ask a question
Journal ArticleDOI

Why Can’t a Student Be More Like an Average Person?: Sampling and Attrition Effects in Social Science Field and Laboratory Experiments

TL;DR: In this article, the authors argue that experimental researchers should pay more attention to the characteristics of participants in their experimental design, despite the obvious practical difficulties this might entail with regard to recruitment and motivation of the participants.
Abstract: In the social sciences, the use of experimental research has expanded greatly in recent years. For various reasons, most experiments rely on convenience samples of undergraduate university students. This practice, however, might endanger the validity of experimental findings, as we can assume that students will react differently to experimental conditions than the general population. We therefore urge experimental researchers to broaden their pool of participants, despite the obvious practical difficulties this might entail with regard to recruitment and motivation of the participants. We report on an experiment comparing the reactions of student and non-student participants, showing clear and significant differences. A related problem is that differential attrition rates might endanger the effects found in long-term research. We argue that experimental researchers should pay more attention to the characteristics of participants in their experimental design.

Summary (2 min read)

Introduction

  • The majority of experimental laboratory research in the social sciences is based on convenience samples.
  • Yet their experience shows that recruitment among non-student groups creates new methodological challenges (i.e., recruitment strategy, setting of the experiments, random assignment, incentives, attrition) that are of interest for researchers who intend to include non-students in their experimental design.
  • The inclusion of non-student samples can affect the results and implications of the research.

Limitations of Student Samples: Research on Youth Mobilization

  • In addition to all the doubts about the exclusive use of student samples, there are obviously specific research questions that require non-student sampling.
  • An example from their own research experience will make this point more clear.
  • Of the few experiments that have been conducted do not use a full experimental design including pre- and post-test and the inclusion of a control group (Beaumount et al.
  • Most importantly though, these studies have typically focused either on high school or on university students (Bernstein and Meizlish 2003; Hansmann et al.
  • As a result, the policy prescriptions that emerge do not adequately address the alarming decline in political participation on the part of those who are leaving school with only a grade 9 or grade 10 education, or those who come from socio-economically disadvantaged backgrounds or who are young unemployed.

Attrition

  • Using a broader recruitment pool would ensure a more diverse sample of participants at the outset of an experiment, but still this offers no guarantee that this level of diversity can be maintained throughout the experiment as some groups have a stronger propensity to drop out than other groups.
  • These differential attrition levels might endanger the validity of the results of especially long-term effect experiments.
  • When a subject does not show up for the second experimental session or does not answer follow-up questionnaires because of medical reasons, the authors might still assume that this condition might be randomly spread across the sample.
  • Returning to the example of the mobilization experiments the authors could expect higher attrition rates among the two extreme participant groups: the respondents for whom global warming was not a topic of great concern on the one hand, and the respondents who were initially already very interested and engaged in the environmental issue on the other hand.
  • Students and non-students have different characteristics, and this implies they will need different incentives to stay in all three waves of the study.

Conclusion

  • In recent years, there has been a call within medical research to diversify the participants of medical experiments.
  • The main reason is that some effects might be found, e.g., among white males, but that there is no reason to assume that the new experimental drug will be just as effective, e.g., among Afro-Americans, Asians or women.
  • Given all the caveats associated with experimental research, it would be unwise to arrive at firm conclusions, based on the results of just one research design.
  • But when the authors want to study a broader social phenomenon or specific effects in other population groups outside the university campus, clearly student samples can tell us only part of the story.
  • Efforts to corroborate experimental findings – e.g., with results from survey research, or from specific small experiments among different target groups – might be used to strengthen the validity of their findings.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

Marc Hooghe, Dietlind Stolle, Valérie-Anne Mahéo & Sara Vissers
Why can’t a Student be more like an Average Person?
Sampling and Attrition Effects in Social Science Field and
Laboratory Experiments
Annals of the American Academy of Political and Social
Sciences, 628(1), 2010, 85-96.
© Sage Publications, Thousand Oaks 2010.
DOI: 10.1177/0002716209351516
Introduction
The majority of experimental laboratory research in the social sciences is based on convenience
samples. Most of these studies recruit among undergraduate university students.
Undergraduates are a convenient target population for researchers as course credits are often
used to render students’ participation compulsory. The use of convenience samples and the
overrepresentation of undergraduate students have raised some concerns about the external
validity of experimental results (Kam et al. 2007; Levitt & List 2007). Although there might be
several research questions for which a student sample is appropriate, research has found that
concerns remain about the generalizability of the results to off-campus populations. Moreover,
to answer some specific research questions might simply require the inclusion of non-student
populations.

2
In this essay, we make a case for the inclusion of non-student samples in field and laboratory
experiments wherever the research question requires it. We particularly focus on the
recruitment of people with lower socio-economic background characteristics, a group that is
largely under-represented in most research and particularly in experimental projects. In
addition, we show that the inclusion of non-student samples can affect the results and
implications of the research. Yet our experience shows that recruitment among non-student
groups creates new methodological challenges (i.e., recruitment strategy, setting of the
experiments, random assignment, incentives, attrition) that are of interest for researchers who
intend to include non-students in their experimental design.
The inclusion of non-student samples can affect the results and implications of the research.
Yet our experience shows that recruitment among non-student groups creates new
methodological challenges.
An associated problem for experimental researchers is that of attrition of participants. Different
populations, e.g. student and non-students might result in different drop out rates, which in turn
requires different strategies to maintain the motivation of participants. Long-term exposure to
experimental manipulations is often rendered difficult because of the tendency of specific
groups within the population to drop out of the experiment. As a result, researchers tend to
concentrate on effects of a relatively brief exposure, but these can be practically and
theoretically less relevant. Retaining participants in a long-lasting experiment, however, is at

3
least as serious a challenge as recruiting a non-compulsory audience for the experiments in the
first place.
(...)
Limitations of Student Samples: Research on Youth Mobilization
In addition to all the doubts about the exclusive use of student samples, there are obviously
specific research questions that require non-student sampling. An example from our own
research experience will make this point more clear. We are interested in the growing concern
about declining turnout across established democracies. Much of this decline can be explained
by generational replacement. The problem is that this generational effect is most visible among
the young and the socially under-privileged groups such as the unemployed (Bennulf &
Hedberg 1999; Hooghe & Stolle 2004; Tam Cho, Gimpel & Wu 2006; McDewitt & Kiousis
2006). For example, since the 1993 Canadian federal election, turnout in the youngest
generation has dropped over 30 points among high school dropouts and 15 points or more
among those who have completed high school and/or some college, but it has held steady
among young university graduates. Similarly, in Europe, young socio-economically
disadvantaged citizens with low levels of education or unemployment are disproportionately
turning out less to the polls (Bennulf & Hedberg 1999). Clearly, efforts to foster political
engagement need to target young people who are not college bound (Pasek et al. 2006).

4
Field and laboratory experiments are vital here because they make it possible to isolate the
effects of different ways of mobilizing young people, free of other confounding factors. Some
of the few experiments that have been conducted do not use a full experimental design
including pre- and post-test and the inclusion of a control group (Beaumount et al. 2006; Bers
and Chou 2006), or do not study long-term effects (Pinkleton 1998; Addonizio 2004;
Hansmann et al. 2005). One of the few exceptions is the quasi-experimental study with
randomly assigned students by Bernstein and Meizlish (2003), which includes a longitudinal
analysis three years after the treatment and the study by Phillips (2004) with mobilizational
treatments of high school juniors. Most importantly though, these studies have typically
focused either on high school or on university students (Bernstein and Meizlish 2003;
Hansmann et al. 2005; Livingston and Kidder 1973; Niemi and Junn 1998; Smith 1999). As a
result, the policy prescriptions that emerge do not adequately address the alarming decline in
political participation on the part of those who are leaving school with only a grade 9 or grade
10 education, or those who come from socio-economically disadvantaged backgrounds or who
are young unemployed. Moreover, many strategies that are effective for university-bound
adolescents may not work for young people who lack equivalent literacy and other political
skills (Nickerson 2006; Tedesco 2007). The theoretical issue and policy puzzle here then is
about finding generalizable political mobilization strategies that work for young people but
which at the same time allow us to address the specific circumstances and needs of populations
from a lower socio-economic background.
In our experiments then we examined the comparative mobilizational potential of face-to-face
versus online campaigns developed around the themes of climate change. Because we expect
differential results from university students and non college bound youth, we sampled

5
participants from high and low socio-economic status backgrounds. All of the roughly 400
participants, between the ages of 18 to 25 were exposed to the same mobilization content,
besides the control group. The non college bound population was reached in three different
ways in disadvantaged neighborhoods: vocational/professional schools, work and employment
programs, and continuing education schools. All participants were exposed to one manipulation
tool, and exposed to a pre-, and two post-tests (immediately after the manipulation and after
three months following the experimental condition).
(...)
Attrition
Using a broader recruitment pool would ensure a more diverse sample of participants at the
outset of an experiment, but still this offers no guarantee that this level of diversity can be
maintained throughout the experiment as some groups have a stronger propensity to drop out
than other groups. These differential attrition levels might endanger the validity of the results
of especially long-term effect experiments. Attrition is a well-known problem in long-term
panel surveys, but also in experimental designs. This time factor in long term experiments
raises an extra problem which does not exist in classical experiments, in which subjects only
have to participate once (Housman & Wise 1979). Attrition occurs when subjects drop out of
the experimental study, for instance after the pre-test or after the first post-test. This can have
several causes, which may have consequences for the internal validity of the experiment and
the interpretation of the results. Attrition by itself does not invalidate experimental results
(Cook & Campbell 1979, Fitzgerald, Gottschalk and Moffitt 1998). Attrition is only a serious

Citations
More filters
Journal ArticleDOI
TL;DR: In this article, a meta-analysis explored the efficacy of the theory of Planned Behaviour (TPB) dependent on behaviour and methodological moderators and found that behavioural type moderated the model; physical activity and diet behaviours were better predicted (23.9% and 21.2% variance explained, respectively).
Abstract: This meta-analysis explored the efficacy of the Theory of Planned Behaviour (TPB) dependent on behaviour and methodological moderators. A lack of hierarchical analysis in previous reviews risks confounding these moderators. Here moderating roles of behaviour type, length of follow-up, sample age and behavioural measure are explored hierarchically amongst prospective tests of the TPB, controlling for past behaviour where possible. Searching identified 237 prospective tests from 206 articles. Random-effects meta-analytic procedures were used to correcting correlations for sampling and measurement error. Behaviour type moderated the model; physical activity and diet behaviours were better predicted (23.9% and 21.2% variance explained, respectively) whilst risk, detection, safer sex and abstinence from drugs were poorly predicted (between 13.8 and 15.3% variance explained). Methodological moderators were also apparent: age of sample moderated relations with student samples better predicted for physical activi...

1,759 citations

Journal ArticleDOI
TL;DR: Although results support theory predictions, effect sizes were small, particularly for the intention-behavior relationship, and the theory explained 33 and 9% of the variance in intention and adherence behavior respectively.
Abstract: Social-cognitive models such as the theory of planned behavior have demonstrated efficacy in predicting behavior, but few studies have examined the theory as a predictor of treatment adherence in chronic illness. We tested the efficacy of the theory for predicting adherence to treatment in chronic illness across multiple studies. A database search identified 27 studies, meeting inclusion criteria. Averaged intercorrelations among theory variables were computed corrected for sampling error using random-effects meta-analysis. Path-analysis using the meta-analytically derived correlations was used to test theory hypotheses and effects of moderators. The theory explained 33 and 9 % of the variance in intention and adherence behavior respectively. Theoretically consistent patterns of effects among the attitude, subjective norm, perceived behavioral control, intention and behavior constructs were found with small-to-medium effect sizes. Effect sizes were invariant across behavior and measurement type. Although results support theory predictions, effect sizes were small, particularly for the intention-behavior relationship.

184 citations

Journal ArticleDOI
TL;DR: The authors examined the medium-specific and spill-over effects from various aspects of using social networking sites (SNS) to other forms of online and offline political participation and found a reciprocal relationship between the posting of Facebook messages and offline protest, confirming that Facebook political activity is both mobilizing and reinforcing.
Abstract: This article examines the medium-specific and spill-over effects from various aspects of using social networking sites (SNS) to other forms of online and offline political participation. The study relies on a two-wave panel survey of undergraduate students at a major Canadian university, which was designed to measure detailed aspects of political participation on social networking sites (SNS) and various other forms of political engagement. Our results show that prior political Facebook participation fosters other forms of online participation, but also reveals some spill-over effects to offline protests. Indeed, we find a reciprocal relationship between the posting of Facebook messages and offline protest, confirming that Facebook political activity is both mobilizing and reinforcing.

96 citations

Journal ArticleDOI
Itamar Shatz1
TL;DR: The article discusses current online recruitment sources and their limitations, provides an overview of Reddit, validates its use for research purposes, examines participation data from previous studies which recruited through Reddit, and suggests guidelines that can improve recruitment and retention rates for scientists looking to use Reddit for their research.
Abstract: Recruiting participants is a necessary step in many studies. With the advent of online research techniques, scientists are looking for new places where participants can be recruited online, in orde...

92 citations


Cites background from "Why Can’t a Student Be More Like an..."

  • ...Thus, while there is attrition among participants, which is problematic from a methodological perspective, it does not seem to be a greater issue here than it is for other source of recruitment (Hooghe et al., 2010; Umbach, 2004)....

    [...]

  • ...Although this is a commonly used convenience sample, the reliance on college students for experiments can lead to systematic biases in findings, an issue known as the ‘‘narrow data base problem’’ (Hooghe et al., 2010; Kam, Wilking, & Zechmeister, 2007; Lane et al., 2015; Sears, 1986)....

    [...]

Journal ArticleDOI
TL;DR: In this paper, the authors applied the theory of planned behaviour (TPB) to predict behavioural intention to adopt cryptocurrency, and found that attitude and perceived behavioural control positively impact the intention to use cryptocurrency.
Abstract: Financial technology innovation within the developed world is driving financial markets, yet its adoption is lagging among consumers in emerging markets. At the same time, most African economies continue to be at the tail end of global financial innovations adoption. Given lagging consumer adoption of cryptocurrency in South Africa, the purpose of this paper is to apply the theory of planned behaviour (TPB) to predict behavioural intention to adopt cryptocurrency.,A survey instrument based on the TPB was used to collect quantitative data for predicting adoption from adult distance students at the Mancosa, Cape Town campus. For data analysis, the two-step structural equation modelling approach was used.,The findings indicate that attitude and perceived behavioural control positively impact the intention to adopt cryptocurrency. Subjective norm showed a negative non-significant influence. Overall, the results of the study show that the model has a good model fit and can be used to explain the theory.,The results of this study may not be generalisable to the wider population as it is only based on a cross-sectional study of a sample of adult students at a single institute in South Africa.,The contribution of this paper is threefold: it is one of a few studies on the behavioural intention to adopt cryptocurrency in South Africa using the TPB model, it contributes towards the use of predictive behavioural economics models in understanding consumer behaviour critical to accelerating the adoption of financial innovations, and the results of the study also inform behaviour change strategies that can be applied by practitioners or policymakers to improve adoption. Studies of this nature may lead to the development of financial innovation in emerging markets through a nuanced understanding of consumer behaviour.

66 citations

References
More filters
01 Jan 2004
TL;DR: Niemi et al. as mentioned in this paper used two field experiments with inner city high school students to test whether extracurricular activities actually affect civic knowledge, attitudes and behavior, and found that the effects of local service learning to be small and elusive.
Abstract: By and large, scholars of electoral behavior and civil society consider education to be the key predictor of civic participation. Indeed, civic education is held to be a fundamental institution of all deliberative democracies. Yet after decades of empirical research, we still know little about the causal mechanisms through which education fosters political participation, and more generally, “good American citizens.” Recently, scholars and educators have suggested that practical lessons about local politics and participation in extracurricular activities – collectively referred to as service learning – are the most promising educational tools for enhancing civic development (Niemi and Junn 1998, ch. 7; Galston 2001, 228-229; Carnegie and CIRCLE 2003, 6). I use two field experiments with inner city high school students to test whether these activities actually affect civic knowledge, attitudes and behavior. The primary advantage of an experimental approach is that it creates the conditions necessary for unbiased causal inference. Accordingly, my results speak to certain issues of cause and effect in civic education with greater certainty than previous research. This study is additionally novel in the measures it uses to gauge civic engagement. Many scholars have characterized minority students as possessing less political knowledge and participating less in civic organizations than their white counterparts (Hodgkinson and Weitzman 1997, Niemi and Junn 1998, Lake et al. 2002). These findings may partially reflect the inability of researchers to measure a broad spectrum of extra-electoral behavior in urban areas. To test this hypothesis, I incorporate more comprehensive and culturally sensitive indicators of civic involvement than those used by prior research. Contrary to the hopes of recent theorists, my findings show the effects of local service learning to be small and elusive. On the other hand, my refined measures of participation reveal some previously neglected, but potentially inspiring sources of civic engagement in the inner city. Only time and future experimentation will tell whether these popular activities represent a more successful means of mobilizing urban youth for political activism.

7 citations

Trending Questions (3)
What are the downsides of student sampling?

The use of student sampling in experiments may raise concerns about the external validity of the results.

Why students feel difficult to design the experimental method?

Students may find it difficult to design experimental methods due to challenges in recruitment, motivation, and attrition rates among different populations.

Why students feel difficult to design experimental method in science?

Students may find it difficult to design experimental methods in science because they often rely on convenience samples of undergraduate university students, which may not accurately represent the general population.