Why Can’t a Student Be More Like an Average Person?: Sampling and Attrition Effects in Social Science Field and Laboratory Experiments

TL;DR: In this article, the authors argue that experimental researchers should pay more attention to the characteristics of participants in their experimental design, despite the obvious practical difficulties this might entail with regard to recruitment and motivation of the participants.

Abstract: In the social sciences, the use of experimental research has expanded greatly in recent years. For various reasons, most experiments rely on convenience samples of undergraduate university students. This practice, however, might endanger the validity of experimental findings, as we can assume that students will react differently to experimental conditions than the general population. We therefore urge experimental researchers to broaden their pool of participants, despite the obvious practical difficulties this might entail with regard to recruitment and motivation of the participants. We report on an experiment comparing the reactions of student and non-student participants, showing clear and significant differences. A related problem is that differential attrition rates might endanger the effects found in long-term research. We argue that experimental researchers should pay more attention to the characteristics of participants in their experimental design.

Summary

Introduction

• The majority of experimental laboratory research in the social sciences is based on convenience samples.
• Yet their experience shows that recruitment among non-student groups creates new methodological challenges (i.e., recruitment strategy, setting of the experiments, random assignment, incentives, attrition) that are of interest for researchers who intend to include non-students in their experimental design.
• The inclusion of non-student samples can affect the results and implications of the research.

Limitations of Student Samples: Research on Youth Mobilization

• In addition to all the doubts about the exclusive use of student samples, there are obviously specific research questions that require non-student sampling.
• An example from their own research experience will make this point more clear.
• Of the few experiments that have been conducted do not use a full experimental design including pre- and post-test and the inclusion of a control group (Beaumount et al.
• Most importantly though, these studies have typically focused either on high school or on university students (Bernstein and Meizlish 2003; Hansmann et al.
• As a result, the policy prescriptions that emerge do not adequately address the alarming decline in political participation on the part of those who are leaving school with only a grade 9 or grade 10 education, or those who come from socio-economically disadvantaged backgrounds or who are young unemployed.

Attrition

• Using a broader recruitment pool would ensure a more diverse sample of participants at the outset of an experiment, but still this offers no guarantee that this level of diversity can be maintained throughout the experiment as some groups have a stronger propensity to drop out than other groups.
• These differential attrition levels might endanger the validity of the results of especially long-term effect experiments.
• When a subject does not show up for the second experimental session or does not answer follow-up questionnaires because of medical reasons, the authors might still assume that this condition might be randomly spread across the sample.
• Returning to the example of the mobilization experiments the authors could expect higher attrition rates among the two extreme participant groups: the respondents for whom global warming was not a topic of great concern on the one hand, and the respondents who were initially already very interested and engaged in the environmental issue on the other hand.
• Students and non-students have different characteristics, and this implies they will need different incentives to stay in all three waves of the study.

Conclusion

• In recent years, there has been a call within medical research to diversify the participants of medical experiments.
• The main reason is that some effects might be found, e.g., among white males, but that there is no reason to assume that the new experimental drug will be just as effective, e.g., among Afro-Americans, Asians or women.
• Given all the caveats associated with experimental research, it would be unwise to arrive at firm conclusions, based on the results of just one research design.
• But when the authors want to study a broader social phenomenon or specific effects in other population groups outside the university campus, clearly student samples can tell us only part of the story.
• Efforts to corroborate experimental findings – e.g., with results from survey research, or from specific small experiments among different target groups – might be used to strengthen the validity of their findings.

Full text

Marc Hooghe, Dietlind Stolle, Valérie-Anne Mahéo & Sara Vissers
Why can’t a Student be more like an Average Person?
Sampling and Attrition Effects in Social Science Field and
Laboratory Experiments
Annals of the American Academy of Political and Social
Sciences, 628(1), 2010, 85-96.
© Sage Publications, Thousand Oaks 2010.
DOI: 10.1177/0002716209351516
Introduction
The majority of experimental laboratory research in the social sciences is based on convenience
samples. Most of these studies recruit among undergraduate university students.
Undergraduates are a convenient target population for researchers as course credits are often
used to render students’ participation compulsory. The use of convenience samples and the
overrepresentation of undergraduate students have raised some concerns about the external
validity of experimental results (Kam et al. 2007; Levitt & List 2007). Although there might be
several research questions for which a student sample is appropriate, research has found that
concerns remain about the generalizability of the results to off-campus populations. Moreover,
to answer some specific research questions might simply require the inclusion of non-student
populations.

2
In this essay, we make a case for the inclusion of non-student samples in field and laboratory
experiments wherever the research question requires it. We particularly focus on the
recruitment of people with lower socio-economic background characteristics, a group that is
largely under-represented in most research and particularly in experimental projects. In
addition, we show that the inclusion of non-student samples can affect the results and
implications of the research. Yet our experience shows that recruitment among non-student
groups creates new methodological challenges (i.e., recruitment strategy, setting of the
experiments, random assignment, incentives, attrition) that are of interest for researchers who
intend to include non-students in their experimental design.
The inclusion of non-student samples can affect the results and implications of the research.
Yet our experience shows that recruitment among non-student groups creates new
methodological challenges.
An associated problem for experimental researchers is that of attrition of participants. Different
populations, e.g. student and non-students might result in different drop out rates, which in turn
requires different strategies to maintain the motivation of participants. Long-term exposure to
experimental manipulations is often rendered difficult because of the tendency of specific
groups within the population to drop out of the experiment. As a result, researchers tend to
concentrate on effects of a relatively brief exposure, but these can be practically and
theoretically less relevant. Retaining participants in a long-lasting experiment, however, is at

3
least as serious a challenge as recruiting a non-compulsory audience for the experiments in the
first place.
(...)
Limitations of Student Samples: Research on Youth Mobilization
In addition to all the doubts about the exclusive use of student samples, there are obviously
specific research questions that require non-student sampling. An example from our own
research experience will make this point more clear. We are interested in the growing concern
about declining turnout across established democracies. Much of this decline can be explained
by generational replacement. The problem is that this generational effect is most visible among
the young and the socially under-privileged groups such as the unemployed (Bennulf &
Hedberg 1999; Hooghe & Stolle 2004; Tam Cho, Gimpel & Wu 2006; McDewitt & Kiousis
2006). For example, since the 1993 Canadian federal election, turnout in the youngest
generation has dropped over 30 points among high school dropouts and 15 points or more
among those who have completed high school and/or some college, but it has held steady
among young university graduates. Similarly, in Europe, young socio-economically
disadvantaged citizens with low levels of education or unemployment are disproportionately
turning out less to the polls (Bennulf & Hedberg 1999). Clearly, efforts to foster political
engagement need to target young people who are not college bound (Pasek et al. 2006).

4
Field and laboratory experiments are vital here because they make it possible to isolate the
effects of different ways of mobilizing young people, free of other confounding factors. Some
of the few experiments that have been conducted do not use a full experimental design
including pre- and post-test and the inclusion of a control group (Beaumount et al. 2006; Bers
and Chou 2006), or do not study long-term effects (Pinkleton 1998; Addonizio 2004;
Hansmann et al. 2005). One of the few exceptions is the quasi-experimental study with
randomly assigned students by Bernstein and Meizlish (2003), which includes a longitudinal
analysis three years after the treatment and the study by Phillips (2004) with mobilizational
treatments of high school juniors. Most importantly though, these studies have typically
focused either on high school or on university students (Bernstein and Meizlish 2003;
Hansmann et al. 2005; Livingston and Kidder 1973; Niemi and Junn 1998; Smith 1999). As a
result, the policy prescriptions that emerge do not adequately address the alarming decline in
political participation on the part of those who are leaving school with only a grade 9 or grade
10 education, or those who come from socio-economically disadvantaged backgrounds or who
are young unemployed. Moreover, many strategies that are effective for university-bound
adolescents may not work for young people who lack equivalent literacy and other political
skills (Nickerson 2006; Tedesco 2007). The theoretical issue and policy puzzle here then is
about finding generalizable political mobilization strategies that work for young people but
which at the same time allow us to address the specific circumstances and needs of populations
from a lower socio-economic background.
In our experiments then we examined the comparative mobilizational potential of face-to-face
versus online campaigns developed around the themes of climate change. Because we expect
differential results from university students and non college bound youth, we sampled

5
participants from high and low socio-economic status backgrounds. All of the roughly 400
participants, between the ages of 18 to 25 were exposed to the same mobilization content,
besides the control group. The non college bound population was reached in three different
ways in disadvantaged neighborhoods: vocational/professional schools, work and employment
programs, and continuing education schools. All participants were exposed to one manipulation
tool, and exposed to a pre-, and two post-tests (immediately after the manipulation and after
three months following the experimental condition).
(...)
Attrition
Using a broader recruitment pool would ensure a more diverse sample of participants at the
outset of an experiment, but still this offers no guarantee that this level of diversity can be
maintained throughout the experiment as some groups have a stronger propensity to drop out
than other groups. These differential attrition levels might endanger the validity of the results
of especially long-term effect experiments. Attrition is a well-known problem in long-term
panel surveys, but also in experimental designs. This time factor in long term experiments
raises an extra problem which does not exist in classical experiments, in which subjects only
have to participate once (Housman & Wise 1979). Attrition occurs when subjects drop out of
the experimental study, for instance after the pre-test or after the first post-test. This can have
several causes, which may have consequences for the internal validity of the experiment and
the interpretation of the results. Attrition by itself does not invalidate experimental results
(Cook & Campbell 1979, Fitzgerald, Gottschalk and Moffitt 1998). Attrition is only a serious