scispace - formally typeset

Journal ArticleDOI

Why Can’t a Student Be More Like an Average Person?: Sampling and Attrition Effects in Social Science Field and Laboratory Experiments

23 Feb 2010-Annals of The American Academy of Political and Social Science (SAGE Publications)-Vol. 628, Iss: 1, pp 85-96

AbstractIn the social sciences, the use of experimental research has expanded greatly in recent years. For various reasons, most experiments rely on convenience samples of undergraduate university students. This practice, however, might endanger the validity of experimental findings, as we can assume that students will react differently to experimental conditions than the general population. We therefore urge experimental researchers to broaden their pool of participants, despite the obvious practical difficulties this might entail with regard to recruitment and motivation of the participants. We report on an experiment comparing the reactions of student and non-student participants, showing clear and significant differences. A related problem is that differential attrition rates might endanger the effects found in long-term research. We argue that experimental researchers should pay more attention to the characteristics of participants in their experimental design.

Topics: Population (52%)

Summary (2 min read)

Introduction

  • The majority of experimental laboratory research in the social sciences is based on convenience samples.
  • Yet their experience shows that recruitment among non-student groups creates new methodological challenges (i.e., recruitment strategy, setting of the experiments, random assignment, incentives, attrition) that are of interest for researchers who intend to include non-students in their experimental design.
  • The inclusion of non-student samples can affect the results and implications of the research.

Limitations of Student Samples: Research on Youth Mobilization

  • In addition to all the doubts about the exclusive use of student samples, there are obviously specific research questions that require non-student sampling.
  • An example from their own research experience will make this point more clear.
  • Of the few experiments that have been conducted do not use a full experimental design including pre- and post-test and the inclusion of a control group (Beaumount et al.
  • Most importantly though, these studies have typically focused either on high school or on university students (Bernstein and Meizlish 2003; Hansmann et al.
  • As a result, the policy prescriptions that emerge do not adequately address the alarming decline in political participation on the part of those who are leaving school with only a grade 9 or grade 10 education, or those who come from socio-economically disadvantaged backgrounds or who are young unemployed.

Attrition

  • Using a broader recruitment pool would ensure a more diverse sample of participants at the outset of an experiment, but still this offers no guarantee that this level of diversity can be maintained throughout the experiment as some groups have a stronger propensity to drop out than other groups.
  • These differential attrition levels might endanger the validity of the results of especially long-term effect experiments.
  • When a subject does not show up for the second experimental session or does not answer follow-up questionnaires because of medical reasons, the authors might still assume that this condition might be randomly spread across the sample.
  • Returning to the example of the mobilization experiments the authors could expect higher attrition rates among the two extreme participant groups: the respondents for whom global warming was not a topic of great concern on the one hand, and the respondents who were initially already very interested and engaged in the environmental issue on the other hand.
  • Students and non-students have different characteristics, and this implies they will need different incentives to stay in all three waves of the study.

Conclusion

  • In recent years, there has been a call within medical research to diversify the participants of medical experiments.
  • The main reason is that some effects might be found, e.g., among white males, but that there is no reason to assume that the new experimental drug will be just as effective, e.g., among Afro-Americans, Asians or women.
  • Given all the caveats associated with experimental research, it would be unwise to arrive at firm conclusions, based on the results of just one research design.
  • But when the authors want to study a broader social phenomenon or specific effects in other population groups outside the university campus, clearly student samples can tell us only part of the story.
  • Efforts to corroborate experimental findings – e.g., with results from survey research, or from specific small experiments among different target groups – might be used to strengthen the validity of their findings.

Did you find this useful? Give us your feedback

...read more

Content maybe subject to copyright    Report

Marc Hooghe, Dietlind Stolle, Valérie-Anne Mahéo & Sara Vissers
Why can’t a Student be more like an Average Person?
Sampling and Attrition Effects in Social Science Field and
Laboratory Experiments
Annals of the American Academy of Political and Social
Sciences, 628(1), 2010, 85-96.
© Sage Publications, Thousand Oaks 2010.
DOI: 10.1177/0002716209351516
Introduction
The majority of experimental laboratory research in the social sciences is based on convenience
samples. Most of these studies recruit among undergraduate university students.
Undergraduates are a convenient target population for researchers as course credits are often
used to render students’ participation compulsory. The use of convenience samples and the
overrepresentation of undergraduate students have raised some concerns about the external
validity of experimental results (Kam et al. 2007; Levitt & List 2007). Although there might be
several research questions for which a student sample is appropriate, research has found that
concerns remain about the generalizability of the results to off-campus populations. Moreover,
to answer some specific research questions might simply require the inclusion of non-student
populations.

2
In this essay, we make a case for the inclusion of non-student samples in field and laboratory
experiments wherever the research question requires it. We particularly focus on the
recruitment of people with lower socio-economic background characteristics, a group that is
largely under-represented in most research and particularly in experimental projects. In
addition, we show that the inclusion of non-student samples can affect the results and
implications of the research. Yet our experience shows that recruitment among non-student
groups creates new methodological challenges (i.e., recruitment strategy, setting of the
experiments, random assignment, incentives, attrition) that are of interest for researchers who
intend to include non-students in their experimental design.
The inclusion of non-student samples can affect the results and implications of the research.
Yet our experience shows that recruitment among non-student groups creates new
methodological challenges.
An associated problem for experimental researchers is that of attrition of participants. Different
populations, e.g. student and non-students might result in different drop out rates, which in turn
requires different strategies to maintain the motivation of participants. Long-term exposure to
experimental manipulations is often rendered difficult because of the tendency of specific
groups within the population to drop out of the experiment. As a result, researchers tend to
concentrate on effects of a relatively brief exposure, but these can be practically and
theoretically less relevant. Retaining participants in a long-lasting experiment, however, is at

3
least as serious a challenge as recruiting a non-compulsory audience for the experiments in the
first place.
(...)
Limitations of Student Samples: Research on Youth Mobilization
In addition to all the doubts about the exclusive use of student samples, there are obviously
specific research questions that require non-student sampling. An example from our own
research experience will make this point more clear. We are interested in the growing concern
about declining turnout across established democracies. Much of this decline can be explained
by generational replacement. The problem is that this generational effect is most visible among
the young and the socially under-privileged groups such as the unemployed (Bennulf &
Hedberg 1999; Hooghe & Stolle 2004; Tam Cho, Gimpel & Wu 2006; McDewitt & Kiousis
2006). For example, since the 1993 Canadian federal election, turnout in the youngest
generation has dropped over 30 points among high school dropouts and 15 points or more
among those who have completed high school and/or some college, but it has held steady
among young university graduates. Similarly, in Europe, young socio-economically
disadvantaged citizens with low levels of education or unemployment are disproportionately
turning out less to the polls (Bennulf & Hedberg 1999). Clearly, efforts to foster political
engagement need to target young people who are not college bound (Pasek et al. 2006).

4
Field and laboratory experiments are vital here because they make it possible to isolate the
effects of different ways of mobilizing young people, free of other confounding factors. Some
of the few experiments that have been conducted do not use a full experimental design
including pre- and post-test and the inclusion of a control group (Beaumount et al. 2006; Bers
and Chou 2006), or do not study long-term effects (Pinkleton 1998; Addonizio 2004;
Hansmann et al. 2005). One of the few exceptions is the quasi-experimental study with
randomly assigned students by Bernstein and Meizlish (2003), which includes a longitudinal
analysis three years after the treatment and the study by Phillips (2004) with mobilizational
treatments of high school juniors. Most importantly though, these studies have typically
focused either on high school or on university students (Bernstein and Meizlish 2003;
Hansmann et al. 2005; Livingston and Kidder 1973; Niemi and Junn 1998; Smith 1999). As a
result, the policy prescriptions that emerge do not adequately address the alarming decline in
political participation on the part of those who are leaving school with only a grade 9 or grade
10 education, or those who come from socio-economically disadvantaged backgrounds or who
are young unemployed. Moreover, many strategies that are effective for university-bound
adolescents may not work for young people who lack equivalent literacy and other political
skills (Nickerson 2006; Tedesco 2007). The theoretical issue and policy puzzle here then is
about finding generalizable political mobilization strategies that work for young people but
which at the same time allow us to address the specific circumstances and needs of populations
from a lower socio-economic background.
In our experiments then we examined the comparative mobilizational potential of face-to-face
versus online campaigns developed around the themes of climate change. Because we expect
differential results from university students and non college bound youth, we sampled

5
participants from high and low socio-economic status backgrounds. All of the roughly 400
participants, between the ages of 18 to 25 were exposed to the same mobilization content,
besides the control group. The non college bound population was reached in three different
ways in disadvantaged neighborhoods: vocational/professional schools, work and employment
programs, and continuing education schools. All participants were exposed to one manipulation
tool, and exposed to a pre-, and two post-tests (immediately after the manipulation and after
three months following the experimental condition).
(...)
Attrition
Using a broader recruitment pool would ensure a more diverse sample of participants at the
outset of an experiment, but still this offers no guarantee that this level of diversity can be
maintained throughout the experiment as some groups have a stronger propensity to drop out
than other groups. These differential attrition levels might endanger the validity of the results
of especially long-term effect experiments. Attrition is a well-known problem in long-term
panel surveys, but also in experimental designs. This time factor in long term experiments
raises an extra problem which does not exist in classical experiments, in which subjects only
have to participate once (Housman & Wise 1979). Attrition occurs when subjects drop out of
the experimental study, for instance after the pre-test or after the first post-test. This can have
several causes, which may have consequences for the internal validity of the experiment and
the interpretation of the results. Attrition by itself does not invalidate experimental results
(Cook & Campbell 1979, Fitzgerald, Gottschalk and Moffitt 1998). Attrition is only a serious

Citations
More filters

Journal ArticleDOI
Abstract: This meta-analysis explored the efficacy of the Theory of Planned Behaviour (TPB) dependent on behaviour and methodological moderators. A lack of hierarchical analysis in previous reviews risks confounding these moderators. Here moderating roles of behaviour type, length of follow-up, sample age and behavioural measure are explored hierarchically amongst prospective tests of the TPB, controlling for past behaviour where possible. Searching identified 237 prospective tests from 206 articles. Random-effects meta-analytic procedures were used to correcting correlations for sampling and measurement error. Behaviour type moderated the model; physical activity and diet behaviours were better predicted (23.9% and 21.2% variance explained, respectively) whilst risk, detection, safer sex and abstinence from drugs were poorly predicted (between 13.8 and 15.3% variance explained). Methodological moderators were also apparent: age of sample moderated relations with student samples better predicted for physical activi...

1,478 citations


Journal ArticleDOI
TL;DR: Although results support theory predictions, effect sizes were small, particularly for the intention-behavior relationship, and the theory explained 33 and 9% of the variance in intention and adherence behavior respectively.
Abstract: Social-cognitive models such as the theory of planned behavior have demonstrated efficacy in predicting behavior, but few studies have examined the theory as a predictor of treatment adherence in chronic illness. We tested the efficacy of the theory for predicting adherence to treatment in chronic illness across multiple studies. A database search identified 27 studies, meeting inclusion criteria. Averaged intercorrelations among theory variables were computed corrected for sampling error using random-effects meta-analysis. Path-analysis using the meta-analytically derived correlations was used to test theory hypotheses and effects of moderators. The theory explained 33 and 9 % of the variance in intention and adherence behavior respectively. Theoretically consistent patterns of effects among the attitude, subjective norm, perceived behavioral control, intention and behavior constructs were found with small-to-medium effect sizes. Effect sizes were invariant across behavior and measurement type. Although results support theory predictions, effect sizes were small, particularly for the intention-behavior relationship.

146 citations


Journal ArticleDOI
Abstract: This article examines the medium-specific and spill-over effects from various aspects of using social networking sites (SNS) to other forms of online and offline political participation. The study relies on a two-wave panel survey of undergraduate students at a major Canadian university, which was designed to measure detailed aspects of political participation on social networking sites (SNS) and various other forms of political engagement. Our results show that prior political Facebook participation fosters other forms of online participation, but also reveals some spill-over effects to offline protests. Indeed, we find a reciprocal relationship between the posting of Facebook messages and offline protest, confirming that Facebook political activity is both mobilizing and reinforcing.

84 citations


Journal ArticleDOI
Abstract: In recent years, voluntary associations and political organizations have increasingly switched to Internet-based mobilization campaigns, replacing traditional forms of face-to-face recruitment and mobilization. The existing body of empirical research on Internet-based mobilization, however, is not conclusive about the effects this form of mobilization might have. In this article, the authors argue that this lack of strong conclusions might be due to the failure to distinguish different behavioral outcomes of mobilization, and more specifically, a distinction between online and off-line forms of participation is missing. In this experimental study, participants were exposed to potentially mobilizing information either by way of face-to-face interaction or by website. The results of the experiment indicate that web-based mobilization only has a significant effect on online participation, whereas face-to-face mobilization has a significant impact on off-line behavior, which would imply that mobilization effects are medium-specific. The authors close with some observations on what these findings might imply for the democratic consequences of the current trend toward an increasing reliance on Internet-based forms of political mobilization.

61 citations


Journal ArticleDOI
Itamar Shatz1
TL;DR: The article discusses current online recruitment sources and their limitations, provides an overview of Reddit, validates its use for research purposes, examines participation data from previous studies which recruited through Reddit, and suggests guidelines that can improve recruitment and retention rates for scientists looking to use Reddit for their research.
Abstract: Recruiting participants is a necessary step in many studies. With the advent of online research techniques, scientists are looking for new places where participants can be recruited online, in orde...

47 citations


Cites background from "Why Can’t a Student Be More Like an..."

  • ...Thus, while there is attrition among participants, which is problematic from a methodological perspective, it does not seem to be a greater issue here than it is for other source of recruitment (Hooghe et al., 2010; Umbach, 2004)....

    [...]

  • ...Although this is a commonly used convenience sample, the reliance on college students for experiments can lead to systematic biases in findings, an issue known as the ‘‘narrow data base problem’’ (Hooghe et al., 2010; Kam, Wilking, & Zechmeister, 2007; Lane et al., 2015; Sears, 1986)....

    [...]


References
More filters

Book
01 Jan 1979

11,861 citations


"Why Can’t a Student Be More Like an..." refers background or result in this paper

  • ...These biases endanger the external validity of any findings as a result of these experiments (Cook and Campbell 1979)....

    [...]

  • ...Attrition by itself does not invalidate experimental results (Cook and Campbell 1979; Fitzgerald, Gottschalk, and Moffitt 1998)....

    [...]


Book
01 Jan 1995
Abstract: * *1. Introduction * Part I: The World of Participation *2. Defining Political Participation *3. Political Participation: How Much? About What? *4. Interpreting Political Activity: A Report from Activists *5. Recruiting Political Activists * Part II: Participation and Representation *6. Thinking about Participatory Representation *7. Who Participates? Economic Circumstances and Needs *8. Who Participates? Race, Ethnicity, and Gender * Part III: The Civic Voluntarism Model *9. Explaining Participation: Introductory Considerations *10. Resources for Politics: Time and Money *11. Resources for Politics: Civic Skills *12. Resources, Engagement, and Political Activity *13. Institutions and Recruitment *14. Participation and the Politics of Issue Engagement *15. From Generation to Generation: The Roots of [incomplete]

6,352 citations


Journal ArticleDOI
Abstract: For the 2 decades prior to 1960, published research in social psychology was based on a wide variety of subjects and research sites. Content analyses show that since then such research has overwhelmingly been based on college students tested in academic laboratories on academiclike tasks. How might this heavy dependence on one narrow data base have biased the main substantive conclusions of sociopsychological research in this era? Research on the full life span suggests that, compared with older adults, college students are likely to have less-crystallized attitudes, less-formulated senses of self, stronger cognitive skills, stronger tendencies to comply with authority, and more unstable peer group relationships. The laboratory setting is likely to exaggerate all these differences. These peculiarities of social psychology's predominant data base may have contributed to central elements of its portrait of human nature. According to this view people (a) are quite compliant and their behavior is easily socially influenced, (b) readily change their attitudes and (c) behave inconsistently with them, and (d) do not rest their self-perceptions on introspection. The narrow data base may also contribute to this portrait of human nature's (e) strong emphasis on cognitive processes and to its lack of emphasis on (f) personality dispositions, (g) material self-interest, (h) emotionally based irrationalities, (i) group norms, and (j) stage-specific phenomena. The analysis implies the need both for more careful examination of sociopsychological propositions for systematic biases introduced by dependence on this narrow data base and for increased reliance on adults tested in their natural habitats with materials drawn from ordinary life.

1,849 citations


Journal ArticleDOI
Abstract: A critical question facing experimental economists is whether behavior inside the laboratory is a good indicator of behavior outside the laboratory. To address that question, we build a model in which the choices that individuals make depend not just on financial implications, but also on the nature and extent of scrutiny by others, the particular context in which a decision is embedded, and the manner in which participants and tasks are selected. We present empirical evidence demonstrating the importance of these various factors. To the extent that lab and naturally occurring environments systematically differ on any of these dimensions, the results obtained inside and outside the lab need not correspond. Focusing on experiments designed to measure social preferences, we discuss the extent to which the existing laboratory results generalize to naturally-occurring markets. We summarize cases where the lab may understate the importance of social preferences as well as instances in which the lab might exaggerate their importance. We conclude by emphasizing the importance of interpreting laboratory and field data through the lens of theory.

1,725 citations


Journal ArticleDOI
Abstract: A second‐order meta‐analysis was conducted to assess the implications of using college student subjects in social science research. Four meta‐analyses investigating response homogeneity (cumulative N > 650,000) and 30 meta‐analyses reporting effect sizes for 65 behavioral or psychological relationships (cumulative N > 350,000) provided comparative data for college student subjects and nonstudent (adult) subjects for the present research. In general, responses of college student subjects were found to be slightly more homogeneous than those of nonstudent subjects. Moreover, effect sizes derived from college student subjects frequently differed from those derived from nonstudent subjects both directionally and in magnitude. Because there was no systematic pattern to the differences observed, caution must be exercised when attempting to extend any relationship found using college student subjects to a nonstudent (adult) population. The results augur in favor of, and emphasize the importance of, replicating r...

1,106 citations