The impact of trial stage, developer involvement and international transferability on universal
social and emotional learning programme outcomes: A meta-analysis
Wigelsworth, M., Lendrum, A., Oldfield, J., Scott, A., ten Bokkel, I., Tate, K., & Emery, C.
The impact of trial stage, developer involvement and international transferability on universal
social and emotional learning programme outcomes: A meta-analysis
Abstract
This study expands upon the extant prior meta-analytic literature by exploring previously theorised
reasons for the failure of school-based, universal social and emotional learning (SEL) programmes to
produce expected results. Eighty-nine studies reporting the effects of school-based, universal SEL
programmes were examined for differential effects on the basis of: 1) stage of evaluation (efficacy or
effectiveness); 2) involvement from the programme developer in the evaluation (led, involved,
independent); and 3) whether the programme was implemented in its country of origin (home or
away). A range of outcomes were assessed including: social-emotional competence, attitudes
towards self, pro-social behaviour, conduct problems, emotional distress, academic achievement
and emotional competence. Differential gains across all three factors were shown, although not
always in the direction hypothesised. The findings from the current study demonstrate a revised
and more complex relationship between identified factors and dictate major new directions for the
field.
Key words: meta-analysis, socio-emotional, efficacy, developer, transferability
Literature Review
There is an emerging consensus that the role of the school should include supporting children’s
emotional education and development (Greenberg, 2010; Weare, 2010). This is often accomplished
through the implementation of universal social and emotional learning (SEL) programmes which aim
to improve learning, promote emotional well-being, and prevent problem behaviours through the
development of social and emotional competencies (Elias et al., 2001; Greenberg et al., 2003).
What is SEL?
Social and emotional learning (SEL) is represented by the promotion of five core competencies: self-
awareness; self-management; social awareness; relationship skills; and responsible decision-making
(Collaborative for Academic, Social and Emotional Learning, 2002). Although a broad definition
serves to encompass many aspects for the effective promotion of SEL, it does little to differentiate or
identify ‘essential ingredients’ of a programme of change. As a result, SEL is implemented through a
variety of formats, differing levels of training and support, varying degrees of intensity, and variation
in regards to the relative importance placed in each of the five core competencies. However, most
SEL programmes typically feature an explicit taught curriculum, delivered by teachers (with or
without additional coaching and technical support), and are delivered during school hours (examples
of SEL programmes can be seen at casel.org).
Currently, a wide range of SEL programmes feature in schools and classrooms across the world,
including in the USA (e.g. Greenberg, Kusche, Cook, & Quamma, 1995), Australia (e.g. Graetz et al.,
2008), across Europe (e.g. Holsen, Smith, & Frey, 2008), and in the UK (e.g. DfES, 2007). Although a
poor tool for assessing the complexity of any specific curriculum intervention or context, meta-
analytic approaches are required to support the hitherto theoretical assumption of the universality
of teachable SEL competencies. Indeed, recent meta-analyses in the United States (Durlak,
Weissberg, Dymicki, Taylor, & Schellinger, 2011) and the Netherlands (Sklad, Diekstra, Ritter, & Ben,
2012) have been used to suggest that high quality, well-implemented universal SEL interventions,
designed to broadly facilitate a range of intra- and inter-personal competencies, can lead to a range
of salient outcomes, including improved social and emotional skills, school attitudes and academic
performance, and reduced
mental health difficulties (Durlak, Weissberg, Dymnicki, Taylor, &
Schellinger, 2011; Sklad, Diekstra, Ritter & Ben, 2012; Wilson & Lipsey, 2007). However, individual
SEL programmes are not always able to produce the same impressive results indicated by these
meta-analyses when adopted and implemented by practitioners in schools (Social and Character
Development Research Consortium, 2010).
Research in prevention science suggests a number of possible reasons for this discrepancy, including
implementation failure (Durlak & DuPre, 2008), a reliance on the results of ‘early’ trials focusing on
the internal logic of intervention, rather than their ‘real world’ applicability (Flay, et al., 2005),
developer involvement in trials (Eisner, 2009) and a lack of cultural transferability of interventions
(Castro, Barrera, & Martinez, 2004). Although implementation fidelity is now recognised as an
important feature in the successful delivery of SEL programmes (included in Durlak et al.’s 2011
meta-analysis), there has been no similar empirical consideration of the other factors. Underlying
such explanations is an implicit assumption of a degree of invariance or ‘treatment’ approach in the
implementation of SEL programmes. Many consumers of educational research will recognise a
‘medical model’ of evaluation (typically involving experimental designs), an approach which is not
without debate (for a brief summary see Evans and Benefield (2001)). Accordingly, prior research in
educational evaluation has noted that such an approach is potentially limited, as associated
methodologies for investigation (neatly described by Elliott and Kushner (2007) as the, “statistical
aggregation of gross yield” (p.324)) will fail to capture complexities of the interactions within specific
contexts required to explain findings. Indeed, lack of process and implementation has been noted in
this particular field (Lendrum & Humphrey, 2010). However, suggested alternate directions (e.g.
anthropological, illuminative, case study (ibid)) can fail to capture the prevalence or magnitude of
trends, and, in their own way, also fail to uncover important lessons as to the successful
implementation of educational interventions. Therefore, there is an opportunity to extend prior
influential work (i.e. Durlak, Weissberg, Dymicki, Taylor, & Schellinger, 2011; Sklad, Diekstra, Ritter,
& Ben, 2012) utilising meta-analytic approaches, to examine key indicators potentially influencing
successful outcomes, and in doing so, to consider extent to which such techniques are useful in this
context.
Key indicators
In attempting to explain the high degree of variation between programmes in achieving successful
outcomes for children, this article now considers the rationale for exploring theoretically important
(e.g. as conceptualised by Lendrum and Wigelsworth (2013)), but often ignored factors, and
hypothesises their likely effect on SEL programme outcomes.
Stage of evaluation: Efficacy vs. effectiveness
An important (and often promoted) indicator as to the potential success of a programme is its
history of development and evaluation. Ideally, an intervention should be tested at several stages
between its initial development and its broad dissemination into routine practice (Greenberg,
Domitrovich, Graczyk, & Zins, 2005) and frameworks have been provided in recent literature to
enable this. For instance, drawn from evaluations of complex health interventions, Campbell et al.
(2000) provide guidance on specific sequential phases for developing interventions: developing
theory (pre-phase), modelling empirical relationships consistent with intended outcomes (phase I),
exploratory trialling (phase II), randomised control trials under optimum conditions (phase III), and
long term implementation in uncontrolled settings (phase IV). An intervention should pass through
all phases to be considered truly effective and evidence-based (Campbell et al., 2000).
An important distinction in Campbell et al.’s framework is the recognition that interventions are
typically first ‘formally’ evaluated under optimal conditions of delivery (phase III) (more broadly
referred to as efficacy trials (Flay, 1986)), such as with the provision of highly-trained and carefully
