Learning analytics to unveil learning strategies in a flipped classroom

TL;DR: Despite the increasing popularity of FL and similar active learning models, there has been limited attention devoted to understanding the reasons for why students may or may not engage in pre-class activities.

Abstract: Prior education studies have consistently emphasized the importance of sustained and active student engagement to aid academic performance and achievement of learning outcomes (e.g., Hockings, Cooke, Yamashita, McGinty, & Bowl, 2008; Michael, 2006). The positive impact of such active learning models on academic outcomes has been well established, particularly, in the STEM (Science, Technology, Engineering andMathematics) disciplines. For example, Freeman et al. (2014) demonstrated that students undertaking STEM courses incorporating active learning models received (on average) higher academic grades and were less likely to fail in comparison to peers in more traditional and lecture based modes of teaching. While active learning has clear benefits for student learning outcomes, the process of implementation is often more complex than first anticipated (Gillies & Boyle, 2010; Hung, 2011). For instance, student engagement in active learning does not occur spontaneously and educators must employ careful consideration of the curriculum design, activity sequencing and progression as well as the diversity of learners, including learners' prior experience and motivation, background and knowledge. Flipped learning (FL) is a form of blended learning that requires students' active participation in learning activities both before and during face-to-face sessions with the teacher (Lage, Platt, & Tregua, 2000). However, students frequently lack the necessary skills, time, and/ormotivation to fully participate in pre-class activities and therefore do not commit to the level of involvement in the learning process that effectively complements the intended design (Lai & Hwang, 2016; Mason, Shuman, & Cook, 2013). Clearly, the reasoning for why students may or may not engage in pre-class activities is complex and multi-dimensional. However, if provided with a deeper insight into the types of learning strategies students employ in such active learning models, teaching staff can make better informed decisions regarding student support and course design processes (Stief & Dollar, 2009). Despite the increasing popularity of FL and similar active learning models, there has been limited attention devoted to understanding

Summary

1. Introduction

• The educational research community offers a diversity of interpretations on what constitutes a learning strategy.
• Husman, & Dierking (2000, p. 227) suggesting that a learning strategy includes "any thoughts, behaviors, beliefs or emotions that facilitate the acquisition, understanding or later transfer of new knowledge and skills".the authors.
• The authors consider students' learning strategies as latent constructs that cannot be directly observed in the collected traces, but have to be mined/detected using appropriate analytical methods and techniques.
• The authors make a combined use of exploratory sequence analysis and agglomerative hierarchical clustering to detect patterns in student behaviour that are indicative of the adopted learning strategies.

1.1 Active learning and Flipped learning

• This model of active learning requires students to be self-regulated learners in order to undertake and complete the preparatory activities (Lai & Hwang, 2016; Mason et al., 2013; Sletten, 2015) .
• Many students have underdeveloped self-regulation skills and need support and scaffolding to manage their learning in less familiar and more intensive settings that often characterize FL designs.
• To address this need, the FL design examined in this paper has a well-defined structure that is consistent throughout the entire course duration (see Section 2.1).

1.2 Learning strategies and Self-regulated learning

• The capacity of a student to choose and adapt their learning strategy in accordance with the requirements of the learning setting is a key self-regulatory skill (Winne, 2006) .
• Furthermore, previous research has shown that learners are not accurate reporters of how they study and what strategies they apply (Zhou & Winne, 2012) .
• According to Winne, to improve learning, students "might profit from (a) feedback that accurately represents how they actually studied and (b) information about tactics and strategies that might be more effective than those they actually used" (Winne, 2013, p.387) .
• This approach would be better complemented by, or substituted with, digital learning traces (Winne, 2013) .

1.4 Learning strategies and academic performance in flipped classroom

• The above given considerations suggest that a FL setting can both positively and negatively (far transfer) affect a student's selection and regulation of learning strategies, and consequently, their academic performance.
• Previous research has shown that when students manage to quickly adjust to the FL model (i.e., resolve the transfer problem), their academic achievements are comparable to or better than that of students attending traditional lecturing model (Mason et al., 2013; McLaughlin et al., 2013) .
• It has not been sufficiently explored how regulation of pre-class activities affect the overall course performance.

2.1 Study context

• Students were provided with real-time feedback on their level of engagement with the preparation activities and their activity scores via an analytics dashboard (Anonymous, 2016a) .
• Through the dashboard, students could monitor their engagement with the video resources, success in answering MCQs that followed the videos, and MCQs that were embedded in the course related documents, as well as the percentage of correctly solved problem sequences.
• Next to the students' personal scores, the dashboard displayed the overall class scores, thus allowing for social comparison.
• The displayed data was updated every 15 minutes, and the magnitudes were reset each week.

2.2 Learning traces

• To gain an insight into the general patterns of learning sessions of the two student groups, the authors removed the outliers.
• In particular, the authors removed overly short sequences, i.e., those comprising of only one event, as well as those that were overly long, i.e., those that were above the 95 th percentile in terms of the number of events.
• After pruning the outliers, the sizes of the two groups were: 786 sequences for the students with the scores above the 90 th percentile, and 684 sequences for the group with scores below the 25 th percentile.

2.3.2 Clustering

• Kruskal Wallis tests followed by Mann Whitney U tests were used to compare the resulting student clusters based on the midterm and final exam scores.
• False Discovery Rate (FDR) was used as a recommended correction for preventing alpha inflation when doing multiple tests (Cramer et al., 2015) .

3.1 Exploratory sequence analysis

• The figures suggest that there is a considerable difference in the distribution of learning actions along learning sequences between the two examined groups.
• High performing students were observed to be giving roughly equal attention to all types of actions throughout their learning sessions.
• In contrast, their lower performing peers were almost exclusively focused on the summative assessment tasks.
• This initial insight suggested that further analysis of students' learning sequences might lead to the identification of patterns in students' learning behaviour, potentially indicative of the adopted learning strategies.
• Learning action abbreviations are outlined in the figure legend and briefly explained in Table 1 .

3.2 Clusters of learning sequences as manifestations of student learning strategies

• Formative assessment actions are also present though they are gradually and mostly towards the end of the sessions substituted by summative assessment actions.
• These seem to be sessions where students were primarily watching videos, then doing the follow-up multiple-choice questions, and finally trying the exercises.

3.3 Clusters of students based on the shared learning strategies

• The degree of variation in learning strategies adopted by students within each cluster was also examined.
• Figure 4 illustrates how the median number of learning sequences per learning strategy (sequence cluster) changed over the 12 weeks of the course.
• The peak of their activities and strategy use was in week 6, right before the midterm exam.
• From week 7, they abandoned strategy 1 (focus on formative assessment), and while still retaining the other three strategies, they showed preference for strategy 2 (focus on summative assessment).

4. Discussion

• An important practical implication of the presented findings is that instructors should occasionally, and especially after the midterm, remind their students about the importance of choosing effective learning strategies, particularly those strategies that rely on active engagement with the learning resources (e.g., different forms of formative assessment).
• To assure the students' attentiveness to such recommendations, instructors should make the students aware of the value and relevance of the recommended strategies for both learning and academic achievement.
• Furthermore, learning strategies are skills, and as all skills they have to be practiced to develop proficiency (Ericsson, Krampe, & Tesch-Romer, 1993; Winne, 2013) .
• Hence, the instructors should consider altering the learning design, in particular the preparation part of the FL design, to scaffold the development of the desired learning strategies.

4.2 RQ2: Association between learning strategies and course performance

• This finding is also consistent with empirical findings of research studies that examined students' approaches to learning and how these approaches impact academic performance.
• Three approaches to learning have been recognized (Biggs, 2012) : i) deep approach, characterized by critical evaluation and syntheses of information, and driven by intrinsic motivation; ii) surface approach, dominated by shallow cognitive strategies and associated with extrinsic motivation; and iii) strategic approach, which assumes alterations between deep and surface approaches, depending on the characteristics of the task at hand.
• Learning strategies practiced by students from the Intensive group (cluster 1) might be considered as indicative of deep approach; clusters 2 and 3 gather strategic learners, whereas the Selective and Highly selective groups seem to be practicing surface approach to learning.
• Course performance of the five clusters is consistent with the performance levels characterizing the three learning approaches.
• Specifically, meta-analysis by Richardson, Abraham, & Bond (2012) demonstrated positive, though small, correlations between students' performance and both deep and strategic approaches to learning, whereas surface approach was found to be negatively correlated with academic performance.

4.3 Limitations and future research

• Collection of data required for identifying students' goal orientation is not straightforward.
• Traditional self-report measures are not capable of capturing the dynamics of students' goals (Zhou & Winne, 2012) , which, although generally stable, can change along with changes in learning tasks (Fryer & Elliot, 2007) .
• In addition, the ability of students to give valid and objective reports on their goal orientations is questionable (Richardson, 2004) .
• Hence there is a need to extend learning environment with instruments that would allow for seamless and unobtrusive collection of data about the dynamics of students' goal orientation.
• An illustrative example is an annotation tool that allows students to associate selected pieces of content with one or more tags (from a predefined tags collection) reflective of their goal orientations (Zhou & Winne, 2012) .

5. Conclusion

• To inform the instructor on whether the deployed FL design was effective in sustaining student engagement and preparing them for active participation in the class (i.e., face-toface session).
• To provide grounds for selective/adaptive inclusion of scaffolds (e.g., hints, guidelines) to help students improve their learning behavior.
• To make students aware of their learning strategies, and how those strategies compare to the strategies of well performing peers.
• Students in a FL setting often require more awareness of their learning process than students in more traditional settings (Frederickson, Reed, & Clifford, 2005) ; they need to reflect on their learning activities in order to properly connect them with the course materials and requirements, and make necessary adjustments in their learning approach (Strayer, 2012) .

Figures

Table 4 Pairwise comparison of clusters with respect to the students’ final exam scores

Table 3 Pairwise comparison of clusters with respect to the students’ midterm exam scores

Figure 1. Examples of learning sequences encoded in the TraMineR format

Table 2 describes the resulting clusters in terms of i) the five variables used for clustering (seq.clust1 – seq.clust4, and seq.total); ii) the midterm exam score (midterm.score); and iii) the final exam score (final.score). For all the variables the table shows the median, 25th and 75th percentiles.

Figure 3. Clusters of learning sequences, indicative of students’ learning strategies. Legend from Figure 2 applies here, as well; interpretation of the axes is also the same as for Figure 2.

Figure 2. Distribution of learning actions throughout learning sequences (sessions) of (a) students with the exam scores above the 90 th percentile, and (b) students with scores below the 25 th percentile. The Y axis represents the proportion of certain type of action in each ‘point’ of the sequences, e.g., in case of top performing students (a), the first action in ~65% of the learning sequences was reading (green color); in ~10% of the learning sequences the first action was successful completion of the summative assessment (yellow); in ~5% of the sequences, the first action was summative assessment done incorrectly (violet), and so on. Learning action abbreviations are outlined in the figure legend and briefly explained in Table 1.

Table 1 Types of learning actions examined in the study

Figure 4. Change in the applied learning strategies for each of the five student clusters over the 12 weeks of the course. Y-axis represents the median number of learning sequences per each of the 4 learning strategies.

Full text

Edinburgh Research Explorer
Learning analytics to unveil learning strategies in a flipped
classroom
Citation for published version:
Jovanovic, J, Gasevic, D, Dawson, S, Pardo, A & Mirriahi, N 2017, 'Learning analytics to unveil learning
strategies in a flipped classroom', Internet and Higher Education, vol. 33, pp. 74–85.
https://doi.org/10.1016/j.iheduc.2017.02.001
Digital Object Identifier (DOI):
10.1016/j.iheduc.2017.02.001
Link:
Link to publication record in Edinburgh Research Explorer
Document Version:
Peer reviewed version
Published In:
Internet and Higher Education
General rights
Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)
and / or other copyright owners and it is a condition of accessing these publications that users recognise and
abide by the legal requirements associated with these rights.
Take down policy
The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer
content complies with UK legislation. If you believe that the public display of this file breaches copyright please
contact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately and
investigate your claim.
Download date: 09. Aug. 2022

Learning Analytics to Unveil Learning Strategies
in a Flipped Classroom
Jelena Jovanović
1
, Dragan Gašević
2
, Abelardo Pardo
3
, Shane Dawson
4
, Negin Mirriahi
5
1
Faculty of Organizational Sciences, University of Belgrade, Serbia
2
Moray House School of Education and School of Informatics, University of Edinburgh, United
Kingdom
3
School of Electrical and Information Engineering, University of Sydney, Australia
4
Teaching Innovation Unit, University of South Australia, Australia
5
School of Education, University of New South Wales, Australia
1. Introduction
Prior education studies have consistently emphasized the importance of sustained and active
student engagement to aid academic performance and achievement of learning outcomes (e.g.,
Michael, 2006; Hockings, Cooke, Yamashita, McGinty, & Bowl, 2008). The positive impact of such
active learning models on academic outcomes has been well established, particularly, in the STEM
disciplines. For example, Freeman et al (2014) demonstrated that students undertaking STEM
courses incorporating active learning models received (on average) higher academic grades and
were less likely to fail in comparison to peers in more traditional and lecture based modes of
teaching. While active learning has clear benefits for student learning outcomes, the process of
implementation is often more complex than first anticipated (Gillies & Boyle, 2010; Hung, 2011). For
instance, student engagement in active learning does not occur spontaneously and educators must
employ careful consideration of the curriculum design, activity sequencing and progression as well
as the diversity of learners, including learners' prior experience and motivation, background and
knowledge.
Flipped learning (FL) is a form of blended learning that requires students’ active participation in
learning activities both before and during face-to-face sessions with the teacher (Lage, Platt, &
Tregua, 2000). However, students frequently lack the necessary skills, time, and/or motivation to
fully participate in pre-class activities and therefore do not commit to the level of involvement in the
learning process that effectively complements the intended design (Lai & Hwang, 2016; Mason,
Shuman, & Cook, 2013). Clearly, the reasoning for why students may or may not engage in pre-
class activities is complex and multi-dimensional. However, if provided with a deeper insight into the
types of learning strategies students employ in such active learning models, teaching staff can make
better informed decisions regarding student support and course design processes (Stief & Dollar,
2009).
Despite the increasing popularity of FL and similar active learning models, there has been limited
attention devoted to understanding the types of learning strategies that students employ when
engaged in this model of education. Studies on FL have to date, primarily focused on examining
students’ satisfaction with this mode of learning and their course performance (O’Flaherty & Phillips,
2015; Bishop & Verleger, 2013). However, considering that FL encourages students' sense of
autonomy and ownership of learning and is quite different to the ‘traditional’ lecture model, it is

important to shed some light on how students approach and manage this new learning setting, and
how they organize and regulate their learning process. The relevance for undertaking such research
is further strengthened by studies noting that students often lack sufficient skills and proficiency to
modify their learning strategies to better suit the specificities of newly encountered learning
situations (Lust, Elen, & Clarebout, 2013a). Consequently, students often employ suboptimal
learning tactics and strategies (Winne & Jamieson-Noel, 2003).
Research into student learning tactics and strategies has primarily relied on self-reports that are
typically collected through questionnaires or think-aloud protocols (Bannert, Reimann, &
Sonnenberg, 2013; Chamot, 2005; Hill & Hannafin, 1997). While these studies have provided
insights into the student learning process, there are several inherent deficiencies that have
effectively limited the generalizability of the findings. For instance, self-reports are often inaccurate
due to the poor recall of prior behavior related to the use of study tactics (Winne & Jamieson-Noel,
2002). Similarly, think aloud protocols are negatively impacted by the increased level of cognitive
load placed on the participants (Winne, 2013). However, given that contemporary FL activities are
typically delivered via an online medium (e.g. Learning Management System - LMS) there is a new
opportunity to draw on alternate analytic approaches derived from the fields of learning analytics
and educational data mining (Siemens, 2013; Gašević, Dawson, & Siemens, 2015). Essentially, the
deficiencies commonly associated with self-report protocols can be overcome by grounding the
analysis in the users’ trace data i.e. data collected from the tools and services the students interact
with during the learning process (Winne, 2013; Stief & Dollar, 2009). Such learning analytic
approaches provide a direct analysis of the users’ “actual” behavior in lieu of the students’
perception and recall of events.
The present study examined students’ learning strategies by using the trace data collected from the
University’s LMS. The study focuses on the trace data originating from the preparatory activities that
students were requested to complete prior to the scheduled face-to-face sessions (i.e., lectures) in a
first-year undergraduate course in computer engineering. The rationale for focusing on this
component of the FL design centers on the importance of the preparation activities to facilitate and
enable student participation in the face-to-face sessions (Rahman, Aris, Rosli, Mohamed, Abdullah,
& Zaid, 2015).
The educational research community offers a diversity of interpretations on what constitutes a
learning strategy. In this work we rely on the broad definition developed by Weinstein, Husman, &
Dierking (2000, p. 227) suggesting that a learning strategy includes “any thoughts, behaviors,
beliefs or emotions that facilitate the acquisition, understanding or later transfer of new knowledge
and skills”. We consider students’ learning strategies as latent constructs that cannot be directly
observed in the collected traces, but have to be mined/detected using appropriate analytical
methods and techniques. Unsupervised methods such as clustering and sequential pattern mining
have proven beneficial for mining latent, unobservable constructs from learning traces (see e.g.,
Perera, Kay, Koprinska, Yacef, & Zaïane, 2009; Jeong, Biswas, Johnson, & Howard, 2010;
Kovanovic, Gašević, Joksimović, Hatala, & Adesope, 2015; Lust et al., 2013a; Blikstein et al., 2014).
In this study, we make a combined use of exploratory sequence analysis and agglomerative
hierarchical clustering to detect patterns in student behaviour that are indicative of the adopted
learning strategies.

1.1 Active learning and Flipped learning
The earlier work of Trigwell, Prosser, & Waterhouse (1999) clearly demonstrated the impact that a
teaching model can play on a student’s approach to learning. In essence, Trigwell, et al. noted that
a student’s choice between a surface or deep approach to learning is dependent on the instructor’s
approach to teaching. For instance, a teacher-focused approach oriented towards information
transmission tends to evoke a surface approach to learning. In contrast, a student-focused
approach aimed at assisting learners in changing their conceptions of the studied phenomena
results in a deeper approach to learning. This latter model of teaching is akin to active learning and
shares a lot of similarities with FL. Hence, the study by Trigwell et al. (1999), with 48 first year
science classes, strongly suggests that active learning strategies can engage students in a deep
approach to learning, and therefore lead to the development of higher learning outcomes (Trigwell &
Prosser, 1991). FL assumes that students are not only actively participating in the classroom
activities, but that they are also actively engaging in pre-class and/or post-class activities. This level
of active engagement in studies throughout the course, leads to improved academic outcomes.
To compare student performance in undergraduate STEM courses with traditional lecturing and
active learning approaches Freeman et al. (2014) undertook a meta-analysis of 225 studies. The
authors examined two outcome measures: the failure rate in courses and student performance on
tests. They observed that students in traditional lecture courses were 1.5 times more likely to fail
than students in courses with an active learning design. Regarding the test performance, the meta-
analysis showed that on average, student performance on identical or comparable tests increased
by about a half a standard deviation when active learning methods were deployed compared to
traditional lectures. The observed benefits of active learning in the Freeman et al. meta-analytic
study were consistent across all STEM disciplines, including different levels of courses, and different
experimental methodologies. The highest impacts were observed in primary studies where the
majority of class time was devoted to active learning. Freeman et al. (2014) also pointed to evidence
that active learning tends to have a greater impact on student mastery of higher versus lower-level
cognitive skills.
Although FL as a form of active learning has been around for over 15 years, it has only recently
seen an increase in adoption and interest within the education community (Bishop & Verleger, 2013;
Hamdan, McKnight, & McKnight, 2013). As such, FL as an approach to enhance student learning
remains under-evaluated and under-researched in general (Abeysekera & Dawson, 2015). Previous
studies examining FL predominantly relied on questionnaires and interviews to collect students’
opinions and perceptions of FL, whereas pre- and post-tests and course grades were used to
assess the extent of improvement in students’ performance (O’Flaherty & Phillips, 2015). The
majority of the reported studies confirmed the educational benefits associated with FL models, such
as increased student satisfaction (e.g., Forsey, Low, M., & Glance, 2013), higher course grades
(e.g., Pierce & Fox, 2012), and increased attendance (e.g., Prober & Khan, 2013). Despite these
noted benefits to learners, O’Flaherty & Phillips (2015) warn fellow educators not to rush to
conclusions regarding the advantages of FL over more traditional lectures. In particular, O’Flaherty
& Phillips found that there were “very few studies that actually demonstrated robust evidence to
support that the flipped learning approach is more effective than conventional teaching methods.”
(p.94). Clearly, further work is required to provide greater methodological rigor associated with such
comparative analyses.
An important and challenging aspect affecting student success in FL setting is the high level of
learner autonomy associated with a FL design (Kim, Kim, Khera, & Getman, 2014). This model of

active learning requires students to be self-regulated learners in order to undertake and complete
the preparatory activities (Lai & Hwang, 2016; Mason et al., 2013; Sletten, 2015). However, many
students have underdeveloped self-regulation skills and need support and scaffolding to manage
their learning in less familiar and more intensive settings that often characterize FL designs. To
address this need, the FL design examined in this paper has a well-defined structure that is
consistent throughout the entire course duration (see Section 2.1).
1.2 Learning strategies and Self-regulated learning
There has been much research undertaken related to student learning strategies. Authors such as
Pask & Scott (1972) examined learning strategies in relation to students' cognitive competences.
The authors identified discrete learning strategies as behavioral patterns that were adopted by
students when attempting to solve a given learning task. Pask and Scott (1972), demonstrated that
the adopted strategies were related to a student's cognitive competence. In particular, they noted
that students with similar cognitive competences tended to adopt similar behavioural patterns (i.e.
learning strategies), and that the students' learning success was dependent on how well the
adopted learning strategy matched the instructor's teaching strategy. Pintrich & de Groot (1990)
examined the relationship between students' motivation, self-regulation, cognitive strategies used,
and performance on classroom academic tasks. They found that self-regulated learning (SRL) was
closely tied to a student’' efficacy beliefs and the intrinsic value they associated with the study tasks.
However, self-efficacy and intrinsic values, as motivational components, are not sufficient to lead to
successful academic performance, but have to be concomitant with SRL components (self-
regulation and cognitive strategy use) as the latter are noted to be more directly implicated in the
students' academic performance. Moreover, Pintrich & de Groot's (1990) findings suggest that the
adopted cognitive strategy must be coupled with self-regulation to aid overall academic
performance. In other words, apart from being aware of possible learning strategies, students must
also know how and when to use specific strategies. This is particularly the case in FL settings where
learners are expected to take control of and be responsible for their own learning, including making
decisions on how to utilize the available learning resources and what strategies to apply (Lai &
Hwang, 2016). Considering these findings, and confirmed by several other research studies (see
Section 1.4), the present study models our understanding of learning strategies through the lens of
SRL. We view SRL as a set of actions and processes that are well thought of, planned and
employed for the purposes of learning new skills and knowledge. The employment of such actions
and processes implies there is a level of learner agency and autonomy to monitor and evaluate the
effectiveness of the adopted learning strategy and modify where necessary (Winne, 2013).
The capacity of a student to choose and adapt their learning strategy in accordance with the
requirements of the learning setting is a key self-regulatory skill (Winne, 2006). Unfortunately,
students often have poorly developed self-regulation skills and tend to choose suboptimal learning
strategies (Winne & Jamieson-Noel, 2003). Furthermore, previous research has shown that learners
are not accurate reporters of how they study and what strategies they apply (Zhou & Winne, 2012).
These findings have two important implications. First, learners would benefit from scaffolds that
make them aware of their learning strategies, so that they can identify if, when and where they can
make adjustments to enhance their learning experience. According to Winne, to improve learning,
students “might profit from (a) feedback that accurately represents how they actually studied and (b)
information about tactics and strategies that might be more effective than those they actually used”
(Winne, 2013, p.387). Second, the inaccuracy of students’ self-reports indicates that such data
collection methods should not be used as the primary or sole source of data for examining students