1
Gaze Tutor: A Gaze-Reactive Intelligent Tutoring System
Sidney D’Mello
1
, Andrew Olney
2
, Claire Williams
2
, and Patrick Hays
2
1
Departments of Computer Science and Psychology
384 Fitzpatrick Hall, University of Notre Dame
Notre Dame, IN 46601, USA
[sdmello@nd.edu]
2
Institute for Intelligent Systems and Psychology Department
202 Psychology Building, University of Memphis,
Memphis, TN 38152, USA.
[aolney|mcwllams|dphays]@memphis.edu
Corresponding author
Sidney D’Mello
384 Fitzpatrick Hall, University of Notre Dame
Notre Dame, IN 46601, USA
Email: sdmello@nd.edu
Telephone: (001) (1) 901-378-0531
ABSTRACT
We developed an Intelligent Tutoring System (ITS) that aims to promote engagement
and learning by dynamically detecting and responding to students’ boredom and
disengagement. The tutor uses a commercial eye tracker to monitor a student’s gaze
patterns and identify when the student is bored, disengaged, and has zoned out. The
tutor then attempts to reengage the student with dialogue moves that direct the student
to reorient his or her attentional patterns towards the animated pedagogical agent
embodying the tutor. We evaluated the efficacy of the gaze-reactive tutor in promoting
learning, motivation, and engagement in a controlled experiment where 48 students
were tutored on four biology topics with both gaze-reactive and non gaze-reactive
(control condition) versions of the tutor. The results indicated that: (a) gaze-sensitive
dialogues were successful in dynamically reorienting students’ attentional patterns to
the important areas of the interface, (b) gaze-reactivity was effective in promoting
learning gains for questions that required deep reasoning, (c) gaze-reactivity had
minimal impact on students’ state motivation and on self-reported engagement, and (d)
individual differences in scholastic aptitude moderated the impact of gaze-reactivity on
overall learning gains. We discuss the implications of our findings, limitations, future
work, and consider the possibility of using gaze-reactive ITSs in classrooms.
Keywords—affective computing, affect-sensitive ITS, boredom, disengagement, eye
tracking, gaze-sensitive dialogues, Intelligent Tutoring Systems (ITSs), zoning out
2
1. INTRODUCTION
Intelligent Tutoring Systems (ITSs) have emerged as effective tools to promote active
knowledge construction by capitalizing on the merits of one-on-one human tutoring in
an automated fashion (Graesser, Conley, & Olney, in press; Psotka, Massey, & Mutter,
1988; Sleeman & Brown, 1982; Woolf, 2009). ITSs are increasingly being used in
classrooms all over the United States, and the ones that have been successfully imple-
mented and tested have produced learning gains with average effect sizes ranging from
0.79 to 1 sigma
1
(Corbett, 2001; Koedinger, Anderson, Hadley, & Mark, 1997;
VanLehn, 2011; VanLehn et al., 2007). When compared to classroom instruction and
other naturalistic controls, the 1.0 effect sizes obtained by ITSs is superior to the .39
sigma effect for computer-based training, the .50 sigma effect for multimedia, and the
.40 sigma effect obtained by novice human tutors (Cohen, Kulik, & Kulik, 1982; Cor-
bett, 2001; Dodds & Fletcher, 2004; Wisher & Fletcher, 2004).
Despite their impressive successes, it is important to note that ITSs are not the pan-
acea for all the problems associated with learning. Although most ITSs are effective at
supporting students’ cognitive needs, until recently, they have made less of an effort to
promote student engagement, motivation, and interest in learning. This is a serious limi-
tation that reduces the efficacy of these systems because engagement, motivation, and
interest are precursors to learning, effortful problem solving, and deep thinking (Ber-
lyne, 1978; Craig, Graesser, Sullins, & Gholson, 2004; Csikszentmihalyi, 1990). Stu-
dents might begin a learning session with an ITS with some level of interest and enthu-
siasm, but boredom inevitably creeps in as the session progresses, when the novelty of
the system and content fades, and when they have difficulty comprehending the materi-
al (Csikszentmihalyi, 1990; D’Mello & Graesser, in press; Larson & Richards, 1991;
Mann & Robinson, 2009; Moss et al., 2008; Pekrun, 2010; Pekrun, Goetz, Daniels,
Stupnisky, & Raymond, 2010). When boredom strikes, students’ interest wanes to a
point where they give up and eventually disengage from the learning session. At this
point, any further instruction is essentially futile.
Recent work, albeit outside of learning contexts, has investigated how engagement
can be maintained in computer based interventions over extended periods of time (e.g.,
months) (Bickmore, Schulman, & Yin, 2010; Bickmore & Picard, 2005). For example,
in a series of studies measuring long term engagement with an animated agent for a
health intervention, Bickmore and colleagues (2010) demonstrated that increasing the
variability of the agent’s behavior and adding a backstory to the agent increased the
amount of time users spent with the system. Some research has also examined whether
polite or direct strategies are more effective at maintaining engagement. When users are
switching tasks, polite interruptions, as measured by the annoyingness of computer
1
An effect-size measures the strength of a relationship between two variables. Cohen’s d (see below) is
a common measure of effect size in standard deviation units between two samples with means and
and standard deviations and . According to (Cohen, 1992), effect sizes approximately equal to .3,
.5, and .8 represent small, medium, and large effects, respectively. . In
learning contexts, an effect size of 1.0 sigma is roughly equivalent to an improvement of one letter grade.
3
beep/alert sounds, has been shown to increase user compliance in a health care inter-
vention, whereas impolite interruptions had the opposite effect (Bickmore, Mauer, Cre-
spo, & Brown, 2007). However, in negotiation settings, angry/threatening statements
such as “This is a ridiculous offer, it really pisses me off” were found to lead to greater
user concessions than neutral or happy/non-threatening statements (de Melo, Carnevale,
& Gratch, 2010). The different effects suggest that task-specific factors, e.g. multi-
tasking vs. single-tasking, may strongly influence whether polite/non-threatening agent
behaviors enhance successful outcomes more than annoying/threatening behaviors.
One perspective on this recent work is to consider the problem of engagement as
two complimentary subproblems operating on different timescales. The first is disen-
gagement repair. Disengagement occurs within a session and prevents the user from
completing that session successfully. Disengagement repair requires refocusing the us-
er’s attention and increasing his or her motivation to complete the task at hand. The
second engagement subproblem is maintaining sustained engagement across multiple
sessions. Sustained engagement requires making the sessions compelling enough so
that the risk of disengagement is minimized within a session and attrition across ses-
sions is low. These two subproblems are related, since a user who becomes disengaged
during a single session may be less likely to engage in a future session. However, work
on sustained engagement in computer-based interventions has not directly addressed
disengagement repair.
The present paper focuses on disengagement repair strategies within the context of
learning environments. Before articulating the specific disengagement-repair strategy
we have implemented, we review some findings on the prevalence, antecedents, and
consequences of boredom and disengagement during learning.
1.1. Boredom and Disengagement during Learning
When compared to cognitive constructs such as attention and memory, or basic emo-
tions such as anger and disgust (Ekman, 1992), the scientific research on boredom dur-
ing complex learning is relatively sparse and scattered. For example, the number of
studies on boredom and engagement in educational contexts is negligible when com-
pared to the approximately 1,000 studies on test anxiety (Hembree, 1988; Pekrun et al.,
2010; Zeidner, 2007). Nevertheless, some theoretical models of the cognitive and affec-
tive processes that underlie boredom have emerged (Larson & Richards, 1991; Mann &
Robinson, 2009). The understimulation model (Perkins & Hill, 1985) posits that bore-
dom arises when the student is physiologically and cognitively underaroused, presuma-
bly due to the monotony of repetitive tasks that have been habituated (e.g., solving nu-
merous algebraic problems once the basic concepts have been mastered). The forced-
effort model (Larson & Richards, 1991; Robinson, 1975) claims that students will expe-
rience more boredom when they are required to invest considerable mental effort in
tasks that are beyond their control (e.g., forced to suffer through a lecture when there is
no intrinsic motivation to learn).
According to Pekrun’s control-value theory of emotions, subjective appraisals of
control and value of a learning activity are the critical predictors of engagement
(Hulleman, Durik, Schweigert, & Harackiewicz, 2008; Pekrun, 2010; Pekrun, Elliot, &
4
Maier, 2006). Subjective control pertains to the perceived influence that a student has
over the activity, while subjective value represents the perceived value of the outcomes
of the activity. Boredom occurs when perceived value or control are low, as would be
the case when an unmotivated student (low value) is attempting to solve math problems
that far exceed his or her ability (low control) (Csikszentmihalyi, 1975). Boredom has
also been hypothesized to occur when control is too high, as is the case when skills
greatly outweigh challenges and the student is understimulated (Pekrun et al., 2010).
In addition to these theoretical perspectives, boredom has recently been gaining
some attention in studies that investigate the links between affect and cognition during
learning (Baker, D'Mello, Rodrigo, & Graesser, 2010; Beck, 2005; Cocea & Weibel-
zahl, 2009; D’Mello & Graesser, in press; Drummond & Litman, 2010; Moss et al.,
2008; Pekrun et al., 2010). Available data suggest a number of conclusions pertaining to
the incidence and effects of boredom during learning. These conclusions are summa-
rized below.
1.1.1. Prevalence of boredom. Boredom is one of the most frequent affective states
that students experience during learning, irrespective of the learning context, content
area, task, student population, and method used to track affect (see D'Mello (in review)
for a meta-analyses). Boredom is not only prevalent with computer learning environ-
ments, but is also observed in human-human tutoring sessions. For example, an analysis
of several tutoring sessions with expert human tutors indicated that students spent a
fourth of the time merely socially attending to the tutor instead of actively learning the
material (Lehman, Matthews, D'Mello, & Person, 2008).
1.1.2. Hindrance to learning and performance. As could be expected, boredom
negatively correlates with learning gains (Craig et al., 2004; D’Mello & Graesser, 2011;
Forbes-Riley & Litman, 2011; Schutz & Pekrun, 2007), presumably because bored stu-
dents have trouble focusing attention (Fisher, 1993; Thackray, 1981) or are simply not
willing to process the material at deeper levels of comprehension.
1.1.3. Persistent temporal quality. Boredom adopts a persistent temporal quality
upon activation (D’Mello & Graesser, 2011), where students wallow in their ennui and
are less likely to be reengage in the material. This form of persistent boredom is a nega-
tive predictor of learning gains. More importantly, the typical tutorial interventions
(e.g., feedback, hints) are not very effective in alleviating boredom, indicating that nov-
el pedagogical and motivational strategies are required to increase task persistence.
1.1.4. Gateway into negative affect. Consistent with predictions of the forced-
effort model (Larson & Richards, 1991), bored students are more likely to transition
into frustration (D'Mello & Graesser, 2010a). Frustration is another affective state that
is harmful to learning (Linnenbrink & Pintrich, 2002). Persistent frustration can also
transition into boredom if the student is stuck and simply gives up.
1.1.5. Catalyst for harmful behaviors. Bored students also engage in problematic
behaviors such as going off-task, zoning out, intentionally misusing the learning envi-
ronment (i.e., gaming the system), or simply becoming careless. These behaviors, and
boredom in general, lead to lower self-efficacy, diminished interest in educational activ-
ities, increased attrition and dropout, and eventually lead to poorer learning (Baker et
al., 2010; Cocea, Hershkovitz, & Baker, 2009; Craig et al., 2004; Drummond & Lit-
man, 2010; Moss et al., 2008).
5
1.1.6. Long-term effects of boredom. In addition to the short-term effects of negli-
gible or even negative learning gains, boredom in educational activities is diagnostic of
lower self-efficacy, lack of motivation in learning, hostility and dissatisfaction towards
school, abnormal behavior in school, lower work satisfaction, and diminished work
output (Fogelman, 1976; McGiboney & Carter, 1988; Perkins & Hill, 1985; Robinson,
1975; Wasson, 1981).
Given this sketch of the harmful effects of boredom on learning, it is important for
ITSs to be more than mere cognitive machines, because preventing waning attention,
zoning out, disengagement, and boredom are critically important for learning (Calvo &
D’Mello, 2010; del Soldato & du Boulay, 1995; Woolf, 2009). Fortunately, as high-
lighted in the next section, there has been a recent emergence of research along this
front.
1.2 Disengagement Diagnosis and Repair
A number of research groups have been addressing the problem of building learning
environments that detect and respond to affective states such as boredom, confusion,
frustration, and anxiety (Afzal & Robinson, 2009; Burleson & Picard, 2007; Chaffar,
Derbali, & Frasson, 2009; Conati & Maclaren, 2009; D'Mello & Graesser, 2010b;
D'Mello, Lehman, Sullins et al., 2010; Forbes-Riley, Rotaru, & Litman, 2008; Robison,
McQuiggan, & Lester, 2009; Woolf et al., 2010). These systems use state-of-the art
sensing technologies and machine learning techniques to automatically detect student
affect by monitoring facial features, speech contours, body language, interaction logs,
language, and peripheral physiology (e.g., electromyography, galvanic skin response)
(see (Calvo & D’Mello, 2010) for an overview). These affect-sensitive systems then
alter their pedagogical and motivational strategies in a manner that is dynamically re-
sponsive to the sensed affective states. Some of the implemented responses to student
affect include affect mirroring (Burleson & Picard, 2007), empathetic responses (Woolf
et al., 2010), and a combination of politeness, empathy, encouragement, and incremen-
tal challenge (D'Mello, Lehman, Sullins et al., 2010).
Although these affective-response strategies have the potential of alleviating certain
negative emotions (e.g., frustration), an effective response to boredom must address
attention due to the inextricable link between attention and engagement (Fisher, 1993;
Pekrun et al., 2010; Thackray, 1981). That is, engagement can be conceptualized as a
state of involvement with a task such that concentration is intense, attention is focused,
and involvement is moderate to complete (Baker et al., 2010; Csikszentmihalyi, 1975;
Csikszentmihalyi, 1990). Engagement is a multifaceted construct encompassing both
cognitive and affective components. Some of the cognitive aspects of engagement in-
clude attention and concentration, while the affective components consist of modula-
tions in arousal and valence (D’Mello, Chipman, & Graesser, 2007; Mandler, 1984;
Pekrun et al., 2010).
Attention, which is one important cognitive component of engagement, is the focus
of the present paper. Attention is critical because maintaining engagement in a learning
activity requires attentional resources. Therefore, developing interventions that monitor
periods of waning attention and attempt to encourage more productive use of attention-
