A conversational intelligent tutoring system to automatically predict learning styles
Annabel Latham
a
, Keeley Crockett
a
, David McLean
a
, Bruce Edmonds
b
a
The Intelligent Systems Group, School of Computing, Mathematics and Digital Technology
b
The Centre for Policy Modelling
The Manchester Metropolitan University, Chester Street, Manchester M1 5GD, UK
Email: a.latham@mmu.ac.uk, k.crockett@mmu.ac.uk, d.mclean@mmu.ac.uk, b.edmonds@mmu.ac.uk
Abstract
This paper proposes a generic methodology and architecture for developing a novel conversational intelligent
tutoring system (CITS) called Oscar that leads a tutoring conversation and dynamically predicts and adapts to a
student’s learning style. Oscar aims to mimic a human tutor by implicitly modelling the learning style during
tutoring, and personalising the tutorial to boost confidence and improve the effectiveness of the learning
experience. Learners can intuitively explore and discuss topics in natural language, helping to establish a deeper
understanding of the topic. The Oscar CITS methodology and architecture are independent of the learning styles
model and tutoring subject domain. Oscar CITS was implemented using the Index of Learning Styles (ILS)
model (Felder & Silverman 1988) to deliver an SQL tutorial. Empirical studies involving real students have
validated the prediction of learning styles in a real-world teaching/learning environment. The results showed
that all learning styles in the ILS model were successfully predicted from a natural language tutoring
conversation, with an accuracy of 61-100%. Participants also found Oscar’s tutoring helpful and achieved an
average learning gain of 13%.
Keywords:
Architectures for educational technology system
Human-computer interface
Intelligent tutoring systems
Interactive learning environments
Teaching/learning strategies
1. Introduction
The widespread use of computers and access to the Internet has created many opportunities for online education,
such as improving distance-learning and classroom support. Intelligent Tutoring Systems (ITS) extend
traditional content-delivery computerised learning systems by adding intelligence to improve the effectiveness
of a learner’s experience (Brusilovsky & Peylo 2003). This normally involves personalising tutoring using
factors such as learner knowledge, emotion or learning style to alter the sequence and style of learning material.
Most ITS are hyperlink menu based (Cha, Kim, Park, Yoon, Jung & Lee 2006; Klasnja-Milicevic, Vesin,
Ivanovic & Budimac 2011; Popescu 2010; Wang, Wang & Huang 2008) and adapt the tutoring by reordering
menu items (Garcia, Amandi, Schiaffino & Campo 2007), allowing learners to manage their own study at a time
and place to suit them. This experience has more in common with computerised textbooks than classroom
tutorials, where human tutors direct the learning. An extension of ITS is Conversational Intelligent Tutoring
Systems (CITS), which integrate natural language interfaces rather than menus, allowing learners to explore
topics through discussion and to construct knowledge as they would in the classroom. However, it is a complex
and time consuming task to develop a CITS which can converse naturally. Consequently only a few CITS exist
at present (D’Mello, Lehman, Sullins, Daigle, Combs, Vogt et al 2010; Arnott, Hastings & Allbritton 2008;
Sarrafzadeh, Alexander, Dadgostar, Fan & Bigdeli 2008).
A CITS that can imitate a human tutor by leading an adaptive tutorial conversation uses a familiar format which
can help improve learner confidence and motivation, leading to a better learning experience. Human tutors adapt
their tutoring style and content based on cues they pick up from learners, such as their level of understanding
and learning style. Learning styles model the way groups of people prefer to learn (Felder & Silverman 1988;
Hsieh, Jang & Hwang 2011), for example by active experimentation or by observation. Some ITS adapt tutoring
to an individual’s learning style, either determined using a formal questionnaire (Papanikolaou, Grigoriadou,
Kornilakis & Magoulas 2003) or by analysing learner behaviour (Kelly & Tangney 2006). However, there are
no tutor-led CITS that can predict and adapt to learning style during the tutoring session like a human tutor.
This paper describes the architecture and methodology for creating a novel CITS called Oscar that dynamically
predicts and adapts to an individual’s preferred learning style during a tutorial conversation. The aim of the
research was to imitate a human tutor by using knowledge of learning styles and learner behaviour to predict
learning style rather than an interface specifically designed to capture learning styles, as in (Cha et al 2006).
Whilst this considerably increases the complexity of predicting learning styles, conversational interfaces are
intuitive to use and the discussion of problems can prompt a deeper understanding of topics. This paper also
Page 1
describes a series of experiments that aim to determine if it is possible to predict learning style from a learner’s
behaviour during a tutorial conversation, and thus validate the proposed methodology and architecture.
In this paper, section 2 introduces some background and related work of intelligent tutoring systems,
conversational agents and the Index of Learning Styles (Felder & Silverman 1988). Section 3 introduces the
Oscar CITS, and Sections 4 and 5 describe a generic methodology and architecture for creating an Oscar CITS.
Section 6 describes the implementation of Oscar CITS and the real-world experiments conducted to investigate
the prediction of learning styles from a natural language tutoring dialogue. Section 7 presents the experimental
results and discussion and Section 8 outlines the conclusions and future work.
2. Related work
2.1. Intelligent tutoring systems
Computerised learning systems were traditionally information-delivery systems developed by converting tutor
or distance-learning material into a computerised format (Brooks, Greer, Melis & Ullrich 2006). The popularity
of the Internet has enhanced the opportunities for e-learning, however most online systems are still teacher-
centred and take little account of individual learner needs (Spallek 2003). Within the field of computerised
learning systems, adaptive educational systems attempt to meet the needs of different students by offering
individualised learning (Brusilovsky & Peylo 2003). Intelligent Tutoring Systems (ITS) are adaptive systems
which use intelligent technologies to personalise learning according to individual student characteristics, such as
knowledge of the subject, mood and emotion (D’Mello et al. 2010) and learning style (Yannibelli, Godoy &
Amandi 2006).
There are three main approaches to intelligent tutoring (Brusilovsky & Peylo 2003):
Curriculum sequencing introduces adaptation by presenting students with learning material in a sequence and
style best suited to their needs (Klasnja-Milicevic et al 2011).
Intelligent solution analysis adds intelligence to ITS by giving students detailed feedback on incomplete or
erroneous solutions, helping them learn from their mistakes (Mitrovic 2003).
Problem solving support techniques offer learners intelligent assistance to reach a solution (Melis, Andres,
Budenbender, Frishauf, Goguadse, Libbrecht et al 2001).
Curriculum sequencing is the most widely used technique (Brusilovsky and Peylo 2003). Traditionally ITS
adapt to existing student knowledge but more recently learner affect factors have been incorporated, such as
emotion (Ammar, Neji, Alimi & Gouarderes 2010), personality (Leontidis & Halatsis 2009) and learning style
(Popescu 2010). Few ITS incorporate all three techniques as they are complex and time-consuming to develop,
but the Oscar CITS presented in this paper will incorporate all three intelligent technologies by personalising
learning material and discussing problems and solutions with students.
ITS are normally menu or hyperlink based, with choices re-ordered or ranked to recommend a particular
sequence to learners (Klasnja-Milicevic et al 2011; Garcia et al 2007). Whilst this design simplifies the capture
of learner behaviour and choices, learners are really being assisted in self-learning rather than tutored, which is
little different from recommending chapters of a book. CITS address this issue by employing natural language
interfaces whose intuitive, dialogue-based tutoring is more comparable to classroom tutorials (Chi, Siler, Jeong,
Yamauchi & Hausmann 2001; D’Mello et al 2010; Sarrafzadeh et al 2008). However, despite their more
instinctive, teacher-led learning experience (which supports the construction of knowledge adopted by human
tutors), it is difficult to automate natural conversations and so CITS are uncommon (D’Mello et al 2010; Woo
Woo, Evens, Freedman, Glass, Seop Shim, Zhang et al 2006; Sarrafzadeh et al 2008).
ITS that adapt to learning styles capture them using a formal questionnaire (Papanikolaou et al 2003) or by
analysing learner behaviour (Cha et al 2006; Garcia et al 2007). Whilst questionnaires are the simplest measure
of learning styles, learners find them onerous and may not lend enough time or attention to complete them
accurately (Yannibelli, Godoy & Amandi 2006). Implicitly modelling learning styles by analysing learner
behaviour history (Garcia et al 2007) removes the requirement for a questionnaire, but delays adaptation until
several modules have been completed. Also, this method does not incorporate changes in learning style which
may occur over time or for different topics. EDUCE (Kelly & Tangney 2006) and WELSA (Popescu 2010) both
estimate learning style dynamically for curriculum sequencing, but do not include a conversational interface or
other intelligent tutoring technologies. The Oscar CITS will dynamically predict learning style throughout the
tutoring conversation and adapt its intelligent tutoring style to suit the learning style predicted.
2.2. Conversational agents
Conversational agents (CAs) are computer programs which allow people to communicate with computer
systems using natural language (O’Shea, Bandar & Crockett 2011). CA interfaces are intuitive to use, and have
been used effectively in many applications, such as web-based guidance (Latham, Crockett & Bandar 2010),
database interfaces (Pudner, Crockett & Bandar 2007) and tutoring (D’Mello et al 2010). CAs can add natural
Page 2
dialogue to ITS, but are used infrequently as they are complex and time-consuming to develop, requiring
expertise in the scripting of dialogues (O’Shea, Bandar & Crockett 2011). ITS which aim to mimic a human
tutor (such as Oscar CITS) need CA interfaces to support the construction of knowledge through discussion (Chi
et al 2001).
Textual CAs usually adopt a pattern matching (Michie 2001) or semantic based (Li, Bandar, McLean & O’Shea
2004; Khoury, Karray & Kamel 2008) approach. Semantic-based CAs seek to understand the meaning of the
natural language whereas pattern-matching CAs use an algorithm to match key words and phrases from the
input to a set of pattern-based rules (Pudner, Crockett & Bandar 2007). As pattern matching CAs match key
words within an utterance, they do not require grammatically correct or complete input. However, there are
usually numerous patterns in a given context (Sammut 2001), leading to many hundreds of rules in the CA’s
knowledge base, which demonstrates the complexity and time required to script rules for a pattern-matching
CA. A pattern matching CA was adopted for Oscar CITS as it must cope with grammatically incomplete or
incorrect utterances that are commonly found in text-based chat by students.
2.3. Index of learning styles
The Index of Learning Styles (ILS) model (Felder & Silverman 1988) describes the teaching and learning styles
in engineering education. The ILS model represents an individual’s learning style as points along four
dimensions that indicate both the strength and the nature of their learning style preference. Each learning style
dimension relates to a step in the process of receiving and processing of information, as illustrated in Fig. 1. The
ILS is assessed using a 44-question forced-choice questionnaire (11 questions per learning style dimension), that
assigns a style and score for each dimension.
Fig. 1. ILS dimensions.
In addition to the formal assessment questionnaire, the ILS model describes typical learner behaviours that can
be used to informally group types of learners. The ILS model was adopted when implementing the Oscar CITS
as engineering students make up the initial experimental groups. However, the Oscar CITS is generic and its
flexible modular structure does not restrict the choice of learning styles model to the ILS.
3. Oscar CITS
The Oscar CITS is a novel conversational intelligent tutoring system which dynamically predicts a student’s
learning style during a tutoring conversation, and adapts its tutoring style appropriately. Oscar’s pedagogical
aim is to provide the learner with the most appropriate learning material for their learning style to promote a
more effective learning experience and a deeper understanding of the topic. Rather than being designed with the
purpose of picking up learning styles (such as Cha et al 2006) the Oscar CITS aims to imitate a human tutor by
leading a two-way discussion and using cues from the student’s dialogue and behaviour to predict and adapt to
their learning style. Oscar CITS incorporates intelligent technologies to sequence the curriculum according to
learner knowledge and learning style, intelligently analyse solutions and give hints to assist learners in
constructing knowledge. Oscar’s natural language interface and classroom tutorial style are modelled on
classroom tutorials (Crown copyright 2004), enabling learners to draw on their experience of face-to-face
tutoring to feel more comfortable and confident in using the CITS. Oscar CITS is an online personal tutor which
can answer questions, provide hints and assistance using natural dialogue, and which favours learning material
to suit each individual’s learning style. The Oscar CITS offers 24-hour personalised learning support at a fixed
cost.
General descriptions of Oscar CITS, including its implementation, example learner dialogue and the results of
initial studies in predicting learning styles, have been reported in Latham, Crockett, McLean, Edmonds &
O’Shea (2010) and Latham, Crockett, McLean & Edmonds (2010). The Oscar CITS adaptation strategies were
described in Latham, Crockett, McLean & Edmonds (2011), which reported empirical results showing that
students whose learning material matched their learning styles performed 12% better than those with unmatched
material.
The rest of this paper will describe an original methodology and architecture for creating an Oscar CITS and the
experiments conducted to investigate its success in predicting learning styles in a real teaching/learning
environment.
4. Predicting learning styles through natural language dialogue
CITS are complex and time-consuming to develop, requiring expertise in knowledge engineering (the capture
and formatting of expert knowledge (O’Shea, Bandar & Crockett 2011), such as tutoring, learning styles and
domain knowledge) and CA scripting. Formalising the development of a CITS which can be applied to different
learning styles models and tutoring domains will help to speed up the development. This section presents a
methodology for creating an Oscar CITS which can predict learning styles from a natural language dialogue.
4.1. Methodology for creating Oscar CITS
Page 3
The methodology for creating an Oscar CITS consists of three phases as shown in Table 1. The first phase of the
methodology relates to the creation of the learning styles module and the second phase to the tutorial subject
domain. The third phase incorporates the learning styles predictor and tutorial conversation into a CITS
architecture. Each phase will now be described.
Table 1.
3-Phase methodology for creating Oscar CITS.
Phase 1: Create the Learning Styles Predictor Module
1.1. Select a Learning Styles Model
a. Reduce the learning styles model if necessary
b. Extract the behaviour characteristics
1.2. Map learning style behaviour to the conversational tutoring style
1.3. Analyse the learning styles model for language traits
1.4. Adapt the generic logic rules to predict learning styles
Phase 2: Design a Tutorial Conversation
2.1. Capture the tutorial scenario and questions (including movies, voice, images, examples, etc.) from
human tutors in a specific domain
2.2. Determine the conversational structure/style
2.3. Map tutorial questions onto the generic question styles and templates
2.4. Script CA natural language dialogue for each tutorial question using the 3-level model
2.5. Link tutorial dialogue to logic rules through CA variables
Phase 3: Construct the CITS Architecture
4.2. Methodology phase 1: create the learning styles predictor module
4.2.1. Step 1.1: select a learning styles model
The first step in creating the learning styles predictor module requires a learning styles model (Felder &
Silverman 1988, Honey & Mumford 1992) to be selected. To illustrate and validate Phase 1 of the methodology,
the ILS model (Felder & Silverman 1988) was selected as the initial experimental group will be university
engineering students. The ILS questionnaire contains 44 questions, which is too many to incorporate into a
single tutoring session without being onerous for students. To reduce the ILS model, a study was undertaken to
investigate which were the best predictor questions (Latham, Crockett, McLean & Edmonds 2009). The study of
103 completed ILS questionnaires found that 17 questions predicted the overall learning style result in at least
75% of cases, with the top three questions predicting the result in 84% of cases. The resulting subset of the best
ILS predictor questions formed the basis of further analysis in developing the Oscar CITS strategy for the
prediction of learning styles.
The ILS model describes typical behaviour characteristics for each learning style. For clarity and ease of
analysis, the behaviour characteristics were extracted from the ILS model and summarised in a table of common
learner behaviour (Table 2).
Page 4
Table 2.
Typical learner behaviour characteristics extracted from the ILS model.
Sensor
Prefer facts, data, experimentation
Prefer solving problems using standard methods
Dislike surprises
Patient with detail
Do not like complications
Good at memorising facts
Careful but slow
Comfortable with symbols (eg. words)
Visual
Remember what they see
Like pictures, diagrams, flow charts, time lines, films
Prefer visual demonstration
Active
Do something with information – discuss/explain/test
Active experimentation
Do not learn much in passive situations (lectures)
Work well in groups
Experimentalists
Process information by setting up an experiment to test an
idea, or try out on a colleague
Sequential
Follow linear reasoning processes
Can work with material they have only partially or
superficially understood
Strong in convergent thinking and analysis
Learn best when information is presented in a steady
progression of complexity and difficulty
Intuitor
Prefer principles and theories
Prefer innovation
Dislike repetition
Bored by detail
Welcome complications
Good at grasping new concepts
Quick but careless
Uncomfortable with symbols
Verbal
Remember what they hear, or what they hear then say
Like discussion
Prefer verbal explanation
Learn by explaining to others
Reflective
Examine and manipulate information introspectively
Reflective observation
Do not learn much if no chance to think (lectures)
Work better alone
Theoreticians
Process information by postulating explanations/interpretations, drawing
analogies, formulating models
Global
Make intuitive leaps
Difficulty working with material not understood
Divergent thinking and synthesis
Sometimes better to jump directly to more complex and difficult material
4.2.2. Step 1.2: map learning style behaviour to the conversational tutoring style
To map learning style behaviour to the conversational tutoring style, each behaviour characteristic extracted in
step 1.1b (in Table 2) is assessed using the following criteria:
1. Is it possible to map the behaviour trait onto a two-way online conversational tutorial?
2. How could the behaviour trait be used to implicitly predict learning styles?
All behaviour traits that can be mapped onto a tutorial conversation and used to predict learning styles should be
included in a summary table along with a description of how they could be used to predict learning styles (Table
3).
Page 5
