IDEA: Identifying Design Principles in
Educational Applets
Jody S. Underwood
Christopher Hoadley
Hollylynne Stohl Lee
Karen Hollebrands
Chris DiGiano
K. Ann Renninger
The Internet is increasingly being used as a
medium for educational software in the form of
miniature applications (e.g., applets) to explore
concepts in a domain. One such effort in
mathematics education, the Educational
Software Components of Tomorrow (ESCOT)
project, created 42 miniature applications each
consisting of a context, a set of questions, and
one or more interactive applets to help
students explore a mathematical concept. They
were designed by experts in interface design,
educational technology, and classroom
teaching. However, some applications were
more successful for fostering student
problem-solving than others. This article
describes the method used to mine a subset
(25) of these applets for design principles that
describe successful learner-centered design by
drawing on such data as videos of students
using the software and summaries of written
student work. Twenty-one design principles
were identified, falling into the categories of
motivation, presentation, and support for
problem solving. The main purpose of this
article is to operationalize a method for post
hoc extraction of design principles from an
existing library of educational software,
although readers may also find the design
principles themselves to be useful.
The Internet is increasingly becoming a vehi-
cle to design, develop, and publish educational
software in the form of miniature applications
(e.g., applets) that can be used to explore specific
problem contexts or concepts in a domain. The
design of technology tools has the potential to
dramatically influence how students interact
with tools, and these interactions in turn may
influence students’ content area understanding
and problem solving. However, the rapid devel-
opment and dissemination of such tools in many
cases occur without an explicit set of design
principles in place. The purpose of this article is
to describe the process that IDEA (Identifying
Design principles in Educational Applets) par-
ticipants used to cull design principles from a
library of applets developed for mathematics
education. We view the process of extracting
these design principles as a first step in a
broader methodology of proposing, refining,
vetting, and using design principles.
Design principles published for educational
software range in their specificity depending on
the tools being analyzed. For example, Sinclaire
(2003) studied student interactions with java-
based dynamic geometry sketches that are
accompanied by a set of questions to guide stu-
dent exploration of the objects in the sketches.
From her analysis of student work with the
sketches, she has extracted a set of guiding prin-
ciples that should inform future development
(e.g., questions should aim to focus student
attention on aspects of the sketch, whereas the
ETR&D, Vol. 53, No. 2, 2005, pp. 99–112 ISSN 1042–1629 99
AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 04-07-2005 / 19:20
sketch must provide the visual stimulus to draw
attention through color, motion and markings).
One example of a general list of design prin-
ciples is published in Clements’s (2000) review
of research of the use of computers in mathemat-
ical problem solving, in which he suggests sev-
eral contributions that the use of computers can
make to facilitate students’ problem solving. As
another general example, Schoenfeld (1985) con-
tributed a framework of factors that affect stu-
dent abilities to solve problems, which in turn
could inform the design of educational software
for problem solving. For example, as students
are solving a problem, they need to implement
strategies, use resources, and evaluate their
progress so that they are aware of and critically
examining their own decision making. In a tech-
nological environment, the resources available
to students include their knowledge of concepts,
facts, and procedures, as well as those offered by
the technology. Students need knowledge of
how to use the technology, and how the various
objects and actions on those objects can aid their
problem solving. The presence of a particular
technology tool also affects the available strate-
gies a student may use during problem solving,
since a tool may afford or constrain certain
actions, which make some strategies more acces-
sible than others.
The National Science Foundation-funded
Educational Software Components of Tomor-
row (ESCOT, Roschelle et al., 1999) project pro-
duced a library of educational software. There
were no preexisting design principles that
ESCOT designers knowingly adopted. Instead,
they drew on the expertise of people in a variety
of complementary fields. Over the course of two
school years (1999– 2001), integrated design
teams consisting of professional programmers,
teachers, mathematics education researchers,
and educational technologists, produced 42
problem contexts with supporting applets that
were intended to facilitate mathematical prob-
lem solving for middle school students. These
problems and applets were published and dis-
seminated through the Math Forum’s existing
“ Problem of the Week” structure (see
http://www.mathforum.org /escotpow/). Stu-
dents from around the world used these prob-
lems and applets, leaving a large database of
students’ problem-solving work.
The ESCOT project resulted in valuable les-
sons learned regarding problem context, ques-
tions, and applet design, as well as the
interactions among these features. The design
principles published prior to 1999 did not
address these features and interactions. The
IDEA project was undertaken as a follow-up
project to ESCOT with the goal of identifying
principles that could guide the design of effec-
tive problem-solving technologies. It focused on
the analysis of the ESCOT products and the
large database of student work with these prod-
ucts. Five members of the IDEA team were part
of the ESCOT project, each with different areas
of expertise— middle school teacher, software
developer, educational technologist, mathemat-
ics educator, and project evaluator. The remain-
ing IDEA team members, not having been part
of ESCOT, brought less subjective views about
the software we set out to evaluate, and had sim-
ilarly broad areas of expertise— teacher, mathe-
matics educator, and technology designer.
In a mature field such as architecture, a spe-
cific template for design is followed: State the
recurring problem that needs to be solved, pres-
ent the design pattern that addresses the prob-
lem, enumerate examples and varieties of
solutions that meet the design pattern (Alexan-
der et al., 1977). In order to create effective edu-
cational software, a similar template could be
followed: (a) Define educational problem, (b)
find recognizable solutions, (c) consider peda-
gogical implications of possible solutions (mis-
conceptions, tradeoffs, intended effects, etc.), (d)
identify principles that are appropriate for the
educational setting and learning goals, (e) craft
the features of the software that would enact
those principles. However, in this early stage of
educational design, until we identify the recog-
nizable solutions, the rest of the process is some-
what halted. By identifying what worked and
what did not work from the ESCOT experience,
we are generating hypotheses about useful
design principles that can be generalized
beyond the specific context of interactive prob-
lems of the week.
AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 04-07-2005 / 19:20
100 ETR&D, Vol. 53, No. 2
A Brief Background of ESCOT
ESCOT tested and disseminated its educa-
tional software via the Math Forum, a large
mathematics education resource portal. In their
well-established Problems of the Week (PoWs;
http://mathforum.org/pow/), students read
the problem, work on a solution either individu-
ally or with a peer or group, and write an expla-
nation of how to obtain a solution. Students
submit their solutions with explanations online,
receive feedback about their work via e-mail,
and are encouraged to submit a revised solution
if there are areas that can be improved. ESCOT
created a number of PoWs (ESCOT PoW, or
EPoW) that followed this same arrangement.
Teachers used the EPoWs in many different
ways: for example, as class requirements, as
extra credit, in small groups, and also individu-
ally. An example EPoW can be found in a later
section of this article.
Findings from Renninger et al.’s (2003) study
of student work with the EPoWs suggests that
these software-enhanced problems were moti-
vating for students. They also found that the
EPoW problem-solving environment appeared
to override differences that would typically be
found as a function of interest and self-efficacy
with respect to student ability to connect to, gen-
erate strategies for, and be autonomous in prob-
lem solving. These differences may be owing to
the design of the EPoWs. Thus, it was a natural
extension of ESCOT to investigate the student
data post hoc to hypothesize critical elements in
the design of those EPoWs that seemed to sup-
port students’ problem solving.
The data collected as a part of the ESCOT
project include hundreds of student solutions to
the EPoWs, summaries of feedback given to stu-
dents, rubrics, teacher support pages, video-
taped sessions of some students using the
EPoWs, and documented design decisions made
by the teams who created the EPoWs (Figure 1
shows the relations between these various
pieces of data). These data informed the effort of
the IDEA group.
We next present an example EPoW and then
describe the mining method and the resulting
design principles in detail.
AN EXAMPLE EPoW: FISH FARM
To situate our discussion, consider the Fish
Farm EPoW (see Table 1 and Figure 2, available
online at http://mathforum.org/escotpow/sol
utions/solution.ehtml?puzzle=40) in which stu-
dents solve a ratio problem. The Fish Farm prob-
lem is flexible in the sense that there are multiple
strategies that can be used to find three possible
correct solutions. The applet was also designed
to give students access to different representa-
tions for making sense of the problem. Although
this problem can be solved using algebraic tech-
niques, the intent of using the problem situation
and tools in the Java applet was to engage stu-
Figure 1 Educational Software Components of Tomorrow (ESCOT) problem-of-the-week data.
AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 04-07-2005 / 19:20
IDENTIFYING DESIGN PRINCIPLES 101
dents in thinking about different strategies and
solution paths, as well as part– part and part–
whole reasoning and equivalent ratios. The
bonus question was designed to induce a pertur-
bation for students about the relation between a
part– part and a part– whole representation of a
ratio. The students are asked to compare the
part– part ratio of 1:2 to a pie graph showing a
part– whole
1
⁄
3
:
2
⁄
3
representation. Many students
intuitively think about a 1:2 ratio as representing
a one-half situation and do not easily make the
transition to a
1
⁄
3
:
2
⁄
3
representation.
The applet was created with a tank on its left
side with 26 fish (13 males, 13 females) that the
user could drag and drop into one of the three
ponds to its right. As a fish is dropped into a
pond, a numerical count and pie graph are
updated to keep tally of the number of females
and males and the percent of females and males
in each pond. Once a fish is dropped into a
pond, it will swim within its boundaries.
Although a user can move the fish without hit-
ting the
RUN button at the bottom of the screen,
the
RUN button is used to activate the applet so
that the updates and swimming occur when a
fish is dropped in a pond. The
STOP button deac-
tivates the update and swimming features. The
CLEAR button will erase all fish from the tank on
the left and the three ponds, whereas the
RESET
button will place all 26 fish back into the tank.
The complete materials associated with this
EPoW include: (a) the problem situation and
questions, (b) an interactive applet, (c) a teacher
support page with suggestions for pre and post
activities, and (d) expected solutions. The solu-
tions were prepared by the ESCOT design team
and used by mentors who provided feedback in
response to student solutions. After all student
work was submitted and scored for each EPoW,
student solutions and mentor feedback and
scores were archived along with comments from
a lead mentor summarizing students’ solution
strategies and difficulties, and sample student
responses.
Below is part of one 13-year-old girl’s solu-
tion to the first question in the Fish Farm EPoW.
Notice that the student described her strategy in
terms of ratios, and used resources in the applet
to help her reflect on her problem solving.
Angel had 8 male fish and 8 female fish in her pond.
Molly had 3 male fish and 1 female fish in her pond.
Gar had 2 male fish and 4 female fish in her pond. I
first put one male and one female in Angel’s pond, 3
male and 1 female in Molly’s pond, and 2 males and 4
females in Gar’s pond. I thought I could put the rest
into Angel’s pond, but I noticed that there was unequal
Table 1 Fish Farm Educational Software Components of Tomorrow problem of the week
(EPoW) text of the problem.
A Fishy Family. For their birthday, the Carp triplets received 26 tropical fish: 13 females and 13 males. They
discussed ways to divide the fish among their three tiny backyard ponds.
Angel said, “ I want the same number of male and female fish in my pond.”
“ Okay,” said Molly. “ I want three times as many males as females in my pond.”
“ Then I want twice as many females as males in my pond,” Gar replied.
Is there a way to put all 26 fish into those three ponds, while giving each triplet what he or she wants? Use
the applet to explore this question.
Questions
1. How many male fish and female fish does each triplet get in his or her pond? Describe the work you did to
find the solution. (Sample questions you can answer: Into which pond did you put fish first? How many fish
of each kind went into that pond? Why? What was your next step? How were you sure a pond had the cor-
rect ratio?)
2. Given the 13 males and 13 females, what are ALL the possible numbers of male and female fish that would
satisfy the ratio of 1 male to 2 female fish in Gar’s pond? Explain why these different amounts are equivalent
to the ratio 1:2.
Bonus: Explain why all possible answers in question 2 result in the same pie graph for Gar’s pond.
AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 04-07-2005 / 19:20
102 ETR&D, Vol. 53, No. 2
amounts of males and females. So to make it equal, I
put one more male fish and 2 more female fish in Gar’s
pond, that would still be the same as 1:2. That left me
with the same amount of male fish and female fish, so
they could all go into Angel’s pond (that would still
equal 1:1). Since I did everything slowly, I made sure
that my amounts of fish were equal to the ratios. All I
did was get the total amounts and then reduce them,
and the reduced number should’ve equaled the ratio. I
knew I got everything right when the bricks turned
green.
From this student’s description, it ap-
peared that several of the design elements in
the applet provided tools for her to complete
the task. Specifically, the displayed ratios
allowed her to compare a pond’s male-to-
female ratio with the desired ratio so she
could check if one ratio reduced to the other. It
appeared that having the bricks turn green
provided closure and confirmation that her
solution was correct. We considered these
types of interaction and response as we mined
the EPoW data to identify design principles and
intended effects of those principles.
METHOD:
MINING DATA FOR DESIGN PRINCIPLES
The types of design principles that the IDEA
team sought were those that successfully sup-
port problem solving and learner-centered
design issues. Given the large amount of data
and diverse backgrounds of the researchers on
the IDEA team, we decided to use a methodol-
ogy inspired by Erickson’s (1986) analytic induc-
tion. We first explored the EPoWs and reviewed
summaries of student work (not raw student
data) to hypothesize design principles, some of
which would finally emerge as a convergence of
the opinions and theoretical underpinnings
from our diverse research and experiential back-
grounds (Phase I). Once these hypothesized
Figure 2 Fish Farm Educational Software Components of Tomorrow problem of the week
(EPoW) applet.
AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 04-07-2005 / 19:20
IDENTIFYING DESIGN PRINCIPLES 103
