978-1-7281-3637-0/19/$31.00 ©2019 IEEE
Improving the Usability of Interactive Systems by
Incorporating Design Thinking into the Engineering
Process: Raising Computer Science Students’
Awareness of Quality versus Quantity in Ideation
Frode Eika Sandnes
Department of Computer Science
Oslo Metropolitan University
Oslo, Norway
frodes@oslomet.no
Department of Technology
Kristiania University College
Oslo, Norway
Evelyn Eika
Department of Computer Science
Oslo Metropolitan University
Oslo, Norway
evelyn.eika@oslomet.no
Fausto Orsi Medola
Department of Design
UNESP-Sao Paulo State University
Bauru, Brazil
fausto.medola@unesp.br
Abstract—Traditional engineering methods are considered
unsuitable for the development of usable and engaging
interactive systems such as online experimentation and
simulation software. For systems involving users, user-centric
design approaches are more appropriate. The ideation stage of
design involves exploring the space of opportunities. One
commonly held view in design disciplines is that quantity leads
to quality. Yet, for related non-design disciplines such as
engineering, quantity is often regarded as a negative
characteristic associated with low quality. Focusing on quality
alone may lead to inferior user experience and ineffective
systems. This study describes an initiative to (1) collect
empirical evidence to support the design-belief that quantity
leads to quality and (2) to use the activity and its results as part
of a pedagogical strategy to enhance students’ awareness of the
connections between quantity and quality during ideation. A
class of 100 computer science students was divided into two
groups. Both groups were given the same task to design a text-
input strategy for individuals with motor disabilities but with
different focuses: one group was asked to focus on quality of
ideas and the other group on the quantity of ideas. The results
show that the students who focused on quantity of ideas
produced better quality concepts compared to the students that
focused on quality. The results were presented to and discussed
with the students as a part of the learning process.
Keywords—ideation, sketching, universal design education,
human computer interaction, text entry, motor disability
I. INTRODUCTION
Systems targeted at end users engineered with traditional
methods are unlikely to achieve full usability and user
satisfaction. Still, many such systems are made by engineers
using traditional methods. A complex device or instrument
will fail in the marketplace if it is not usable by the
customers. E-learning tools such as simulators and online
experimentation will not lead to the desired learning effects
if they are not engaging the students. One reason for this
situation is that design thinking is typically not taught in
engineering programs, and teachers of engineering courses
are rarely trained in design thinking.
This study focuses on one aspect of design thinking,
namely the achievement of quality through quantity. In most
domains, quality is often viewed as the opposite of quantity
in that quantity leads to poorer quality while quality is
achieved through focus and low quantity (see, for instance,
Becker and Lewis [1] and Motta [2]). It is thus a curious fact
that designers often argue differently that quantity is needed
to achieve quality. During ideation it is necessary to explore
as much as possible of the design space thereby considering
many poor ideas in order to discover the really good ideas.
The claim that quantity is needed for quality thereby
challenges the widely believed myth that quantity leads to
poor quality. However, for someone with a basic
understanding of design thinking the notion of quantity
leading to quality is both intuitive and logical. Yet, we have
been unable to identify any empirical evidence for this claim.
The only exception being Keller and Staelin’s study of the
impact of information quality and information quantity for
consumer decisions [3]. Other somewhat related studies
include Shah, Smith and Vargas-Hernandez’s [4] assessment
of four metrics for measuring ideation, i.e., quantity, quality,
novelty and variety, with the goal of assessing the
effectiveness of different ideation techniques. Chan, Dow
and Schunn [5] explored the claim that conceptually distant
sources of inspiration is the most valuable for design
processes. Their results supported the view that conceptual
distance is not important.
This study is inspired by a text in the book entitled Art
and Fear by Bayles and Orland [6] who gives the following
account of an “experiment” to demonstrate that quantity
leads to quality:
“The ceramics teacher accounted on opening day that he
was dividing the class into groups. All those on the left
side of the studio, he said, would be graded solely on the
quantity of work they produced, all those on the right
solely on its quality. His procedure was simple: on the
final day of class he would bring in his bathroom scales
and weigh the work of the “quantity” group: fifty pounds
of pots rated an “A”, forty pounds a “B” and so on.
Those being graded on “quality”, however, needed to
produce only one pot – albeit a perfect one – to get an
“A”. Well, came grading time and a curious fact
emerged: the works of highest quality were all produced
by the group being graded for quantity. It seems that
while the “quantity” group was busily churning out piles
of work – and learning from their mistakes – the
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses,
in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works,
for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
“quality” group had sat theorizing about perfection, and
in the end had little more to show for their efforts than
grandiose theories and a pile of dead clay.”
Later, Buxton [7] discussed the same principle in the
domain of user experience design with reference to the same
text. It is not clear from Bayles and Orland whether this
ceramics class actually took place or if it is just fictitious
event constructed to help readers more easily visualise the
point. This study thus set out collect empirical evidence
based on Bayles and Orland’s class experiment for a task of
designing assistive technology to assess whether a focus on
quantity leads to better quality in practice.
The implications of this study can be illustrated by
revisiting the example of online experimentation. Online
Experimentation (OE) [8, 9] involves learning through online
and/or remote access to experiments. These can be virtual 2D
and 3D simulations [10, 11, 12], augmented reality
simulations or virtual reality simulations. Typically, such
systems include various types of sensors at remote sites to
make the simulation more real. Online Experimentation is
often made available through collaborative web platforms
[13, 14]. Typically, such a system will be engineered using
the creators’ first and best idea, and then improved in
incremental steps until the result is satisfactory to the
creators of the tool or framework. We argue that instead one
should start more broadly and explore many possible ways of
realising the online experience before committing to a
particular approach. At each step the validity of the ideas
should be tested on representatives from the target group (in
this case students) thereby ending up with a more usable and
engaging system that leads to improved learning effect
among students.
II. METHOD
A. Experimental design
A between groups design was chosen with design
approach as independent variable with the two the levels
quantitative goal and quality goal and frequency of solutions
as dependent variables.
B. Participants
The class comprised a total of N = 106 bachelor students
in their second year (fourth semester) studying computer
engineering, information technology and applied computer
science. Most of the students were applied computer science
students. The main difference between these study
programmes is the amount of mathematics and physics in the
curriculum where applied computer science has the least of
the traditional sciences.
The class was divided into two groups according to the
following criterion: As each enrolled student is assigned a
running 6-digit student number, this number was used as
filtering criteria. Students with an odd student number were
to solve the problem with quality as the main objective and
was named yellow project. Students with an even student
numbers were to solve the problem with quantity of solutions
as the main objective. This was named the red project. In
theory there should have been approximately a 50/50
students solving red and yellow projects, however, there
were 62 students (58.5%) in the yellow group and 44
students (41.5%) in the red group.
Most of the students in the class were in their 20s, while a
handful of students were older. The gender balance was as
follows: there were 16 females (15.1%) and 90 males
(84.9%) in the class. Of these, 6 females (13.4%) solved the
red project and 10 females (16.1%) solved the yellow project.
C. Task
The project was set as an individual assignment and
conducted as their first hand-in in the course Human
Computer Interaction, which is assessed using portfolios
comprising one individual and two group projects.
Two separate problem descriptions were developed based
on the same basic problem. The problem was to develop a
concept for text entry using just one key [15]. The students
were not given any pointers to the literature. This problem is
easy to understand and narrow in scope, yet few students
have been exposed to this problem before. A bi-product of
working with this problem is that students get familiarity
with design for diversity and marginalized groups. Another
advantage of this problem is the large number of possible
solutions with varying levels of quality.
The students assigned the yellow project were
specifically instructed to develop the most efficient text entry
method possible, while the students assigned the red project
were specifically instructed to generate as many concepts as
possible.
The students were also told to analyse their designs using
MacKenzie’s theoretical steps per character model [16]
which uses the frequency of the alphabetic characters to
estimate the average number of steps to input a character.
To balance the workload the students solving the yellow
project also had to develop a working prototype and perform
a simple usability test of their method. The projects were
documented as written reports.
D. Analysis
The written reports were manually analysed during the
assessment phase, and the approaches devised by the
students were categorized into several classes and coded for
subsequent analysis. Although the two methods were not
completely identical, they were considered one category if
the principles were sufficiently similar to the given input
strategy.
The following main categories were identified: linear
multitap where users press the button to move the letter
forward one step [17], Morse code with long and short
presses [18, 19], multitap with grouping, binary codes,
binary search [20], Huffman codes as used in compression
[21], knock codes (also known as Russian tap codes) [22],
linear scanning [23], row-column scanning [24]. Morse
code, Huffman codes and binary codes are considered
impractical early stage ideas which are theoretically efficient
from an information theoretic perspective, but impractical
from a cognitive and ergonomic perspective as they impose
high memory demands on the users. Multitap, multitap with
grouping and knock codes are more practical as they rely on
the visual system instead of memory, while linear scanning
and row-column scanning are considered the optimal designs
as they require little work and little demand on memory, but
instead takes longer.
4.9%
2.4%
19.5%
22.0% 22.0%
7.3% 7.3%
4.9%
7.3%
2.4%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
1 2 3 4 5 6 7 8 9 10
Frequency (%)
No. solutions
Fig. 1. Distribution of solutions generated by the students focusing on
quantity.
These also represent the state of the art as assistive
technologies used with switches for users having reduced
physical function [25, 26, 27].
The data was gathered using Microsoft Excel and
analysed using the open source statistics software JASP
version 0.9.0.1 [28].
E. Ethics
This study was carried out during the autumn of 2017
before the introduction of the General Data Protection
Regulation (GDPR). The identities of the participants were
anonymized as the analysis described herein was conducted
after the course had finished.
III. RESULTS
As instructed, students focusing on quality reported on a
single solution, while most of the students who were asked to
focus on quantity generated more than one solution (M = 4.9,
SD = 2.2). Fig. 1 shows the distribution of the number of
concepts generated by the students that focused on quantity.
Most of these groups produced 3-5 solutions. Only 4.9%
documented just one solution while 2.4% of the students
documented 10 concepts or more.
Fig. 2 shows the distribution of the different concepts for
the two groups. As expected, the frequencies are higher for
all the categories for students focusing on quantity. The
entropy of the distributions (a measure of spread for
categorical data) shows that the students focusing on quantity
explored more solutions (Entropy = 2.84) than students
focusing on quality (Entropy = 1.66). A contingency table
analysis shows that these distributions for the wo groups are
significantly different (χ
2
(9) = 24.0, p = .004).
Among the group focusing on quantity, 68.2%
documented the basic multitap method, which was only
documented by 25.8% of the student focusing on quality.
Among the students focusing on quality, row/column
multitap was the most frequently reported with 32.3%, while
36.4% of the other group reported similar techniques.
Fig. 2 also shows that among the methods that are
considered optimal, 29.5% among the quantity group
reported linear scanning, while only 3.2% ended up with the
linear scanning concept among those focusing on quality.
0.0%
3.2%
0.0%
4.8%
4.8%
14.5%
3.2%
32.3%
4.8%
25.8%
2.3%
4.5%
15.9%
18.2%
25.0%
27.3%
29.5%
36.4%
56.8%
68.2%
0.0% 20.0% 40.0% 60.0% 80.0%
Huffman code
knock code
binary code
binary search
scanning group
group
scanning linear
row/column multitap
morse
linear multitap
Frequency (%)
Quantity (red)
Quality (yellow)
Fig. 2. Distribution of solution types
For what is considered the optimal solution, that is
row/column scanning, 25.0% of the quantity group reported
this method, while only 4.8% ended up with this concept
among the quality group. These results thus agree with the
claim that quantity lead to better quality than quality by
itself. By focusing on quantity, the student had a 25%
chance of detecting the optimal solution, whilst when
focusing on quality the students had less than 5% chance of
finding the optimal solution.
Each student in the two groups were assessed based on
the objectives of the respective two problem descriptions. To
confirm that the type of assignment would not affect the
students grade, the grades of the two groups were compared.
A Mann-Whitney U Test shows that the grades for the two
groups were not statistically different (W = 1127, p = .969).
In order to identify any possible relationship between the
quantity of solutions and the quality of the solutions, the
quality of the solutions were categorized according to the
following ordinal scale in increasing order of quality: 0: code
based solutions (Morse, Huffman, binary), 1: binary search,
2: linear multitap, group, 3: row/column multitap, Russian
tap-code, 4: linear scanning and 5: row/column scanning.
The quality of the concept designed by each student/group
was therefore represented by the solution with the maximum
quality. This quality was then correlated with the number of
concepts designed by the students. A Spearman correlation
shows that there is a strong positive and significant
correlation between quantity of solutions and quality of
solutions (r
s
(44) = .514, p < .001). The correlation is
depicted in Fig. 3.
IV. DISCUSSION
The results agree with the designers’ belief that quantity
leads to quality and disagrees with the technological belief
that quantity reduces quality. Clearly, the claim that quantity
leads to quality is specific to the context where one is
seeking the solution to a new and unknown problem, the
process which designers call design. Unfortunately, the term
design is often used differently by engineers and
technologists to mean solving a known problem with specific
methods, i.e., such as designing the thickness of a weight
carrying concrete beam.
y = 0.3874x + 1.3316
0
1
2
3
4
5
6
0 2 4 6 8 10 12
quality
quantity
Fig. 3. Correlation between quantity and quality (with dithering)
Technologists, especially computer scientists often come
across problems where there are no straightforward methods
for finding the solutions. We therefore argue for developing
engineers and technology students design thinking skills.
One of the major challenges is not to make students delve
into a technical solution too quickly but rather explore other
options. One approach is to specifically instruct students to
present several alternatives and to make students reflect over
the quality gains through the process.
Another approach specific to computer science students
is to impose a programming ban in a course, forcing students
to find other ways to test out ideas, that is, low-fidelity
prototyping with simple means.
It is our experience that the notion of design thinking is
also not widely accepted among the more computationally
oriented faculty members. One way to persuade such
individuals about the benefits of quantity of solutions may be
to draw parallels to stochastic processes such as Monte Carlo
methods [29, 30, 31] and stochastic optimization [32, 33]
where many pseudo-random solutions are generated by
computers to find high quality solutions to complex
problems.
This study evolved around a case involving the design of
assistive technology for individuals with reduced motor
function. This is just an example. The notion of quality
through quantity can be applied to nearly any development
process and target group, including platforms for online
experimentation targeting students. We argue that adopting a
design thinking approach to the development of online
experimentation platforms where one refines an array of
ideas instead of improving upon a single idea will lead to
more engaging and effective learning experiences for
students.
V. CONCLUSIONS
The paper described an experiment to explore the effect
of emphasizing quality or quantity in a student design
assignment. The results show that when students were asked
to focus on quantity the resulting quality was better than
when students focused on quality. The notion of quality
through quantity is well established in the design disciplines
while equating quantity with low quality is still a frequently
believed myth within engineering and technological
disciplines. Design thinking is increasingly important in
order for engineers and technologists to solve tomorrow’s
challenges. Students therefore need to be trained to explore a
larger portion of the design space before settling on a
specific engineering solution.
REFERENCES
[1] G. S. Becker and H. G. Lewis, “On the Interaction between the
Quantity and Quality of Children,” Journal of political Economy, vol.
81, no. 2 part 2, pp. 279-288, 1973.
[2] M. Motta, “Endogenous quality choice: price vs. quantity
competition,” The journal of industrial economics, pp. 113-131, 1993.
[3] K.L. Keller and R. Staelin, “Effects of quality and quantity of
information on decision effectiveness,” Journal of consumer research,
vol. 14, no. 2, pp. 200-213, 1987.
[4] J. J. Shah, S. M. Smith, and N. Vargas-Hernandez, “Metrics for
measuring ideation effectiveness,” Design studies, vol. 24, no. 2, pp.
111-134, 2003.
[5] J. Chan, S. P. Dow, and C. D. Schunn, “Do the best design ideas
(really) come from conceptually distant sources of inspiration?” in
Engineering a Better Future, Springer, Cham. 2018, pp. 111-139.
[6] D. Bayles and T. Orland, “Art & fear: Observations on the perils (and
rewards) of artmaking,” Image Continuum Press, 2001, p. 29.
[7] B. Buxton, Sketching user experiences: getting the design right and
the right design. Morgan Kaufmann. 2007.
[8] T. Restivo and A. Cardoso, GUEST EDITORIAL, Vol. 8, 2012, SI
(exp. at'11), 2012, http://dx.doi.org/10.3991/ijoe.v8iS1.1931
[9] T. Restivo and A. Cardoso, GUEST EDITORIAL, Vol. 9, 2013, SI
(exp. at'13), 2013, http://dx.doi.org/10.3991/ijoe.v9iS8.3448
[10] F. E. Sandnes, “Communicating panoramic 360 degree immersed
experiences: a simple technique for sketching in 3D,” in International
Conference on Universal Access in Human-Computer Interaction,
Springer, Cham., 2016, pp. 338-346.
[11] F. E. Sandnes, “PanoramaGrid: a graph paper tracing framework for
sketching 360-degree immersed experiences,” in Proceedings of the
International Working Conference on Advanced Visual Interfaces,
ACM, 2016, pp. 342-343.
[12] F. E. Sandnes, “Sketching 3D immersed experiences rapidly by hand
through 2D cross sections,” in Online Engineering & Internet of
Things, Springer, Cham, 2018, pp. 1001-1013.
[13] F. E. Sandnes, Y. Lianguzov, O. V. Rodrigues, H. Lieng, F. O.
Medola, and N. Pavel, “Supporting collaborative ideation through
freehand sketching of 3D-shapes in 2D using colour,” in International
Conference on Cooperative Design, Visualization and Engineering,
Springer, Cham., 2017, pp. 123-134.
[14] F. E. Sandnes and Y. Lianguzov, “A simple browser-based 3D-
sketching framework for novice and infrequent users,” in 2017 4th
Experiment@ International Conference (exp. at'17), IEEE, 2017, pp.
155-156.
[15] I. S. MacKenzie, “The one-key challenge: searching for a fast one-
key text entry method,” in Proceedings of the 11th international ACM
SIGACCESS conference on Computers and accessibility, ACM,
2009, pp. 91-98.
[16] I. S. MacKenzie, “Modeling text input for single-switch scanning,” In
International Conference on Computers for Handicapped Persons,
Springer, Berlin, Heidelberg, 2012, pp. 423-430.
[17] C. L. James and K. M. Reischel, “Text input for mobile devices:
comparing model prediction to actual performance,” in Proceedings
of the SIGCHI conference on Human factors in computing systems,
ACM, 2001, pp. 365-371.
[18] T. Felzer and R. Nordmann, “Alternative text entry using different
input methods,” in Proceedings of the 8th international ACM
SIGACCESS conference on Computers and accessibility, ACM,
2006, pp. 10-17.
[19] K. Partridge, S. Chatterjee, V. Sazawal, G. Borriello, and R. Want,
“TiltType: accelerometer-supported text entry for very small
devices,” in Proceedings of the 15th annual ACM symposium on User
interface software and technology, ACM, 2002, pp. 201-204.
[20] F. E. Sandnes, H. W. Thorkildssen, A. Arvei, and J. O. Buverad,
“Techniques for fast and easy mobile text-entry with three-keys,” in
Proceedings of the 37th Annual Hawaii International Conference on
System Sciences, IEEE. 2004.
[21] B. Bajer, I. S. MacKenzie, and M. Baljko, “Huffman base-4 text entry
glove (H4 TEG),” in 2012 16th International Symposium on
Wearable Computers, IEEE, 2012, pp. 41-47.
[22] F. E. Sandnes and F. O. Medola, “Exploring russian tap-code text
entry adaptions for users with reduced target hitting accuracy,” in
Proceedings of the 7th International Conference on Software
Development and Technologies for Enhancing Accessibility and
Fighting Info-exclusion, ACM, 2016, pp. 33-38.
[23] S. Bhattacharya, D. Samanta, and A. Basu, “Performance models for
automatic evaluation of virtual scanning keyboards,” IEEE
Transactions on neural systems and rehabilitation engineering, vol.
16, no. 5, p. 510, 2008.
[24] O. Polacek, A. J. Sporka, and P. Slavik, “Text input for motor-
impaired people,” Universal Access in the Information Society, vol.
16, no. 1, pp. 51-72, 2017.
[25] F. E. Sandnes and F. O. Medola, “Effects of Optimizing the Scan-
Path on Scanning Keyboards with QWERTY-Layout for English
Text,” Studies in health technology and informatics, vol. 242, pp.
930-938, 2017.
[26] F. E. Sandnes, E. Eika, F. O. Medola, “Reducing Scanning Keyboard
Input Errors with Extended Start Dwell-Time,” in International
Conference on Applied Human Factors and Ergonomics, Springer,
Cham., 2018, pp. 808-817.
[27] F. E. Sandnes, E. Eika, and F. O. Medola, “Multi-switch Scanning
Keyboards: A Theoretical Study of Simultaneous Parallel Scans with
QWERTY Layout,” in International Conference on Universal Access
in Human-Computer Interaction, Springer, Cham., 2018, pp. 287-298.
[28] JASP Team (2018). JASP (Version 0.9)[Computer software].
[29] J. S. Liu and R. Chen, “Sequential Monte Carlo methods for dynamic
systems,” Journal of the American statistical association, vol. 93, no.
443, pp. 1032-1044, 1998.
[30] J. H. Halton, “A retrospective and prospective survey of the Monte
Carlo method,” Siam review, vol. 12, no. 1, pp. 1-63, 1970.
[31] E. Yigit and F. Gokpinar, “A simulation study on tests for one-way
ANOVA under the unequal variance assumption,” Commun Fac Sci
Univ Ankara, Ser A, vol. 1, pp. 15-34, 2010.
[32] L. Bianchi, M. Dorigo, L. M. Gambardella, and W. J. Gutjahr, “A
survey on metaheuristics for stochastic combinatorial optimization,”
Natural Computing, vol. 8, no. 2, pp. 239-287, 2009.
[33] M. Srinivas and L. M. Patnaik, Genetic algorithms: A survey.
computer, vol. 27, no. 6, pp. 17-26, 1994.
