Using a fuzzy comprehensive evaluation method to determine product usability
A proposed theoretical framework
Zhou, Ronggang; Chan, Alan H. S.
Published in:
Work
Published: 01/01/2017
Document Version:
Final Published version, also known as Publisher’s PDF, Publisher’s Final version or Version of Record
License:
CC BY-NC
Publication record in CityU Scholars:
Go to record
Published version (DOI):
10.3233/WOR-162474
Publication details:
Zhou, R., & Chan, A. H. S. (2017). Using a fuzzy comprehensive evaluation method to determine product
usability: A proposed theoretical framework.
Work
,
56
(1), 9-19. https://doi.org/10.3233/WOR-162474
Citing this paper
Please note that where the full-text provided on CityU Scholars is the Post-print version (also known as Accepted Author
Manuscript, Peer-reviewed or Author Final version), it may differ from the Final Published version. When citing, ensure that
you check and use the publisher's definitive version for pagination and other details.
General rights
Copyright for the publications made accessible via the CityU Scholars portal is retained by the author(s) and/or other
copyright owners and it is a condition of accessing these publications that users recognise and abide by the legal
requirements associated with these rights. Users may not further distribute the material or use it for any profit-making activity
or commercial gain.
Publisher permission
Permission for previously published items are in accordance with publisher's copyright policies sourced from the SHERPA
RoMEO database. Links to full text versions (either Published or Post-print) are only available if corresponding publishers
allow open access.
Take down policy
Contact lbscholars@cityu.edu.hk if you believe that this document breaches copyright and provide us with details. We will
remove access to the work immediately and investigate your claim.
Download date: 26/08/2022
Work 56 (2017) 9–19
DOI:10.3233/WOR-162474
IOS Press
9
Using a fuzzy comprehensive evaluation
method to determine product usability:
A proposed theoretical framework
Ronggang Zhou
a,∗
and Alan H. S. Chan
b
a
School of Economics and Management, Beihang University, Beijing, P. R. China
b
Department of Systems Engineering and Engineering Management, City University of Hong Kong,
Hong Kong, P. R. China
Received 11 May 2015
Accepted 19 May 2016
Abstract.
BACKGROUND: In order to compare existing usability data to ideal goals or to that for other products, usability practitioners
have tried to develop a framework for deriving an integrated metric. However, most current usability methods with this aim
rely heavily on human judgment about the various attributes of a product, but often fail to take into account of the inherent
uncertainties in these judgments in the evaluation process.
OBJECTIVE: This paper presents a universal method of usability evaluation by combining the analytic hierarchical process
(AHP) and the fuzzy evaluation method. By integrating multiple sources of uncertain information during product usability
evaluation, the method proposed here aims to derive an index that is structured hierarchically in terms of the three usability
components of effectiveness, efficiency, and user satisfaction of a product.
METHODS: With consideration of the theoretical basis of fuzzy evaluation, a two-layer comprehensive evaluation index
was first constructed. After the membership functions were determined by an expert panel, the evaluation appraisals were
computed by using the fuzzy comprehensive evaluation technique model to characterize fuzzy human judgments. Then with
the use of AHP, the weights of usability components were elicited from these experts.
RESULTS AND CONCLUSIONS: Compared to traditional usability evaluation methods, the major strength of the fuzzy
method is that it captures the fuzziness and uncertainties in human judgments and provides an integrated framework that
combines the vague judgments from multiple stages of a product evaluation process.
Keywords: Usability, fuzzy comprehensive evaluation, analytic hierarchy process (AHP)
1. Introduction
Usability has become an increasingly important
factor that influences how consumers and designers
choose among different systems or products [1–3].
Usability evaluation is a specialized process that has
been shown to require expertise from a wide range
of knowledge domains [4]. However, according to
Hornbæk’s paper on current practice in measuring
∗
Address for correspondence: Ronggang Zhou, School of
Economics and Management, Beihang University, Beijing, P. R.
China. Tel.: +86 10 8231 6083; Fax: +86 10 8233 9338; E-mail:
zhrg@buaa.edu.cn.
usability [5], a major weakness of current methods
is that there is no principal technique that addresses
the vagueness and uncertainties inherent in the var-
ious components that contribute to the concept of
usability. Indeed, most usability methods rely heavily
on human judgment about the various attributes of a
product, but often fail to take account of the inherent
uncertainties in these judgments in the evaluation pro-
cess. The main goal of this study was to demonstrate
how these uncertainties can be elicited, captured
and combined by using a fuzzy method integrated
with an analytic hierarchy process (AHP) method.
Section 2 provides a brief review of existing usability
1051-9815/17/$35.00 © 2017 – IOS Press and the authors. All rights reserved
This article is published online with Open Access and distributed under the terms of the Creative Commons Attribution Non-Commercial License (CC BY-NC 4.0).
10 R. Zhou and A.H.S. Chan / A fuzzy usability evaluation technique: Theoretical framework
evaluation techniques such as those conducted using
general mathematical methods, questionnaires, the
AHP method, and fuzzy approach. In Section 3, the
general methodological steps of how to use fuzzy
evaluation and AHP method will be described. Sec-
tion 4, will consider the theoretical framework of the
proposed fuzzy usability evaluation technique based
on the AHP method. A discussion is provided in
Section 5.
2. A brief review of existing usability
evaluation techniques
As a core term in human factors and ergonomics,
usability has been defined by researchers in different
ways [4, 6–10]. By focusing on product perception
and acceptance, Shackel proposed an operational def-
inition of usability which provided a set of usability
criteria, including effectiveness (level of interaction
in terms of speed and errors), learnability (level of
learning needed to accomplish a task), flexibility
(level of adaptation to various tasks) and attitude
(level of user satisfaction with a system) [6]. This def-
inition has been generally accepted by the usability
community [11]. Another well-accepted definition of
usability was offered by Nielsen [4], which described
usability as ‘the measure of the quality of the user
experience when interacting with something whether
a Web site, a traditional software application, or
any other device the user can operate in some way
or another’ [7]. Nielsen suggested several oper-
ational usability dimensions such as learnability,
memorability, efficiency, user satisfaction (subjec-
tive assessment of how pleasurable it is to use) and
error (number of errors, ability to recover from errors,
existence of serious errors) [4]. To consolidate the
definitions, the International Organization for Stan-
dardization (ISO) defined usability as ‘the extent to
which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency
and satisfaction in a specified context of use [8].
However, these various definitions of usability con-
tain concepts that are far from concrete, and they are
often highly context dependent in the sense that no
single set of measurements can be applied to all prod-
ucts or services. Indeed, many practitioners lament
that usability can mean different things to different
people, even when it is defined, and it still remains
intuitive, uncertain, and ambiguous [5, 6, 12]. There-
fore, in the usability community, in order to compare
existing usability data to ideal goals or that of other
products, practitioners have tried to develop a frame-
work for deriving a single or integrated metric from
the various aforementioned metrics with the use of
different evaluation techniques [13, 14].
2.1. The general weighted additive method
The definition of usability is highly dependent on
the measurement method. One of the most direct mea-
surements is the method of user performance which
is used widely for evaluating product usability. Prac-
titioners can easily measure the task success rate of
users in actually using a product and derive an average
accuracy or error rate that reflects product effective-
ness [9]. However, different products may require
different sets of metrics to measure their effective-
ness and it is always difficult to make comparisons
between evaluations for different products. A num-
ber of attempts have been made to derive a single
usability score that combines the different metrics in
order to facilitate comparisons. Babiker et al. sug-
gested assigning different weights to a set of metrics
such as ease of access, navigation, orientation, and
user interaction for evaluating usability of hypertext
systems, and then integrating these metrics into a
simple weighted additive score [15]. Although they
found that the combined metric correlated well to
subjective assessment measures, whether the method
could be easily generalized or transferred to other
systems is questionable, because the weights were
based on the specific assessment criteria of a prod-
uct use. Other methods based on this kind of weighted
additive model have been used by various researchers
[5, 13, 16]. However, a common problem with this
method is that the measurements depend critically
on specific products and on the practitioner’s subjec-
tive judgment. Also, there is always the problem that
it may be too simplistic to assume that a single weight
can be assigned to each of the evaluated attributes.
2.2. The questionnaire method
In contrast to objective performance measure-
ments, usability evaluation can be made with
subjective evaluations [17–19]. In the ergonomics
community, several well-known subjective usabil-
ity questionnaires have been developed based on
user personal interactive product experience. These
methods include the Post-Study System Usabil-
ity Questionnaire (PSSUQ) [17, 18], the Software
Usability Measurement Inventory (SUMI) [20, 21],
and the Questionnaire for User Interaction Satisfac-
R. Zhou and A.H.S. Chan / A fuzzy usability evaluation technique: Theoretical framework 11
tion (QUIS) [22, 23]. The primary advantage of using
questionnaires over other usability evaluation meth-
ods is that they can be readily applied and have a high
benefit to cost ratio. All three questionnaire methods
are claimed to have high reliability and validity for
usability testing in practice. However, as found in
the weighted additive method, these questionnaires
suffer from the same problem that it is not clear
how multiple metrics (either subjective or objective)
derived from the responses can be weighted and com-
bined to provide an overall product usability index.
2.3. The analytic hierarchy process (AHP)
method
The AHP method was developed by Saaty [24]
and has been generally accepted as a robust and flex-
ible multi-criteria decision-making tool for dealing
with complex decision problems in various research
domains [25–27]. In usability engineering, the AHP
method has been used to determine the weights of dif-
ferent components during the evaluation process as
well as to conduct synthetic comparative evaluation
for multiple products or prototypes [28, 29]. With a
structurally hierarchical model, this method requires
experts to provide only the rank orders of different
metrics of usability, such as the learnability and ease
of use, and the corresponding weights for these met-
rics can then be derived. The AHP is a technique that
focuses directly on deriving the appropriate weights
based on expert judgments. It is well suited to com-
paring the relative usability of different alternatives,
and thus is a powerful multi-criteria decision-making
tool for usability testing purpose. In later sections it
will be shown how this method can be coupled with
a fuzzy approach to enhance its ability to capture the
uncertainties and vagueness of usability perceptions
expressed by the experts.
2.4. The fuzzy evaluation method
In the discipline of ergonomics there is a good
understanding of the role of fuzzy set theory in show-
ing a quantifiable degree of uncertainty in human
judgment [26, 30, 31]. The fuzzy evaluation method
is based on fuzzy set theory developed by Zadeh [32]
for capturing the uncertainties inherent in a system.
As discussed above, the processes in usability eval-
uation inherently involve fuzzy, uncertain, dynamic,
and changing information. In the usability engineer-
ing field, some early attempts at using the fuzzy
evaluation method were made. Cai et al. applied the
method to capture the perceived shape and color aes-
thetics of different products [31]. To compare design
alternatives, the imprecise preference structures of
the alternatives were modeled by a set of fuzzy pref-
erence relations. These relations not only specified
whether one attribute was preferred over another
attribute, but also how confidently this particular pref-
erence order was expressed by the user. For Web page
design, Hsiao et al. proposed a Gestalt-like percep-
tual measure method by combining Gestalt grouping
principles and fuzzy entropy [26]. They developed
a set of fuzzy relations that captured the layout of
graphics, arrangement of texts, and combinations of
colors. Both studies showed that the fuzzy evaluation
approach can provide a powerful mathematical tool to
quantify imprecise information in human judgments.
3. The methodological framework
Based on these previous efforts in structuring user
experience or usability evaluations, in this paper, a
universal method of usability evaluation for products
will be proposed. This universal method will involve
combining the AHP and fuzzy evaluation methods
for synthesizing performance data and subjective
response data. The aim for this universal method is
to derive an index that is structured hierarchically
within the framework of ISO 9241 part 11 [8], which
define usability in terms of three major components,
viz. effectiveness, efficiency, and user satisfaction.
An additional goal is to demonstrate the generality of
the fuzzy usability evaluation method by showing that
any set of standard usability attributes can be adopted
and the same process can be applied to obtain a com-
prehensive evaluation. The general methodological
framework will be described in the next section.
3.1. The general fuzzy evaluation model
The general fuzzy evaluation model aims at provid-
ing a fuzzy mapping between each of the evaluation
factors e.g. effectiveness, efficiency and user satisfac-
tion, to a set of categorical appraisal grades e.g. good,
excellent. The idea is to define fuzzy sets for the eval-
uation factors, such that for a particular a usability
rating e.g.a5ona7-point scale, could belong to the
both the grades ‘good’ and ‘excellent’. However, the
extent to which the usability rating belongs to each
grade may vary i.e. different degrees of membership
to each grade, depending on the weights given to each
evaluation factor and the average ratings given by
12 R. Zhou and A.H.S. Chan / A fuzzy usability evaluation technique: Theoretical framework
different raters. In the above example, one may find
that a rating of 5 for effectiveness can be mapped
to the fuzzy sets ‘fair’, ‘good’, and ‘excellent’ with
degrees of membership of 0.2, 0.7, and 0.5 respec-
tively. By assigning membership degree to multiple
‘fuzzy grades’, more of the uncertainties inherent
in the rating process can be captured and retained,
which will be particularly useful for comparing two
products. The formal procedures of the general fuzzy
evaluation model can be described by the following
steps.
Step 1: Determining the set of evaluation factors
Evaluation factors can be defined according to
the objectives of the product evaluation process.
A set of n evaluation factors can be represented as
a vector U =
{
u
1
,u
2
, ...., u
n
}
. For example, one
can define U =
{
effectiveness, efficiency, user
satisfaction
}
such that different measurements will
be conducted to evaluate the product based on these
three factors.
Step 2: Determining the set of appraisal grades
The appraisal set can be represented as a vector
V =
{
v
1
, v
2
, ...., v
m
}
, in which m represents the
number of levels in the appraisal. For example, if
m = 5, the appraisal vector can be represented as V =
{
very poor, poor, medium, good, excellent
}
.
Step 3: Setting the fuzzy mapping matrix
The goal of the evaluation process is to provide a
mapping from U to V. For a specific factor u
i
the fuzzy
mapping to the appraisal vector V can be represented
by the vector R
i
=
{
r
i1
,r
i2
, ..., r
ik
, ..., r
im
}
,in
which m represents the number of levels in the
appraisal (see step 2), and r
ik
represents the fuzzy
membership degree of appraisal factor i to grade k.
Using the example from step 1 and 2, if R
1
=
{
0, 0, 0.3, 0.7, 0
}
, then the measurement on the
evaluation factor “effectiveness” i.e., u
1
, on this prod-
uct has a fuzzy membership of 0.3 in the grade
“medium”, and a fuzzy membership of 0.7 in the
grade “good”, respectively.
In general, the fuzzy appraisal matrix of all n fac-
tors can be derived and represented as a matrix R, such
that if there are n factors and m levels of appraisal
grades:
R =
⎡
⎢
⎢
⎢
⎢
⎢
⎣
r
11
r
12
··· r
1m
r
21
r
22
··· r
2m
.
.
.
.
.
.
.
.
.
r
n1
r
n2
··· r
nm
⎤
⎥
⎥
⎥
⎥
⎥
⎦
(1)
In the above matrix notation for R, each row repre-
sents the set of appraisal membership degrees to the
corresponding appraisal vectorVfor each evaluation
factor u
i
in the evaluation vector U.
Step 4: Determining the weight of each evaluation
factor
To obtain a comprehensive usability evaluation, the
relative importance of each evaluation factor on the
overall grading of the product should be quantified.
The weight vector can be represented by W, which
can be formulated by the AHP method, as described
in the next subsection. As above, for n evaluation
factors, the weight can be represented by the vector
W =
(
W
1
,W
2
, ..., W
n
)
, in which the sum of all ele-
ments equal 1. From the example discussed earlier,
if it is determined that W =
(
0.3, 0.3, 0.4
)
, then the
relative weights for effectiveness, efficiency, and user
satisfaction will be 0.3, 0.3, and 0.4 respectively.
Step 5: Getting the overall appraisal result
The overall appraisal result can be obtained by
taking into the account the relative weights of each
evaluation factor, such that a single vector with the
same level of appraisal grades m (see step 2) can be
represented by:
B =
(
b
1
,b
2
, ..., b
m
)
= W ◦ R(2)
Where ‘◦’ is a composition operator,b
j
could be
operated by a number of possible models [33, 34].
The different composition operations will affect how
the final appraisal vector B will be changed by differ-
ent distributions of the weights i.e., in vector W. The
choice of composition operators is clearly beyond the
scope of this paper (for a comprehensive set of oper-
ators and when they should be used see, for example
[33, 34]). For the current purpose, we assume that
all evaluation factors should be considered, such that
no single factor is significantly selected or ignored
more than others. We therefore choose to use the com-
position operator that calculates each element b
j
of
the final appraisal vector by the following formula,
which is suitable for evaluations in which all weights
of factors must be accommodated:
b
j
= min
1,
n
i=1
W
i
r
ij
(i, j = 1, 2, ··· ,n) (3)
3.2. Determining the weight vector by AHP
The weight vector W (see step 4 in subsection 3.1)
can be determined by consulting expert opinions,
by conducting empirical and/or field studies, or by
adopting an existing theoretical framework [26, 27].