Copyright © 2010 IEEE. Citation for the published paper:
Fiedler, Markus; Hossfeld, Tobias; Tran-Gia, Phuoc
“A generic quantitative relationship between Quality of Experience and
Quality of Service”
IEEE Network, 24(2):36—41, March 2010
This material is posted here with permission of the IEEE. Such permission of the
IEEE does not in any way imply IEEE endorsement of any of BTH's products or
services Internal or personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or promotional
purposes or for creating new collective works for resale or redistribution must be
obtained from the IEEE by sending a blank email message to pubs-
permissions@ieee.org.
By choosing to view this document, you agree to all provisions of the copyright
laws protecting it.
IEEE Network • March/April 2010
36
0890-8044/10/$25.00 © 2010 IEEE
nformation and communications technology (ICT) users
find an ever growing set of IP-based applications and
access networks from which to choose. With the availabili-
ty of residential broadband, IP-based digital TV is also
offered via Ethernet and asymmetric digital subscriber line
(ADSL). Such a multitude of offers makes prices decrease,
and competition between service and/or network providers
increase. The customer finds itself in a strong position, being
able to choose between different competing providers. Given
similar pricing schemes, which are a primary decision aid for
many users, their subsequent choices are then likely to be
influenced by expected and experienced quality (i.e., through
personal ratings of the perception and price-worthiness of a
service). Consequently, the providers’ interest in how users
perceive usability, reliability, quality, and price-worthiness has
increased. A provider needs to be able to observe and react
quickly on quality problems, at best before the customer per-
ceives them and considers churn. Facing this kind of quality
competition, the concept of quality of experience (QoE)
emerged, combining user perception, experience, and expecta-
tions with non-technical and technical parameters such as
application- and network-level quality of service (QoS) [1].
We observe a non-uniform view of QoS by different stake-
holders. While the International Telecommunication Union
(ITU) standards focus on service quality toward the end user,
the Internet Engineering Task Force’s (IETF’s) understanding
of QoS relates to the capabilities of the network to provide
packet transfer in a better-than-best-effort way. While the
ITU view on QoS is user-centric [1], the IETF view on QoS is
network-centric. This raises the question of how network-level
QoS measurements and control relate to the user perception
of a service. In order to choose appropriate measures to keep
user-perceived service quality above an acceptance threshold,
a provider needs to know how network-level QoS parameters
translate into user-level QoE perception and vice versa.
Obviously, QoE-QoS relationships depend on many aspects
and parameters. A typical user-related measure is the mean
opinion score (MOS), which can be determined from subjec-
tive ratings by real users or predicted from objective measure-
ments of properties of the delivered goods such as audio,
video, or files. From the area of usability engineering, reaction
time thresholds for user perception are known as follows: 100
ms is roughly the boundary at which a user feels the system
reacting instantaneously; less than 1 s keeps the user’s
thoughts, although a delay is perceived; less than 10 s keeps
the user’s attention, while exceeding 10 s implies the risk of
the user abandoning the activity. The order of magnitude of
the latter threshold has been confirmed by studies of user
patience regarding web surfing [2].
Yet, no matter the networked application, users expect
timely and complete data delivery. Ideally, a networked appli-
cation should behave as if the enabling network was complete-
ly transparent. Streaming applications must maintain fluidity,
conversations and interactive applications shall expose short
response times, and downloads need to be able to efficiently
stream large amounts of data in order to minimize waiting
times [3]. Basic QoS problems on the network level relate to
non- or late delivery, which might entail re-ordering. Thus,
I
I
Markus Fiedler, Blekinge Institute of Technology
Tobias Hossfeld and Phuoc Tran-Gia, University of Würzburg
Abstract
Quality of experience ties together user perception, experience, and expectations
to application and network performance, typically expressed by quality of service
parameters. Quantitative relationships between QoE and QoS are required in
order to be able to build effective QoE control mechanisms onto measurable QoS
parameters. Against this background, this article proposes a generic formula in
which QoE and QoS parameters are connected through an exponential relation-
ship, called IQX hypothesis. The formula relates changes of QoE with respect to
QoS to the current level of QoE, is simple to match, and its limit behaviors are
straightforward to interpret. It validates the IQX hypothesis for streaming services,
where QoE in terms of Mean Opinion Scores is expressed as functions of loss and
reordering ratio, the latter of which is caused by jitter. For web surfing as the sec-
ond application area, matchings provided by the IQX hypothesis are shown to out-
perform previously published logarithmic functions. We conclude that the IQX
hypothesis is a strong candidate to be taken into account when deriving relation-
ships between QoE and QoS parameters.
A Generic Quantitative Relationship
between Quality of Experience and
Quality of Service
FIEDLER LAYOUT 3/3/10 9:38 AM Page 36
IEEE Network • March/April 2010
37
they affect the timely behavior of the application and possibly
even the appearance of the content.
Obviously, generic QoS problems (e.g., loss, delay, jitter, re-
ordering, throughput limitations) imply generic QoE problems
(e.g., glitches, artifacts, excessive waiting times). Thus, it is of
interest to investigate generic relationships between QoE and
QoS, which is the core contribution of this article. It presents
a nice, simple, unified, easy-to-match, and practicable formula
expressing an exponential dependency of QoE on QoS. It is
thus applicable for online in-service classification of QoE
problems based on QoS observations, which is of interest to
service providers and network operators. Our results enable
QoE control mechanisms that build on QoS monitoring.
The remainder of the article is structured as follows. The
next section elaborates on different types of quality metrics
relevant to QoE and QoS evaluations. We then present and
discuss the qualitative relationship between QoE and QoS.
We then quantify the latter into an exponential formula and
validate it for three case studies addressing different QoE
parameters such as voice quality and user reactions to down-
load times and throughput limitations. In the latter two cases,
the original logarithmic matching formulae are outperformed
by the proposed exponential matching. The final section con-
cludes and points out directions for future work.
Quality Comparisons and Classification of
Metrics
The derivation of QoE-QoS relationships builds on quality
comparisons between:
• The so-called reference, by which we mean undistorted con-
tent such as image or video, or an undistorted service such
as a download activity
• The outcome of the transmission in form of a potentially
distorted image or video, or a delayed download activity
A distortion of the outcome may impact the quality of the
content (e.g., image quality) and/or timing (e.g., fluidity of a
video, download times, service activation time). Then the QoE
relates to the remaining quality of the outcome after such a
distortion.
References play an important role when it comes to rating
the quality of the outcome. Evidently, the closer the quality of
the outcome comes to that of the reference, the better the
QoE. In the optimal case both match, and the network
between can be considered transparent. The impact of distor-
tions can, among other methods, be expressed with aid of util-
ity functions [4].
For the sake of quality comparison, there are different
measurement methods and observation levels. We can distin-
guish between:
•A communication situation, in which merely the outcome is
available
•A laboratory situation, in which both reference and outcome
are available and can be compared offline with great effort
and in great detail
There are two basic measurement options. Subjective tests are
carried out by a test panel of real users. While many and possi-
bly even diverging views on the quality of the outcome can be
taken into account — entailing accurate results and a good
understanding of the QoE and its sensitivity — this type of test
can be time-consuming and costly, since the tests have to be
conducted by a large number of users for statistically relevant
results. Objective tests are carried out by an algorithm on behalf
of a real user, trying to imitate or predict user perception based
on key properties of the reference and/or the outcome. Typical-
ly, subjective quality tests form the basis for perceptual objec-
tive test methods. Objective tests can follow psychophysical
approaches and engineering approaches, a detailed description
of which is found in [5]. For voice over IP (VoIP), the Percep-
tual Evaluation of Speech Quality (PESQ) standard objectively
evaluates and quantifies voice quality of voice-band speech
codecs. It uses a psycho-acoustic and cognitive model to ana-
lyze and compare the reference and the outcome. PESQ allows
for repeatable and automated measurement processes, yielding
statistically significant results [6].
Depending on the object of interest, we can observe con-
tent and related network traffic on different levels. Observa-
tion on the application level implies examination of the
payload, which makes it possible to get a detailed picture of
the content, and on the timing of reference and outcome.
Problems with the latter may arise from network nodes and
links and the network stacks in the end systems, as well as
from the implementation of the application. Additionally,
measurements on the network level may be conducted. This
means investigation of the flow of packets in terms of com-
pleteness, timeliness, and pattern analysis regarding bursty
losses or correlated delays. For testing the IQX hypothesis, we
observe both levels to derive QoE-QoS relationships.
Metrics can be classified according to the following scheme [5]:
Full reference (FR) metrics: Both reference and outcome
are available, and allow for detailed subjective and objective
comparisons of images, videos, download times (on the appli-
cation level), packet traces (on the network level), and so on.
Concretely, this means extraction, evaluation, and comparison
of QoE and QoS parameters on any level in an offline man-
ner, which is most interesting for deriving QoE-QoS relation-
ships. A prominent example of an FR metric is PESQ [6].
No reference (NR) metrics: Quality information has to be
extracted from the outcome, as no reference is available. This
is a typical online situation with focus solely on the resulting
quality as perceived by the end user (evaluated through obser-
vations and questions) or its representative (an algorithm). In
a networking context, NR metrics usually lack the possibility of
discerning between quality problems stemming from the very
reference and additional disturbance by the network. This is a
hindrance to deriving QoE-QoS relationships aiming at captur-
ing the impact of the network. On the other hand, user-accep-
tance-related QoE parameters building on generalizable
experiences such as image quality (exposure, sharpness, con-
trast, colors) or response times may well serve as a basis for
QoE-QoS relationships; see the investigations made in [3, 7].
Reduced reference (RR) metrics: For reference and out-
come, the same set of parameters are derived and compared.
For instance, QoE on the application level can be described
by the hybrid image quality metric (HIQM) [5], and QoS on
the network level can be represented by throughput variations
and losses [4]. Such parameters often have their roots in FR
research as a means of summarizing and interpreting results.
However, as they represent key QoE and QoS parameters in a
very condensed manner, they can be applied in an online in-
service scenario by transmitting them between source and
sink, and subsequently comparing them in order to find out
Figure 1. Illustration of the different quality metrics to derive
QoE-QoS relationships.
Reference A
FR(A,B)
Outcome BNR(B)IP network
Measurement
X
RR(B,X)
FIEDLER LAYOUT 3/3/10 9:38 AM Page 37
IEEE Network • March/April 2010
38
about quality problems. Thus, they represent promising candi-
dates to formulate QoE-QoS relationships [4].
Figure 1 illustrates the different FR, NR, and RR quality
metrics and their required inputs. For the FR metric, the ref-
erence A as well as the outcome B are available, allowing to
estimate the QoE by FR(A, B). For the NR metric, only the
outcome B is available, yielding NR(B). For the RR metric, in
addition to the outcome B, the measured parameters X are
available on the receiver side and potentially even communi-
cated from the sender side [4]. Thus, the quality is estimated
as RR(B, X).
Qualitative Relationship between QoE and
QoS
We now turn our focus onto a qualitative schematic relation-
ship describing the impact of QoS problems on QoE, illustrat-
ed in Fig. 2. On the x-axis, the QoS disturbance is denoted,
while the y-axis indicates a QoE value (e.g., an opinion score).
Although this relationship is basically independent of the type
of metric discussed above, we now focus on a situation in
which the network accounts for QoE reductions between ref-
erence and outcome [4].
The QoE of the outcome of the transmission as a function
of QoS disturbance is split in several areas, separated by
thresholds x
1
and x
2
.
Area 1: constant optimal QoE — For a vanishing QoS distur-
bance (i.e., in case of a transparent network), the user considers
the QoE equivalent to that of the reference. A slight growth of
the QoS disturbance may not affect the QoE at all. For
instance, small delays and delay variations may be eliminated
by a jitter buffer, without the user noticing the additional delay.
A similar effect is observed for web surfing delays up to half a
second. Even if web pages were delivered faster than what is
given by threshold x
1
, to the average user it would not matter.
Area 2: sinking QoE — When the QoS disturbance exceeds
a certain threshold x
1
,the former quasi-optimal QoE level can-
not be maintained anymore. As the QoS disturbance grows,
the QoE and thus the user satisfaction sinks. In case of a high
QoE, a certain additional QoS disturbance might have a con-
siderable impact on the QoE, while for low QoE, the particu-
lar additional QoS disturbance might not be that critical
anymore. Consequently, as the QoE sinks, its negative gradi-
ent is expected to do so as well.
Area 3: unacceptable QoE — As soon as the QoS distur-
bance reaches another threshold, x
2
, the outcome of the trans-
mission might become unacceptably bad in quality, or the ser-
vice might stop working because of technical constraints such
as timeouts. A user might give up using the service at that
point; this is illustrated by the dashed line.
While threshold x
1
, due to its technical nature, represents a
sharp threshold that very well may be collocated with the y-
axis, threshold x
2
may be user-dependent, an example of
which is discussed for the cancellation rate of web browsing
users in the next section.
The Exponential Relationship between QoE
and QoS
In this section we present, demonstrate, and discuss a funda-
mental functional relationship between QoE and QoS param-
eters, the IQX hypothesis (exponential interdependency of
quality of experience and quality of service). It is motivated
and presented later. The use of such a QoE-QoS relationship
is straightforward; by inserting measured QoS values into the
corresponding exponential formula, their impact on QoE can
be assessed immediately. Thus, the formulae presented and
discussed in the sequel enable QoE threshold surveillance and
control.
In the first place, the IQX hypothesis is formulated as
described earlier with the QoE parameter representing the
level of satisfaction and the QoS parameter reflecting the
level of disturbance. Typically, the QoE parameter and user
perception decrease when the QoS parameter increases. The
IQX hypothesis is tested for two different services, VoIP and
web browsing, and the corresponding results are shown later
in this section for the impact of loss, jitter, and re-ordering on
VoIP, and for the impact of response times on user ratings.
In the second place, the QoE parameter reflects a negative
user reaction, while the QoS parameter is associated with the
availability of resources. Also, here, the QoE parameter sinks
when the QoS parameter increases, but this stands for
improved user perception. In this case we test the IQX
hypothesis with the cancellation rate as the QoE parameter
and throughput as the QoS parameter.
The IQX Hypothesis for QoE
At large, QoE = Φ(I
1
, I
2
, …, I
n
) is a function of n influence
factors I
j
. In this contribution we focus on a single influence
factor, I = QoS, in order to derive the fundamental relation-
ship QoE = f(QoS).
In general, the subjective sensibility of the QoE is more
pronounced the higher this experienced quality is. If the QoE
is very high, a small disturbance will strongly decrease the
QoE. On the other hand, if the QoE is already low, a further
disturbance is not perceived significantly. This relationship
can be motivated considering a restaurant QoE: If we dined
in a five-star restaurant, a single spot on the clean white table-
cloth would strongly disturb the atmosphere. The same inci-
dent would go unnoticed in a simple tavern.
Against this background, we assume that the change of
QoE depends on the current level of QoE, given the same
amount of change of the QoS value, but with a different sign.
Assuming a linear dependence on the QoE level, we arrive at
the following differential equation:
(1)
The solution to this equation is easily found as an exponential
function, which expresses the basic relation of the IQX
hypothesis:
∂
∂
−−
()
QoE
QoS
QoE∼ γ .
Figure 2. General shape of the mapping curve between QoS and
QoE.
QoS disturbance
x
1
1
QoE value
2
3
4
5
x
2
21 3
Area 1: No distortion
Area 2: User disturbed
Area 3: User gives up
FIEDLER LAYOUT 3/3/10 9:38 AM Page 38
IEEE Network • March/April 2010
39
QoE = α⋅e
–β⋅QoS
+ γ. (2)
At this point, it is observed that the logarithmic approaches
found in [3, 7] and discussed later imply the differential equa-
tion
(3)
In contrast to the IQX hypothesis (Eq. 1), the change of QoE
depends on the reciprocal QoS value.
Voice Quality Affected by Loss, Jitter, and Reordering
For demonstrating the mapping between QoE and QoS of a
VoIP service, we use as an example the free Internet low bit
rate codec (iLBC) [8] used by Skype. The measurement ses-
sions are detailed in [9]. For emulating loss and jitter in the
network, we used Nistnet v. 2.0.12c (http://www-
x.antd.nist.gov/nistnet/). The potentially disturbed audio
stream is captured from the audio application SJPhone v.
160.299 (available at http: //www.sjlabs.com/sjp.html) into an
audio .wav file. This outcome is compared to the undistorted
reference audio file using the PESQ method. The resulting
PESQ value is mapped into a subjective MOS value according
to ITU-T Recommendation ITU-T P.862.1 [6], taking on the
values 1 = bad, 2 = poor, 3 = fair, 4 = good, and 5 = excel-
lent. While PESQ is a full-reference method FR(A, B), the
IQX hypothesis yields a reduced reference metric RR(B, X),
where X represents the QoS distortion and is derived from
comparing the amount of data received with the amount of
data sent, the difference of which indicates loss of data on its
way from user A to user B. We vary the packet loss probabili-
ty from 0 to 90 percent in steps of 0.9 percent.
Figure 3 shows the obtained MOS values dependent on the
measured packet loss ratio p
loss
for the conducted experiments
and for the applied model f
exp
(p
loss
). Each dot represents a sin-
gle MOS-from-PESQ measurement for a given packet loss
probability. For p
loss
→ 1, the QoE in terms of MOS approach-
es its minimum of one from above. The model function
f
exp
(p
loss
) is retrieved by means of nonlinear regression. We
used the optimization toolbox of Matlab to find an optimal fit-
ting function (i.e., the unknown parameters α, β, γ in Eq. 2)
such that the normal error E is minimized. The normal error is
defined as the sum of the residuals r
i
for all n measurements
(x
i
, y
i
) with a measured packet loss x
i
and a measured MOS y
i
:
(4)
We obtain the following fit for the chosen iLBC voice
codec according to the IQX hypothesis:
QoE = 3.010 ⋅ e
–4.473 ⋅ p
loss
+ 1.065. (5)
The goodness of fit for the model function f
exp
(x) can be
measured with different metrics. The coefficient of correlation
R between the model function and the measured data or the
coefficient of determination R
2
should approach one for a
close-to-perfect match. In contrast, the mean squared error
or the normalized mean squared error NMSE = MSE/Var[y
i
]
should approach zero. In fact, Eq. 5 yields R = 0.998, R
2
=
0.995, MSE = 0.003, and NMSE = 0.005, meaning that all
metrics indicate an almost perfect match. Further details are
found in [9], where we also show well fitting exponential
results for the G.711 voice codec.
It has to be noted that the packet loss is only one impairment
factor indicating QoS problems. For a general quantification of
the QoE, additional factors such as jitter have to be considered.
In the context of this work, jitter is used as a general term to
describe the variation in the end-to-end delays of IP packets
between sender and receiver, a common effect in packet-switch-
ing networks and quite disturbing in real-time communications.
Although jitter does not cause a packet to get lost in the net-
work, it is possible that the packet is received too late at the
application layer and thus considered lost from the application’s
perspective. We can thus expect excessive jitter to degrade the
QoE in a similar way as real data loss. In order to defeat such
problems and guarantee continuous playout of audio transmit-
ted over the network, jitter buffers are used at the receiver. The
maximum jitter that can be countered is equal to the buffering
delay introduced before starting the playout of the voice stream.
Nistnet is configured with the average and standard devia-
tion of the end-to-end delay as input parameters and delays
each individual packet according to a normal distribution with
the given parameters. It has to be noted that the generated
end-to-end delays of consecutive packets are independent of
each other. Hence, it is possible that packets overtake each
other, which means that a packet q sent after packet p arrives
at the receiver before packet p (i.e., t
q
< t
p
). We use the type-
p reordered ratio [10] to quantify the jitter, defined as the per-
centage of packets in the received stream that are reordered
and obtained from comparing the sequence numbers X at
sender and receiver, respectively. Voice codec and speech
samples were the same as described above. In our measure-
ments we use an average end-to-end delay of 90 ms and vary
its standard deviation σ from 0 to 90 ms. Figure 4 shows the
measurement results. Each dot in the diagram represents a
single measurement. The x-value shows the measured type-p
reordered ratio and the y-value the corresponding MOS. We
use the same nonlinear regression techniques mentioned earli-
er to obtain an optimal fitting function f
exp
(x) between the
QoE in terms of MOS and the QoS in terms of type-p
reordered ratio. The coefficient of correlation R = 0.993
shows a very good match between the measurement data and
the applied exponential model. We get similar results when
applying different metrics for quantifying the jitter of the end-
to-end path. Such metrics are, for example, the standard devi-
ation of the one-way delays, the interpacket delay variation, or
more complex reordering metrics like type-p reord-late-time.
MSE
n
r
i
i
n
=
=
∑
1
2
1
Errfxy
i
i
n
iii
==
()
−
=
∑
,.
exp
1
∂
∂
−
QoE
QoS QoS
∼
1
.
Figure 3. Measurement results and obtained mapping function
f
exp
(p
loss
) between packet loss ratio p
loss
and QoE for the iLBC
codec.
Packet loss ratio
0.20
1.5
1
Mean opinion score
2
2.5
3
3.5
4
4.5
0.4 0.6 0.8 1
Measurement exponential: R = 0.998
f
exp
(x) = 3.010 • exp(–4.473 • x) + 1.065
FIEDLER LAYOUT 3/3/10 9:38 AM Page 39
![Figure 5. Measurement results for web browsing in a fast network taken from G.1030 [3] and comparison of logarithmic model](/figures/figure-5-measurement-results-for-web-browsing-in-a-fast-7jd0epjr.webp)
![Figure 6. Measurement results for web browsing taken from Khirman and Henriksen [7] and comparison of logarithmic model](/figures/figure-6-measurement-results-for-web-browsing-taken-from-gt9irtu4.webp)
