QUALITY PREDICTION OF ASYMMETRICALLY DISTORTED STEREOSCOPIC
IMAGES FROM SINGLE VIEWS
Jiheng Wang, Kai Zeng and Zhou Wang
Dept. of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, N2L 3G1, Canada
Emails: {jiheng.wang, kzeng, zhou.wang}@uwaterloo.ca
ABSTRACT
Objective quality assessment of distorted stereoscopic images is a
challenging problem. Existing studies suggest that simply averaging
the quality of the left- and right-views well predicts the quality of
symmetrically distorted stereoscopic images, but generates substan-
tial prediction bias when applied to asymmetrically distorted stereo-
scopic images. In this study, we first carry out a subjective test,
where we find that the prediction bias could lean towards opposite
directions, largely depending on the distortion types. We then de-
velop an information-content and divisive normalization based pool-
ing scheme that improves upon SSIM in estimating the quality of
single view images. Finally, we propose a binocular rivalry in-
spired model to predict the quality of stereoscopic images based
on that of the single view images. Our results show that the pro-
posed model, without explicitly identifying image distortion types,
successfully eliminates the prediction bias, leading to significantly
improved quality prediction of stereoscopic images.
Index Terms— image quality assessment, stereoscopic image,
3D image, asymmetric distortion, SSIM, divisive normalization
1. INTRODUCTION
With the fast development of 3D acquisition, communication, pro-
cessing and display technologies, automatic quality assessment of
3D images and videos has become ever important. Nevertheless,
recent process on 3D image quality assessment (IQA) remains lim-
ited. In the literature, a majority of studies have been focused on
extending existing 2D-IQA methods to stereoscopic images. These
methods can be grouped into two categories based on whether depth
or disparity information is explicitly employed. In [1, 2], 2D-IQA
measures are applied to the left- and right-views separately and then
combined to a 3D quality score. In [3, 4, 5], disparity maps between
left- and right-views are estimated, followed by 2D quality assess-
ment of disparity quality, which is subsequently combined with 2D
image quality to produce an overall 3D quality score.
Recent subjective studies suggested that in the case of symmetric
distortion of both views, simply averaging state-of-the-art 2D-IQA
measures of both views is sufficient to provide reasonably accurate
quality predictions of stereoscopic images. In particular, in [6], it
was shown that averaging peak-signal-to-noise ratio (PSNR), struc-
tural similarity (SSIM) [7] and multi-scale SSIM (MS-SSIM) [8]
measurements of left- and right-views performs equally well or bet-
ter than the advanced 3D-IQA models [2, 3, 4]. Similar results
were also observed in [9], where averaging universal quality index
(UQI) [10], MS-SSIM [8] and visual information fidelity (VIF) [11]
of both views all outperformed advanced 3D-IQA models [2, 3, 4, 5].
Compared with the case of symmetric distortions, quality assess-
ment of asymmetrically distorted stereoscopic images is a much
more challenging problem. In [6], it was reported that there is a
large drop in the performance of both 2D-IQA and 3D-IQA mod-
els from quality predictions of symmetrically to asymmetrically dis-
torted stereoscopic images. Previous studies exhibit somewhat con-
flict observations and opinions regarding the effect of asymmetric
distortions. For image blur, evidence in [12] shows that the qual-
ity of asymmetric blurred images is largely dominated by the higher
quality view, a result generally agrees with [13]. For image blocki-
ness, it was reported in [14] that 3D quality has a tendency towards
the lower quality view, while in [12], it was claimed that it should be
approximated by averaging the quality of both views. In [13], it was
suggested that the best strategy of asymmetric quality assessment for
compressed images should be content and texture dependent.
Subjective data is essential in understanding the impact of asym-
metric distortions on the quality of stereoscopic images. Ideally, we
would need a complete set of subjective test on an image database
that contains both single-views and stereoscopic images, both sym-
metrically and asymmetrically distorted images at different distor-
tion levels, as well as both single- and mixed-distortion types. Ex-
isting 3D databases are highly valuable but limited in one aspect or
another. Specifically, IRCCyN/IVC 3D Images Database [4], Tianjin
Database [5], Ningbo Database Phase II [15], and LIVE 3D Image
Database Phase I [9] only include symmetrically distorted stereo-
scopic images. Ningbo Database Phase I [13] only includes asym-
metrically distorted stereoscopic images. MICT 3D Image Evalu-
ation Database [16] contains both cases but only for JPEG com-
pressed images. The most recent LIVE 3D Database Phase II [6]
includes both symmetric and asymmetric cases with five distortion
types. Unfortunately, subjective 2D-IQA of single-view images is
still missing, making it difficult to directly examine the relationship
between the perceptual quality of single-view and stereoscopic im-
ages. Moreover, asymmetric distortions with mixed distortion types
are missing in all existing databases, making it hard to validate the
generalization capability of 3D quality prediction models.
Meanwhile, studying the impact of asymmetric distortions on
the quality of stereoscopic images not only has scientific values in
understanding the human visual system, but is also desirable in the
practice of 3D video compression and transmission, where it has
been hypothesis that only one of the two views need to be coded
at high rate, and thus significant bandwidth can be saved by coding
the other view with low rate. However, previously reported test-
ing on this hypothesis has been controversial [17, 14], because of a
lack of accurate quality model. It is worth noting that the distortions
involved in 3D video coding/communication are not only compres-
sion artifacts. The practical encoder/decoder also needs to decide on
whether deblocking filters need to be turned on, and whether mixed-
resolutions of the left/right views should be used. Thus the actual
asymmetric distortions in practice could be mixed and complicated.
In this work, we first carry out a subjective quality assessment
2014 IEEE International Conference on Multimedia and Expo (ICME'14), Chengdu, China, July 2014
experiment on a database that contains both single-views and stereo-
scopic images with symmetric and asymmetric distortion types and
levels. This database allows us to directly study the quality pre-
diction performance from single-views to stereoscopic images, for
which we observe that simply averaging the quality of both views
creates substantial bias on asymmetrically distorted stereoscopic im-
ages, and interestingly, the bias could lean towards opposite di-
rections, largely depending on the distortion types. We then de-
velop an information-content and divisive normalization based pool-
ing scheme that improves upon SSIM in estimating the quality of
single-views. Furthermore, we propose a binocular rivalry inspired
model to account for the bias, which not only results in better quality
prediction of stereoscopic images with asymmetric distortion levels,
but also well generalizes to the case of asymmetric distortions with
mixed distortion types.
2. SUBJECTIVE STUDY
2.1. WATERLOO IVC 3D Image Quality Database
The new Waterloo IVC 3D Image Quality Database is created from
6 pristine stereopairs (and thus their corresponding single-views),
all collected from the Middlebury Stereo 2005 Datasets [18]. Each
single-view was altered by three types of distortions: noise contam-
ination, blur, and JPEG compression and each distortion type had
four distortion levels. The single-views are employed to generate
distorted stereopairs, either symmetrically or asymmetrically. Ta-
ble 1 categorizes these images into seven groups. There are two
unique features of the new database when compared with existing
3D-IQA databases. First, this is the only database that performs sub-
jective test on both 2D and 3D images. The inclusion of 2D images
allows us to directly examine the relationship between the perceptual
quality of stereoscopic images and that of the its single-view images.
Second, this is the only database that contains mixed distortion types
in asymmetrically distorted images. This provides the potential of a
much stronger test on 3D-IQA models on their generalizability.
Table 1. Categories of test images
Group # of images Description
Group 2D.0 6 × 1 Pristine single-view images
Group 2D.1 6 × 12 Distorted single-view images
Group 3D.0 6 × 1 Pristine stereopairs
Group 3D.1 6 × 12 Symmetrically distorted stereopairs
with the same distortion type and
distortion level
Group 3D.2 6 × 12 Asymmetrically distorted stere-
opairs with distortion on one view
only
Group 3D.3 6 × 18 Asymmetrically distorted stere-
opairs with the same distortion type
but different levels
Group 3D.4 6 × 12 Asymmetrically distorted stere-
opairs with mixed distortion types
and levels
Single stimulus continuous quality scale protocol was adopted
in the subjective test. An ASUS 27” VG278H 3D LED monitor with
NVIDIA 3D Vision
TM
2 active shutter glasses is used for the test.
Twenty-four naive subjects, 14 males and 10 females aged from 22 to
45, participated in the study. Following previous works [19, 20, 21],
the subjects were asked to evaluate four aspects of their 3D viewing
experience, including the perception of 3D image quality (3DIQ),
depth quality (DQ), visual comfort (VC) and overall 3D quality of
experience (3DQoE). The detailed descriptions of each aspects of
visual experience are elaborated in Table 2. The rest of the paper
focuses on the relationship between single-view image quality and
the 3DIQ scores. More detailed descriptions of our database and
analysis of the other aspects of the subjective experiments will be
reported in future publications.
Table 2. Description of visual experience criteria
Criterion Description
3DIQ The image content quality without considering 3D
viewing experience
DQ The amount, naturalness and clearness of depth per-
ception experience
VC The comfortness when viewing stereoscopic images
3DQoE The overall 3D viewing experience
2.2. Key Observations
The raw 3DIQ scores given by each subject were converted to Z-
scores and the entire data set was rescaled to fill the range from 1 to
100. The mean opinion score (MOS) for each image was then com-
puted after removing outliers. Given the subjective data, the main
question we would like to ask first is how the single-view 2D image
quality predicts the 3D image quality, for which the most straight-
forward prediction method is to directly average the MOSs of the
left- and right-views. The left column of Fig. 1 shows the scatter
plots of using averaging 2D-MOS to predict 3D-MOS for different
distortion types. Quantitative measures of Spearman rank-order cor-
relation coefficient (SRCC) can be found in Table 3, which shows
that the best prediction occurs when the distortions are symmetric
(consistent with the literature [9, 6]). By contrast, the SRCC values
drop significantly with asymmetrical distortions. The drops are also
reflected in the scatter plots in Fig. 1, where the direct average pre-
diction model generates substantial bias of many stereopairs. Most
interestingly, this bias leans towards opposite directions, largely de-
pending on the distortion types. In particular, for noise contamina-
tion and JPEG compression, direct average prediction overestimates
the 3D quality of many images, while for blur distortion, direct av-
erage prediction often underestimates the 3D image quality.
It is interesting to compare our observations regarding distortion
type dependency with those published in the literature. For image
blur, it was reported in [12, 13] that 3D quality is less affected by the
view with lower quality, which is consistent with our result. For im-
age blockiness, [14] and [12] reported somewhat conflicting results.
The former concluded that 3D quality is mainly dependent on the
view with lower quality, and the latter suggested that averaging the
quality of both views is a better choice. These seemingly controver-
sial results are well explained by the scatter plot shown in Fig. 1(g),
where the bias of the averaging prediction model increases with the
level of distortions, and thus whether the bias is pronounced depends
on the quality range being investigated.
3. 2D-TO-3D QUALITY PREDICTION METHOD
We opt to use a two-stage approach in the design of an objective
3DIQ predictor. The first stage aims to evaluate the perceptual qual-
ity of single-view images, while in the second stage, a binocular
rivalry inspired model is developed to combine 2D image quality of
both views into a quality estimation of 3D image quality.
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
(a) All Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
(b) All Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Noisy Images
Other Images
(c) Noisy Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Noisy Images
Other Images
(d) Noisy Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Blurred Images
Other Images
(e) Blurred Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Blurred Images
Other Images
(f) Blurred Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
JPEG Images
Other Images
(g) JPEG Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
JPEG Images
Other Images
(h) JPEG Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Mixed Distortion Images
Other Images
(i) Mixed Distortion Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Mixed Distortion Images
Other Images
(j) Mixed Distortion Images
Fig. 1. Scatter plots of 3D image quality MOS score versus predic-
tions from the MOS scores of 2D left- and right-views. Left column,
prediction by averaging the MOS scores of both views; right column,
prediction by the proposed method.
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
(a) All Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
(b) All Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Noisy Images
Other Images
(c) Noisy Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Noisy Images
Other Images
(d) Noisy Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Blurred Images
Other Images
(e) Blurred Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Blurred Images
Other Images
(f) Blurred Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
JPEG Images
Other Images
(g) JPEG Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
JPEG Images
Other Images
(h) JPEG Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Direct Averaging
Mixed Distortion Images
Other Images
(i) Mixed Distortion Images
0 20 40 60 80 100
0
20
40
60
80
100
3D MOS
Proposed
Mixed Distortion Images
Other Images
(j) Mixed Distortion Images
Fig. 2. Scatter plots of 3D image quality MOS score versus predic-
tions from the IDW-SSIM scores of 2D left- and right-views. Left
column, prediction by averaging IDW-SSIM of both views; right col-
umn, prediction by the proposed method.
Table 3. SRCC performance of 2D-to-3D quality prediction models
on WATERLOO IVC 3D database
All images Symmetric Asymmetric
2D-MOS with
0.8765 0.9657 0.8471
direct average
2D-MOS with
0.9484 0.9657 0.9405
proposed weighting
You [3] 0.5968 0.7517 0.5706
Benoit [4] 0.5852 0.7275 0.5766
Yang [5] 0.6106 0.6668 0.6108
Proposed 0.8836 0.9462 0.8846
3.1. Objective 2D Quality Assessment
In the literature, the SSIM index [7] and its derivatives [8, 22]
have demonstrated competitive performance in 2D objective IQA
tests [22]. An advantage of the SSIM approach is that it provides
a quality map that indicates the variations of image quality over
space [7]. It was shown that spatial pooling built upon the qual-
ity map based on information-content or distortion weighting further
improves the performance [22]. When testing SSIM and its deriva-
tives on the 2D image datasets of our new database, we find although
they have high correlations with subjective quality on images with
the same distortion types, their performance drops when all distor-
tion types are mixed. Therefore, here we build our 2D-IQA model
upon SSIM, but improve it further by incorporating an information-
content and divisive normalization based pooling scheme. A general
form of spatially weighted pooling is given by
Q
2D
=
P
N
i=1
w
i
q
i
P
N
i=1
w
i
, (1)
where q
i
and w
i
are the local quality value (e.g., local SSIM value)
and the weight assigned to the i-th spatial location, respectively. The
assumption behind information-content weighted pooling is that the
spatial locations that contain more information are more likely to at-
tract visual attention, and thus should be given larger weights. Let x
i
and y
i
be the local image patches extracted around the i-th spatial
location from the reference and the distorted images, respectively.
Following the information content evaluation method in [23], we
compute the weighting factor by
w
(ic),i
= log
1 +
σ
2
x
C
1 +
σ
2
y
C
, (2)
where σ
x
and σ
y
are the standard deviations of x
i
and y
i
, respec-
tively, and C is the noisy visual channel power. Another useful pool-
ing strategy is distortion-weighted pooling, which is based on the
intuitive idea that the spatial locations that contain more distortions
are more likely to attract visual attention, and thus should be given
more weights. Since the local quality has been gauged by q
i
(e.g.,
the SSIM value at location i), we can easily obtain a local distortion
measure, for example, by d
i
= 1 − SSIM
i
. Divisive normaliza-
tion has been recognized as a perceptually and statistically motivated
non-linear transformation [24]. We apply divisive normalization to
the local distortion map and define a normalized distortion based
weighting factor by
w
(d),i
=
d
i
q
P
j∈N
i
d
2
j
+ D
0
, (3)
where N
i
denotes the set of neighboring pixels surrounding the i-th
spatial location, and D
0
is a stability constant. The final weight-
ing factor is obtained by combining information content and divisive
normalization-based distortion weighting factors
w
i
= max{w
2
(ic),i
, w
2
(d),i
} . (4)
Applying this weighted pooling approach to the SSIM map, we ob-
tain an information-content and distortion weighted SSIM (IDW-
SSIM) measure. This has led to significant performance improve-
ment when tested using the single-view images in our new database.
Specifically, the SRCC with respect to the MOS scores has been im-
proved from 0.4800 for PSNR and 0.7066 for SSIM to 0.9381 for
IDW-SSIM.
3.2. 2D-to-3D Quality Prediction Model
The competition between binocular fusion and binocular rivalry [25]
provides a potential theory to develop 2D-to-3D quality prediction
models. When the left- and right-views are consistent, they are fused
in the visual system to a single percept of the scene, known as binoc-
ular fusion. On the other hand, when the two views are inconsistent,
instead of being seen superimposed, one of them may dominate or
two views may be seen alternately, known as binocular rivalry [25].
Although there is a rich literature on binocular fusion and rivalry in
biological vision science [25], how to apply the principle to 3D-IQA
remains an active research topic. Since in 3D-IQA we need to work
on complicated scenes and distortions, simplifications are essential
to create practical solutions.
Our work is motivated by existing vision studies on binocular
rivalry [26, 27], where it was found that for simple ideal stimuli, an
increasing contrast increases the predominance of one view against
the other. Also note that in complicated scenes the contrast of a sig-
nal increases with its signal strength measured using energy. This
inspires us to hypothesize that the level of view dominance in binoc-
ular rivalry of stereoscopic images is monotonically increasing with
the relative energy of the two views. The diagram of the proposed
method is shown in Fig. 3. Let (I
r,l
, I
r,r
) and (I
d,l
, I
d,r
) be the
left- and right-view image pairs of the reference and distorted stereo-
scopic images, respectively. We first create their local energy maps
by computing the local variances, i.e., the variances of local image
patches extracted around each spatial location from the reference or
the distorted images are computed, for which a sliding Gaussian win-
dow with standard deviation of 1.5 is employed. The resulting en-
ergy maps are denoted as E
r,l
, E
r,r
, E
d,l
and E
d,r
, respectively.
Assume that the reference stereopair has perfect quality with strong
3D effect, where binocular fusion prevails. When at least one of the
single-view images is distorted at some spatial locations, the distor-
tion may affect the consistency between the image structures from
the two views, and thus binocular rivalry prevails. As a result, one
view may dominate the other at any time instance. Based on our
hypothesis, we compute the local energy ratio maps in both views:
R
l
=
E
d,l
E
r,l
and R
r
=
E
d,r
E
r,r
. (5)
The energy ratio maps provide useful local information, which may
be combined with the qualities of single-view images to predict 3D
image quality. A pooling stage is necessary for this purpose. To em-
phasize on the importance of high-energy image regions, we adopt
an energy weighted pooling method given by
g
l
=
P
E
d,l
R
l
P
E
d,l
and g
r
=
P
E
d,r
R
r
P
E
d,r
, (6)
LocalEnergy
Estimation
LocalEnergy
Estimation
LocalEnergy
Estimation
LocalEnergy
Estimation
RatioMap
Computation
RatioMap
Computation
Spatial
Pooling
Spatial
Pooling
Weight
Computation
Reference
Stereoscopic
Images
Distorted
Stereoscopic
Images
Weighted
Average
3DQuality
Measure
2DLeftViewQuality
2DRightViewQuality
LeftView
RightView
LeftView
2DQuality
Asessment
2DQuality
Assessment
RightView
Fig. 3. Diagram of the proposed 2D-to-3D quality prediction model.
Table 4. SRCC performance of 2D-to-3D prediction models
Database 2D-IQA Method Direct Average Proposed
Waterloo IVC
2D-MOS 0.8765 0.9484
PSNR 0.5213 0.5240
SSIM 0.5976 0.6580
IDW-SSIM 0.7761 0.8967
LIVE Phase II
PSNR 0.7303 0.7638
SSIM 0.7925 0.8560
IDW-SSIM 0.7886 0.9145
where the summations are over the full energy and ratio maps. The
weights assigned to the left- and right-views are then given by
w
l
=
g
2
l
g
2
l
+ g
2
r
and w
r
=
g
2
r
g
2
l
+ g
2
r
. (7)
Finally, the overall prediction of 3D image quality is calculated by a
weighted average of the left-view and right-view image quality:
Q
3D
= w
l
Q
l
2D
+ w
r
Q
r
2D
, (8)
where Q
l
2D
and Q
r
2D
denote the 2D image quality of the left- and
right-views, respectively.
3.3. Testing
We use two 3D image quality databases to test the proposed algo-
rithm, which are our WATERLOO IVC 3D database and the LIVE
3D Phase II database [6]. The latter is a most recent database that
contains both symmetrically and asymmetrically distorted images.
Note that the parameters of the proposed method is selected em-
pirically when working with our new database, but are completely
independent of the LIVE database. Due to space limit, only SRCC
results are reported here, but highly consistent results are also ob-
tained in our analysis based on Pearson linear correlation coefficient
and Kendall rank-order correlation coefficient.
Table 5. SRCC performance of 2D-to-3D quality prediction models
on LIVE 3D Phase II database
All images Symmetric Asymmetric
You [3] 0.7924 0.8030 0.7721
Benoit [4] 0.7436 0.6959 0.7474
Yang [5] 0.7210 0.7608 0.6960
Chen [6] 0.8800 0.9180 0.8340
Proposed 0.9145 0.9190 0.9030
We test the proposed 2D-to-3D quality prediction model by ap-
plying it to different base 2D-IQA approaches on both databases.
The comparison results with the direct averaging method are shown
in Table 4, where it can be seen that the proposed method signif-
icantly improves most base 2D-IQA methods. The only exception
is PSNR, which might be due to its poor performance in 2D image
quality assessment, and thus merely changing 2D to 3D prediction
method would not lead to any meaningful result. By comparing the
left and right columns of Figs. 1 and 2, we observe how the proposed
2D-to-3D prediction model affects each image distortion type, for
the cases of using 2D-MOS and IDW-SSIM as the base IQA meth-
ods, respectively. Most importantly, for different distortion types,
although the direct averaging method produces different levels of
quality prediction biases towards different directions, the proposed
method, which does not attempt to identify the distortion types or
give any specific treatment for any specific distortion type, removes
or significantly reduces the prediction biases for all distortion types.
Moreover, the mixed distortion case provides the strongest test on the
generalization ability of the model, for which the proposed method
maintains consistent improvement.
We have also compared the proposed method with state-of-the-
art 3D-IQA approaches [3, 4, 5, 6] using both databases, and the
results are shown in Tables 3 and 5, respectively. The proposed
method achieves the best performance in both databases among all
full objective IQA methods. The highly competitive performance in
the LIVE database is a more convincing result because the test is
