Stereoscopic video quality assessment (SVQA) is a challenging problem. It has not been well investigated on how to measure depth perception quality independently under different distortion categories and degrees, especially exploit the depth perception to assist the overall quality assessment of 3D videos. In this paper, we propose a new depth perception quality metric (DPQM) and verify that it outperforms existing metrics on our published 3D video extension of High Efficiency Video Coding (3D-HEVC) video database. Furthermore, we validate its effectiveness by applying the crucial part of the DPQM to a novel blind stereoscopic video quality evaluator (BSVQE) for overall 3D video quality assessment. In the DPQM, we introduce the feature of auto-regressive prediction-based disparity entropy (ARDE) measurement and the feature of energy weighted video content measurement, which are inspired by the free-energy principle and the binocular vision mechanism. In the BSVQE, the binocular summation and difference operations are integrated together with the fusion natural scene statistic measurement and the ARDE measurement to reveal the key influence from texture and disparity. Experimental results on three stereoscopic video databases demonstrate that our method outperforms state-of-the-art SVQA algorithms for both symmetrically and asymmetrically distorted stereoscopic video pairs of various distortion types.

Blind Stereoscopic Video Quality Assessment: From Depth Perception to Overall Experience

Image quality is important not only for the viewing experience, but also for the performance of image processing algorithms. Image quality assessment (IQA) has been a topic of intense research in the fields of image processing and computer vision. In this paper, we first analyze the factors that affect two-dimensional (2D) and three-dimensional (3D) image quality, and then provide an up-to-date overview on IQA for each main factor. The main factors that affect 2D image quality are fidelity and aesthetics. Another main factor that affects stereoscopic 3D image quality is visual comfort. We also describe the IQA databases and give the experimental results on representative IQA metrics. Finally, we discuss the challenges for IQA, including the influence of different factors on each other, the performance of IQA metrics in real applications, and the combination of quality assessment, restoration, and enhancement.

/pdf/2d-and-3d-image-quality-assessment-a-survey-of-metrics-and-5ej1ev4vs2.pdf

2D and 3D Image Quality Assessment: A Survey of Metrics and Challenges

https://nottingham-repository.worktribe.com/preview/743551/JLR%20entry%20exit%20paper_ePrints.pdf

The Use of Virtual Reality and Physical Tools in the Development and Validation of Ease of Entry and Exit in Passenger Vehicles

In this paper, we propose a novel deep learning-based visual comfort assessment (VCA) for stereoscopic images. To assess the overall degree of visual discomfort in stereoscopic viewing, we devise a binocular fusion deep network (BFN) learning binocular characteristics between stereoscopic images. The proposed BFN learns the latent binocular feature representations for the visual comfort score prediction. In the BFN, the binocular feature is encoded by fusing the spatial features extracted from left and right views. Finally, the visual comfort score is predicted by projecting the binocular feature onto the subjective score space. In addition, we devise a disparity regularization network (DRN) for improving the prediction results. The proposed DRN takes the binocular feature from the BFN and estimates disparity maps from the feature in order to embed disparity relations between left and right views into the deep network. The proposed deep network with BFN and DRN is end-to-end trained in a unified framework in which the DRN acts as disparity regularization. We evaluated the prediction performance of the proposed deep network for VCA by the comparison of existing objective VCA metrics. Further, we demonstrated that the proposed BFN showed various factors causing visual discomfort by using network visualization.

Binocular Fusion Net: Deep Learning Visual Comfort Assessment for Stereoscopic 3D

When compared to conventional 2-D video, multiview video can significantly enhance the visual 3-D experience in 3-D applications by offering horizontal parallax. However, when processing images originating from different views, it is common that the colors between the different cameras are not well- calibrated . To solve this problem, a novel energy function -based color correction method for multiview camera setups is proposed to enforce that colors are as close as possible to those in the reference image but also that the overall structural information is well-preserved. The proposed system introduces a spatio-temporal correspondence matching method to ensure that each pixel in the input image gets bijectively mapped to a reference pixel. By combining this mapping with the original structural information, we construct a global optimization algorithm in a Laplacian matrix formulation and solve it using a sparse matrix solver. We further introduce a novel forward-reverse objective evaluation model to overcome the problem of lack of ground truth in this field. The visual comparisons are shown to outperform state-of-the-art multiview color correction methods, while the objective evaluation reports PSNR gains of up to 1.34 dB and SSIM gains of up to 3.2%, respectively.

/pdf/spatio-temporally-consistent-color-and-structure-31cu1vhj0k.pdf

Spatio-Temporally Consistent Color and Structure Optimization for Multiview Video Color Correction

Almost all existing 3D visual discomfort prediction models are based, at least in part, on features that are extracted from computed disparity maps. These include such estimated quantities such as the maximum disparity, disparity range, disparity energy and other measures of the disparity distribution. A common first step when implementing a 3D visual discomfort model is some form of disparity calculation, whence the accuracy of prediction largely depends on the accuracy of the disparity result. Unfortunately, most algorithms that compute disparity maps are expensive, and are not guaranteed to deliver sufficiently accurate or perceptually relevant disparity data. This raises the question of whether it is possible to build a 3D discomfort prediction model without explicit disparity calculation. Towards this possibility, we have developed a new feature map, called the percentage of un-linked pixels (PUP), that is descriptive of the presence of disparity, and which can be used to accurately predict experienced 3D visual discomfort without the need for actually calculating disparity values. Instead, PUP features are extracted by predicting the percentage of un-linked pixels in corresponding retinal patches of image pairs. The un-linked pixels are determined by feature classification on orientation and luminance distributions. Calculation of PUP maps is much faster than traditional disparity computation, and the experimental results demonstrate that the predictive power attained using the PUP map is highly competitive with prior models that rely on computed disparity maps. HighlightsA first of a kind 3D discomfort model without disparity calculation is proposed.A new feature map, the percentage of un-linked pixels is developed in the model.PUP map is superior to prior models that rely on computed disparity maps.

/pdf/visual-discomfort-prediction-on-stereoscopic-3d-images-1pgehecnw6.pdf

Visual discomfort prediction on stereoscopic 3D images without explicit disparities

This paper focuses on discomfort assessment issue for S3D images with multiple salient objects, and proposes four kinds of visual features of S3D images that have potential correlations with visual discomfort, including disparity distribution features, disparity jump features, object distribution features as well as object width features. A stereoscopic image database is designed focusing on multiple salient object scenarios and prediction model is built using support vector regression based on the proposed features. Prediction performance evaluation result shows that the proper combinations of the proposed features can achieve effective visual discomfort prediction for these types of pictures, and also shows that the convergence adjustments induced by the disparity discrepancies among the salient objects do cause visual discomfort in these scenarios.

Visual comfort assessment of stereoscopic images with multiple salient objects

Luminance discrepancies between image pairs occur owing to inconsistent parameters between stereoscopic camera devices and from imperfect capture conditions. Such discrepancies induce binocular mismatches and affect the visual comfort that is felt by viewers, as well as their ability to fuse stereoscopic. To better understand and observe this effect, we built a stereoscopic images database of 240 luminance discrepancy images and 30 natural images with subjective scores of visual discomfort and fusion difficulty. Two features, binocular contrast and luminance similarity were extracted to analyze the relationship between the subjective scores and the luminance discrepancies. Structural dissimilarity and average luminance are used to predict the effects of binocular mismatches. The experimental results show that the combination of binocular contrast, structural dissimilarity and average luminance exhibits high consistency with subjective scores of visual discomfort, fusion difficulty and overall binocular mismatches in terms of Spearman's Rank Ordered Correlation Coefficient.

Binocular mismatch induced by luminance discrepancies on stereoscopic images

Three-dimensional television (3D-TV) adopts two-view stereoscopic 3D (S3D) images with a special watching mechanism. Perfect stereoscopic visual comfort experience can bring enhanced depth perception. Actually, there are numerous potential sources of adverse effects make visual discomfort. Border effects are the usual scenarios that lead to visual discomfort due to the depth conflict between the border objects and screen border in the 3D perceptual space. In this paper, we focus on how the border effects affect visual discomfort and fatigue. We investigate the visual features of border objects qualify the the visual comfort, including border object disparity, border object thickness, border object blur and border object color contrast. The visual features of border objects are extracted to qualify the visual comfort level induced by border effects. To have a better performance evaluation on visual comfort of border effects, we design our stereoscopic images database VBED (Visual Border Effect Database) focusing on various border objects dependent visual features. Experimental results demonstrate that the combined use of various object dependent visual features can achieve effective border effects prediction.

Comfort assessment of border effects in stereoscopic images

More and more stereoscopic film makers have begun to use fully integrated 3D camcorders. These 3D camcorders provide a stable shooting style and enable easy adjustment of the convergence point. In this paper, we analyzed the visual discomfort induced by adjustment of the convergence point in a video scene. We captured outdoor and indoor scenes and conducted a subjective assessment to investigate the degree of visual discomfort caused by the adjustment and the level of difficulty to recognize the variation of depth induced by the adjustment. As results of the subjective assessment, we observed that human eyes could be easily affected by sudden change of convergence point. But slowly gradual variation of convergence point will bring only little visual discomfort and it is really difficult for people to recognize the variation of depth even with rapid adjustment.

Jianyu Chen

Papers

Visual discomfort prediction on stereoscopic 3D images without explicit disparities

Visual comfort assessment of stereoscopic images with multiple salient objects

Binocular mismatch induced by luminance discrepancies on stereoscopic images

Comfort assessment of border effects in stereoscopic images

Visual discomfort induced by adjustment of convergence distance in stereoscopic video