In order to provide an immersive visual experience, modern displays require head mounting, high image resolution, low latency, as well as high refresh rate. This poses a challenging computational problem. On the other hand, the human visual system can consume only a tiny fraction of this video stream due to the drastic acuity loss in the peripheral vision. Foveated rendering and compression can save computations by reducing the image quality in the peripheral vision. However, this can cause noticeable artifacts in the periphery, or, if done conservatively, would provide only modest savings. In this work, we explore a novel foveated reconstruction method that employs the recent advances in generative adversarial neural networks. We reconstruct a plausible peripheral video from a small fraction of pixels provided every frame. The reconstruction is done by finding the closest matching video to this sparse input stream of pixels on the learned manifold of natural videos. Our method is more efficient than the state-of-the-art foveated rendering, while providing the visual experience with no noticeable quality degradation. We conducted a user study to validate our reconstruction method and compare it against existing foveated rendering and video compression techniques. Our method is fast enough to drive gaze-contingent head-mounted displays in real time on modern hardware. We plan to publish the trained network to establish a new quality bar for foveated rendering and compression as well as encourage follow-up research.

https://dl.acm.org/doi/pdf/10.1145/3355089.3356557

DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos

An ensemble method for full-reference image quality assessment (IQA) based on the parallel boosting (ParaBoost) idea is proposed in this paper. We first extract features from existing image quality metrics and train them to form basic image quality scorers (BIQSs). Then, we select additional features to address specific distortion types and train them to construct auxiliary image quality scorers (AIQSs). Both BIQSs and AIQSs are trained on small image subsets of certain distortion types and, as a result, they are weak performers with respect to a wide variety of distortions. Finally, we adopt the ParaBoost framework, which is a statistical scorer selection scheme for support vector regression (SVR), to fuse the scores of BIQSs and AIQSs to evaluate the images containing a wide range of distortion types. This ParaBoost methodology can be easily extended to images of new distortion types. Extensive experiments are conducted to demonstrate the superior performance of the ParaBoost method, which outperforms existing IQA methods by a significant margin. Specifically, the Spearman rank order correlation coefficients (SROCCs) of the ParaBoost method with respect to the LIVE, CSIQ, TID2008, and TID2013 image quality databases are 0.98, 0.97, 0.98, and 0.96, respectively.

A ParaBoost Method to Image Quality Assessment

In this work, we study the visual quality of streaming video and propose a fusion-based video quality assessment (FVQA) index to predict its quality. In the first step, video sequences are grouped according to their content complexity to reduce content diversity within each group. Then, at the second step, several existing video quality assessment methods are fused to provide the final video quality score, where fusion coefficients are learned from training video samples in the same group. We demonstrate the superior performance of FVQA as compared with other video quality assessment methods using the MCL-V video quality database.

A fusion-based video quality assessment (fvqa) index

A critical survey of state-of-the-art image inpainting quality assessment metrics

A no-reference (NR) learning-based approach to assess image quality is presented in this paper. The devised features are extracted from wide perceptual domains, including brightness, contrast, color, distortion, and texture. These features are used to train a model (scorer) which can predict scores. The scorer selection algorithms are utilized to help simplify the proposed system. In the final stage, the ensemble method is used to combine the prediction results from selected scorers. Two multiple-scale versions of the proposed approach are also presented along with the single-scale one. They turn out to have better performances than the original single-scale method. Because of having features from five different domains at multiple image scales and using the outputs (scores) from selected score prediction models as features for multi-scale or cross-scale fusion (i.e., ensemble), the proposed NR image quality assessment models are robust with respect to more than 24 image distortion types. They also can be used on the evaluation of images with authentic distortions. The extensive experiments on three well-known and representative databases confirm the performance robustness of our proposed model.

No-Reference Image Quality Assessment by Wide-Perceptual-Domain Scorer Ensemble Method

Research on visual quality assessment has been active during the last decade. In this work, we provide an in-depth review of recent developments in the field. As compared with existing survey papers, our current work has several unique contributions. First, besides image quality databases and metrics, we put equal emphasis on video quality databases and metrics as this is a less investigated area. Second, we discuss the application of visual quality evaluation to perceptual coding as an example for applications. Third, we benchmark the performance of state-of-the-art visual quality metrics with experiments. Finally, future trends in visual quality assessment are discussed.

/pdf/visual-quality-assessment-recent-developments-coding-2pm0xio8ki.pdf

Yu-Chieh Lin

Papers

Visual quality assessment : recent developments, coding applications and future trends