Mean opinion score (MOS) has become a very popular indicator of perceived media quality. While there is a clear benefit to such a "reference quality indicator" and its widespread acceptance, MOS is often applied without sufficient consideration of its scope or limitations. In this paper, we critically examine MOS and the various ways it is being used today. We highlight common issues with both subjective and objective MOS and discuss a variety of alternative approaches that have been proposed for media quality measurement.

Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives

Quality of experience (QoE) is the perceptual quality of service (QoS) from the users' perspective. For video service, the relationship between QoE and QoS (such as coding parameters and network statistics) is complicated because users' perceptual video quality is subjective and diversified in different environments. Traditionally, QoE is obtained from subjective test, where human viewers evaluate the quality of tested videos under a laboratory environment. To avoid high cost and offline nature of such tests, objective quality models are developed to predict QoE based on objective QoS parameters, but it is still an indirect way to estimate QoE. With the rising popularity of video streaming over the Internet, data-driven QoE analysis models have newly emerged due to availability of large-scale data. In this paper, we give a comprehensive survey of the evolution of video quality assessment methods, analyzing their characteristics, advantages, and drawbacks. We also introduce QoE-based video applications and, finally, identify the future research directions of QoE.

From QoS to QoE: A Tutorial on Video Quality Assessment

http://ilab.usc.edu/publications/doc/Li_etal11ivc.pdf

Visual attention guided bit allocation in video compression

The performance evaluation of video transport mechanisms becomes increasingly important as encoded video accounts for growing portions of the network traffic. Compared to the widely studied MPEG-4 encoded video, the recently adopted H.264 video coding standards include novel mechanisms, such as hierarchical B frame prediction structures and highly efficient quality scalable coding, that have important implications for network transport. This tutorial introduces a trace-based evaluation methodology for the network transport of H.264 encoded video. We first give an overview of H.264 video coding, and then present the trace structures for capturing the characteristics of H.264 encoded video. We give an overview of the typical video traffic and quality characteristics of H.264 encoded video. Finally, we explain how to account for the H.264 specific coding mechanisms, such as hierarchical B frames, in networking studies.

/pdf/video-transport-evaluation-with-h-264-video-traces-2ztswbnb2c.pdf

Video Transport Evaluation With H.264 Video Traces

This paper presents a new database, CID2013, to address the issue of using no-reference (NR) image quality assessment algorithms on images with multiple distortions. Current NR algorithms struggle to handle images with many concurrent distortion types, such as real photographic images captured by different digital cameras. The database consists of six image sets; on average, 30 subjects have evaluated 12–14 devices depicting eight different scenes for a total of 79 different cameras, 480 images, and 188 subjects (67% female). The subjective evaluation method was a hybrid absolute category rating-pair comparison developed for the study and presented in this paper. This method utilizes a slideshow of all images within a scene to allow the test images to work as references to each other. In addition to mean opinion score value, the images are also rated using sharpness, graininess, lightness, and color saturation scales. The CID2013 database contains images used in the experiments with the full subjective data plus extensive background information from the subjects. The database is made freely available for the research community.

CID2013: A Database for Evaluating No-Reference Image Quality Assessment Algorithms

The use of video-based applications has increased in recent years owing to the development of video technology as well as the widespread use of the Internet. Thus the evaluation of perceptual video quality has become very important and numerous video quality assessment (VQA) metrics have been developed over the past years. In this paper, we give a classification and a short review of objective VQA metrics, with a focus on the full reference metrics. With the aim of conducting a reliable test of the VQA metrics performances, we made two databases, each of them including 90 distorted video sequences. We carried out a subjective quality evaluation on these databases and the data were made available to the research community. Furthermore, we compared the performance of nine different, freely available, objective VQA metrics by using three different databases in different resolutions: LIVE Video Quality Database (768x432 resolution) and our two newly created databases for progressively scanned videos, i.e. ETFOS CIF Video Quality (ECVQ) database and ETFOS VGA Video Quality (EVVQ) database. Five different distortion types were used and the total number of 330 video sequences was evaluated. A comparison of metrics was done with respect to accuracy, monotonicity, stability, as well as complexity vs. accuracy criteria. The results show that the resolution, the content of the sequence and the distortion type have a significant influence on the performances of VQA metrics. Metrics that generally achieve a high correlation with subjective results for all databases and all distortion types are MOtion-based Video Integrity Evaluation (MOVIE) and Foveated Mean Squared Error (FMSE), but MOVIE has significantly higher complexity than FMSE. An exception is the distortion caused by IP transmission for which none of the analyzed metrics has shown satisfying accuracy and stability.

Review of objective video quality metrics and performance comparison using different databases

An overview of a new scalable video coding (SVC) extension of the H.264/AVC standard is given in this paper. A performance analysis is presented for quality and spatial scalability. Influence of the inter-layer prediction and the number of scalable layers to coding efficiency is shown. Results for CGS (coarse-grain quality scalability) and MGS (medium-grain quality scalability) coding are also compared.

Scalable Video Coding extension of the H.264/AVC standard

Efficiency of a video coding process, as well as accuracy of an objective video quality evaluation can be significantly improved by introduction of the human visual system (HVS) characteristics. In this paper we analyze one of these characteristics; namely, visual acuity reduction due to the foveated vision and object movements in a video sequence. We propose a new video quality metric called Foveated Mean Squared Error (FMSE) that takes into account a variable resolution of the HVS across the visual field. The highest visual acuity is at the point of fixation that falls into fovea, an area at retina with the highest density of photoreceptors. Visual acuity decreases rapidly for image regions which are further with respect to the fixation point. FMSE also utilizes the effect of additional spatial acuity reduction due to motion in a video sequence. The quality measures calculated by FMSE have shown a high correlation with experimental results obtained by subjective video quality assessment.

Foveated mean squared error--a novel video quality metric

Objective video quality evaluation incorporates spatial and temporal degradation effects to calculate quality grade for each video frame alone, as well as the overall quality grade for the whole sequence. Distortion visibility depends not only on the level of degradation but also on video content and viewer's preferences. Temporal pooling is a method that collapses series of frame quality scores to one quality score for whole video sequence and it should be aware of an influence of the video quality variation to the human overall judgments of quality. In this paper six pooling methods applied to five objective video quality metrics are compared. The best prediction capability has been obtained by those pooling methods that reflect two phenomena: a recency effect and the worst quality section influence. Moreover, the statistics of individual frames scores tends to be of high importance and strongly influences pooling method capabilities.

Influence of temporal pooling method on the objective video quality evaluation

This paper discusses methods used in different objective video quality metrics. An experimental comparison of different objective methods is also conducted. This experiment shows the importance of video content for a subjective quality evaluation not comprised well by the objective metrics used.

/pdf/objective-video-quality-metrics-3jkvyn4y61.pdf

Mario Vranjes

Papers

Review of objective video quality metrics and performance comparison using different databases

Scalable Video Coding extension of the H.264/AVC standard

Foveated mean squared error--a novel video quality metric

Influence of temporal pooling method on the objective video quality evaluation

Objective video quality metrics