Perceptual quality assessment of high frame rate video
03 Dec 2015-pp 1-6
TL;DR: It is observed that perceived video quality generally increases with frame rate, but the gain saturates at high rates, and such gain also depends on the interactions between quantization level, spatial resolution, and spatial and motion complexities.
Abstract: High frame rate video has been a hot topic in the past few years driven by a strong need in the entertainment and gaming industry. Nevertheless, progress on perceptual quality assessment of high frame rate video remains limited, making it difficult to evaluate the exact perceptual gain by switching from low to high frame rates. In this work, we first conduct a subjective quality assessment experiment on a database that contains videos compressed at different frame rates, quantization levels and spatial resolutions. We then carry out a series of analysis on the subjective data to investigate the impact of frame rate on perceived video quality and its interplay with quantization level, spatial resolution, spatial complexity, and motion complexity. We observe that perceived video quality generally increases with frame rate, but the gain saturates at high rates. Such gain also depends on the interactions between quantization level, spatial resolution, and spatial and motion complexities.
Citations
More filters
TL;DR: A comprehensive survey of the works that have been carried out over recent decades in perceptual audio, video, and joint audio-visual quality assessments is provided, describing existing methodologies in terms of requirement of a reference signal, feature extraction, feature mapping, and classification schemes.
Abstract: Measuring perceived quality of audio-visual signals at the end-user has become an important parameter in many multimedia networks and applications. It plays a crucial role in shaping audio-visual processing, compression, transmission and systems, along with their implementation, optimization, and testing. Service providers are enacting different quality of service (QoS) solutions to issue the best quality of experience (QoE) to their customers. Thus, devising precise perception-based quality metrics will greatly help improving multimedia services over wired and wireless networks. In this paper, we provide a comprehensive survey of the works that have been carried out over recent decades in perceptual audio, video, and joint audio-visual quality assessments, describing existing methodologies in terms of requirement of a reference signal, feature extraction, feature mapping, and classification schemes. In this context, an overview of quality formation and perception, QoS, QoE as well as quality of perception is also presented. Finally, open issues and challenges in audio-visual quality assessment are highlighted and potential future research directions are discussed.
70 citations
TL;DR: Temporal down-sampling is utilized to enable both subjective and objective comparisons across a range frame rates and shows that those which explicitly account for temporal distortions provide improved correlation with subjective opinions compared to generic quality metrics such as PSNR.
Abstract: High frame rates are acknowledged to increase the perceived quality of certain video content. However, the lack of high frame rate test content has previously restricted the scope of research in this area—especially in the context of immersive video formats. This problem has been addressed through the publication of a high frame rate video database BVI-HFR, which was captured natively at 120 fps. BVI-HFR spans a variety of scenes, motions, and colors, and is shown to be representative of BBC broadcast content. In this paper, temporal down-sampling is utilized to enable both subjective and objective comparisons across a range frame rates. A large-scale subjective experiment has demonstrated that high frame rates lead to increases in perceived quality, and that a degree of content dependence exists—notably related to camera motion. Various image and video quality metrics have been benchmarked on these subjective evaluations, and analysis shows that those which explicitly account for temporal distortions (e.g., FRQM) provide improved correlation with subjective opinions compared to generic quality metrics such as PSNR.
50 citations
Cites background from "Perceptual quality assessment of hi..."
...Previous research has demonstrated that there are a number of clear benefits associated with increased frame rates, including: the visibility of temporal aliasing artefacts being diminished [10], [21]–[26]; a reduction in perceptible motion blur [23]–[25], [27]–[29]; increased realism, smoother motion, improved depth perception for both expert [30] and nonexpert [31] viewers (especially when tracking using smooth pursuit eye movements); more realistic motion image quality (confirmed using EEG data) [32]; a reduction in viewer stress levels [33] (signified by a lower blinking frequency [34]); an improvement in speed and spatial discrimination, and reading ability [35]; and an increase in perceptual quality [18], [36], at least up to 240 fps [29]....
[...]
TL;DR: A new subjective resource, called the LIVE-YouTube-H FR (LIVE-YT-HFR) dataset, which is comprised of 480 videos having 6 different frame rates, obtained from 16 diverse contents, and is made available online for public use and evaluation purposes.
Abstract: High frame rate (HFR) videos are becoming increasingly common with the tremendous popularity of live, high-action streaming content such as sports. Although HFR contents are generally of very high quality, high bandwidth requirements make them challenging to deliver efficiently, while simultaneously maintaining their quality. To optimize trade-offs between bandwidth requirements and video quality, in terms of frame rate adaptation, it is imperative to understand the intricate relationship between frame rate and perceptual video quality. Towards advancing progression in this direction we designed a new subjective resource, called the LIVE-YouTube-HFR (LIVE-YT-HFR) dataset, which is comprised of 480 videos having 6 different frame rates, obtained from 16 diverse contents. In order to understand the combined effects of compression and frame rate adjustment, we also processed videos at 5 compression levels at each frame rate. To obtain subjective labels on the videos, we conducted a human study yielding 19,000 human quality ratings obtained from a pool of 85 human subjects. We also conducted a holistic evaluation of existing state-of-the-art Full and No-Reference video quality algorithms, and statistically benchmarked their performance on the new database. The LIVE-YT-HFR database has been made available online for public use and evaluation purposes, with hopes that it will help advance research in this exciting video technology direction. It may be obtained at https://live.ece.utexas.edu/research/LIVE_YT_HFR/LIVE_YT_HFR/index.html .
32 citations
TL;DR: An objective VQA model called Space-Time GeneRalized Entropic Difference (GREED) is devised which analyzes the statistics of spatial and temporal band-pass video coefficients and achieves state-of-the-art performance on the LIVE-YT-HFR Database when compared with existing V QA models.
Abstract: We consider the problem of conducting frame rate dependent video quality assessment (VQA) on videos of diverse frame rates, including high frame rate (HFR) videos. More generally, we study how perceptual quality is affected by frame rate, and how frame rate and compression combine to affect perceived quality. We devise an objective VQA model called Space-Time GeneRalized Entropic Difference (GREED) which analyzes the statistics of spatial and temporal band-pass video coefficients. A generalized Gaussian distribution (GGD) is used to model band-pass responses, while entropy variations between reference and distorted videos under the GGD model are used to capture video quality variations arising from frame rate changes. The entropic differences are calculated across multiple temporal and spatial subbands, and merged using a learned regressor. We show through extensive experiments that GREED achieves state-of-the-art performance on the LIVE-YT-HFR Database when compared with existing VQA models. The features used in GREED are highly generalizable and obtain competitive performance even on standard, non-HFR VQA databases. The implementation of GREED has been made available online: this https URL
30 citations
Cites methods from "Perceptual quality assessment of hi..."
...The problem of HFR-VQA has been previously attempted by analyzing databases like Waterloo HFR [1] and BVI-HFR [2], which primarily address HFR content quality....
[...]
01 Sep 2017
TL;DR: An objective quality metric (FRQM), which characterises the relationship between variations in frame rate and perceptual video quality, and offers significant improvement over all other tested quality assessment methods with relatively low complexity.
Abstract: This paper presents an objective quality metric (FRQM), which characterises the relationship between variations in frame rate and perceptual video quality. The proposed method estimates the relative quality of a low frame rate video with respect to its higher frame rate counterpart, through temporal wavelet decomposition, subband combination and spatiotemporal pooling. FRQM was tested alongside six commonly used quality metrics (two of which explicitly relate frame rate variation to perceptual quality), on the publicly available BVI-HFR video database, that spans a diverse range of scenes and frame rates, up to 120fps. Results show that FRQM offers significant improvement over all other tested quality assessment methods with relatively low complexity.
28 citations
References
More filters
TL;DR: A new philosophy in designing image and video quality metrics is followed, which uses structural dis- tortion as an estimate of perceived visual distortion as part of full-reference (FR) video quality assessment.
Abstract: Objective image and video quality measures play important roles in a variety of image and video pro- cessing applications, such as compression, communication, printing, analysis, registration, restoration, enhancement and watermarking. Most proposed quality assessment ap- proaches in the literature are error sensitivity-based meth- ods. In this paper, we follow a new philosophy in designing image and video quality metrics, which uses structural dis- tortion as an estimate of perceived visual distortion. A com- putationally ecient approach is developed for full-reference (FR) video quality assessment. The algorithm is tested on the video quality experts group (VQEG) Phase I FR-TV test data set. Keywords—Image quality assessment, video quality assess- ment, human visual system, error sensitivity, structural dis- tortion, video quality experts group (VQEG)
1,083 citations
"Perceptual quality assessment of hi..." refers background in this paper
...First, the uncertainty of human visual perception increases with the speed of motion [20, 21]....
[...]
TL;DR: Improved video quality assessment algorithms are obtained by incorporating a recent model of human visual speed perception and incorporating the model as spatiotemporal weighting factors, where the weight increases with the information content and decreases with the perceptual uncertainty in video signals.
Abstract: Motion is one of the most important types of information contained in natural video, but direct use of motion information in the design of video quality assessment algorithms has not been deeply investigated. Here we propose to incorporate a recent model of human visual speed perception [Nat. Neurosci. 9, 578 (2006)] and model visual perception in an information communication framework. This allows us to estimate both the motion information content and the perceptual uncertainty in video signals. Improved video quality assessment algorithms are obtained by incorporating the model as spatiotemporal weighting factors, where the weight increases with the information content and decreases with the perceptual uncertainty. Consistent improvement over existing video quality assessment algorithms is observed in our validation with the video quality experts group Phase I test data set.
224 citations
"Perceptual quality assessment of hi..." refers background in this paper
...First, the uncertainty of human visual perception increases with the speed of motion [20, 21]....
[...]
25 Apr 2004
TL;DR: Contrary to existing guidelines, it is found that users prefer high-resolution images to high frame rate, and the rule "high motion = highframe rate" does not apply to small screens.
Abstract: We introduce a new methodology to evaluate the perceived quality of video with variable physical quality. The methodology is used to evaluate existing guidelines - that high frame rate is more important than quantization when watching high motion video, such as sports coverage. We test this claim in two studies that examine the relationship between these physical quality metrics and perceived quality. In Study 1, 41 soccer fans viewed CIF-sized images on a desktop computer. Study 2 repeated the experiment with 37 soccer fans, viewing the same content, in QCIF size, on a palmtop device. Contrary to existing guidelines, we found that users prefer high-resolution images to high frame rate. We conclude that the rule "high motion = high frame rate" does not apply to small screens. With small screen devices, reducing quantization removes important information about the players and the ball. These findings have important implications for service providers and designers of streamed video applications.
204 citations
TL;DR: It is found that the temporal correction factor follows closely an inverted falling exponential function, whereas the quantization effect on the coded frames can be captured accurately by a sigmoid function of the peak signal-to-noise ratio.
Abstract: In this paper, we explore the impact of frame rate and quantization on perceptual quality of a video. We propose to use the product of a spatial quality factor that assesses the quality of decoded frames without considering the frame rate effect and a temporal correction factor, which reduces the quality assigned by the first factor according to the actual frame rate. We find that the temporal correction factor follows closely an inverted falling exponential function, whereas the quantization effect on the coded frames can be captured accurately by a sigmoid function of the peak signal-to-noise ratio. The proposed model is analytically simple, with each function requiring only a single content-dependent parameter. The proposed overall metric has been validated using both our subjective test scores as well as those reported by others. For all seven data sets examined, our model yields high Pearson correlation (higher than 0.9) with measured mean opinion score (MOS). We further investigate how to predict parameters of our proposed model using content features derived from the original videos. Using predicted parameters from content features, our model still fits with measured MOS with high correlation.
174 citations
"Perceptual quality assessment of hi..." refers background or methods or result in this paper
...It can be observed that there is a significant improvement in terms of MOS values from 5fps to 30fps, which is consistent with previous results [3, 15]....
[...]
...In [3], the impact of frame rate (30, 15, 7....
[...]
...Similarly, in [2, 3, 5, 4, 18], only small resolution videos (CIF or QCIF size) were employed....
[...]
TL;DR: A new content-based, non-intrusive quality of experience (QoE) prediction model for low bitrate and resolution (QCIF) H.264 encoded videos and its application in video quality adaptation over Universal Mobile Telecommunication Systems (UMTS) networks is illustrated.
Abstract: The primary aim of this paper is to present a new content-based, non-intrusive quality of experience (QoE) prediction model for low bitrate and resolution (QCIF) H.264 encoded videos and to illustrate its application in video quality adaptation over Universal Mobile Telecommunication Systems (UMTS) networks. The success of video applications over UMTS networks very much depends on meeting the QoE requirements of users. Thus, it is highly desirable to be able to predict and, if appropriate, to control video quality to meet such QoE requirements. Video quality is affected by distortions caused both by the encoder and the UMTS access network. The impact of these distortions is content dependent, but this feature is not widely used in non-intrusive video quality prediction models. In the new model, we chose four key parameters that can impact video quality and hence the QoE-content type, sender bitrate, block error rate and mean burst length. The video quality was predicted in terms of the mean opinion score (MOS). Subjective quality tests were carried out to develop and evaluate the model. The performance of the model was evaluated with unseen dataset with good prediction accuracy ( ~ 93%). The model also performed well with the LIVE database which was recently made available to the research community. We illustrate the application of the new model in a novel QoE-driven adaptation scheme at the pre-encoding stage in a UMTS network. Simulation results in NS2 demonstrate the effectiveness of the proposed adaptation scheme, especially at the UMTS access network which is a bottleneck. An advantage of the model is that it is light weight (and so it can be implemented for real-time monitoring), and it provides a measure of user-perceived quality, but without requiring time-consuming subjective tests. The model has potential applications in several other areas, including QoE control and optimization in network planning and content provisioning for network/service providers.
169 citations
"Perceptual quality assessment of hi..." refers background in this paper
...In [7, 12], temporal resolution was considered as a quality factor in specific applications of video broadcasting and distribution over the network....
[...]