HTTP adaptive streaming is being increasingly deployed by network content providers, such as Netflix and YouTube. By dividing video content into data chunks encoded at different bitrates, a client is able to request the appropriate bitrate for the segment to be played next based on the estimated network conditions. However, this can introduce a number of impairments, including compression artifacts and rebuffering events, which can severely impact an end-user’s quality of experience (QoE). We have recently created a new video quality database, which simulates a typical video streaming application, using long video sequences and interesting Netflix content. Going beyond previous efforts, the new database contains highly diverse and contemporary content, and it includes the subjective opinions of a sizable number of human subjects regarding the effects on QoE of both rebuffering and compression distortions. We observed that rebuffering is always obvious and unpleasant to subjects, while bitrate changes may be less obvious due to content-related dependencies. Transient bitrate drops were preferable over rebuffering only on low complexity video content, while consistently low bitrates were poorly tolerated. We evaluated different objective video quality assessment algorithms on our database and found that objective video quality models are unreliable for QoE prediction on videos suffering from both rebuffering events and bitrate changes. This implies the need for more general QoE models that take into account objective quality models, rebuffering-aware information, and memory. The publicly available video content as well as metadata for all of the videos in the new database can be found at http://live.ece.utexas.edu/research/LIVE_NFLXStudy/nflx_index.html .

Study of Temporal Effects on Subjective Video Quality of Experience

With the rapid growth of streaming media applications, there has been a strong demand of quality-of-experience (QoE) measurement and QoE-driven video delivery technologies. Most existing methods rely on bitrate and global statistics of stalling events for QoE prediction. This is problematic for two reasons. First, using the same bitrate to encode different video content results in drastically different presentation quality. Second, the interactions between video presentation quality and playback stalling experiences are not accounted for. In this work, we first build a streaming video database and carry out a subjective user study to investigate the human responses to the combined effect of video compression, initial buffering, and stalling. We then propose a novel QoE prediction approach named Streaming QoE Index that accounts for the instantaneous quality degradation due to perceptual video presentation impairment, the playback stalling events, and the instantaneous interactions between them. Experimental results show that the proposed model is in close agreement with subjective opinions and significantly outperforms existing QoE models. The proposed model provides a highly effective and efficient meanings for QoE prediction in video streaming services. 1

A Quality-of-Experience Index for Streaming Video

With the recent increased usage of video services, the focus has recently shifted from the traditional quality of service-based video delivery to quality of experience (QoE)-based video delivery. Over the past 15 years, many video quality assessment metrics have been proposed with the goal to predict the video quality as perceived by the end user. HTTP adaptive streaming (HAS) has recently gained much attention and is currently used by the majority of video streaming services, such as Netflix and YouTube. HAS, using reliable transport protocols, such as TCP, does not suffer from image artifacts due to packet losses, which are common in traditional streaming technologies. Hence, the QoE models developed for other streaming technologies alone are not sufficient. Recently, many works have focused on developing QoE models targeting HAS-based applications. Also, the recently published ITU-T Recommendation series P.1203 proposes a parametric bitstream-based model for the quality assessment of progressive download and adaptive audiovisual streaming services over a reliable transport. The main contribution of this paper is to present a comprehensive overview of recent and currently undergoing works in the field of QoE modeling for HAS. The HAS QoE models, influence factors, and subjective test methodologies are discussed, as well as existing challenges and shortcomings. The survey can serve as a guideline for researchers interested in QoE modeling for HAS and also discusses possible future work.

/pdf/qoe-modeling-for-http-adaptive-video-streaming-a-survey-and-4ugy26rj1k.pdf

QoE Modeling for HTTP Adaptive Video Streaming–A Survey and Open Challenges

The dynamic adaptive streaming over HTTP provides an inter-operable solution to overcome the volatile network conditions, but its complex characteristic brings new challenges to objective video quality-of-experience (QoE) measurement. To test the generalizability and to facilitate the wide usage of QoE measurement techniques in real-world applications, we establish a new database named Waterloo Streaming QoE Database III (SQoE-III). Unlike existing databases constructed with hand-crafted test sequences, the SQoE-III database, so far the largest and most realistic of its kind, consists of a total of 450 streaming videos created from diverse source content and diverse distortion patterns, with six adaptation algorithms of diverse characteristics under 13 representative network conditions. All streaming videos are assessed by 34 subjects, and a comprehensive evaluation is conducted on the performance of 15 objective QoE models from four categories with regards to their efficacy in predicting subjective QoE. Detailed correlation analysis and statistical hypothesis testing are carried out. The results of this paper shed light on the future development of adaptive bitrate streaming algorithm and video QoE monitoring system. The subjective database is available online at https://ece.uwaterloo.ca/~zduanmu/tbc2018qoe/ .

A Quality-of-Experience Database for Adaptive Video Streaming

Over-the-top mobile adaptive video streaming is invariably influenced by volatile network conditions, which can cause playback interruptions (stalling or rebuffering events) and bitrate fluctuations, thereby impairing users’ quality of experience (QoE). Video quality assessment models that can accurately predict users’ QoE under such volatile network conditions are rapidly gaining attention, since these methods could enable more efficient design of quality control protocols for media-driven services such as YouTube, Amazon, Netflix, and many others. However, the development of improved QoE prediction models requires data sets of videos afflicted with diverse stalling events that have been labeled with ground-truth subjective opinion scores. Toward this end, we have created a new mobile video quality database that we call LIVE Mobile Stall Video Database-II. Our database contains a total of 174 videos afflicted with distortions caused by 26 different stalling patterns. We describe the way we simulated the diverse stalling events to create a corpus of distorted videos, and we detail the human study we conducted to obtain continuous-time subjective scores from 54 subjects. We also present the outcomes of our comprehensive analysis of the impact of several factors that influence subjective QoE, and report the performance of existing QoE-prediction models on our data set. We are making the database (videos, subjective data, and video metadata) publicly available in order to help the advance state-of-the-art research on user-centric mobile network planning and management. The database may be accessed at http://live.ece.utexas.edu/research/LIVEStallStudy/liveMobile.html .

A Subjective and Objective Study of Stalling Events in Mobile Streaming Videos

There is a wide range of video services over complex transmission networks, and in some cases end users fail to receive an acceptable quality level. In this paper, the different factors that degrade users' quality of experience (QoE) in video streaming service that use TCP as transmission protocol are studied. In this specific service, impairment factors are: number of pauses, their duration and temporal location. In order to measure the effect that each temporal segment has in the overall video quality, subjective tests. Because current subjective test methodologies are not adequate to assess video streaming over TCP, some recommendations are provided here. At the application layer, a customized player is used to evaluate the behavior of player buffer, and consequently, the end user QoE. Video subjective test results demonstrate that there is a close correlation between application parameters and subjective scores. Based on this fact, a new metrics named VsQM is defined, which considers the importance of temporal location of pauses to assess the user QoE of video streaming service. A useful application scenario is also presented, in which the metrics proposed herein is used to improve video services.

Quality metric to assess video streaming service over TCP considering temporal location of pauses

There is a wide range of telecommunications services to transmit voice, video and data through complex networks of transmission and in some cases the service has not an acceptable quality by the end user. This work presents in a practical way as a video service in an IP network can be affected by adverse conditions on the network, mainly in cases of packet loss, point-topoint delay and delay variation. For the development of the tests was mounted on a scenario of emulation of isolated network, in which there are parameterized different conditions of network, causing different qualities of video observed by the user that uses a service of streaming video. Keywords—Video Quality, Objective Video Metrics, FR, PSNR, SSIM, VQM. I. INTRODUCAO A avaliacao da qualidade de video e realizada por metricas objetivas e subjetivas. As metricas subjetivas sao baseadas na percepcao humana. As metricas objetivas sao baseadas em modelos matematicos para estimar a media das opinioes dos usuarios. As metricas objetivas podem ser classificadas de acordo com a disponibilidade do video original, que e considerado livre de distorcao ou de alta qualidade: FR (Full Reference) Referencia Completa [2], RR (Reduced Reference) Referencia Reduzida [2]e os NR (No-Reference) Sem Referencia [2]. O modelo de avaliacao de video no presente artigo e do tipo FR [3], porque utilizamos duas entradas (video original e video degradado) para obter a qualidade de video observada pelo usuario (observador). Para comparar o video original e o video degradado utilizamos as metricas objetivas: PSNR (Peak Signal to Noise Ratio) Relacao Sinal Ruido de Pico [4] [5] [6], SSIM (Structural Similarity Index) Indice de Similaridade Estrutural [4] [5] [6] e VQM (Video Quality Measure) Medicao de Qualidade de Video [4] [6]. A. Metrica Objetiva PSNR A metrica PSNR, chamada de Relacao Sinal Ruido de Pico define a relacao entre a maxima energia possivel de um sinal e o ruido que afeta a representacao do sinal, e representada pela seguinte equacao:

/pdf/avaliacao-de-qualidade-de-video-sobre-uma-rede-ip-usando-4gpq6q2s8c.pdf

Avaliação de qualidade de vídeo sobre uma rede IP usando métricas objetivas

There are several subjective test methodologies for assessing video quality, being that each of these methods was developed by considering specific degradation factors in video quality. Therefore, for video streaming services that use TCP as transport protocol different evaluation criteria should be considered as compared to current methods, because the impairment are only pauses. In this paper, the concepts of the mechanisms that integrate perceptual and cognitive processes are studied, relating them to a test environment for subjective evaluation of video, which is limited to streaming video service transmitted via HTTP / TCP. For this reason, some recommendations to perform subjective assessment of streaming video service are proposed, taking into account perceptual and cognitive criteria. In this work, subjective tests are conducted, evaluators with different content preferences assess 3 different video content. Results show that the assessors' QoE depends strongly on video content preferences. Thus, assessors who are interested in the video content are more drastic in the ratings. In the case of sport videos, the Mean Opinion Score (MOS) value of assesors with explicit preference in sports reachs only 54% of the MOS obtained by assessors without interest in sports. Furthermore, the roll of the instructor during the test is also evaluated.

Video quality subjective assessment considering cognitive criteria and user preferences on video content

There are many factors that can affect the users' quality of experience (QoE) in Voice over IP (VoIP) services, especially in wireless networks. In speech communication systems there are different impairment factors because physical phenomena that occur in the wired and wireless transmission channel. In this context, a non-intrusive speech quality classifier based on Discriminative Restricted Boltzmann Machines (DRBM) is proposed, which considers speech signals degradations caused by wired and wireless network degradations. To accomplish this goal, a test scenario is implemented, in which the speech signal is coded by the AMR-WB and transmitted using both a lossy wired channel and the Rayleigh fading model. Also, different modulation schemes and channel degradations, such as packet loss rate, signal-to-noise ratio and Doppler shifts are implemented. As a result, a speech database is built that is used to train different machine learning algorithms. Experimental results demonstrated that the DRBM reached the best results. Performance assessment results show that the proposed classifier based on DRBM overcomes the current standardized algorithm described in ITU-T Rec. P.563.

A Speech Quality Classifier based on Signal Information that Considers Wired and Wireless Degradations

Modern communication systems, such as 5G networks promise to meet the crescent demand of users for high data rates in order to support new technologies. However, the frequency range of operation of these networks deserves special attention, since the high frequencies are more susceptible to interferences caused by atmospheric phenomena. The recommendations ITU-R P.838-3 and ITU-R P.676-11 describe methodologies to estimate signal degradations caused by phenomena related to rain and atmospheric gases, respectively. In this work, the impact of such phenomena on speech signal quality is investigated. The perceptual speech quality is evaluated using the algorithm described in ITU-T Rec. P.862. The experimental results show that the higher the frequency of operation, the higher the level of signal degradation for the same atmospheric conditions. Based on these results a parametric speech quality assessment model is proposed that uses an artificial neural network. Performance evaluation results demonstrated a high correlation between the proposed model and subjective test results, reaching an PCC and an RMSE of 0.9869 and 0.4773, respectively. Hence, the proposed model intends to be useful for network planning tasks.

Dante Coaquira Begazo

Papers

Quality metric to assess video streaming service over TCP considering temporal location of pauses

Avaliação de qualidade de vídeo sobre uma rede IP usando métricas objetivas

Video quality subjective assessment considering cognitive criteria and user preferences on video content

A Speech Quality Classifier based on Signal Information that Considers Wired and Wireless Degradations

Evaluation of Speech Quality Degradation due to Atmospheric Phenomena