Christian Keimel

TL;DR: The concepts and ideas cited in this paper mainly refer to the Quality of Experience of multimedia communication systems, but may be helpful also for other areas where QoE is an issue, and the document will not reflect the opinion of each individual person at all points.

TL;DR: The focus of this article is on the issue of reliability and the use of video quality assessment as an example for the proposed best practices, showing that the recommended two-stage QoE crowdtesting design leads to more reliable results.

TL;DR: This contribution proposes the QualityCrowd framework, which allows codec independent quality assessment with a simple web interface, usable with common web browsers, and compared the results from an online subjective test using this framework with theresults from a test in a standardized environment, showing that qualityCrowd delivers equivalent results within the acceptable inter-lab correlation.

TL;DR: High correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal is found, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.

TL;DR: This white paper summarizes the recommendations and best practices for crowdsourced quality assessment of multimedia applications from the Qualinet Task Force on “Crowdsourcing” and resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data.