THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

/pdf/the-unreasonable-effectiveness-of-deep-features-as-a-10k66u5qas.pdf

This paper describes a recently created image database, TID2013, intended for evaluation of full-reference visual quality assessment metrics. With respect to TID2008, the new database contains a larger number (3000) of test images obtained from 25 reference images, 24 types of distortions for each reference image, and 5 levels for each type of distortion. Motivations for introducing 7 new types of distortions and one additional level of distortions are given; examples of distorted images are presented. Mean opinion scores (MOS) for the new database have been collected by performing 985 subjective experiments with volunteers (observers) from five countries (Finland, France, Italy, Ukraine, and USA). The availability of MOS allows the use of the designed database as a fundamental tool for assessing the effectiveness of visual quality. Furthermore, existing visual quality metrics have been tested with the proposed database and the collected results have been analyzed using rank order correlation coefficients between MOS and considered metrics. These correlation indices have been obtained both considering the full set of distorted images and specific image subsets, for highlighting advantages and drawbacks of existing, state of the art, quality metrics. Approaches to thorough performance analysis for a given metric are presented to detect practical situations or distortion types for which this metric is not adequate enough to human perception. The created image database and the collected MOS values are freely available for downloading and utilization for scientific purposes. We have created a new large database.This database contains larger number of distorted images and distortion types.MOS values for all images are obtained and provided.Analysis of correlation between MOS and a wide set of existing metrics is carried out.Methodology for determining drawbacks of existing visual quality metrics is described.

https://hal.archives-ouvertes.fr/hal-01109219/document

Image database TID2013

Visual quality of color images is an important aspect in various applications of digital image processing and multimedia. A large number of visual quality metrics (indices) has been proposed recently. In order to assess their reliability, several databases of color images with various sets of distortions have been exploited. Here we present a new database called TID2013 that contains a larger number of images. Compared to its predecessor TID2008, seven new types and one more level of distortions are included. The need for considering these new types of distortions is briefly described. Besides, preliminary results of experiments with a large number of volunteers for determining the mean opinion score (MOS) are presented. Spearman and Kendall rank order correlation factors between MOS and a set of popular metrics are calculated and presented. Their analysis shows that adequateness of the existing metrics is worth improving. Special attention is to be paid to accounting for color information and observers focus of attention to locally active areas in images.

http://www.comlab.uniroma3.it/BattistiPapers/EUVIP2013_Battisti.pdf

Color image database TID2013: Peculiarities and preliminary results

Measuring perceptual contrast in digital images

The evaluation of perceived image quality in color prints is a complex task due to its subjectivity and dimensionality. The perceived quality of an image is influenced by a number of different quality attributes. It is difficult and complicated to evaluate the influ- ence of all attributes on overall image quality, and their influence on other attributes. Because of this difficulty, the most important at- tributes of a color image should be identified to achieve a more efficient and manageable evaluation of the image's quality. Based on a survey of the existing literature and a psychophysical experi- ment, we identify and categorize existing image quality attributes to propose a refined selection of meaningful ones for the evaluation of color prints. © 2010 SPIE and IS&T. DOI: 10.1117/1.3277145

Attributes of image quality for color prints

A large number of Image Quality (IQ) metrics have been developed over the last decades and the number continues to grow. For development and evaluation of such metrics, IQ databases with reference images, distortions, and perceptual quality data, is very useful. However, existing IQ databases have some drawbacks, making them incapable of evaluating properly all aspects of IQ metrics. The lack of reference image design principles; limited distortion aspects; and uncontrolled viewing conditions. Furthermore, same sets of images are always used for evaluating IQ metrics, so more images are needed. These are some of the reasons why a newly developed IQ database is desired. In this study we propose a new IQ database, Colourlab Image Database: Image Quality (CID:IQ), for which we have proposed methods to design reference images, and different types of distortions have been applied. Another new feature with our database is that we have conducted the perceptual experiments at two viewing distances. The CID:IQ database is available at http://www.colourlab.no/cid .

/pdf/cid-iq-a-new-image-quality-database-34wgcbvys9.pdf

CID:IQ – A New Image Quality Database

This paper aims to give a survey of full-reference image quality metrics, including metrics specifically designed to evaluate image quality, but also metrics for image difference, image fidelity, and more. These metrics have in common that they try to predict the perceived difference between an original image and a modified version of it, this modification can typically be compression, halftoning and blurring. They output one numerical value and/or an image difference map. More than 100 image quality metrics have been reviewed and categorized, and short descriptions and analysis of all metrics and their relationships are given. This should prove valuable to researchers in various fields, to find the most appropriate metric for their application, and to give a better understanding of the state of the art of the field of image quality metrics.

https://brage.bibsys.no/xmlui/bitstream/11250/144194/1/rapport052009_elektroniskversjon.pdf

Survey of full-reference image quality metrics

/pdf/image-quality-assessment-by-comparing-cnn-features-between-jlpg38hy7k.pdf

Marius Pedersen

Papers

Measuring perceptual contrast in digital images

Attributes of image quality for color prints

CID:IQ – A New Image Quality Database

Survey of full-reference image quality metrics

Image Quality Assessment by Comparing CNN Features between Images