Visual Equivalence: an Object-based Approach to Image Quality.

TLDR: This paper explores how object geometry, material, and illumination interact to produce images that are visually equivalent, and identifies how two kinds of transformations on illumination fields (blurring and warping) influence observers’ judgments of equivalence.

Abstract: In this paper we introduce a new approach to characterizing image quality: visual equivalence. Images are visually equivalent if they convey the same information about object appearance even if they are visibly different. In a series of psychophysical experiments we explore how object geometry, material, and illumination interact to produce images that are visually equivalent, and we identify how two kinds of transformations on illumination fields (blurring and warping) influence observers’ judgments of equivalence. We use the results of the experiments to derive metrics that can serve as visual equivalence predictors (VEPs) and we generalize these metrics so they can be applied to novel objects and scenes. Finally we validate the predictors in a confirmatory study, and show that they reliably predict observer’s judgments of equivalence. Visual equivalence is a significant new approach to measuring image quality that goes beyond existing visible difference metrics by leveraging the fact that some kinds of image differences do not matter to human observers. By taking advantage of higher order aspects of visual object coding, visual equivalence metrics should enable the development of powerful new classes of image capture, compression, rendering, and display algorithms. Introduction Measuring image differences is an important aspect of image quality evaluation, and a variety of metrics have been developed for this purpose. Numerical metrics measure physical differences between a reference image and test image and characterize quality in terms of the distance from the reference to the test. Well known numerical metrics include mean squared error (MSE) and peak signal to noise ratio (PSNR). Although these metrics are easy to compute, they often do not correlate well with observers’ judgments of image differences. For this reason, perceptual metrics have been developed that incorporate computational models of human visual processing. In these metrics visual models are used to represent an observer’s responses to the reference and test images and then these responses are compared to identify visible differences. Popular perceptual metrics include Daly’s Visible Differences Predictor (VDP) and the Lubin/Sarnoff model. These metrics typically do a better job at predicting the visual impact of common imaging artifacts such as noise and quantization on perceived image quality, and many researchers have successfully applied these perceptual metrics to important problems in digital imaging. However current metrics have an interesting limitation that is illustrated in Figure 1. Figure 1a and 1b show two computer-generated images of a tabletop scene. Figure 1a was rendered using path tracing, a physically accurate but computationally intensive algorithm that can produce faithful simulations of environmental light reflection and transport. It can take hours to render a single image using path tracing. In contrast, Figure 1b was rendered using environment mapping, a fast but approximate rendering technique that uses an image of the surround rather than the model of the surround to illuminate the objects on the tabletop. Environment mapping is a standard feature of commodity graphics hardware and can render images like the one shown in Figure 1b at interactive rates. One consequence the approximations used in environment mapping is that illumination features such as surface reflections are warped with respect to the geometrically correct features produced by path tracing. This can be seen by comparing the images reflected by the two teapots. If we take the path traced image as the reference, and the environment mapped image as the test, and process the images with one of the standard perceptual metrics (in this case an implementation of Daly’s VDP), the metric produces the difference map shown in Figure 1c which correctly indicates that the images are visibly different (green and red pixels 75% and 95% probability of detection respectively). However an important question is: are these meaningful image differences? When we look at images we don’t see pixels. Rather, we see objects with recognizable shapes, sizes, and materials, at specific spatial locations, lit by distinct patterns of illumination. From this perspective the two images shown in Figure 1 are much more similar than they are different. For example, the shapes, sizes, and locations of the objects shown in the images appear the same; the objects appear to have the same material properties; and the lighting in the scenes seems the same. Although the images are visibly different they are visually equivalent as representations of object appearance. The existence of images like these coupled with the growing range of image transformations used in computer graphics, digital imaging, and computational photography points to the need for a new kind of image difference/quality metric that can predict when different classes of imaging algorithms produce images that are visually equivalent. Figure 2: Factors that affect object appearance. Dynamics and viewpoint can also play significant roles. Understanding Object Appearance The concept of visual equivalence is based on the premise that two visibly different images can convey the same information about object appearance to a human observer. To develop metrics for predicting when images will be visually equivalent we need to understand the factors that influence object appearance. Figure 2 shows a computer-generated image of a chrome bunny. We perceive the properties of this and other objects based on the patterns of light they reflect to our eyes. For a fixed object and viewpoint these patterns are determined by the geometry of the object, its material properties, and the illumination it receives. The perception of each of these properties is the subject of an extensive literature that will only be briefly reviewed here. More comprehensive introductions on these subtopics are available in the papers cited. Shape perception: The central problem in shape perception is how the visual system recovers 3D object shape from the 2D retinal images. Many sources of information for shape have been identified including stereopsis, surface shading, shadows, texture, perspective, motion, and occlusion Recent work has tried to characterize the effectiveness of these different sources and to model how they combine to provide reliable shape percepts. Material perception: Although there is significant interest in industry on the topic of material perception, there has been relatively little basic research on the topic. This situation is changing with the development of advanced computer graphics techniques that allow the accurate simulation and systematic manipulation of realistic materials in complex scenes. Active research topics include the perception of 3D surface lightness and color, gloss perception, perception of translucency , and 3D texture appearance. Illumination perception: Historically, illumination has been regarded as a factor that needs to be discounted to achieve shape and lightness/color constancy, but recently there has been interest in understanding illumination perception itself. Recent studies include the characterization of natural illumination statistics and surface illuminance flow, the perception of illumination directionality and complexity, and tolerance for illumination inconsistencies. So an object’s appearance is based on the images it reflects to our eyes, and these images are determined by the object’s geometry, material, and illumination properties. How the visual system disentangles the image information to perceive these object properties is one of the great unsolved problems in vision research. Although eventually we would like to understand this process, the goal of this paper is more immediate: to develop metrics that can predict when visibly different images are equivalent as representations of object appearance. To achieve this goal conducted a series of experiments that investigated when different configurations of object geometry, material, and illumination produce visually equivalent images. Experiments Even for a single object, the space of images spanned by all possible variations in object geometry, material properties, and scene illumination is vast. To begin to quantify the phenomenon of visual equivalence we had to constrain the scope of our studies. Starting from the proof-of-concept demonstration shown in Figure 1, we decided to study visual equivalence across two kinds of illumination transformations (blurring and warping) for objects with different geometric and material properties. The following sections describe our methods and procedures. Stimuli To conduct the experiments we created a set of images that Figure 3. The geometries and materials of the objects used in the experiments. Parameters were chosen to be perceptually uniform in both surface “bumpiness” and surface reflectance. would allow us to systematically explore the interactions between object geometry, material, and illumination. To accomplish this we used computer-generated images. Figure 3 shows representative images from our stimulus set. The scene consisted of a bumpy ball-like test object on a brick patio flanked by two pairs of children's blocks. The following paragraphs describe the parameters we used to generate the images. Geometry: The four object geometries (G0-G3) shown in the rows of Figure 3 were defined as follows. Object G0 was a geodesically tesselated sphere with 164 vertices. Objects G1 though G3 were generated by placing the sphere in a cube of Perlin noise and displacing the vertices according to the 3d noise function. By varying the size of the cube relative to the sphere (scale factors of 2,1, 1/2,1/2, 1/8 respectively) it was possible to produce random surface displacements that were constant in amplitude but varied in spatial frequency bandwidth. In pre-testing the objects were informally judged to be