Humans perceive the three-dimensional structure of the world with apparent ease. However, despite all of the recent advances in computer vision research, the dream of having a computer interpret an image at the same level as a two-year old remains elusive. Why is computer vision such a challenging problem and what is the current state of the art? Computer Vision: Algorithms and Applications explores the variety of techniques commonly used to analyze and interpret images. It also describes challenging real-world applications where vision is being successfully used, both for specialized applications such as medical imaging, and for fun, consumer-level tasks such as image editing and stitching, which students can apply to their own personal photos and videos. More than just a source of recipes, this exceptionally authoritative and comprehensive textbook/reference also takes a scientific approach to basic vision problems, formulating physical models of the imaging process before inverting them to produce descriptions of a scene. These problems are also analyzed using statistical models and solved using rigorous engineering techniques Topics and features: structured to support active curricula and project-oriented courses, with tips in the Introduction for using the book in a variety of customized courses; presents exercises at the end of each chapter with a heavy emphasis on testing algorithms and containing numerous suggestions for small mid-term projects; provides additional material and more detailed mathematical topics in the Appendices, which cover linear algebra, numerical techniques, and Bayesian estimation theory; suggests additional reading at the end of each chapter, including the latest research in each sub-field, in addition to a full Bibliography at the end of the book; supplies supplementary course material for students at the associated website, http://szeliski.org/Book/. Suitable for an upper-level undergraduate or graduate-level course in computer science or engineering, this textbook focuses on basic techniques that work under real-world conditions and encourages students to push their creative boundaries. Its design and exposition also make it eminently suitable as a unique reference to the fundamental techniques and current research literature in computer vision.

/pdf/computer-vision-algorithms-and-applications-25dn6wu83j.pdf

Computer Vision: Algorithms and Applications

This paper presents a new reflectance model for rendering computer synthesized images. The model accounts for the relative brightness of different materials and light sources in the same scene. It describes the directional distribution of the reflected light and a color shift that occurs as the reflectance changes with incidence angle. The paper presents a method for obtaining the spectral energy distribution of the light reflected from an object made of a specific real material and discusses a procedure for accurately reproducing the color associated with the spectral energy distribution. The model is applied to the simulation of a metal and a plastic.

A reflectance model for computer graphics

We present a method to acquire the reflectance field of a human face and use these measurements to render the face under arbitrary changes in lighting and viewpoint. We first acquire images of the face from a small set of viewpoints under a dense sampling of incident illumination directions using a light stage. We then construct a reflectance function image for each observed image pixel from its values over the space of illumination directions. From the reflectance functions, we can directly generate images of the face from the original viewpoints in any form of sampled or computed illumination. To change the viewpoint, we use a model of skin reflectance to estimate the appearance of the reflectance functions for novel viewpoints. We demonstrate the technique with synthetic renderings of a person's face under novel illumination and viewpoints.

/pdf/acquiring-the-reflectance-field-of-a-human-face-1q7nifn0px.pdf

Acquiring the reflectance field of a human face

We present a generative model for isotropic bidirectional reflectance distribution functions (BRDFs) based on acquired reflectance data. Instead of using analytical reflectance models, we represent each BRDF as a dense set of measurements. This allows us to interpolate and extrapolate in the space of acquired BRDFs to create new BRDFs. We treat each acquired BRDF as a single high-dimensional vector taken from a space of all possible BRDFs. We apply both linear (subspace) and non-linear (manifold) dimensionality reduction tools in an effort to discover a lower-dimensional representation that characterizes our measurements. We let users define perceptually meaningful parametrization directions to navigate in the reduced-dimension BRDF space. On the low-dimensional manifold, movement along these directions produces novel but valid BRDFs.

/pdf/a-data-driven-reflectance-model-4at07m3un0.pdf

A data-driven reflectance model

Light transport algorithms generate realistic images by simulating the emission and scattering of light in an artificial environment. Applications include lighting design, architecture, and computer animation, while related engineering disciplines include neutron transport and radiative heat transfer. The main challenge with these algorithms is the high complexity of the geometric, scattering, and illumination models that are typically used. In this dissertation, we develop new Monte Carlo techniques that greatly extend the range of input models for which light transport simulations are practical. Our contributions include new theoretical models, statistical methods, and rendering algorithms. We start by developing a rigorous theoretical basis for bidirectional light transport algorithms (those that combine direct and adjoint techniques). First, we propose a linear operator formulation that does not depend on any assumptions about the physical validity of the input scene. We show how to obtain mathematically correct results using a variety of bidirectional techniques. Next we derive a different formulation, such that for any physically valid input scene, the transport operators are symmetric. This symmetry is important for both theory and implementations, and is based on a new reciprocity condition that we derive for transmissive materials. Finally, we show how light transport can be formulated as an integral over a space of paths. This framework allows new sampling and integration techniques to be applied, such as the Metropolis sampling algorithm. We also use this model to investigate the limitations of unbiased Monte Carlo methods, and to show that certain kinds of paths cannot be sampled. Our statistical contributions include a new technique called multiple importance sampling, which can greatly increase the robustness of Monte Carlo integration. It uses more than one sampling technique to evaluate an integral, and then combines these samples in a

/pdf/robust-monte-carlo-methods-for-light-transport-simulation-23mfnk1eki.pdf

Robust Monte Carlo methods for light transport simulation

We introduce a new class of primitive functions with non-linear parameters for representing light reflectance functions. The functions are reciprocal, energy-conserving and expressive. They can capture important phenomena such as off-specular reflection, increasing reflectance and retro-reflection. We demonstrate this by fitting sums of primitive functions to a physically-based model and to actual measurements. The resulting representation is simple, compact and uniform. It can be applied efficiently in analytical and Monte Carlo computations. CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism; I.3.3 [Computer Graphics]: Picture/Image Generation

Non-linear approximation of reflectance functions

In this paper we present a new Monte Carlo rendering algorithm that seamlessly integrates the ideas of shooting and gathering power to create photorealistic images The algorithm can be explained as a generalisation of the well-known path tracing algorithm Test results show that it performs signicantly better for typical indoor scenes where indirect lighting is important

Bi-directional path tracing

We present a new image-based process for measuring the bidirectional reflectance of homogeneous surfaces rapidly, completely, and accurately. For simple sample shapes (spheres and cylinders) the method requires only a digital camera and a stable light source. Adding a 3D scanner allows a wide class of curved near-convex objects to be measured. With measurements for a variety of materials from paints to human skin, we demonstrate the new method's ability to achieve high resolution and accuracy over a large domain of illumination and reflection directions. We verify our measurements by tests of internal consistency and by comparison against measurements made using a gonioreflectometer.

/pdf/image-based-brdf-measurement-including-human-skin-3l5eittk5y.pdf

Image-based BRDF measurement including human skin

Our goal is to develop physically based lighting models and perceptually based rendering procedures for computer graphics that will produce synthetic images that are visually and measurably indistinguishable from real-world images. Fidelity of the physical simulation is of primary concern. Our research framework is subdivided
into three sub-sections: the local light reflection model, the energy transport simulation, and the visual display algorithms. The first two subsections are physically based, and the last is perceptually based.

We emphasize the comparisons between simulations and actual
measurements, the difficulties encountered, and the need to utilize the vast amount of psychophysical research already conducted. Future research directions are enumerated. We hope that results of this research will help establish a more fundamental, scientific approach
for future rendering algorithms. This presentation describes a chronology of past research in global illumination and how parts of our
new system are currently being developed.

/pdf/a-framework-for-realistic-image-synthesis-3jl4uhnebl.pdf

A framework for realistic image synthesis

Algorithms for image synthesis render photo-realistic imagery, given the description of a scene. Physically based rendering specifically stresses the physical correctness of the algorithms and their results. The algorithms perform an accurate simulation of the behaviour of light, in order to faithfully render global illumination effects such as soft shadows, glossy reflections and indirect illumination. In this dissertation we investigate image-based Monte Carlo rendering algorithms. We pay special attention to their correctness, their versatility and their efficiency. First of all, we discuss theoretical frameworks that describe the global illumination problem. These formal mathematical models are the first step to ensure the correctness of the eventual results. Moreover, they allow to apply standard numerical techniques to compute a solution. We give an overview of existing models, which are based on the rendering equation and the potential equation. We then introduce a model based on a new concept, called the global reflectance distribution function. It combines the ideas of radiance and potential into a single function, which is defined by a set of two integral equations. We later show that, while the three models are equivalent, a straightforward Monte Carlo approach to solve them leads to entirely different rendering algorithms. We chose to apply Monte Carlo methods because of their versatility. First, we give an overview of Monte Carlo techniques in general. The variance of a technique provides a measure for the stochastic errors on its results. The basic strategy of Monte Carlo methods is to reduce the variance by averaging the results of large numbers of samples. Convergence is slow, however. Variance reduction techniques therefore try to improve the efficacy of the individual samples, by taking into account information about the integrand. We discuss techniques such as stratified sampling, importance sampling, the combining of estimators, control variates, Russian roulette and next event estimation. We stress their unbiasedness, which ensures that the estimators always converge to the exact solution. We specifically analyse aspects that are of importance for the global illumination problem and we present a few improvements to existing techniques. Finally, we apply the Monte Carlo methods to the mathematical models of the global illumination problem. The different models give rise to entirely different algorithms. The rendering equation leads to the well-known path tracing algorithm, while the potential equation leads to the light tracing algorithm. We discuss their strengths and weaknesses. We study the application of the variance reduction techniques and present results from practical experiments. The global reflectance distribution function and its equations give rise to a new algorithm, which we have called bidirectional path tracing. This algorithm proves to have superior qualities for rendering typical indirectly illuminated interior scenes.

Eric P. Lafortune

Papers

Non-linear approximation of reflectance functions

Bi-directional path tracing

Image-based BRDF measurement including human skin

A framework for realistic image synthesis

Mathematical Models and Monte Carlo Algorithms for Physically Based Rendering