Showing papers by "Paul Debevec published in 2009"

Dynamic shape capture using multi-view photometric stereo

Abstract: We describe a system for high-resolution capture of moving 3D geometry, beginning with dynamic normal maps from multiple views. The normal maps are captured using active shape-from-shading (photometric stereo), with a large lighting dome providing a series of novel spherical lighting configurations. To compensate for low-frequency deformation, we perform multi-view matching and thin-plate spline deformation on the initial surfaces obtained by integrating the normal maps. Next, the corrected meshes are merged into a single mesh using a volumetric method. The final output is a set of meshes, which were impossible to produce with previous methods. The meshes exhibit details on the order of a few millimeters, and represent the performance over human-size working volumes at a temporal resolution of 60Hz.

Compressive light transport sensing

Abstract: In this article we propose a new framework for capturing light transport data of a real scene, based on the recently developed theory of compressive sensing. Compressive sensing offers a solid mathematical framework to infer a sparse signal from a limited number of nonadaptive measurements. Besides introducing compressive sensing for fast acquisition of light transport to computer graphics, we develop several innovations that address specific challenges for image-based relighting, and which may have broader implications. We develop a novel hierarchical decoding algorithm that improves reconstruction quality by exploiting interpixel coherency relations. Additionally, we design new nonadaptive illumination patterns that minimize measurement noise and further improve reconstruction quality. We illustrate our framework by capturing detailed high-resolution reflectance fields for image-based relighting.

Achieving eye contact in a one-to-many 3D video teleconferencing system

Abstract: We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant's face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180° field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast vertex shader based on a 6D lookup table for projecting 3D scene vertices to a range of subject angles, heights, and distances. We generalize the projection mathematics to arbitrarily shaped display surfaces, which allows us to employ a curved concave display surface to focus the high speed imagery to individual observers. To achieve two-way eye contact, we capture 2D video from a cross-polarized camera reflected to the position of the virtual participant's eyes, and display this 2D video feed on a large screen in front of the real participant, replicating the viewpoint of their virtual self. To achieve correct vertical perspective, we further leverage this image to track the position of each audience member's eyes, allowing the 3D display to render correct vertical perspective for each of the viewers around the device. The result is a one-to-many 3D teleconferencing system able to reproduce the effects of gaze, attention, and eye contact generally missing in traditional teleconferencing systems.

The Digital Emily project: photoreal facial modeling and animation

Abstract: This course describes how high-resolution face scanning, advanced character rigging, and performance-driven facial animation were combined to create Digital Emily, a believably photorealistic digital actor. Actress Emily O'Brien was scanned in the USC ICT light stage in 35 different facial poses using a new high-resolution face-scanning process capable of capturing geometry and textures down to the level of skin pores and fine wrinkles. These scans were assembled into a rigged digital character, which could then be driven by Image Metrics video-based facial animation technology. The real Emily was captured speaking on a small set, and her movements were used to drive a complete digital face replacement of her character, including its diffuse, specular, and animated displacement maps. HDRI lighting reconstruction techniques were used to reproduce the lighting on her original performance. The most recent results show new real-time animation and rendering research for the Digital Emily character.

Estimating specular roughness and anisotropy from second order spherical gradient illumination

Abstract: This paper presents a novel method for estimating specular roughness and tangent vectors, per surface point, from polarized second order spherical gradient illumination patterns. We demonstrate that for isotropic BRDFs, only three second order spherical gradients are sufficient to robustly estimate spatially varying specular roughness. For anisotropic BRDFs, an additional two measurements yield specular roughness and tangent vectors per surface point. We verify our approach with different illumination configurations which project both discrete and continuous fields of gradient illumination. Our technique provides a direct estimate of the per-pixel specular roughness and thus does not require off-line numerical optimization that is typical for the measure-and-fit approach to classical BRDF modeling.

Creating a Photoreal Digital Actor: The Digital Emily Project

Abstract: The Digital Emily Project is a collaboration between facial animation company Image Metrics and the Graphics Laboratory at the University of Southern California’s Institute for Creative Technologies to achieve one of the world’s first photorealistic digital facial performances. The project leverages latest-generation techniques in high-resolution face scanning, character rigging, video-based facial animation, and compositing. An actress was first filmed on a studio set speaking emotive lines of dialog in high definition. The lighting on the set was captured as a high dynamic range light probe image. The actress’ face was then three-dimensionally scanned in thirty-three facial expressions showing different emotions and mouth and eye movements using a high-resolution facial scanning process accurate to the level of skin pores and fine wrinkles. Lighting-independent diffuse and specular reflectance maps were also acquired as part of the scanning process. Correspondences between the 3D expression scans were formed using a semi-automatic process, allowing a blendshape facial animation rig to be constructed whose expressions closely mirrored the shapes observed in the rich set of facial scans; animated eyes and teeth were also added to the model. Skin texture detail showing dynamic wrinkling was converted into multiresolution displacement maps also driven by the blend shapes. A semi-automatic video-based facial animation system was then used to animate the 3D face rig to match the performance seen in the original video, and this performance was tracked onto the facial motion in the studio video. The final face was illuminated by the captured studio illumination and shading using the acquired reflectance maps with a skin translucency shading algorithm. Using this process, the project was able to render a synthetic facial performance which was generally accepted as being a real face.

Image-based separation of diffuse and specular reflections using environmental structured illumination

Abstract: We present an image-based method for separating diffuse and specular reflections using environmental structured illumination. Two types of structured illumination are discussed: phase-shifted sine wave patterns, and phase-shifted binary stripe patterns. In both cases the low-pass filtering nature of diffuse reflections is utilized to separate the reflection components. We illustrate our method on a wide range of example scenes and applications.

Estimating pose of photographic images in 3d earth model using human assistance

Abstract: The pose of a photographic image of a portion of Earth may be estimated using human assistance. A 3D graphics engine may render a virtual image of Earth from a controllable viewpoint based on 3D data that is representative of a 3D model of at least a portion of Earth. A user may locate and display a corresponding virtual image of Earth at a viewpoint that approximately corresponds to the pose of the photographic image by manipulating user controls. The photographic image and the corresponding virtual image may be overlaid on one another so that both images can be seen at the same time. The user may adjust the pose of one of the images while overlaid on the other image by manipulating user controls so that both images appear to substantially align with one another. The settings of the user controls may be converted to pose data that is representative of the pose of the photographic image within the 3D model.

Variance minimization light probe sampling

Abstract: We present a technique for sampling the light probe image using variance minimization. The technique modifies median cut algorithm for light probe sampling [Debevec 2005] so that the variance of each region is minimized. The algorithm is fast, efficient, and easy to implement.

Cosine Lobe Based Relighting from Gradient Illumination Photographs

Abstract: We present an image-based method for relighting a scene by analytically fitting a cosine lobe to the reflectance function at each pixel, based on gradient illumination photographs. Realistic relighting results for many materials are obtained using a single per-pixel cosine lobe obtained from just two color photographs: one under uniform white illumination and the other under colored gradient illumination. For materials with wavelength-dependent scattering, a better fit can be obtained using independent cosine lobes for the red, green, and blue channels, obtained from three monochromatic gradient illumination conditions instead of the colored gradient condition. We explore two cosine lobe reflectance functions, both of which allow an analytic fit to the gradient conditions. One is non-zero over half the sphere of lighting directions, which works well for diffuse and specular materials, but fails for materials with broader scattering such as fur. The other is non-zero everywhere, which works well for broadly scattering materials and still produces visually plausible results for diffuse and specular materials. Additionally, we estimate scene geometry from the photometric normals to produce hard shadows cast by the geometry, while still reconstructing the input photographs exactly.

Image-based Separation of Diffuse and Specular Reflections using Environmental Structured Illumination, Supplemental Material

Abstract: We present an image-based method for separating diffuse and specular reflections using environmental structured illumination. Two types of structured illumination are discussed: phase-shifted sine wave patterns, and phase-shifted binary stripe patterns. In both cases the low-pass filtering nature of diffuse reflections is utilized to separate the reflection components. We illustrate our method on a wide range of example scenes and applications.

HeadSPIN: a one-to-many 3D video teleconferencing system

Abstract: When people communicate in person, numerous cues of attention, eye contact, and gaze direction provide important additional channels of information, making in-person meetings more efficient and effective than telephone conversations and 2D teleconferences. Two-dimensional video teleconferencing precludes the impression of accurate eye contact: when a participant looks into the camera, everyone seeing their video stream sees the participant looking toward them; when the participant looks away from the camera (for example, toward other participants in the meeting), no one sees the participant looking at them. In this work, we develop a one-to-many teleconferencing system which uses 3D acquisition, transmission, and display technologies to achieve accurate reproduction of gaze and eye contact. In this system, the face of a single remote participant is scanned at interactive rates using structured light while the participant watches a large 2D screen showing an angularly correct view of the audience. The scanned participant's geometry is then shown on the 3D display to the audience.

Estimating specular roughness from polarized second order spherical gradient illumination

Abstract: Measurement of spatially varying BRDFs of real world materials has been an active area of research in computer graphics with image-based measurements being the preferred approach in practice. In order to restrict the total number of measurements, existing techniques typically trade spatial variation for angular variation of the surface BRDF [Marschner et al. 1999]. Recently, Ma et al. [2007] introduced a technique for estimating high quality specular normals and albedo (Fig. 1, (a) & (b) respectively) of a specular object using polarized first order spherical gradient illumination conditions. In this work, we extend this technique to estimate per pixel specular roughness using polarized second order spherical gradients as a measure of the variance about the mean (reflection vector). We demonstrate that for isotropic BRDFs, only three spherical gradient illumination patterns related to the second order spherical harmonics are sufficient for a robust estimate of per pixel specular roughness (Fig. 1, (c)). Thus, we go further than previous work on image-based measurement of specular BRDFs that typically obtain sparse estimates of the spatial variation. Our technique also provides a direct estimate of the per pixel specular roughness and hence has the added advantage of not requiring any off-line numerical optimization that is typical of the measure and fit approach to BRDF modeling.

Computation & cultural heritage: fundamentals and applications

Abstract: This course surveys several practical techniques advanced by graphics and vision researchers for applications in cultural heritage, archeology, and art history. Topics include: efficient techniques for digital capture of heritage objects, appropriate uses in the heritage field, an end-to-end pipeline for processing archeological reconstructions (with special attention to incorporating archeological data and review throughout the process), how digital techniques are actually used in cultural heritage projects, and an honest evaluation of progress and challenges in this field.

Data-driven diffuse-specular separation of spherical gradient illumination

Abstract: Separation of the diffuse and specular components of observed reflectance has been an active area of research in computer graphics and vision, with major applications in reflectance modeling and scene analysis. Traditionally, researchers have investigated diffusespecular separation under point or directional illumination conditions while employing polarization and, in the case of dielectric materials, color space analysis techniques. Recently, Ma et al. [2007] introduced a technique for estimating high quality diffuse and specular normals and albedo maps (see Fig. 1, (a) & (d)) of a specular object using polarized spherical gradient illumination. However, the employed polarization technique imposes view-point restriction, and results in insufficient light levels for performance capture with high speed acquisition. Hence, in this work, we look into an alternate diffuse-specular separation technique for spherical gradients based on a data-driven reflectance model. Traditional separation techniques based on color space analysis focus on removing specular reflections from the observation for scene analysis [Mallick et al. 2005]. In contrast, we focus on obtaining high quality estimates of both the diffuse and the specular reflectance components.

Practical modeling and acquisition layered facial reflectance

Abstract: Techniques are described for modeling layered facial reflectance consisting of specular reflectance, single scattering, and shallow and deep subsurface scattering. Parameters of appropriate reflectance models can be estimated for each of these layers, e.g., from just 20 photographs recorded in a few seconds from a single view-point. Spatially-varying specular reflectance and single-scattering parameters can be extracted from polarization-difference images under spherical and point source illumination. Direct-indirect separation can be employed to decompose the remaining multiple scattering observed under cross-polarization into shallow and deep scattering components to model the light transport through multiple layers of skin. Appropriate diffusion models can be matched to the extracted shallow and deep scattering components for different regions on the face. The techniques were validated by comparing renderings of subjects to reference photographs recorded from novel viewpoints and under novel illumination conditions. Related geometry acquisition systems and software products are also described.

Interactive lighting manipulation application on GPU

Abstract: We present a technique for relighting an image such that different areas of the image are illuminated with different combinations of lighting directions. The key idea is to capture illumination data using a lighting apparatus system such as Hawkins et al. [2004], calculate radial basis function interpolation of light constraints specified by users and render the calculated illumination result in realtime using GPU. The application can simulate the result of unnatural lighting conditions, for example, the image of a whole face lit from per pixel view dependence reflection angles or from gazing angles (see Fig. 1, a). The application can also render a high-resolution result at 1920 x 1080 in three to four minutes.

Compact Representation of Reflectance Fields using Clustered Sparse

Abstract: We present a novel compression method for fixed viewpoint reflectance fields, captured for example by a Light Stage. Our compressed representation consists of a global approximation that exploits the similarities between the reflectance functions of different pixels, and a local approximation that encodes the per-pixel residual with the global approximation. Key to our method is a clustered sparse residual factorization. This sparse residual factorization ensures that the per-pixel residual matrix is as sparse as possible, enabling a compact local approximation. Finally, we demonstrate that the presented compact representation is well suited for high-quality real-time rendering.

2D and 3D facial correspondences via photometric alignment

Abstract: Capturing facial geometry that is high-resolution, yet easy to animate, remains a difficult challenge. While a single scanned geometry may be straightforward to animate smoothly, it may not always yield realistic fine scale detail when deformed into different facial expressions. Combining scans of multiple facial expressions, however, is only practical if geometrical correspondences between the different scanned expressions are available. Correspondences obtained based on locations of facial landmarks or of placed markers are often sparse, especially compared to fine-scale structures such as individual skin pores. The resulting misalignment of fine detail can introduce artifacts or blur out details we wish to preserve.

Compact Representation of Reflectance Fields using Clustered Sparse Residual Factorization

Abstract: We present a novel compression method for fixed viewpoint reflectance fields, captured for example by a Light Stage. Our compressed representation consists of a global approximation that exploits the similarities between the reflectance functions of different pixels, and a local approximation that encodes the per-pixel residual with the global approximation. Key to our method is a clustered sparse residual factorization. This sparse residual factorization ensures that the per-pixel residual matrix is as sparse as possible, enabling a compact local approximation. Finally, we demonstrate that the presented compact representation is well suited for high-quality real-time rendering.