A morphable model for the synthesis of 3D faces

This paper presents a new reflectance model for rendering computer synthesized images. The model accounts for the relative brightness of different materials and light sources in the same scene. It describes the directional distribution of the reflected light and a color shift that occurs as the reflectance changes with incidence angle. The paper presents a method for obtaining the spectral energy distribution of the light reflected from an object made of a specific real material and discusses a procedure for accurately reproducing the color associated with the spectral energy distribution. The model is applied to the simulation of a metal and a plastic.

A reflectance model for computer graphics

Current methods for combining two different images produce visible artifacts when the sources have very different textures and structures. We present a new method for synthesizing a transition region between two source images, such that inconsistent color, texture, and structural properties all change gradually from one source to the other. We call this process image melding. Our method builds upon a patch-based optimization foundation with three key generalizations: First, we enrich the patch search space with additional geometric and photometric transformations. Second, we integrate image gradients into the patch representation and replace the usual color averaging with a screened Poisson equation solver. And third, we propose a new energy based on mixed L2/L0 norms for colors and gradients that produces a gradual transition between sources without sacrificing texture sharpness. Together, all three generalizations enable patch-based solutions to a broad class of image melding problems involving inconsistent sources: object cloning, stitching challenging panoramas, hole filling from multiple photos, and image harmonization. In several cases, our unified method outperforms previous state-of-the-art methods specifically designed for those applications.

Image melding: combining inconsistent images using patch-based synthesis

Stereo techniques have witnessed tremendous progress over the last decades, yet some aspects of the problem still remain challenging today. Striking examples are reflecting and textureless surfaces which cannot easily be recovered using traditional local regularizers. In this paper, we therefore propose to regularize over larger distances using object-category specific disparity proposals (displets) which we sample using inverse graphics techniques based on a sparse disparity estimate and a semantic segmentation of the image. The proposed displets encode the fact that objects of certain categories are not arbitrarily shaped but typically exhibit regular structures. We integrate them as non-local regularizer for the challenging object class ‘car’ into a superpixel based CRF framework and demonstrate its benefits on the KITTI stereo evaluation. At time of submission, our approach ranks first across all KITTI stereo leaderboards.

/pdf/displets-resolving-stereo-ambiguities-using-object-knowledge-2jne8swqmk.pdf

Displets: Resolving stereo ambiguities using object knowledge

We present an image-based relighting method that can synthesize scene appearance under novel, distant illumination from the visible hemisphere, from only five images captured under pre-defined directional lights. Our method uses a deep convolutional neural network to regress the relit image from these five images; this relighting network is trained on a large synthetic dataset comprised of procedurally generated shapes with real-world reflectances. We show that by combining a custom-designed sampling network with the relighting network, we can jointly learn both the optimal input light directions and the relighting function. We present an extensive evaluation of our network, including an empirical analysis of reconstruction quality, optimal lighting configurations for different scenarios, and alternative network architectures. We demonstrate, on both synthetic and real scenes, that our method is able to reproduce complex, high-frequency lighting effects like specularities and cast shadows, and outperforms other image-based relighting methods that require an order of magnitude more images.

Deep image-based relighting from optimal sparse samples

Current research trends demonstrate that, for a wide range of applications in cultural heritage, 3D shape acquisition alone is not sufficient. To generate a digital replica of a real world object the digitized geometric models have to be complemented with information pertaining to optical properties of the object surface. We therefore propose an integrated system for acquiring both the 3D shape and reflectance properties necessary for obtaining a photo-realistic digital replica. The proposed method is suitable for the digitization of objects showing the complex reflectance behavior, for example specularities and meso-scale interreflections, often encountered in the field of cultural heritage. We demonstrate the performance of our system with four challenging examples. By using Bidirectional Texture Functions, our structured light based approach is able to achieve good geometric precision while preserving tiny details such as scratches and engravings.

Integrated high-quality acquisition of geometry and appearance for cultural heritage

In this paper, we present a novel compression technique for Bidirectional Texture Functions based on a sparse tensor decomposition. We apply the K-SVD algorithm along two different modes of a tensor to decompose it into a small dictionary and two sparse tensors. This representation is very compact, allowing for considerably better compression ratios at the same RMS error than possible with current compression techniques like PCA, N-mode SVD and Per Cluster Factorization. In contrast to other tensor decomposition based techniques, the use of a sparse representation achieves a rendering performance that is at high compression ratios similar to PCA based methods.

BTF compression via sparse tensor decomposition

In this paper, we present a novel, robust multi-view normal field integration technique for reconstructing the full 3D shape of mirroring objects. We employ a turntable-based setup with several cameras and displays. These are used to display illumination patterns which are reflected by the object surface. The pattern information observed in the cameras enables the calculation of individual volumetric normal fields for each combination of camera, display and turntable angle. As the pattern information might be blurred depending on the surface curvature or due to non-perfect mirroring surface characteristics, we locally adapt the decoding to the finest still resolvable pattern resolution. In complex real-world scenarios, the normal fields contain regions without observations due to occlusions and outliers due to interreflections and noise. Therefore, a robust reconstruction using only normal information is challenging. Via a non-parametric clustering of normal hypotheses derived for each point in the scene, we obtain both the most likely local surface normal and a local surface consistency estimate. This information is utilized in an iterative min-cut based variational approach to reconstruct the surface geometry.

/pdf/multi-view-normal-field-integration-for-3d-reconstruction-of-5txrj0mmzg.pdf

Multi-view Normal Field Integration for 3D Reconstruction of Mirroring Objects

In this work, we present a framework for multi-camera, multi-projector object acquisition based on structured light. This approach allows the reconstruction of an object without moving either the object or the acquisition setup, avoiding any registration of independent measurements. To overcome the resolution limitations of the individual projectors, we introduce a novel super-resolution scheme. By exploiting high dynamic range imaging, we are able to handle even complicated objects, exhibiting strong specularities. We show that, combined with an iterated bundle adjustment, these improvements increase the accuracy of the obtained point cloud.

https://cg.cs.uni-bonn.de/aigaion2root/attachments/Presentation.pdf

A Multi-camera, Multi-projector Super-Resolution Framework for Structured Light

In this paper, we present a novel exemplar-based technique for the interpolation between two textures that combines patch-based and statistical approaches. Motivated by the notion of texture as a largely local phenomenon, we warp and blend small image neighborhoods prior to patch-based texture synthesis. In addition, interpolating and enforcing characteristic image statistics faithfully handles high frequency detail. We are able to create both intermediate textures as well as continuous transitions. In contrast to previous techniques computing a global morphing transformation on the entire input exemplar images, our localized and patch-based approach allows us to successfully interpolate between textures with considerable differences in feature topology for which no smooth global warping field exists.

Roland Ruiters

Papers

Integrated high-quality acquisition of geometry and appearance for cultural heritage

BTF compression via sparse tensor decomposition

Multi-view Normal Field Integration for 3D Reconstruction of Mirroring Objects

A Multi-camera, Multi-projector Super-Resolution Framework for Structured Light

Patch-based texture interpolation