Image-based rendering (IBR) has attracted a lot of research interest recently. In this paper, we survey the various techniques developed for IBR, including representation, sampling and compression. The goal is to provide an overview of research for IBR in a complete and systematic manner. We observe that essentially all the IBR representations are derived from the plenoptic function, which is seven dimensional and difficult to handle. We classify various IBR representations into two categories based on how the plenoptic function is simplified, namely restraining the viewing space and introducing source descriptions. In the former category, we summarize six common assumptions that were often made in various approaches and discuss how the dimension of the plenoptic function can be reduced based on these assumptions. In the latter category, we further categorize the methods based on what kind of source description was introduced, such as scene geometry, texture map or reflection model. Sampling and compression are also discussed respectively for both categories.

/pdf/a-survey-on-image-based-rendering-representation-sampling-4xo0wqvjkp.pdf

A survey on image-based rendering—representation, sampling and compression

In this article, we present an image-based modeling and rendering system, which we call pop-up light field, that models a sparse light field using a set of coherent layers. In our system, the user specifies how many coherent layers should be modeled or popped up according to the scene complexity. A coherent layer is defined as a collection of corresponding planar regions in the light field images. A coherent layer can be rendered free of aliasing all by itself, or against other background layers. To construct coherent layers, we introduce a Bayesian approach, coherence matting, to estimate alpha matting around segmented layer boundaries by incorporating a coherence prior in order to maintain coherence across images.We have developed an intuitive and easy-to-use user interface (UI) to facilitate pop-up light field construction. The key to our UI is the concept of human-in-the-loop where the user specifies where aliasing occurs in the rendered image. The user input is reflected in the input light field images where pop-up layers can be modified. The user feedback is instant through a hardware-accelerated real-time pop-up light field renderer. Experimental results demonstrate that our system is capable of rendering anti-aliased novel views from a sparse light field.

/pdf/pop-up-light-field-an-interactive-image-based-modeling-and-hf5y8uca12.pdf

Pop-up light field: An interactive image-based modeling and rendering system

Image-based rendering (IBR) is unique in that it requires computer graphics, computer vision, and image processing to join forces to solve a common goal, namely photorealistic rendering through the use of images. IBR as an area of research has been around for about ten years, and substantial progress has been achieved in effiectively capturing, representing, and rendering scenes. In this article, we survey the techniques used in IBR. Our survey shows that representations and rendering techniques can differ radically, depending on design decisions related to ease of capture, use of geometry, accuracy of geometry (if used), number and distribution of source images, degrees of freedom for virtual navigation, and expected scene complexity.

/pdf/image-based-rendering-2ajv8z1bew.pdf

Image-based rendering

We present a technique for capturing the extreme dynamic range of natural illumination environments that include the sun and sky, which has presented a challenge for traditional high dynamic range photography processes. We find that through careful selection of exposure times, aperture, and neutral density filters that this full range can be covered in seven exposures with a standard digital camera. We discuss the particular calibration issues such as lens vignetting, infrared sensitivity, and spectral transmission of neutral density filters which must be addressed. We present an adaptive exposure range adjustment technique for minimizing the number of exposures necessary. We demonstrate our results by showing time-lapse renderings of a complex scene illuminated by high-resolution, high dynamic range natural illumination environments.

https://ict.usc.edu/pubs/Direct%20HDR%20Capture%20of%20the%20Sun%20and%20Sky.pdf

Direct HDR capture of the sun and sky

To store and transmit the large amount of image data necessary for Image-based Rendering (IBR), efficient coding schemes are required. This paper presents two different approaches which exploit three-dimensional scene geometry for multi-view compression. In texture-based coding, images are converted to view-dependent texture maps for compression. In model-aided predictive coding, scene geometry is used for disparity compensation and occlusion detection between images. While both coding strategies are able to attain compression ratios exceeding 2000:1, individual coding performance is found to depend on the accuracy of the available geometry model. Experiments with real-world as well as synthetic image sets show that texture-based coding is more sensitive to geometry inaccuracies than predictive coding. A rate-distortion theoretical analysis of both schemes supports these findings. For reconstructed approximate geometry models, model-aided predictive coding performs best, while texture-based coding yields superior coding results if scene geometry is exactly known.

/pdf/multi-view-coding-for-image-based-rendering-using-3-d-scene-52y1b6szfl.pdf

Multi-view coding for image-based rendering using 3-D scene geometry

High-Quality Interactive Lumigraph Rendering Through Warping

Hierarchical radiosity with clustering has positioned itself as one of the most efﬁcient algorithms for computingglobal illumination in non-trivial environments. However, using hierarchical radiosity for complex scenes is stillproblematic due to the necessity of storing a large number of transport coefﬁcients between surfaces in the formof links. In this paper, we eliminate the need for storage of links through the use of a modiﬁed shooting methodfor solving the radiosity equation. By distributing only unshot radiosity in each step of the iteration, the numberof links decreases exponentially. Recomputing these links instead of storing them increases computation time, butreduces memory consumption dramatically. Caching may be used to reduce the time overhead. We analyze theerror behavior of the new algorithm in comparison with the normal gathering approach for hierarchical radiosity.In particular, we consider the relation between the global error of a hierarchical radiosity solution and the localerror threshold for each link.1. IntroductionGlobal illumination has attracted much attention during thelast years. Starting as a purely academic research topic in thebeginning, it now has become a state-of-the-art means forgenerating realistic synthetic images. Radiosity has placeditself as one of the most popular illumination methods forcomputing indirect lighting and producing such effects ascolor bleeding and soft shadows.With the quickly growing number of radiosity applica-tions today, the size of typical scenes as well as the complex-ity of the corresponding illumination simulation increasesdrastically, resulting in large numbers of patch-to-patchlinksthat represent the energy exchange between ﬁnite surface el -ements in the scene.From among the existing radiosity algorithms, hierarchi-cal methods have proved to be robust and easy to use. Formoderately complex scenes, they are also most efﬁcient. Forcomplex scenes, the immense number of links requires alarge amount of memory, which can be considered the majordrawback of existing hierarchical algorithms

Getting Rid of Links in Hierarchical Radiosity

We introduce an algorithm for high-quality, interactive light field rendering from only a small number of input images with dense depth information. The algorithm bridges the gap between image warping and interpolation from image databases, which represent the two major approaches in image based rendering. By warping and blending only the necessary parts of each reference image, we are able to generate a single view-corrected texture for every output frame at interactive rates. In contrast to previous light field rendering approaches, our warping-based algorithm is able to fully exploit perpixel depth information in order to depth-correct the light field samples with maximum accuracy. The complexity of the proposed algorithm is nearly independent of the number of stored reference images and of the final screen resolution. It performs with only small overhead and very few visible artifacts. We demonstrate the visual fidelity as well as the performance of our method through various examples.

High-Quality Interactive Lumigraph Rendering Through Warping.

We show a simple and efficient way for rendering arbitrary views from so-called free-form light fields , employing a convex free form camera surface and a set of arbitrarily oriented camera planes. This way directionally varying real-world imagery can be displayed without intermediate resampling steps, and yet rendering of free form light fields can be performed as efficiently as for two-planeparameterized light fields using texture mapping graphics hardware. Comparable to sphere-based parameterizations, a single free form light field can represent all possible views of the scene without the need for multiple slabs, and it allows for relatively uniform sampling. Furthermore, we extend the rendering algorithm to account for occlusion in certain input views. We apply our method to synthetic and real-world datasets with and without additional geometric information and compare the resulting rendering performance and quality to twoplane-parameterized light field rendering.

https://domino.mpi-inf.mpg.de/intranet/ag4/ag4publ.nsf/93832a04987390a3c12567530068622d/8320de37b415569fc1256a99002e9e69/$FILE/schirmacher_2001_eff.pdf

Efficient Free Form Light Field Rendering

Lighting algorithms for outdoor scenes suffer from the sheer geometric and lighting complexity of such environments. In this paper we introduce an effi­cient, hierarchical solution to the problem of outdoor illumination. Data structures and sampling algorithms are presented, permitting the integration of complex and natural objects in a hierarchical radiosity simulation system. This new approach allows the hierarchical simulation of radiant energy exchanges in outdoor scenes for the first time, including terrain and botanical models as well as sunlight and skylight. This is accomplished by providing the necessary tools to treat terrain meshes as a hierarchy of light-exchanging objects, as well as an efficient hierar­chical representation for the sky dome. In addition, refinement criteria are adapted to the particular characteristics of natural lighting. Results of our implementation are presented including naturally-lit images of terrain-maps, trees and buildings.

/pdf/hierachical-lighting-simulation-for-outdoor-scenes-3kd1njqmtp.pdf

Hartmut Schirmacher

Papers

High-Quality Interactive Lumigraph Rendering Through Warping

Getting Rid of Links in Hierarchical Radiosity

High-Quality Interactive Lumigraph Rendering Through Warping.

Efficient Free Form Light Field Rendering

Hierachical Lighting Simulation for Outdoor Scenes