Advances in 3D scanning technologies have enabled the practical creation of meshes with hundreds of millions of polygons. Traditional algorithms for display, simplification, and progressive transmission of meshes are impractical for data sets of this size. We describe a system for representing and progressively displaying these meshes that combines a multiresolution hierarchy based on bounding spheres with a rendering system based on points. A single data structure is used for view frustum culling, backface culling, level-of-detail selection, and rendering. The representation is compact and can be computed quickly, making it suitable for large data sets. Our implementation, written for use in a large-scale 3D digitization project, launches quickly, maintains a user-settable interactive frame rate regardless of object complexity or camera position, yields reasonable image quality during motion, and refines progressively when idle to a high final image quality. We have demonstrated the system on scanned models containing hundreds of millions of samples.

/pdf/qsplat-a-multiresolution-point-rendering-system-for-large-1j69vm9agr.pdf

QSplat: a multiresolution point rendering system for large meshes

Surface elements (surfels) are a powerful paradigm to efficiently render complex geometric objects at interactive frame rates. Unlike classical surface discretizations, i.e., triangles or quadrilateral meshes, surfels are point primitives without explicit connectivity. Surfel attributes comprise depth, texture color, normal, and others. As a pre-process, an octree-based surfel representation of a geometric object is computed. During sampling, surfel positions and normals are optionally perturbed, and different levels of texture colors are prefiltered and stored per surfel. During rendering, a hierarchical forward warping algorithm projects surfels to a z-buffer. A novel method called visibility splatting determines visible surfels and holes in the z-buffer. Visible surfels are shaded using texture filtering, Phong illumination, and environment mapping using per-surfel normals. Several methods of image reconstruction, including supersampling, offer flexible speed-quality tradeoffs. Due to the simplicity of the operations, the surfel rendering pipeline is amenable for hardware implementation. Surfel objects offer complex shape, low rendering cost and high image quality, which makes them specifically suited for low-cost, real-time graphics, such as games.

/pdf/surfels-surface-elements-as-rendering-primitives-5a74bn8wzw.pdf

Surfels: surface elements as rendering primitives

This paper studies the problem of plenoptic sampling in image-based rendering (IBR). From a spectral analysis of light field signals and using the sampling theorem, we mathematically derive the analytical functions to determine the minimum sampling rate for light field rendering. The spectral support of a light field signal is bounded by the minimum and maximum depths only, no matter how complicated the spectral support might be because of depth variations in the scene. The minimum sampling rate for light field rendering is obtained by compacting the replicas of the spectral support of the sampled light field within the smallest interval. Given the minimum and maximum depths, a reconstruction filter with an optimal and constant depth can be designed to achieve anti-aliased light field rendering.Plenoptic sampling goes beyond the minimum number of images needed for anti-aliased light field rendering. More significantly, it utilizes the scene depth information to determine the minimum sampling curve in the joint image and geometry space. The minimum sampling curve quantitatively describes the relationship among three key elements in IBR systems: scene complexity (geometrical and textural information), the number of image samples, and the output resolution. Therefore, plenoptic sampling bridges the gap between image-based rendering and traditional geometry-based rendering. Experimental results demonstrate the effectiveness of our approach.

Plenoptic sampling

Developing visual perception models for active agents and sensorimotor control in the physical world are cumbersome as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we investigate developing real-world perception for active agents, propose Gibson Environment for this purpose, and showcase a set of perceptual tasks learned therein. Gibson is based upon virtualizing real spaces, rather than artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism "Goggles" enabling deploying the trained models in real-world without needing domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

/pdf/gibson-env-real-world-perception-for-embodied-agents-3hvwk4uu4r.pdf

Gibson Env: Real-World Perception for Embodied Agents

Modern laser range and optical scanners need rendering techniques that can handle millions of points with high resolution textures. This paper describes a point rendering and texture filtering technique called surface splatting which directly renders opaque and transparent surfaces from point clouds without connectivity. It is based on a novel screen space formulation of the Elliptical Weighted Average (EWA) filter. Our rigorous mathematical analysis extends the texture resampling framework of Heckbert to irregularly spaced point samples. To render the points, we develop a surface splat primitive that implements the screen space EWA filter. Moreover, we show how to optimally sample image and procedural textures to irregular point data during pre-processing. We also compare the optimal algorithm with a more efficient view-independent EWA pre-filter. Surface splatting makes the benefits of EWA texture filtering available to point-based rendering. It provides high quality anisotropic texture filtering, hidden surface removal, edge anti-aliasing, and order-independent transparency.

Surface splatting

Using multiple reference images in 3D image warping has been a challenging problem. Recently, the Layered Depth Image (LDI) was proposed by Shade et al. to merge multiple reference images under a single center of projection, while maintaining the simplicity of warping a single reference image. However it does not consider the issue of sampling rate. We present the LDI tree, which combines a hierarchical space partitioning scheme with the concept of the LDI. It preserves the sampling rates of the reference images by adaptively selecting an LDI in the LDI tree for each pixel. While rendering from the LDI tree, we only have to traverse the LDI tree to the levels that are comparable to the sampling rate of the output image. We also present a progressive refinement feature and a “gap filling” algorithm implemented by pre-filtering the LDI tree. We show that the amount of memory required has the same order of growth as the 2D reference images. This also bounds the complexity of rendering time to be less than directly rendering from all reference images. CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation Viewing Algorithms; I.3.6 [Computer Graphics] Methodology and Techniques Graphics data structures and data types; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism. Additional

/pdf/ldi-tree-a-hierarchical-representation-for-image-based-4y3r8734ih.pdf

LDI tree: a hierarchical representation for image-based rendering

A method and an apparatus determine a color for pixels in a graphics system in which images are defined by pixels. Multiple fragments of an image may be visible in any given pixel. Each visible fragment has a fragment value that includes the color of that fragment. For such given pixel, up to a predetermined number of the fragment values are stored. When a new fragment is visible in the given pixel, one of the fragment values is discarded to determine which fragment values are stored and subsequently used to generate the color of the pixel. The discarded fragment value may be the new fragment value or one of the stored fragment values. Various strategies can be used to determine which fragment value is discarded. One such scheme selects the stored fragment value with the greatest Z-depth. Another scheme selects the stored fragment value that produces the smallest color difference from the new fragment value. Still another scheme selects the new fragment value when one of the fragments is in front of the new fragment and the stored fragment value of that fragment produces the smallest color difference from the new fragment value.

Method and apparatus for compositing colors of images with memory constraints for storing pixel data

Compared to a personal computer, mobile devices typically have weaker processing power, less memory capacity, and lower resolution of display. While the former two factors are clearly disadvantages for 3D graphics applications running on mobile devices, the display factor could be turned into an advantage instead. However the traditional 3D graphics pipeline cannot take advantage of the smaller display because its run time depends mostly on the number of polygons to be rendered. In contrast, the run time of image-based rendering methods depends mainly on the display resolution. Therefore it is well suited for mobile devices. Furthermore, we may use the network connection to build a client-server framework, which allows us to integrate with nonimage-based rendering programs. We present our system framework and the experiment results on PocketPC® based devices in this work.

/pdf/enhancing-3d-graphics-on-mobile-devices-by-image-based-19yotum062.pdf

Enhancing 3D Graphics on Mobile Devices by Image-Based Rendering

In this paper we present an algorithm for low-cost hardware antialiasing and transparency. This technique keeps a central Z value along with 8-bit floating-point Z gradients in the X and Y dimensions for each fragment within a pixel (hence the name Z 3). It uses a small fixed amount of storage per pixel. If there are more fragments generated for a pixel than the space available, it merges only as many fragments as necessary in order to fit in the available per-pixel memory. The merging occurs on those fragments having the closest Z values. This combines different fragments from the same surface, resulting in both storage and processing efficiency. When operating with opaque surfaces, Z 3 can provide superior image quality over sparse supersampling methods that use eight samples per pixel while using storage for only three fragments. Z 3 also makes the use of large numbers of samples (e.g., 16) feasible in inexpensive hardware, enabling higher quality images. It is simple to implement because it uses a small fixed number of fragments per pixel. Z can also provide order-independent transparency even if many transparent surfaces are present. Moreover, unlike the original A-buffer algorithm it correctly antialiases interpenetrating transparent surfaces because it has three-dimensional Z information within each pixel.

Z3: an economical hardware technique for high-quality antialiasing and transparency

Revision control is a vital component of digital project management and has been widely deployed for text files Binary files, on the other hand, have received relatively less attention This can be inconvenient for graphics applications that use a significant amount of binary data, such as images, videos, meshes, and animations Existing strategies such as storing whole files for individual revisions or simple binary deltas could consume significant storage and obscure vital semantic information We present a nonlinear revision control system for images, designed with the common digital editing and sketching workflows in mind We use DAG (directed acyclic graph) as the core structure, with DAG nodes representing editing operations and DAG edges the corresponding spatial, temporal and semantic relationships We visualize our DAG in RevG (revision graph), which provides not only as a meaningful display of the revision history but also an intuitive interface for common revision control operations such as review, replay, diff, addition, branching, merging, and conflict resolving Beyond revision control, our system also facilitates artistic creation processes in common image editing and digital painting workflows We have built a prototype system upon GIMP, an open source image editor, and demonstrate its effectiveness through formative user study and comparisons with alternative revision control systems

Chun-Fa Chang

Papers

LDI tree: a hierarchical representation for image-based rendering

Method and apparatus for compositing colors of images with memory constraints for storing pixel data

Enhancing 3D Graphics on Mobile Devices by Image-Based Rendering

Z3: an economical hardware technique for high-quality antialiasing and transparency

Nonlinear revision control for images