Showing papers by "Marc Levoy published in 2005"

Light field photography with a hand-held plenoptic camera

Abstract: This paper presents a camera that samples the 4D light field on its sensor in a single photographic exposure. This is achieved by inserting a microlens array between the sensor and main lens, creating a plenoptic camera. Each microlens measures not just the total amount of light deposited at that location, but how much light arrives along each ray. By re-sorting the measured rays of light to where they would have terminated in slightly different, synthetic cameras, we can compute sharp photographs focused at different depths. We show that a linear increase in the resolution of images under each microlens results in a linear increase in the sharpness of the refocused photographs. This property allows us to extend the depth of field of the camera without reducing the aperture, enabling shorter exposures and lower image noise. Especially in the macrophotography regime, we demonstrate that we can also compute synthetic photographs from a range of different viewpoints. These capabilities argue for a different strategy in designing photographic imaging systems. To the photographer, the plenoptic camera operates exactly like an ordinary hand-held camera. We have used our prototype to take hundreds of light field photographs, and we present examples of portraits, high-speed action and macro close-ups.

High performance imaging using large camera arrays

Abstract: The advent of inexpensive digital image sensors and the ability to create photographs that combine information from a number of sensed images are changing the way we think about photography. In this paper, we describe a unique array of 100 custom video cameras that we have built, and we summarize our experiences using this array in a range of imaging applications. Our goal was to explore the capabilities of a system that would be inexpensive to produce in the future. With this in mind, we used simple cameras, lenses, and mountings, and we assumed that processing large numbers of images would eventually be easy and cheap. The applications we have explored include approximating a conventional single center of projection video camera with high performance along one or more axes, such as resolution, dynamic range, frame rate, and/or large aperture, and using multiple cameras to approximate a video camera with a large synthetic aperture. This permits us to capture a video light field, to which we can apply spatiotemporal view interpolation algorithms in order to digitally simulate time dilation and camera motion. It also permits us to create video sequences using custom non-uniform synthetic apertures.

Imaging arrangements and methods therefor

Abstract: Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

Dual photography

Abstract: We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate pictures from the viewpoint of the projector, even though no camera was present at that location. The technique is completely image-based, requiring no knowledge of scene geometry or surface properties, and by its nature automatically includes all transport paths, including shadows, inter-reflections and caustics. In its simplest form, the technique can be used to take photographs without a camera; we demonstrate this by capturing a photograph using a projector and a photo-resistor. If the photo-resistor is replaced by a camera, we can produce a 4D dataset that allows for relighting with 2D incident illumination. Using an array of cameras we can produce a 6D slice of the 8D reflectance field that allows for relighting with arbitrary light fields. Since an array of cameras can operate in parallel without interference, whereas an array of light sources cannot, dual photography is fundamentally a more efficient way to capture such a 6D dataset than a system based on multiple projectors and one camera. As an example, we show how dual photography can be used to capture and relight scenes.

Synthetic Aperture Focusing using a Shear-Warp Factorization of the Viewing Transform

Abstract: Synthetic aperture focusing consists of warping and adding together the images in a 4D light field so that objects lying on a specified surface are aligned and thus in focus, while objects lying of this surface are misaligned and hence blurred. This provides the ability to see through partial occluders such as foliage and crowds, making it a potentially powerful tool for surveillance. If the cameras lie on a plane, it has been previously shown that after an initial homography, one can move the focus through a family of planes that are parallel to the camera plane by merely shifting and adding the images. In this paper, we analyze the warps required for tilted focal planes and arbitrary camera configurations. We characterize the warps using a new rank- 1 constraint that lets us focus on any plane, without having to perform a metric calibration of the cameras. We also show that there are camera configurations and families of tilted focal planes for which the warps can be factorized into an initial homography followed by shifts. This shear-warp factorization permits these tilted focal planes to be synthesized as efficiently as frontoparallel planes. Being able to vary the focus by simply shifting and adding images is relatively simple to implement in hardware and facilitates a real-time implementation. We demonstrate this using an array of 30 videoresolution cameras; initial homographies and shifts are performed on per-camera FPGAs, and additions and a final warp are performed on 3 PCs.

Interactive deformation of light fields

Abstract: We present a software pipeline that enables an animator to deform light fields. The pipeline can be used to deform complex objects, such as furry toys, while maintaining photo-realistic quality. Our pipeline consists of three stages. First, we split the light field into sub-light fields. To facilitate splitting of complex objects, we employ a novel technique based on projected light patterns. Second, we deform each sub-light field. To do this, we provide the animator with controls similar to volumetric free-form deformation. Third, we recombine and render each sub-light field. Our rendering technique properly handles visibility changes due to occlusion among sub-light fields. To ensure consistent illumination of objects after they have been deformed, our light fields are captured with the light source fixed to the camera, rather than being fixed to the object. We demonstrate our deformation pipeline using synthetic and photographically acquired light fields. Potential applications include animation, interior design, and interactive gaming.

High-speed videography using a dense camera array

Abstract: We demonstrate a system for capturing multi-thousand frame-per-second (fps) video using a dense array of cheap 30fps CMOS image sensors. A benefit of using a camera array to capture high-speed video is that we can scale to higher speeds by simply adding more cameras. Even at extremely high frame rates, our array architecture supports continuous streaming to disk from all of the cameras. This allows us to record unpredictable events, in which nothing occurs before the event of interest that could be used to trigger the beginning of recording. Synthesizing one high-speed video sequence using images from an array of cameras requires methods to calibrate and correct those cameras' varying radiometric and geometric properties. We assume that our scene is either relatively planar or is very far away from the camera and that the images can therefore be aligned using projective transforms. We analyze the errors from this assumption and present methods to make them less visually objectionable. We also present a new method to automatically color match our sensors. Finally, we demonstrate how to compensate for spatial and temporal distortions caused by the electronic rolling shutter, a common feature of low-end CMOS sensors.

Protecting 3d graphics content

Abstract: The digital rights management problem of protecting data from theft and misuse has been addressed for many information types, including software code, digital images, and audio files. Few technological solutions are designed specifically to protect interactive 3D graphics content.Demand for ways to protect 3D graphical models is significant and growing. Contemporary 3D digitization technologies allow the efficient creation of accurate 3D models of many physical objects. For example, our Stanford Digital Michelangelo Project [3] has developed a high-resolution digital archive of 10 of Michelangelo's large statues, including the David (see the sidebar "Generating a Replica of Michelangelo's David"). These statues represent the artistic patrimony of Italy's cultural institutions, and our contract with the Italian authorities permits distribution of the 3D models only to established scholars for noncommercial use. Though everyone involved would like the models to be available for any constructive purpose, the digital 3D model of the David would quickly be pirated if it were distributed without protection; simulated marble replicas would be manufactured outside the provisions of the parties authorizing creation of the model.

Automatic Color Calibration for Large Camera

Abstract: We present a color calibration pipeline for large camera arrays. We assume static lighting conditions for each camera, such as studio lighting or a stationary array outdoors. We also assume we can place a planar calibration target so it is visible from every camera. Our goal is uniform camera color responses, not absolute color accuracy, so we match the cameras to each other instead of to a color standard. We first iteratively adjust the color channel gains and offsets for each camera to make their responses as similar as possible. This step white balances the cameras, and for studio applications, ensures that the range of intensities in the scene are mapped to the usable output range of the cameras. Residual errors are then calibrated in post-processing. We present results calibrating an array of 100 CMOS image sensors in different physical configurations, including closely or widely spaced cameras with overlapping fields of views, and tightly packed cameras with non-overlapping fields of view. The process is entirely automatic, and the camera configuration runs in less than five minutes on the 100 camera array.

Imaging arrangement and method therefor

Abstract: A digital imaging system for generating a virtual image of a scene, wherein the virtual image includes a plurality of pixels, the system comprising: a photosensor array, having a plurality of photosensors, configured to detect light including a plurality of light rays from the scene, wherein each photosensor, in response to detecting the light incident thereon, generates light data, the photosensor array further configured to detect a set of the plurality of light rays that concurrently arrive at the particular portion of the physical focal plane at different angles of incidence;an optics arrangement, having a physical focus, configured to direct light rays from the scene to the photosensor array via a physical focal plane; and processing circuitry to compute a pixel value of each pixel of the virtual image by combining light data from selected photosensors based on angles of incidence of the light rays detected by each photosensor, as characterized by the location of each photosensor relative to the physical focal plane, the virtual image having a virtual focus and a virtual focal plane that are respectively different from the physical focus and the physical focal plane, wherein the virtual focal plane comprises a planar focal plane and a non-planar focal plane, wherein the processing circuitry further determines the angles of incidence of light upon each photosensor using a position of each photosensor relative to the microlens.