Light-field cameras have recently become available to the consumer market. An array of micro-lenses captures enough information that one can refocus images after acquisition, as well as shift one's viewpoint within the sub-apertures of the main lens, effectively obtaining multiple views. Thus, depth cues from both defocus and correspondence are available simultaneously in a single capture. Previously, defocus could be achieved only through multiple image exposures focused at different depths, while correspondence cues needed multiple exposures at different viewpoints or multiple cameras, moreover, both cues could not easily be obtained together. In this paper, we present a novel simple and principled algorithm that computes dense depth estimation by combining both defocus and correspondence depth cues. We analyze the x-u 2D epipolar image (EPI), where by convention we assume the spatial x coordinate is horizontal and the angular u coordinate is vertical (our final algorithm uses the full 4D EPI). We show that defocus depth cues are obtained by computing the horizontal (spatial) variance after vertical (angular) integration, and correspondence depth cues by computing the vertical (angular) variance. We then show how to combine the two cues into a high quality depth map, suitable for computer vision applications such as matting, full control of depth-of-field, and surface reconstruction.

/pdf/depth-from-combining-defocus-and-correspondence-using-light-19oa2755wf.pdf

Depth from Combining Defocus and Correspondence Using Light-Field Cameras

This paper introduces an algorithm that accurately estimates depth maps using a lenslet light field camera. The proposed algorithm estimates the multi-view stereo correspondences with sub-pixel accuracy using the cost volume. The foundation for constructing accurate costs is threefold. First, the sub-aperture images are displaced using the phase shift theorem. Second, the gradient costs are adaptively aggregated using the angular coordinates of the light field. Third, the feature correspondences between the sub-aperture images are used as additional constraints. With the cost volume, the multi-label optimization propagates and corrects the depth map in the weak texture regions. Finally, the local depth map is iteratively refined through fitting the local quadratic function to estimate a non-discrete depth map. Because micro-lens images contain unexpected distortions, a method is also proposed that corrects this error. The effectiveness of the proposed algorithm is demonstrated through challenging real world examples and including comparisons with the performance of advanced depth estimation algorithms.

/pdf/accurate-depth-map-estimation-from-a-lenslet-light-field-3n918eox5r.pdf

Accurate depth map estimation from a lenslet light field camera

Light field photography has gained a significant research interest in the last two decades; today, commercial light field cameras are widely available. Nevertheless, most existing acquisition approaches either multiplex a low-resolution light field into a single 2D sensor image or require multiple photographs to be taken for acquiring a high-resolution light field. We propose a compressive light field camera architecture that allows for higher-resolution light fields to be recovered than previously possible from a single image. The proposed architecture comprises three key components: light field atoms as a sparse representation of natural light fields, an optical design that allows for capturing optimized 2D light field projections, and robust sparse reconstruction methods to recover a 4D light field from a single coded 2D projection. In addition, we demonstrate a variety of other applications for light field atoms and sparse coding, including 4D light field compression and denoising.

/pdf/compressive-light-field-photography-using-overcomplete-3n0z4j8gdl.pdf

Compressive light field photography using overcomplete dictionaries and optimized projections

Consumer-level and high-end light-field cameras are now widely available. Recent work has demonstrated practical methods for passive depth estimation from light-field images. However, most previous approaches do not explicitly model occlusions, and therefore cannot capture sharp transitions around object boundaries. A common assumption is that a pixel exhibits photo-consistency when focused to its correct depth, i.e., all viewpoints converge to a single (Lambertian) point in the scene. This assumption does not hold in the presence of occlusions, making most current approaches unreliable precisely where accurate depth information is most important - at depth discontinuities. In this paper, we develop a depth estimation algorithm that treats occlusion explicitly, the method also enables identification of occlusion edges, which may be useful in other applications. We show that, although pixels at occlusions do not preserve photo-consistency in general, they are still consistent in approximately half the viewpoints. Moreover, the line separating the two view regions (correct depth vs. occluder) has the same orientation as the occlusion edge has in the spatial domain. By treating these two regions separately, depth estimation can be improved. Occlusion predictions can also be computed and used for regularization. Experimental results show that our method outperforms current state-of-the-art light-field depth estimation algorithms, especially near occlusion boundaries.

/pdf/occlusion-aware-depth-estimation-using-light-field-cameras-4i8yjtcy05.pdf

Occlusion-Aware Depth Estimation Using Light-Field Cameras

How do neurons in the brain process information from the senses and drive complex behaviors? This question has fascinated neuroscientists for many years. It is currently not possible to record the electrical activities of all of the 100 billion neurons in a human brain. Yet, in the last decade, it has become possible to genetically engineer some neurons in animals to produce fluorescence reporters that change their brightness in response to brain activity and then monitor them under a microscope. In small animals such as zebrafish larvae, this method makes it possible to monitor the activities of all the neurons in the brain if the animal’s head is held still. However, many behaviors – for example, catching prey – require movement, and no existing technique could image brain activity in enough detail if the animal’s head was moving. Cong, Wang, Chai, Hang et al. have now made progress towards this goal by developing a new technique to image neural activity across the whole brain of a zebrafish larva as it swims freely in a small water-filled chamber. The technique uses high-speed cameras and computer software to track the movements of the fish in three dimensions, and then automatically moves the chamber under the microscope such that the animal’s brain is constantly kept in focus. The newly developed microscope can capture changes in neural activity across a large volume all at the same time. It is then further adapted to overcome problems caused by sudden or swift movements, which would normally result in motion blur. With this microscope set up, Cong et al. were able to capture, for the first time, activity from all the neurons in a zebrafish larva’s brain as it pursued and caught its prey. This technique provides a new window into how brain activity changes when animals are behaving naturally. In the future, this technique could help link the activities of neurons to different behaviors in several popular model organisms including fish, worms and fruit flies.

Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (Danio rerio).

Placing a micro lens array in front of an image sensor transforms a normal camera into a single lens 3D camera,
which also allows the user to change the focus and the point of view after a picture has been taken. While
the concept of such plenoptic cameras is known since 1908, only recently the increased computing power of
low-cost hardware and the advances in micro lens array production, have made the application of plenoptic
cameras feasible. This text presents a detailed analysis of plenoptic cameras as well as introducing a new type
of plenoptic camera with an extended depth of field and a maximal effective resolution of up to a quarter of the
sensor resolution.

Christian Perwass

Papers

Single lens 3D-camera with extended depth-of-field