Showing papers by "Wolfgang Heidrich published in 2014"

FlexISP: a flexible camera image processing framework

Abstract: Conventional pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, each responsible for addressing a particular problem. While this divide-and-conquer approach offers many benefits, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the original sensor data. We propose an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation (e.g., YUV, DCT). Our system is flexible and we demonstrate it on regular Bayer images as well as images from custom sensors. In all cases, we achieve large improvements in image quality and signal reconstruction compared to state-of-the-art techniques. Finally, we show that our approach is capable of very efficiently handling high-resolution images, making even mobile implementations feasible.

Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors

Abstract: The functional difference between a diffuse wall and a mirror is well understood: one scatters back into all directions, and the other one preserves the directionality of reflected light. The temporal structure of the light, however, is left intact by both: assuming simple surface reflection, photons that arrive first are reflected first. In this paper, we exploit this insight to recover objects outside the line of sight from second-order diffuse reflections, effectively turning walls into mirrors. We formulate the reconstruction task as a linear inverse problem on the transient response of a scene, which we acquire using an affordable setup consisting of a modulated light source and a time-of-flight image sensor. By exploiting sparsity in the reconstruction domain, we achieve resolutions in the order of a few centimeters for object shape (depth and laterally) and albedo. Our method is robust to ambient light and works for large room-sized scenes. It is drastically faster and less expensive than previous approaches using femtosecond lasers and streak cameras, and does not require any moving parts.

Temporal frequency probing for 5D transient analysis of global light transport

Abstract: We analyze light propagation in an unknown scene using projectors and cameras that operate at transient timescales. In this new photography regime, the projector emits a spatio-temporal 3D signal and the camera receives a transformed version of it, determined by the set of all light transport paths through the scene and the time delays they induce. The underlying 3D-to-3D transformation encodes scene geometry and global transport in great detail, but individual transport components (e.g., direct reflections, inter-reflections, caustics, etc.) are coupled nontrivially in both space and time. To overcome this complexity, we observe that transient light transport is always separable in the temporal frequency domain. This makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport. We use this idea in a prototype that offers three never-seen-before abilities: (1) acquiring time-of-flight depth images that are robust to general indirect transport, such as interreflections and caustics; (2) distinguishing between direct views of objects and their mirror reflection; and (3) using a photonic mixer device to capture sharp, evolving wavefronts of "light-in-flight".

Imaging in scattering media using correlation image sensors and sparse convolutional coding

Abstract: Correlation image sensors have recently become popular low-cost devices for time-of-flight, or range cameras. They usually operate under the assumption of a single light path contributing to each pixel. We show that a more thorough analysis of the sensor data from correlation sensors can be used can be used to analyze the light transport in much more complex environments, including applications for imaging through scattering and turbid media. The key of our method is a new convolutional sparse coding approach for recovering transient (light-in-flight) images from correlation image sensors. This approach is enabled by an analysis of sparsity in complex transient images, and the derivation of a new physically-motivated model for transient images with drastically improved sparsity.

From capture to simulation: connecting forward and inverse problems in fluids

Abstract: We explore the connection between fluid capture, simulation and proximal methods, a class of algorithms commonly used for inverse problems in image processing and computer vision. Our key finding is that the proximal operator constraining fluid velocities to be divergence-free is directly equivalent to the pressure-projection methods commonly used in incompressible flow solvers. This observation lets us treat the inverse problem of fluid tracking as a constrained flow problem all while working in an efficient, modular framework. In addition it lets us tightly couple fluid simulation into flow tracking, providing a global prior that significantly increases tracking accuracy and temporal coherence as compared to previous techniques. We demonstrate how we can use these improved results for a variety of applications, such as re-simulation, detail enhancement, and domain modification. We furthermore give an outlook of the applications beyond fluid tracking that our proximal operator framework could enable by exploring the connection of deblurring and fluid guiding.

Compressive multi-mode superresolution display.

Abstract: Compressive displays are an emerging technology exploring the co-design of new optical device configurations and compressive computation. Previously, research has shown how to improve the dynamic range of displays and facilitate high-quality light field or glasses-free 3D image synthesis. In this paper, we introduce a new multi-mode compressive display architecture that supports switching between 3D and high dynamic range (HDR) modes as well as a new super-resolution mode. The proposed hardware consists of readily-available components and is driven by a novel splitting algorithm that computes the pixel states from a target high-resolution image. In effect, the display pixels present a compressed representation of the target image that is perceived as a single, high resolution image.

Methods and apparatus for visual display

Abstract: In exemplary implementations of this invention, light from a backlight is transmitted through two stacked LCDs and then through a diffuser. The front side of the diffuser displays a time-varying sequence of 2D images. Processors execute an optimization algorithm to compute optimal pixel states in the first and second LCDs, respectively, such that for each respective image in the sequence, the optimal pixel states minimize, subject to one or more constraints, a difference between a target image and the respective image. The processors output signals to control actual pixel states in the LCDs, based on the computed optimal pixel states. The 2D images displayed by the diffuser have a higher spatial resolution than the native spatial resolution of the LCDs. Alternatively, the diffuser may be switched off, and the device may display either (a) 2D images with a higher dynamic range than the LCDs, or (b) an automultiscopic display.

A Compressive Multi-Mode Superresolution Display

Abstract: Compressive displays are an emerging technology exploring the co-design of new optical device configurations and compressive computation. Previously, research has shown how to improve the dynamic range of displays and facilitate high-quality light field or glasses-free 3D image synthesis. In this paper, we introduce a new multi-mode compressive display architecture that supports switching between 3D and high dynamic range (HDR) modes as well as a new super-resolution mode. The proposed hardware consists of readily-available components and is driven by a novel splitting algorithm that computes the pixel states from a target high-resolution image. In effect, the display pixels present a compressed representation of the target image that is perceived as a single, high resolution image.

System and method for solving inverse imaging problems

Abstract: There is provided a computer-implemented method for solving inverse imaging problems to compensate for distortions in an image. The method comprises: minimizing a cost objective function containing a data fitting term and one or more image prior terms to each of the plurality of channels, the one or more image prior terms comprising cross-channel information for a plurality of channels derived from the image.

Computational Schlieren Photography with Light Field Probes

Abstract: We introduce a new approach to capturing refraction in transparent media, which we call light field background oriented Schlieren photography. By optically coding the locations and directions of light rays emerging from a light field probe, we can capture changes of the refractive index field between the probe and a camera or an observer. Our prototype capture setup consists of inexpensive off-the-shelf hardware, including inkjet-printed transparencies, lenslet arrays, and a conventional camera. By carefully encoding the color and intensity variations of 4D light field probes, we show how to code both spatial and angular information of refractive phenomena. Such coding schemes are demonstrated to allow for a new, single image approach to reconstructing transparent surfaces, such as thin solids or surfaces of fluids. The captured visual information is used to reconstruct refractive surface normals and a sparse set of control points independently from a single photograph.

A Compressive Superresolution Display

Abstract: In this paper, we introduce a new compressive display architecture for superresolution image presentation that exploits co-design of the optical device configuration and compressive computation Our display allows for superresolution, HDR, or glasses-free 3D presentation

High dynamic range camera having multiple sensors

Abstract: PROBLEM TO BE SOLVED: To allow two or more image sensor arrays to capture pixel data and read out the pixel data from the two or more image sensor arrays.SOLUTION: An electronic camera has two or more image sensor arrays. At least one of the image sensor arrays has a high dynamic range. The camera also has a shutter for selectively allowing light to reach the two or more image sensor arrays, a readout circuit for selectively reading out pixel data from the image sensor arrays, and a controller configured to control the shutter and the readout circuit. The controller has a processor and a memory having computer-readable codes embodied therein. The computer-readable codes, when executed by the processor, cause the controller to open the shutter for an image capture period.

Computational methods for aberration correction in simple lens imaging

Abstract: The complexity of camera optics has greatly increased in recent decades. The lenses of modern single-lens reflex (SLR) cameras may contain a dozen or more individual lens elements, which are used to optimize the light efficiency of optical systems while minimizing the imperfections inherent in them. Geometric distortions, chromatic aberrations, and spherical aberrations are prevalent in simple lens systems and cause blurring and loss of detail. Unfortunately, complex optical designs come at a significant cost and weight. Instead of developing ever more complex optics, we propose1 an alternative approach using much simpler optics of the type used for hundreds of years,2 while correcting for the ensuing aberrations computationally. Although using computational methods for aberration correction has a long history,3–7 these methods are most effective in the removal of residual aberrations found in already well-corrected optical systems. A combination of large-aperture simple lens optics with modern high-resolution image sensors can result in very large wavelength-dependent blur kernels (i.e., point spread functions or PSFs: the response of an imaging system to a point source), with disk-shaped supports of 50–100 pixels diameter (see Figure 1). Such large PSFs destroy high-frequency image information, which cannot be recovered using existing methods. The fundamental insight of our work is that the chromatic part of the lens aberration (which occurs when colors are not focused to the same convergence point) can be used to our advantage, since the wavelength dependence of the blur means that different spatial frequencies are preserved in different color channels. Moreover, combining information from different color channels and enforcing consistency of edges (and similar image features) Figure 1. Calibrated point spread functions (PSFs) for two simple lenses, for different regions on the image sensor. (a) PSF of a 100mm biconvex lens at f/2.0. (b) PSF of a 130mm plano-convex lens at f/4.5.