Camera sensors can only capture a limited range of luminance simultaneously, and in order to create high dynamic range (HDR) images a set of different exposures are typically combined. In this paper we address the problem of predicting information that have been lost in saturated image areas, in order to enable HDR reconstruction from a single exposure. We show that this problem is well-suited for deep learning algorithms, and propose a deep convolutional neural network (CNN) that is specifically designed taking into account the challenges in predicting HDR values. To train the CNN we gather a large dataset of HDR images, which we augment by simulating sensor saturation for a range of cameras. To further boost robustness, we pre-train the CNN on a simulated HDR dataset created from a subset of the MIT Places database. We demonstrate that our approach can reconstruct high-resolution visually convincing HDR results in a wide range of situations, and that it generalizes well to reconstruction of images captured with arbitrary and low-end cameras that use unknown camera response functions and post-processing. Furthermore, we compare to existing methods for HDR expansion, and show high quality results also for image based lighting. Finally, we evaluate the results in a subjective experiment performed on an HDR display. This shows that the reconstructed HDR images are visually convincing, with large improvements as compared to existing methods.

HDR image reconstruction from a single exposure using deep CNNs

Inferring a high dynamic range (HDR) image from a single low dynamic range (LDR) input is an ill-posed problem where we must compensate lost data caused by under-/over-exposure and color quantization. To tackle this, we propose the first deep-learning-based approach for fully automatic inference using convolutional neural networks. Because a naive way of directly inferring a 32-bit HDR image from an 8-bit LDR image is intractable due to the difficulty of training, we take an indirect approach; the key idea of our method is to synthesize LDR images taken with different exposures (i.e., bracketed images) based on supervised learning, and then reconstruct an HDR image by merging them. By learning the relative changes of pixel values due to increased/decreased exposures using 3D deconvolutional networks, our method can reproduce not only natural tones without introducing visible noise but also the colors of saturated pixels. We demonstrate the effectiveness of our method by comparing our results not only with those of conventional methods but also with ground-truth HDR images.

Deep reverse tone mapping

Recovering a high dynamic range (HDR) image from a single low dynamic range (LDR) input image is challenging due to missing details in under-/over-exposed regions caused by quantization and saturation of camera sensors. In contrast to existing learning-based methods, our core idea is to incorporate the domain knowledge of the LDR image formation pipeline into our model. We model the HDR-to-LDR image formation pipeline as the (1) dynamic range clipping, (2) non-linear mapping from a camera response function, and (3) quantization. We then propose to learn three specialized CNNs to reverse these steps. By decomposing the problem into specific sub-tasks, we impose effective physical constraints to facilitate the training of individual sub-networks. Finally, we jointly fine-tune the entire model end-to-end to reduce error accumulation. With extensive quantitative and qualitative experiments on diverse image datasets, we demonstrate that the proposed method performs favorably against state-of-the-art single-image HDR reconstruction algorithms.

/pdf/single-image-hdr-reconstruction-by-learning-to-reverse-the-4gk5rm2nhi.pdf

Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline

High dynamic range (HDR) imaging provides the capability of handling real world lighting as opposed to the traditional low dynamic range (LDR) which struggles to accurately represent images with higher dynamic range. However, most imaging content is still available only in LDR. This paper presents a method for generating HDR content from LDR content based on deep Convolutional Neural Networks (CNNs) termed ExpandNet. ExpandNet accepts LDR images as input and generates images with an expanded range in an end-to-end fashion. The model attempts to reconstruct missing information that was lost from the original signal due to quantization, clipping, tone mapping or gamma correction. The added information is reconstructed from learned features, as the network is trained in a supervised fashion using a dataset of HDR images. The approach is fully automatic and data driven; it does not require any heuristics or human expertise. ExpandNet uses a multiscale architecture which avoids the use of upsampling layers to improve image quality. The method performs well compared to expansion/inverse tone mapping operators quantitatively on multiple metrics, even for badly exposed inputs.

/pdf/expandnet-a-deep-convolutional-neural-network-for-high-4lfoaoljnx.pdf

ExpandNet : a deep convolutional neural network for high dynamic range expansion from low dynamic range content

In the last few years, researchers in the field of High Dynamic Range (HDR) Imaging have focused on providing tools for expanding Low Dynamic Range (LDR) content for the generation of HDR images due to the growing popularity of HDR in applications, such as photography and rendering via Image-Based Lighting, and the imminent arrival of HDR displays to the consumer market. LDR content expansion is required due to the lack of fast and reliable consumer level HDR capture for still images and videos. Furthermore, LDR content expansion, will allow the re-use of legacy LDR stills, videos and LDR applications created, over the last century and more, to be widely available. The use of certain LDR expansion methods, those that are based on the inversion of Tone Mapping Operators (TMOs), has made it possible to create novel compression algorithms that tackle the problem of the size of HDR content storage, which remains one of the major obstacles to be overcome for the adoption of HDR. These methods are used in conjunction with traditional LDR compression methods and can evolve accordingly. The goal of this report is to provide a comprehensive overview on HDR Imaging, and an in depth review on these emerging topics.

http://domino.mpi-inf.mpg.de/intranet/ag4/ag4publ.nsf/0/C54556685BC86D61C125755C005A9EBC/$file/HDRIandLDRExpansion.pdf

High Dynamic Range Imaging and Low Dynamic Range Expansion for Generating HDR Content

In recent years many tone mapping operators (TMOs) have been presented in order to display high dynamic range images (HDRI) on typical display devices. TMOs compress the luminance range while trying to maintain contrast. The inverse of tone mapping, inverse tone mapping, expands a low dynamic range image (LDRI) into an HDRI. HDRIs contain a broader range of physical values that can be perceived by the human visual system. We propose a new framework that approximates a solution to this problem. Our framework uses importance sampling of light sources to find the areas considered to be of high luminance and subsequently applies density estimation to generate an expand map in order to extend the range in the high luminance areas using an inverse tone mapping operator. The majority of today’s media is stored in the low dynamic range. Inverse tone mapping operators (iTMOs) could thus potentially revive all of this content for use in high dynamic range display and image based lighting (IBL). Moreover, we show another application that benefits quick capture of HDRIs for use in IBL.

A framework for inverse tone mapping

In this paper we introduce an algorithm and related methods that expand the contrast range of Low Dynamic Range (LDR) videos in order to regenerate missing High Dynamic Range (HDR) data. For content generated from single exposure LDR sequences, this is clearly an under constrained problem. We achieved the expansion by inverting established tone mapping operator, a process we term inverse tone mapping. This approach is augmented by a number of methods which help expand the luminance for the required pixels while avoiding artifacts. These methods may be used to convert the large libraries of available legacy LDR content for use, for instance, on new content-starved HDR devices. Moreover, these same methods may be used to provide animated emissive surfaces for image based lighting (IBL). We demonstrate results for all the above applications and validate the resultant HDR videos with original HDR references using the HDR Visual Difference Predictor (HDR-VDP) image metric.

/pdf/expanding-low-dynamic-range-videos-for-high-dynamic-range-2e5p18j70n.pdf

Expanding low dynamic range videos for high dynamic range applications

In recent years inverse tone mapping techniques have been proposed for enhancing low-dynamic range (LDR) content for a high-dynamic range (HDR) experience on HDR displays, and for image based lighting. In this paper, we present a psychophysical study to evaluate the performance of inverse (reverse) tone mapping algorithms. Some of these techniques are computationally expensive because they need to resolve quantization problems that can occur when expanding an LDR image. Even if they can be implemented efficiently on hardware, the computational cost can still be high. An alternative is to utilize less complex operators; although these may suffer in terms of accuracy. Our study investigates, firstly, if a high level of complexity is needed for inverse tone mapping and, secondly, if a correlation exists between image content and quality. Two main applications have been considered: visualization on an HDR monitor and image-based lighting.

A psychophysical evaluation of inverse tone mapping techniques

A method of producing a compressed video data stream ( 12 ) by compressing a stream of relatively high dynamic range video frames ( 2 ). A bilateral filter ( 3 ) extracts a base frame ( 4 ) with luminance information. The base frame ( 4 ) and the original frame ( 2 ) are used to provide a detail frame ( 5 ) with chroma information. A tone mapping operation ( 6 ) is selected ( 7, 11 ) and applied to the base frame to generate a relatively low dynamic range base frame ( 8 ), which is then compressed ( 9 ). The detail frame ( 5 ) is compressed separately. Final frame data ( 12 ) is then created, consisting of the compressed relatively low dynamic range base frame, the compressed detail frame, and stored information in respect of the tone mapping operation that had been applied to the base frame.

Patrick Ledda

Papers

High Dynamic Range Imaging and Low Dynamic Range Expansion for Generating HDR Content

A framework for inverse tone mapping

Expanding low dynamic range videos for high dynamic range applications

A psychophysical evaluation of inverse tone mapping techniques

Video Data Compression