Inferring a high dynamic range (HDR) image from a single low dynamic range (LDR) input is an ill-posed problem where we must compensate lost data caused by under-/over-exposure and color quantization. To tackle this, we propose the first deep-learning-based approach for fully automatic inference using convolutional neural networks. Because a naive way of directly inferring a 32-bit HDR image from an 8-bit LDR image is intractable due to the difficulty of training, we take an indirect approach; the key idea of our method is to synthesize LDR images taken with different exposures (i.e., bracketed images) based on supervised learning, and then reconstruct an HDR image by merging them. By learning the relative changes of pixel values due to increased/decreased exposures using 3D deconvolutional networks, our method can reproduce not only natural tones without introducing visible noise but also the colors of saturated pixels. We demonstrate the effectiveness of our method by comparing our results not only with those of conventional methods but also with ground-truth HDR images.

Deep reverse tone mapping

We study the problem of multi-focus image fusion, where the key challenge is detecting the focused regions accurately among multiple partially focused source images. Inspired by the conditional generative adversarial network (cGAN) to image-to-image task, we propose a novel FuseGAN to fulfill the images-to-image for multi-focus image fusion. To satisfy the requirement of dual input-to-one output, the encoder of the generator in FuseGAN is designed as a Siamese network. The least square GAN objective is employed to enhance the training stability of FuseGAN, resulting in an accurate confidence map for focus region detection. Also, we exploit the convolutional conditional random fields technique on the confidence map to reach a refined final decision map for better focus region detection. Moreover, due to the lack of a large-scale standard dataset, we synthesize a large enough multi-focus image dataset based on a public natural image dataset PASCAL VOC 2012, where we utilize a normalized disk point spread function to simulate the defocus and separate the background and foreground in the synthesis for each image. We conduct extensive experiments on two public datasets to verify the effectiveness of the proposed method. Results demonstrate that the proposed method presents accurate decision maps for focus regions in multi-focus images, such that the fused images are superior to 11 recent state-of-the-art algorithms, not only in visual perception, but also in quantitative analysis in terms of five metrics.

FuseGAN: Learning to Fuse Multi-Focus Image via Conditional Generative Adversarial Network

Multi-scale exposure fusion is an effective image enhancement technique for a high dynamic range (HDR) scene. In this paper, a new multi-scale exposure fusion algorithm is proposed to merge differently exposed low dynamic range (LDR) images by using the weighted guided image filter to smooth the Gaussian pyramids of weight maps for all the LDR images. Details in the brightest and darkest regions of the HDR scene are preserved better by the proposed algorithm without relative brightness change in the fused image. In addition, a new weighted structure tensor is introduced to the differently exposed images and it is adopted to design a detail extraction component for the proposed fusion algorithm, such that users are allowed to manipulate fine details in the enhanced image according to their preference. The proposed multi-scale exposure fusion algorithm is also applied to design a simple single image brightening algorithm for both low-light imaging and back-light imaging.

Detail-Enhanced Multi-Scale Exposure Fusion

High dynamic range images contain luminance information of the physical world and provide more realistic experience than conventional low dynamic range images. Because most images have a low dynamic range, recovering the lost dynamic range from a single low dynamic range image is still prevalent. We propose a novel method for restoring the lost dynamic range from a single low dynamic range image through a deep neural network. The proposed method is the first framework to create high dynamic range images based on the estimated multi-exposure stack using the conditional generative adversarial network structure. In this architecture, we train the network by setting an objective function that is a combination of L1 loss and generative adversarial network loss. In addition, this architecture has a simplified structure than the existing networks. In the experimental results, the proposed network generated a multi-exposure stack consisting of realistic images with varying exposure values while avoiding artifacts on public benchmarks, compared with the existing methods. In addition, both the multi-exposure stacks and high dynamic range images estimated by the proposed method are significantly similar to the ground truth than other state-of-the-art algorithms.

/pdf/deep-recursive-hdri-inverse-tone-mapping-using-generative-3y5bpsoyxw.pdf

Deep Recursive HDRI: Inverse Tone Mapping using Generative Adversarial Networks

Recently, high dynamic range (HDR) imaging has attracted much attention as a technology to reflect human visual characteristics owing to the development of the display and camera technology. This paper proposes a novel deep neural network model that reconstructs an HDR image from a single low dynamic range (LDR) image. The proposed model is based on a convolutional neural network composed of dilated convolutional layers and infers LDR images with various exposures and illumination from a single LDR image of the same scene. Then, the final HDR image can be formed by merging these inference results. It is relatively simple for the proposed method to find the mapping between the LDR and an HDR with a different bit depth because of the chaining structure inferring the relationship between the LDR images with brighter (or darker) exposures from a given LDR image. The method not only extends the range but also has the advantage of restoring the light information of the actual physical world. The proposed method is an end-to-end reconstruction process, and it has the advantage of being able to easily combine a network to extend an additional range. In the experimental results, the proposed method shows quantitative and qualitative improvement in performance, compared with the conventional algorithms.

Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image

New generations of display technologies provide a significantly improved dynamic range compared to conventional display devices. Inverse tone mapping methods have been proposed to convert low dynamic range (LDR) images to HDR ones, and several of them require multiple exposure LDR images of the same scene as inputs. However, the vast majority of LDR images and videos available have only one single exposure. In this paper, we propose a region-based enhancement of the pseudo-exposures to generate an HDR image. First, we present an exposure dependent $S$ curve to convert one LDR image to the pseudo-multiple-exposures. Only certain regions of the pseudo-exposures contain noticeable detail information. We propose a region-based enhancement on the pseudo-exposures to boost details in the most distinct region. Thereby the region-enhanced pseudo-exposures are fused into an HDR image. The fused image thus enhances details in the bright region of the dark image and the dark region of the bright image. Compared with other inverse tone mapped methods, our method generates lower total contrast error measured under the dynamic range independent image quality assessment method in .

Pseudo-Multiple-Exposure-Based Tone Fusion With Local Region Adjustment

Due to recent advances in high dynamic range (HDR) technologies, the ability to display HDR images or videos on conventional LCD devices has become more and more important. Many tone-mapping algorithms have been proposed to meet this end, the choice of which depends on display characteristics such as luminance range, contrast ratio and gamma correction. An ideal HDR tone-mapping processor should have a robust core functionality, high flexibility, and low area consumption, and therefore an ARM-core-based system-on-chip (SOC) platform with a HDR tone-mapping application-specific integrated circuit (ASIC) is suitable for such applications. In this paper, we present a systematic methodology for the development of a tone-mapping processor of optimized architecture using an ARM SOC platform, and illustrate the use of this novel HDR tone-mapping processor for both photographic and gradient compression. Optimization is achieved through four major steps: common module extraction, computation power enhancement, hardware/software partition, and cost function analysis. Based on the proposed scheme, we present an integrated photographic and gradient tone-mapping processor that can be configured for different applications. This newly-developed processor can process 1,024 × 768 images at 60 fps, runs at 100 MHz clock and consumes a core area of 8.1 mm2 under TSMC 0.13 μm technology, resulting in a 50% improvement in speed and area as compared with previously-described processors.

Real-Time Tone-Mapping Processor with Integrated Photographic and Gradient Compression using 0.13 μm Technology on an Arm Soc Platform

As the development in high dynamic range (HDR) video capture technologies, the bit-depth video encoding and decoding has become an interesting topic. In this paper, we show that the real-time HDR video display is possible. A tone mapping based HDR video architecture pipelined with a video CODEC is presented. The HDR video is compressed by the tone mapping processor. The compressed HDR video can be encoded and decoded by the video standards, such as MPEG2, MPEG4 or H.264 for transmission and display. We propose and implement a modified photographic tone mapping algorithm for the tone mapping processor. The required luminance wordlength in the processor is analyzed and the quantization error is estimated. We also develop the digit-by-digit exponent and logarithm hardware architecture for the tone mapping processor. The synthesized results show that our real-time tone mapping processor can process a NTSC video with 720*480 resolution at 30 frames per second.

/pdf/design-and-implementation-of-a-real-time-global-tone-mapping-4nnfarezfy.pdf

Design and Implementation of a Real-Time Global Tone Mapping Processor for High Dynamic Range Video

Due to progress in high dynamic range (HDR) capture technologies, the HDR image or video display on conventional LCD devices has become an important topic. Many tone mapping algorithms are proposed for rendering HDR images on conventional displays, but intensive computation time makes them impractical for video applications. In this paper, we present a real-time block-based gradient domain HDR compression for image or video applications. The gradient domain HDR compression is selected as our tone mapping scheme for its ability to compress and preserve details. We divide one HDR image/frame into several equal blocks and process each by the modified gradient domain HDR compression. The gradients of smaller magnitudes are attenuated less in each block to maintain local contrast and thus expose details. By solving the Poisson equation on the attenuated gradient field block by block, we are able to reconstruct a low dynamic range image. A real-time Discrete Sine Transform (DST) architecture is proposed and developed to solve the Poisson equation. Our synthesis results show that our DST Poisson solver can run at 50 MHz clock and consume area of 9 mm2 under TSMC 0.18 um technology.

/pdf/block-based-gradient-domain-high-dynamic-range-compression-1p33oe4hv2.pdf

Block-Based Gradient Domain High Dynamic Range Compression Design for Real-Time Applications

New generations of display technologies provide signicantly improved dynamic range over conventional display devices. Current inverse tone mapping schemes require multiple exposure low dynamic range (LDR) images to generate high dynamic range (HDR) images. In this work, we propose an exposure dependent S curve to convert one optimized LDR image to multiple images with different brightness, which are then fused into a virtual real scene HDR image with wide dynamic range. According to our implementation results, the dynamic range can reach about 105. The synthesizer is robust and temporally coherent, and does not require image specific parameter adjustment. This paper also presents the image quality assessment with HDR visual difference predictor (HDR-VDP) and relative entropy contrast, and our work has better performance than other inverse tone mapping operators (iTMOs) for the both image quality assessments.

Tsun-Hsien Wang

Papers

Pseudo-Multiple-Exposure-Based Tone Fusion With Local Region Adjustment

Real-Time Tone-Mapping Processor with Integrated Photographic and Gradient Compression using 0.13 μm Technology on an Arm Soc Platform

Design and Implementation of a Real-Time Global Tone Mapping Processor for High Dynamic Range Video

Block-Based Gradient Domain High Dynamic Range Compression Design for Real-Time Applications

Low visual difference virtual high dynamic range image synthesizer from a single legacy image