Showing papers on "Tone mapping published in 2016"

Burst photography for high dynamic range and low-light imaging on mobile cameras

Abstract: Cell phone cameras have small apertures, which limits the number of photons they can gather, leading to noisy images in low light. They also have small sensor pixels, which limits the number of electrons each pixel can store, leading to limited dynamic range. We describe a computational photography pipeline that captures, aligns, and merges a burst of frames to reduce noise and increase dynamic range. Our system has several key features that help make it robust and efficient. First, we do not use bracketed exposures. Instead, we capture frames of constant exposure, which makes alignment more robust, and we set this exposure low enough to avoid blowing out highlights. The resulting merged image has clean shadows and high bit depth, allowing us to apply standard HDR tone mapping methods. Second, we begin from Bayer raw frames rather than the demosaicked RGB (or YUV) frames produced by hardware Image Signal Processors (ISPs) common on mobile platforms. This gives us more bits per pixel and allows us to circumvent the ISP's unwanted tone mapping and spatial denoising. Third, we use a novel FFT-based alignment algorithm and a hybrid 2D/3D Wiener filter to denoise and merge the frames in a burst. Our implementation is built atop Android's Camera2 API, which provides per-frame camera control and access to raw imagery, and is written in the Halide domain-specific language (DSL). It runs in 4 seconds on device (for a 12 Mpix image), requires no user intervention, and ships on several mass-produced cell phones.

Blind Quality Assessment of Tone-Mapped Images Via Analysis of Information, Naturalness, and Structure

Abstract: High dynamic range (HDR) imaging techniques have been working constantly, actively, and validly in the fault detection and disease diagnosis in the astronomical and medical fields, and currently they have also gained much more attention from digital image processing and computer vision communities. While HDR imaging devices are starting to have friendly prices, HDR display devices are still out of reach of typical consumers. Due to the limited availability of HDR display devices, in most cases tone mapping operators (TMOs) are used to convert HDR images to standard low dynamic range (LDR) images for visualization. But existing TMOs cannot work effectively for all kinds of HDR images, with their performance largely depending on brightness, contrast, and structure properties of a scene. To accurately measure and compare the performance of distinct TMOs, in this paper develop an effective and efficient no-reference objective quality metric which can automatically assess LDR images created by different TMOs without access to the original HDR images. Our model is shown to be statistically superior to recent full- and no-reference quality measures on the existing tone-mapped image database and a new relevant database built in this work.

Bilateral guided upsampling

Abstract: We present an algorithm to accelerate a large class of image processing operators. Given a low-resolution reference input and output pair, we model the operator by fitting local curves that map the input to the output. We can then produce a full-resolution output by evaluating these low-resolution curves on the full-resolution input. We demonstrate that this faithfully models state-of-the-art operators for tone mapping, style transfer, and recoloring. The curves are computed by lifting the input into a bilateral grid and then solving for the 3D array of affine matrices that best maps input color to output color per x, y, intensity bin. We enforce a smoothness term on the matrices which prevents false edges and noise amplification. We can either globally optimize this energy, or quickly approximate a solution by locally fitting matrices and then enforcing smoothness by blurring in grid space. This latter option reduces to joint bilateral upsampling [Kopf et al. 2007] or the guided filter [He et al. 2013], depending on the choice of parameters. The cost of running the algorithm is reduced to the cost of running the original algorithm at greatly reduced resolution, as fitting the curves takes about 10 ms on mobile devices, and 1--2 ms on desktop CPUs, and evaluating the curves can be done with a simple GPU shader.

High Dynamic Range Video - From Acquisition, to Display and Applications

Abstract: At the time of rapid technological progress and uptake of High Dynamic Range (HDR) video content in numerous sectors, this book provides an overview of the key supporting technologies, discusses the effectiveness of various techniques, reviews the initial standardization efforts and explores new research directions in all aspects involved in HDR video systems. Topics addressed include content acquisition and production, tone mapping and inverse tone mapping operators, coding, quality of experience, and display technologies. This book also explores a number of applications using HDR video technologies in the automotive industry, medical imaging, spacecraft imaging, driving simulation and watermarking.

Gradient-Domain Image Reconstruction Framework with Intensity-Range and Base-Structure Constraints

Abstract: This paper presents a novel unified gradient-domain image reconstruction framework with intensity-range constraint and base-structure constraint. The existing method for manipulating base structures and detailed textures are classifiable into two major approaches: i) gradient-domain and ii) layer-decomposition. To generate detail-preserving and artifact-free output images, we combine the benefits of the two approaches into the proposed framework by introducing the intensity-range constraint and the base-structure constraint. To preserve details of the input image, the proposed method takes advantage of reconstructing the output image in the gradient domain, while the output intensity is guaranteed to lie within the specified intensity range, e.g. 0-to-255, by the intensity-range constraint. In addition, the reconstructed image lies close to the base structure by the base-structure constraint, which is effective for restraining artifacts. Experimental results show that the proposed framework is effective for various applications such as tone mapping, seamless image cloning, detail enhancement, and image restoration.

Multi-camera platform for panoramic real-time HDR video construction and rendering

Abstract: High dynamic range (HDR) images are usually obtained by capturing several images of the scene at different exposures. Previous HDR video techniques adopted the same principle by stacking HDR frames in time domain. We designed a new multi-camera platform which is able to construct and render HDR panoramic video in real time, with $$1{,}024 \times 256$$1,024?256 resolution and a frame rate of 25 fps. We exploit the overlapping fields of view between the cameras with different exposures to create an HDR radiance map. We propose a method for HDR frame reconstruction which merges the previous HDR imaging techniques with the algorithms for panorama reconstruction. The developed FPGA-based processing system is able to reconstruct the HDR frame using the proposed method and tone map the resulting image using a hardware-adapted global operator. The measured throughput of the system is 245 MB/s, which is, up to our knowledge, among the fastest HDR video processing system.

A Gamut-Mapping Framework for Color-Accurate Reproduction of HDR Images.

Abstract: Few tone mapping operators (TMOs) take color management into consideration, limiting compression to luminance values only. This could lead to changes in image chroma and hues, which are typically managed with a post-processing step. However, current post-processing techniques for tone reproduction do not explicitly consider the target display gamut. Gamut mapping, on the other hand, deals with mapping images from one color gamut to another, usually smaller, gamut but has traditionally focused on smaller scale, chromatic changes. The authors present a combined gamut- and tone-management framework for color-accurate reproduction of high dynamic range images that can prevent hue and luminance shifts while taking gamut boundaries into consideration. Their approach is conceptually and computationally simple, parameter-free, and compatible with existing TMOs.

System gamma as a function of image- and monitor-dynamic range.

Abstract: System gamma is the end-to-end exponent that describes the relationship between the relative luminance values at capture and the reproduced image. The system gamma preferred by subjects is known to vary with the background luminance condition and the image in question. We confirm the previous two findings using an image database with both high and low dynamic range images (from 102 to 107), but also find that the preferred system gamma varies with the dynamic range of the monitor (CRT, LCD, or OLED). We find that the preferred system gamma can be predicted in all conditions and for all images by a simple model that searches for the value that best flattens the lightness distribution, where lightness is modeled as a power law of onscreen luminance. To account for the data, the exponent must vary with the viewing conditions. The method presented allows the inference of lightness perception in natural scenes without direct measurement and makes testable predictions for how lightness perception varies with the viewing condition and the distribution of luminance values in a scene. The data from this paper has been made available online.

Puma: A high-quality retinex-based tone mapping operator

Abstract: Tone mapping is the process of compressing high dynamic range (HDR) images to obtain low dynamic range (LDR) images in order to display them on standard display devices. The methods that perform tone mapping also known as tone mapping operators (TMOs) can be global and process all luminances in the same way, or they can be local and process the luminances with respect to their closer neighborhood. While the former tend to be faster, the latter are known to produce results of significantly higher quality. In this paper perceptually-based tone mapping is combined with one of the latest Retinex-based methods to create a high-quality TMO. The new TMO requires only a constant number of steps per pixel and experimental results show that it outperforms all but one state-of-the-art TMOs in terms of tone mapped LDR image quality. The source code is available at http://www.fer.unizg.hr/ipg/resources/color_constancy/.

Methods, apparatus, and systems for extended high dynamic range ("hdr") hdr to hdr tone mapping

Abstract: Aspects of present principles are directed to methods and apparatus for tone mapping a high dynamic range image. The apparatus includes a processor for performing the following and the method includes the following: obtaining a luminance component of the high dynamic range image; determining an HDR to HDR tone mapper curve; determining a tone compressed image by applying the HDR to HDR tone mapper curve to the luminance component of the high dynamic range image; wherein the HDR to HDR tone mapper curve comprises a first part for dark and mid-tone levels, and a second part for highlights.

Tone mapping with contrast preservation and lightness correction in high dynamic range imaging

Abstract: In real-world environments, the human visual system perceives a wide range of luminance in a scene. However, the full range of tones in a high dynamic range (HDR) scene cannot be displayed on standard display devices due to their low dynamic range (LDR). Therefore, tone mapping is necessary to faithfully display a HDR scene on an LDR display device. Existing tone mapping methods have problems because details and contrast in a scene are not preserved faithfully, and they also distort the colors in a scene. Thus, we propose a tone mapping method for preserving the detail in an HDR scene using a weighted least squares filter, which preserves the global contrast in a scene by using a competitive learning neural network, before applying a tone reproduction operator to preserve the color without shifting the lightness. According to the Helmholtz–Kohlrausch effect, the perception of brightness depends on the lightness, chroma, and hue of a color. For example, the luminance of pixels with specific colors such as red and blue is low in an HDR scene. The proposed method corrects the lightness of pixels according to the color (i.e., lightness, chroma, and hue) of a tone-mapped image. Experimental results with several test sets of images demonstrated that the proposed tone mapping method with contrast preservation and lightness correction is more suitable for dynamic range compression than existing tone mapping methods, while it also preserves the color of a scene in an effective way.

Parallel HDR tone mapping and auto-focus on a cellular processor array vision chip

Abstract: To improve computational efficiency, it may be advantageous to transfer part of the intelligence lying in the core of a system to its sensors. Vision sensors equipped with small programmable processors at each pixel allow us to follow this principle in so-called near-focal plane processing, which is performed on-chip directly where light is being collected. Such devices need then only to communicate relevant pre-processed visual information to other parts of the system. In this work, we demonstrate how two classical problems, namely high dynamic range imaging and auto-focus, can be solved efficiently using two simple parallel algorithms implemented on such a chip. We illustrate with these two examples that embedding uncomplicated algorithms on-chip, directly where information acquisition takes place can replace more complex dedicated post-processing. Adapting data acquisition by bringing processing at the sensor level allows us to explore solutions that would not be feasible in a conventional sensor-ADC-processor pipeline.

Evaluation of the effectiveness of HDR tone-mapping operators for photogrammetric applications

Abstract: The ability of High Dynamic Range (HDR) imaging to capture the full range of lighting in a scene has meant that it is being increasingly used for Cultural Heritage (CH) applications. Photogrammetric techniques allow the semi-automatic production of 3D models from a sequence of images. Current photogrammetric methods are not always effective in reconstructing images under harsh lighting conditions, as significant geometric details may not have been captured accurately within under- and over-exposed regions of the image. HDR imaging offers the possibility to overcome this limitation, however the HDR images need to be tone mapped before they can be used within existing photogrammetric algorithms. In this paper we evaluate four different HDR tone-mapping operators (TMOs) that have been used to convert raw HDR images into a format suitable for state-of-the-art algorithms, and in particular keypoint detection techniques. The evaluation criteria used are the number of keypoints, the number of valid matches achieved and the repeatability rate. The comparison considers two local and two global TMOs. HDR data from four CH sites were used: Kaisariani Monastery (Greece), Asinou Church (Cyprus), Château des Baux (France) and Buonconsiglio Castle (Italy).

A Retina Inspired Model for High Dynamic Range Image Rendering

Abstract: We propose a new tone mapping model to render high dynamic range (HDR) images in limited dynamic range devices in this paper. This neural network model is inspired by the retinal information processing mechanisms of the biological visual system, including the adaptive gap junction between horizontal cells (HCs), the negative HC-cone feedback pathway, and the center-surround antagonistic receptive fields of bipolar cells (BCs). The key novelty of the proposed model lies in the adaptive adjustment of the receptive field size of HCs based on the local brightness, which simulates the dynamic gap junction between HCs. This enables the brightness of distinct regions to be recovered into clearly visible ranges while reducing halo artifacts common to other methods. The BCs serve to enhance the local contrast with their center-surround RF structure. By comparing with the state-of-the-art tone mapping methods qualitatively and quantitatively, our method shows competitive performance in term of improving details in both dark and bright areas.

High Dynamic Range Image Analysis through Various Tone Mapping Techniques

Abstract: image quality is improved drastically with the increase of the technology. The conventional display devices may not be suitable for these High dynamic range images. The tone mapping is the process to show the good quality image in the normal LDR display devices. This paper presents a review of the tone mapping algorithms. It provides the methodology on Tone Mapped Image Quality Index (TMIQI) and the Blind Quality Assessment of Tone-Mapped Images (BTMQI). The region is basically expanded and compressed to visualize properly. Thereby the region-enhanced pseudo-exposures are fused into an HDR image. The image quality of BTMQI is comparatively higher than the TMIQI method. The low dynamic range images are suitable to both the conventional and advance display devices. Keywordsdynamic range imaging, structural preservation, tone mapping, perceptual image processing, structural similarity

An evaluation of HDR image matching under extreme illumination changes

Abstract: High dynamic range (HDR) imaging has potential to facilitate computer vision tasks such as image matching where lighting transformations hinder the matching performance. However, little has been done to quantify the gains with different possible HDR representations for vision algorithms like feature extraction. In this paper, we evaluate the performance of the full feature extraction pipeline, including detection and description, on ten different image representations: low dynamic range (LDR), seven different tone mapped (TM) HDR and two HDR imaging (linear and log encoded) representations. We measure the impact of using these different representations for feature matching using mean average precision (mAP) scores on four illumination change datasets. We perform feature extraction using four popular schemes in the literature: SIFT, SURF, BRISK, FREAK. With respect to previous studies, our observations confirm the advantages of HDR over conventional LDR imagery, and the fact that HDR linear values are not appropriate for vision tasks. However, HDR representations that work best for keypoint detection are not necessarily optimal when the full feature extraction is taken into account.

Optimizing tone mapping operators for keypoint detection under illumination changes

Abstract: Tone mapping operators (TMO) have recently raised interest for their capability to handle illumination changes. However, these TMOs are optimized with respect to perception rather than image analysis tasks like key point detection. Moreover, no work has been done to analyze the factors affecting the optimization of TMOs for such tasks. In this paper, we investigate the influence of two factors-Correlation Coefficient (CC) and Repeatability Rate (RR) of the tone mapped images for the optimization of classical Retinex based models to enhance key point detection under illumination changes. CC-based optimized models aim at increasing the similarity of the tone mapped images. Conversely, RR-based optimized models quantify the optimal detection performance gains. By considering two simple Retinex based models, i.e., Gaussian and bilateral filtering, we show that estimating as precisely as possible the illumination, CC-based optimized models do not necessarily bring to optimal key point detection performance. We conclude that, instead, other criteria specific to RR-based optimized models should be taken into account. Moreover, large gains in performance with respect to existing popular TMOs motivate further research towards optimal tone mapping technique for computer vision applications.

Fine-tuning JPEG-XT compression performance using large-scale objective quality testing

Abstract: The upcoming JPEG XT standard for High Dynamic Range (HDR) images defines a common framework for the lossy and lossless representation of high-dynamic range images. It describes the decoding process as the combination of various processing tools that can be combined freely. In this paper we analyze the coding efficiency of different decoding tools through a large scale objective quality testing using the HDR-VDP 2.2 objective metric. This evaluation is performed on a large database of 337 images, testing the effect of global and local tone mapping operators for various configurations, and for multiple combinations of quality parameters. The main findings are that using an inverse tone mapping operator for creating an HDR precursor image works well for global, but not for local operators, and that including refinement scans to increase the bit-depth of the extension layer provides substantial improvements for one of the encoding profiles and higher bit-rates.

Perception-based Histogram Equalization for tone mapping applications

Abstract: Due to the ever increasing commercial availability of High Dynamic Range (HDR) content and displays, backward compatibility of HDR content with Standard Dynamic Range displays is currently a topic of high importance. Over the years, a significant amount of Tone Mapping Operators (TMOs) have been proposed to adapt HDR content to the restricted capabilities of SDR displays. Among them, the Histogram Equalization (HE) is considered to provide good results for a wide set of images. However, the naive application of HE results either in banding artifacts or noise amplification when the HDR image has large unified areas (i.e. sky). In order to differentiate relevant information from noise in a uniform background, or in dark areas, the authors proposed a ceiling function. Their method results in noise-free but dim images. In this paper we propose a novel ceiling function which is based on the Perceptual Quantizer (PQ) function. Our method uses as threshold the number of code-words that PQ assigns on a luminance range in the original HDR image and the corresponding number of code-words in the resulting SDR image. We limit the number of code-words on SDR to be equal or less than the HDR. The saved code-words during the ceiling operation are redistributed to increase the contrast as well as the brightness of the final image. Results shows that provided SDR images are noise-free and brighter than the one obtained with prior HE operators. Finally since the proposed method is a Global TMO, it is thereby of low complexity and suitable for real time applications.

Rate-distortion optimization of a tone mapping with SDR quality constraint for backward-compatible high dynamic range compression

Abstract: This paper addresses the problem of designing a global tone mapping operator for rate-distortion optimized backward compatible compression of HDR images. We consider a two layer coding scheme in which a base SDR layer is coded with HEVC, inverse tone mapped and subtracted from the input HDR signal to yield the enhancement HDR layer. The tone mapping curve design is formulated as the minimization of the distortion on the reconstructed HDR signal under the constraint of a total rate cost on both layers, while preserving a good quality for the SDR signal. We first demonstrate that the optimum tone mapping function only depends on the rate of the base SDR layer and that the minimization problem can be separated in two consecutive minimization steps. Experimental results show that the proposed tone mapping optimization yields the best trade-off between rate-distortion performance and quality preservation of the coded SDR.

Inter-Layer Prediction of Color in High Dynamic Range Image Scalable Compression

Abstract: This paper presents a color inter-layer prediction (ILP) method for scalable coding of high dynamic range (HDR) video content with a low dynamic range (LDR) base layer. Relying on the assumption of hue preservation between the colors of an HDR image and its LDR tone mapped version, we derived equations for predicting the chromatic components of the HDR layer given the decoded LDR layer. Two color representations are studied. In a first encoding scheme, the HDR image is represented in the classical Y’CbCr format. In addition, a second scheme is proposed using a colorspace based on the CIE u’v’ uniform chromaticity scale diagram. In each case, different prediction equations are derived based on a color model ensuring the hue preservation. Our experiments highlight several advantages of using a CIE u’v’-based colorspace for the compression of HDR content, especially in a scalable context. In addition, our ILP scheme using this color representation improves on the state-of-the-art ILP method, which directly predicts the HDR layer u’v’ components by computing the LDR layers u’v’ values of each pixel.

Performance evaluation of high dynamic range image tone mapping operators based on separable non-linear multiresolution families

Abstract: This paper addresses the conversion problem of High Dynamic Range (HDR) images into Low Dynamic Range (LDR) images. In this objective, separable non-linear multiresolution approaches are exploited as Image Tone Mapping Operators (TMOs). They are related on: (i) Essentially Non-Oscillatory (ENO) interpolation strategy developed by Harten namely Point-Value (PV) multiresolution family and Cell-Average (CA) multiresolution family; and (ii) Power-P multiresolution family introduced by Amat. These approaches have the advantage to take into account the singularities, such as edge points of the image, in the mathematical model thus preserving the structural information of the HDR images. Moreover the Gibbs phenomenon, harmful in tone mapped images, is avoided. The quality assessment of the tone mapped images is measured according to the TMQI metric. Simulation results show that the proposed TMOs provide good results compared to traditional TMO strategies.

Objective quality assessment of tone-mapped videos

Abstract: With the fast advances in video acquisition, computational imaging, and display technologies, there has been a growing interest in high dynamic range (HDR) videos. Tone mapping operators (TMOs) that convert HDR content to low dynamic range (LDR) ones provide a practically useful solution for the visualization of HDR videos on standard LDR displays, where the user experience highly depends on the performance of the TMOs being used. Without an appropriate perceptual quality measure, different TMOs cannot be compared. Subjective experiments may be a reliable solution, but is time consuming, expensive, and difficult to be embedded into optimization processes. Here we make one of the first attempts to develop an objective quality assessment model for tone-mapped videos that incorporates structural fidelity, statistical naturalness and memory effect. Validation using subject-rated tone-mapped videos show that the proposed method is well-correlated with subjective scores.

Enhancement of high dynamic range images using variational calculus regularizer with stochastic resonance

Abstract: While capturing pictures by a simple camera in a scene with the presence of harsh or strong lighting like a full sunny day, we often find loss of highlight detail information (overexposure) in the bright regions and loss of shadow detail information (underexposure) in dark regions. In this manuscript, a classical method for retrieval of minute information from the high dynamic range image has been proposed. Our technique is based on variational calculus and dynamic stochastic resonance (DSR). We use a regularizer function, which has been added in order to optimise the correct estimation of the lost details from the overexposed or underexposed region of the image. We suppress the dynamic range of the luminance image by attenuating large gradient with the large magnitude and low gradient with low magnitude. At the same time, dynamic stochastic resonance (DSR) has been used to improve the underexposed region of the image. The experimental results of our proposed technique are capable of enhancing the quality of images in both overexposed and underexposed regions. The proposed technique is compared with most of the state-of-the-art techniques and it has been observed that the proposed technique is better or at most comparable to the existing techniques.

Event-Based Tone Mapping for Asynchronous Time-Based Image Sensor.

Abstract: The asynchronous time-based neuromorphic image sensor ATIS is an array of autonomously operating pixels able to encode luminance information with an exceptionally high dynamic range (143 dB). This paper introduces an event-based methodology to display data from this type of event-based imagers, taking into account the large dynamic range and high temporal accuracy that go beyond available mainstream display technologies. We introduce an event-based tone mapping methodology for asynchronously acquired time encoded gray-level data. A global and a local tone mapping operator are proposed. Both are designed to operate on a stream of incoming events rather than on time frame windows. Experimental results on real outdoor scenes are presented to evaluate the performance of the tone mapping operators in terms of quality, temporal stability, adaptation capability, and computational time.

Tone mastering system with creative intent metadata

Abstract: A method, apparatus, and non-transitory computer readable medium for video tone mapping. The method includes receiving the video and metadata at least partially representative of a global tone mapping function. The method further includes determining the global tone mapping function based at least on the metadata and a characteristic of the display and generating a tone mapped video by applying the global tone mapping function to the video and using the peak luminance value of the display in the application of the global tone mapping function.

Rapid tone mapping system and method based on multi-scale Gauss filters

Abstract: The invention relates to a rapid tone mapping system and method based on multi-scale Gauss filters The system comprises a multi-scale decomposition module, a rough layer module, a detail layer module, a fusion module, a chroma processing module, a gamma correction module and a terminal display module The multi-scale Gauss filters are used to decompose images of high dynamic range to obtain rough images and detail images; the fusion module combines the rough images with the detail images linearly to form novel images of low dynamic range; the chroma processing module compensates chroma information; and the gamma correction module uses gamma correction to compensate the nonlinear relation between input signals and output signals of a display system in advance According to the invention, the image of high dynamic range can be effectively compressed, image information is effectively reserved, the algorithm efficiency is high, consumption time is short, and the tone mapping system and method are suitable for mobile phones

Towards HDR Based Facial Expression Recognition under Complex Lighting

Abstract: A Facial Expression Recognition (FER) study is conducted investigating whether High Dynamic Range (HDR) tone mapped images can improve FER performance under complex lighting conditions. For this purpose, we created a new straightforward facial expression dataset of HDR images, a collection of faces under different lighting contrasts. Our approach applies a machine learning method, Support Vector Machines (SVMs) to texture-based, Local Binary Pattern (LBP) and appearance based, Speeded-Up Robust Feature (SURF) image representations to conduct emotion recognition experiments. We run comparisons using HDR tone mapped images and more traditional pre-processing methods of reducing the impact of large lighting variations. The best of the adopted methods achieved 75% and 79.8% accuracy which is significantly better than the more traditional pre-processing methods tested.