Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to hallucinate plausible textures, producing results with artifacts in the saturated areas. In this paper, we present a novel learning-based approach to reconstruct an HDR image by recovering the saturated pixels of an input LDR image in a visually pleasing way. Previous deep learning-based methods apply the same convolutional filters on wellexposed and saturated pixels, creating ambiguity during training and leading to checkerboard and halo artifacts. To overcome this problem, we propose a feature masking mechanism that reduces the contribution of the features from the saturated areas. Moreover, we adapt the VGG-based perceptual loss function to our application to be able to synthesize visually pleasing textures. Since the number of HDR images for training is limited, we propose to train our system in two stages. Specifically, we first train our system on a large number of images for image inpainting task and then fine-tune it on HDR reconstruction. Since most of the HDR examples contain smooth regions that are simple to reconstruct, we propose a sampling strategy to select challenging training patches during the HDR fine-tuning stage. We demonstrate through experimental results that our approach can reconstruct visually pleasing HDR results, better than the current state of the art on a wide range of scenes.

Single image HDR reconstruction using a CNN with masked features and perceptual loss

Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to hallucinate plausible textures, producing results with artifacts in the saturated areas. In this paper, we present a novel learning-based approach to reconstruct an HDR image by recovering the saturated pixels of an input LDR image in a visually pleasing way. Previous deep learning-based methods apply the same convolutional filters on well-exposed and saturated pixels, creating ambiguity during training and leading to checkerboard and halo artifacts. To overcome this problem, we propose a feature masking mechanism that reduces the contribution of the features from the saturated areas. Moreover, we adapt the VGG-based perceptual loss function to our application to be able to synthesize visually pleasing textures. Since the number of HDR images for training is limited, we propose to train our system in two stages. Specifically, we first train our system on a large number of images for image inpainting task and then fine-tune it on HDR reconstruction. Since most of the HDR examples contain smooth regions that are simple to reconstruct, we propose a sampling strategy to select challenging training patches during the HDR fine-tuning stage. We demonstrate through experimental results that our approach can reconstruct visually pleasing HDR results, better than the current state of the art on a wide range of scenes.

Single Image HDR Reconstruction Using a CNN with Masked Features and Perceptual Loss

High Dynamic Range (HDR) technology offers a higher visual quality compared to its Standard Dynamic Range (SDR) counterpart, as it tries to imitate the way our eyes perceive brightness and color information. Converting SDR content to HDR format - using inverse Tone Mapping Operators (iTMOs) - to take advantage of the superior visual quality offered by HDR displays, is an attractive proposition to SDR content owners and real-time broadcasters. In this paper, we propose a novel content adaptive iTMO that works in the perceptual domain to model the sensitivity of the human eye to brightness changes in different areas of a scene. To preserve the overall visual impression, our proposed iTMO utilizes an entropy-based brightness segmentation, which also makes our method content adaptive. In addition, we propose a novel perception-based color adjustment method that can maintain the color accuracy between input SDR and generated HDR frames. By performing the color adjustment in the perceptual domain, our iTMO prevents hue shifts and generates HDR colors that closely follow their SDR counterparts. Our subjective evaluations indicate that our proposed method outperforms other state-of-the-art methods by an average of 81% in terms of visual quality, and 76% in terms of how closely the HDR colors match their SDR counterparts. In addition to subjective evaluations, we also performed objective evaluations using the HDR-VDP 2.2 and PU-SSIM metrics and concluded that, on average, our proposed iTMO outperforms the state-of-the-art methods in terms of these two metrics.

A Fully Automatic Content Adaptive Inverse Tone Mapping Operator With Improved Color Accuracy

Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers.

/pdf/vision-models-fine-tuned-by-cinema-professionals-for-high-13dl5jr8va.pdf

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

We propose a novel and efficient algorithm for detection of specular reflections and light sources (highlights) in cinematic content. The detection of highlights is important for reconstructing them properly in the conversion of the low dynamic range (LDR) to high dynamic range (HDR) content. Highlights are often difficult to be distinguished from bright diffuse surfaces, due to their brightness being reduced in the conventional LDR content production. Moreover, the cinematic LDR content is subject to the artistic use of effects that change the apparent brightness of certain image regions (e.g. limiting depth of field, grading, complex multi-lighting setup, etc.). To ensure the robustness of highlights detection to these effects, the proposed algorithm goes beyond considering only absolute brightness and considers five different features. These features are: the size of the highlight relative to the size of the surrounding image structures, the relative contrast in the surrounding of the highlight, its absolute brightness expressed through the luminance (luma feature), through the saturation in the color space (maxRGB feature) and through the saturation in white (minRGB feature). We evaluate the algorithm on two different image data-sets. The first one is a publicly available LDR image data-set without cinematic content, which allows comparison to the broader State of the art. Additionally, for the evaluation on cinematic content, we create an image data-set consisted of manually annotated cinematic frames and real-world images. For the purpose of demonstrating the proposed highlights detection algorithm in a complete LDR-to-HDR conversion pipeline, we additionally propose a simple inverse-tone-mapping algorithm. The experimental analysis shows that the proposed approach outperforms conventional highlights detection algorithms on both image data-sets, achieves high quality reconstruction of the HDR content and is suited for use in LDR-to-HDR conversion.

/pdf/highlights-analysis-system-hans-for-low-dynamic-range-to-4goq7vvvrw.pdf

Highlights Analysis System (HAnS) for Low Dynamic Range to High Dynamic Range Conversion of Cinematic Low Dynamic Range Content

High Dynamic Range (HDR) displays can show images with higher color contrast levels and peak luminosities than the common Low Dynamic Range (LDR) displays However, most existing video content is recorded and/or graded in LDR format To show this LDR content on HDR displays, a dynamic range expansion by using an Inverse Tone Mapped Operator (iTMO) is required In addition to requiring human intervention for tuning, most of the iTMOs don’t consider artistic intentions inherent to the HDR domain Furthermore, the quality of their results decays with peak brightness above 1000 nits In this paper, we propose a fully-automatic inverse tone mapping operator based on mid-level mapping This allows expanding LDR images into HDR with peak brightness over 1000 nits, preserving the artistic intentions inherent to the HDR domain We assessed our results using full-reference objective quality metrics as HDR- VDP-22 and DRIM Experimental results demonstrate that our proposed method outperforms the current state of the art

/pdf/fully-automatic-inverse-tone-mapping-preserving-the-content-2jiwujerb2.pdf

Fully-Automatic Inverse Tone Mapping Preserving the Content Creator’s Artistic Intentions

High Dynamic Range (HDR) displays can show images with higher color contrast levels and peak luminosities than the common Low Dynamic Range (LDR) displays. However, most existing video content is recorded and/or graded in LDR format. To show LDR content on HDR displays, it needs to be up-scaled using a so-called inverse tone mapping algorithm. Several techniques for inverse tone mapping have been proposed in the last years, going from simple approaches based on global and local operators to more advanced algorithms such as neural networks. Some of the drawbacks of existing techniques for inverse tone mapping are the need for human intervention, the high computation time for more advanced algorithms, limited low peak brightness, and the lack of the preservation of the artistic intentions. In this paper, we propose a fully-automatic inverse tone mapping operator based on mid-level mapping capable of real-time video processing. Our proposed algorithm allows expanding LDR images into HDR images with peak brightness over 1000 nits, preserving the artistic intentions inherent to the HDR domain. We assessed our results using the full-reference objective quality metrics HDR-VDP-2.2 and DRIM, and carrying out a subjective pair-wise comparison experiment. We compared our results with those obtained with the most recent methods found in the literature. Experimental results demonstrate that our proposed method outperforms the current state-of-the-art of simple inverse tone mapping methods and its performance is similar to other more complex and time-consuming advanced techniques.

https://biblio.ugent.be/publication/8642785/file/8649059.pdf

Fully-automatic inverse tone mapping algorithm based on dynamic mid-level tone mapping

High Dynamic Range (HDR) television promises to display higher brightness and deeper black levels and thus more vivid and realistic images. However, home video distribution and video broadcasting were historically designed for what we now call standard dynamic range screens (SDR). In order to display SDR content on an HDR screen, it is explicitly or implicitly converted, in a process called inverse tone mapping (iTMO). This paper’s goal is to assess the perceived quality of converted SDR content in comparison to natively graded HDR content. In doing so, this paper aims to enable content creators/distributors to make informed choices between creating/broadcasting HDR content or relying on conversion. To this end, a psychophysical experiment was performed to tests how viewers evaluate the difference between natively graded HDR and a set of SDR to HDR conversion options in a television setup. Results indicate that viewers prefer natively graded HDR content, followed by inverse tone mapping algorithms starting from videos with a compressed dynamic range. When comparing conversion options, users clearly prefer conversion from ‘compressed dynamic range’ SDR over ‘clipped dynamic range’ SDR. Users disliked videos that were naively stretched from standard SDR. In addition, a significant effect of type of sequence was found, with a preference for light scenes with low contrast.

/pdf/an-experimental-study-on-the-perceived-quality-of-natively-24s9qivnkv.pdf

An experimental study on the perceived quality of natively graded versus inverse tone mapped high dynamic range video content on television

Straight cuts are a storytelling tool that is used to direct the viewer’s attention towards the main events of the plot. This research evaluated the effect of the frequency of straight cuts that are controlled by the system (rather than by the viewer) on the viewer’s experience of VR (i.e., presence, immersion, sense of agency, simulator sickness, and enjoyment). We used both quantitative (online questionnaire) and qualitative (semi-structured interviews) methods to examine the viewer’s experience and understanding of a one-minute long VR movie in three versions (i.e., 15 cuts/minute; 8 cuts/minute; no cuts). The study illustrates that sense of agency, enjoyment, and understanding of the story are the main aspects of the viewer’s experience in VR that are influenced by straight cuts. Also, straight cuts focus viewers on events in the story, which in turn helps them understand the plot better. However, the number of cuts must be balanced so that viewers do not feel a lack of sense of agency. Finally, the study highlights that the users’ experience is a result of a dynamic interplay between different facets such as narrative immersion, sense of agency, and presence.

Sven Rousseaux

Papers

Fully-Automatic Inverse Tone Mapping Preserving the Content Creator’s Artistic Intentions

Fully-automatic inverse tone mapping algorithm based on dynamic mid-level tone mapping

An experimental study on the perceived quality of natively graded versus inverse tone mapped high dynamic range video content on television

Can You Make the Cut? Exploring the Effect of Frequency of Cuts in Virtual Reality Storytelling