Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moire or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

Deep joint demosaicking and denoising

In recent years, many applications using a set of red-green-blue (RGB) and near-infrared (NIR) images, also called an RGB-NIR image, have been proposed. However, RGB-NIR imaging, i.e., simultaneous acquisition of RGB and NIR images, is still a laborious task because existing acquisition systems typically require two sensors or shots. In contrast, single-sensor RGB-NIR imaging using an RGB-NIR sensor, which is composed of a mosaic of RGB and NIR pixels, provides a practical and low-cost way of one-shot RGB-NIR image acquisition. In this paper, we investigate high-quality system designs for single-sensor RGB-NIR imaging. We first present a system evaluation framework using a new hyperspectral image data set we constructed. Different from existing work, our framework takes both the RGB-NIR sensor characteristics and the RGB-NIR imaging pipeline into account. Based on the evaluation framework, we then design each imaging factor that affects the RGB-NIR imaging quality and propose the best-performed system design. We finally present the configuration of our developed prototype RGB-NIR camera, which was implemented based on the best system design, and demonstrate several potential applications using the prototype.

Single-Sensor RGB-NIR Imaging: High-Quality System Design and Prototype Implementation

A conventional camera performs various signal processing steps sequentially to reconstruct an image from a raw Bayer image. When performing these processing in multiple stages the residual error from each stage accumulates in the image and degrades the quality of the final reconstructed image. In this paper, we present a fully convolutional neural network (CNN) to perform defect pixel correction, denoising, white balancing, exposure correction, demosaicing, color transform, and gamma encoding. To our knowledge, this is the first CNN trained end-to-end to perform the entire image signal processing pipeline in a camera. The neural network was trained using a large image database of raw Bayer images. Through extensive experiments, we show that the proposed CNN based image signal processing system performs better than the conventional signal processing pipelines that perform the processing sequentially.

Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing

Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data. In this paper we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from RAW images, thus enabling the use of real data. In addition we show that for this application fine-tuning a network to a specific burst improves the quality of restoration for both demosaicking and denoising.

Joint Demosaicking and Denoising by Fine-Tuning of Bursts of Raw Images

Joint demosaicing and denoising by overfitting of bursts of raw images.

Most demosaicking algorithms only focus on handling noise-free CFA raw data. In practice, the CFA raw data are corrupted by noise, which degrades demosaicking performance. Full-color image quality strongly depends on the performance of the demosaicking. Here, we propose a CFA raw data denoising algorithm. In the proposed algorithm, the CFA raw data is converted to a pseudo four-channel image by rearranging pixels. Then, the four-channel data are transformed based on the principal component analysis (PCA). Existing high-performance gray image denoising algorithm is applied to each transformed image. Finally, the denoised data is rearranged to obtain denoised CFA raw data. We evaluate both the denoised CFA raw data as well as the full-color image reconstructed with the noisy CFA raw data. Experimental comparisons demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms.

Hiroki Akiyama

Papers

Pseudo four-channel image denoising for noisy CFA raw data