Hyperspectral images provide higher spectral resolution than typical RGB images by including per-pixel irradiance measurements in a number of narrow bands of wavelength in the visible spectrum. The additional spectral resolution may be useful for many visual tasks, including segmentation, recognition, and relighting. Vision systems that seek to capture and exploit hyperspectral data should benefit from statistical models of natural hyperspectral images, but at present, relatively little is known about their structure. Using a new collection of fifty hyperspectral images of indoor and outdoor scenes, we derive an optimized “spatio-spectral basis” for representing hyperspectral image patches, and explore statistical models for the coefficients in this basis.

/pdf/statistics-of-real-world-hyperspectral-images-3yao09c4ya.pdf

Statistics of real-world hyperspectral images

Hyperspectral imaging is an important visual modality with growing interest and range of applications. The latter, however, is hindered by the fact that existing devices are limited in either spatial, spectral, and/or temporal resolution, while yet being both complicated and expensive. We present a low cost and fast method to recover high quality hyperspectral images directly from RGB. Our approach first leverages hyperspectral prior in order to create a sparse dictionary of hyperspectral signatures and their corresponding RGB projections. Describing novel RGB images via the latter then facilitates reconstruction of the hyperspectral image via the former. A novel, larger-than-ever database of hyperspectral images serves as a hyperspectral prior. This database further allows for evaluation of our methodology at an unprecedented scale, and is provided for the benefit of the research community. Our approach is fast, accurate, and provides high resolution hyperspectral cubes despite using RGB-only input.

/pdf/sparse-recovery-of-hyperspectral-signal-from-natural-rgb-3dta4t2exb.pdf

Sparse Recovery of Hyperspectral Signal from Natural RGB Images

Hyperspectral imaging has many applications from agriculture and astronomy to surveillance and mineralogy. However, it is often challenging to obtain high-resolution (HR) hyperspectral images using existing hyperspectral imaging techniques due to various hardware limitations. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) image and a HR reference image of the same scene. The estimation of the HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the prior knowledge of the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary representing prototype reflectance spectra vectors of the scene is first learned from the input LR image. Specifically, an efficient non-negative dictionary learning algorithm using the block-coordinate descent optimization technique is proposed. Then, the sparse codes of the desired HR hyperspectral image with respect to learned hyperspectral basis are estimated from the pair of LR and HR reference images. To improve the accuracy of non-negative sparse coding, a clustering-based structured sparse coding method is proposed to exploit the spatial correlation among the learned sparse codes. The experimental results on both public datasets and real LR hypspectral images suggest that the proposed method substantially outperforms several existing HR hyperspectral image recovery techniques in the literature in terms of both objective quality metrics and computational efficiency.

Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation

As compared to the conventional RGB or gray-scale images, multispectral images (MSI) can deliver more faithful representation for real scenes, and enhance the performance of many computer vision tasks. In practice, however, an MSI is always corrupted by various noises. In this paper we propose an effective MSI denoising approach by combinatorially considering two intrinsic characteristics underlying an MSI: the nonlocal similarity over space and the global correlation across spectrum. In specific, by explicitly considering spatial self-similarity of an MSI we construct a nonlocal tensor dictionary learning model with a group-block-sparsity constraint, which makes similar full-band patches (FBP) share the same atoms from the spatial and spectral dictionaries. Furthermore, through exploiting spectral correlation of an MSI and assuming over-redundancy of dictionaries, the constrained nonlocal MSI dictionary learning model can be decomposed into a series of unconstrained low-rank tensor approximation problems, which can be readily solved by off-the-shelf higher order statistics. Experimental results show that our method outperforms all state-of-the-art MSI denoising methods under comprehensive quantitative performance measures.

/pdf/decomposable-nonlocal-tensor-dictionary-learning-for-40mjh9ngt3.pdf

Decomposable Nonlocal Tensor Dictionary Learning for Multispectral Image Denoising

Hyperspectral cameras capture images with many narrow spectral channels, which densely sample the electromagnetic spectrum. The detailed spectral resolution is useful for many image analysis problems, but it comes at the cost of much lower spatial resolution. Hyperspectral super-resolution addresses this problem, by fusing a low-resolution hyperspectral image and a conventional high-resolution image into a product of both high spatial and high spectral resolution. In this paper, we propose a method which performs hyperspectral super-resolution by jointly unmixing the two input images into the pure reflectance spectra of the observed materials and the associated mixing coefficients. The formulation leads to a coupled matrix factorisation problem, with a number of useful constraints imposed by elementary physical properties of spectral mixing. In experiments with two benchmark datasets we show that the proposed approach delivers improved hyperspectral super-resolution.

/pdf/hyperspectral-super-resolution-by-coupled-spectral-unmixing-3dmd6bmvlf.pdf

Hyperspectral Super-Resolution by Coupled Spectral Unmixing

We propose the concept of a generalized assorted pixel (GAP) camera, which enables the user to capture a single image of a scene and, after the fact, control the tradeoff between spatial resolution, dynamic range and spectral detail. The GAP camera uses a complex array (or mosaic) of color filters. A major problem with using such an array is that the captured image is severely under-sampled for at least some of the filter types. This leads to reconstructed images with strong aliasing. We make four contributions in this paper: 1) we present a comprehensive optimization method to arrive at the spatial and spectral layout of the color filter array of a GAP camera. 2) We develop a novel algorithm for reconstructing the under-sampled channels of the image while minimizing aliasing artifacts. 3) We demonstrate how the user can capture a single image and then control the tradeoff of spatial resolution to generate a variety of images, including monochrome, high dynamic range (HDR) monochrome, RGB, HDR RGB, and multispectral images. 4) Finally, the performance of our GAP camera has been verified using extensive simulations that use multispectral images of real world scenes. A large database of these multispectral images has been made available at http://wwwl.cs.columbia.edu/ CAVE/projects/gap_camera/ for use by the research community.

/pdf/generalized-assorted-pixel-camera-postcapture-control-of-4mrfe9pr06.pdf

Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum

Exposure bracketing for high dynamic range (HDR) imaging involves capturing several images of the scene at different exposures. If either the camera or the scene moves during capture, the captured images must be registered. Large exposure differences between bracketed images lead to inaccurate registration, resulting in artifacts such as ghosting (multiple copies of scene objects) and blur. We present two techniques, one for image capture (Fibonacci exposure bracketing) and one for image registration (generalized registration), to prevent such motion-related artifacts. Fibonacci bracketing involves capturing a sequence of images such that each exposure time is the sum of the previous N(N > 1) exposures. Generalized registration involves estimating motion between sums of contiguous sets of frames, instead of between individual frames. Together, the two techniques ensure that motion is always estimated between frames of the same total exposure time. This results in HDR images and videos which have both a large dynamic range and minimal motion-related artifacts. We show, by results for several real-world indoor and outdoor scenes, that the proposed approach significantly outperforms several existing bracketing schemes.

/pdf/fibonacci-exposure-bracketing-for-high-dynamic-range-imaging-2gtiz05pvy.pdf

Fibonacci Exposure Bracketing for High Dynamic Range Imaging

PROBLEM TO BE SOLVED: To facilitate the setting of an access right according to the degree of importance of a file. SOLUTION: A character string Fn indicating an access right range and numerical values F1 indicating access right levels are applied to an information file 107 as file attribute information 108, and a character string Pn indicating the access right range and numerical values P1 indicating access right levels are given to a program file 109 as program attribute information 110. When a file access the information file 107 of a higher-order program process 102 is issued, an access control program process 105 allows access to the program file 109 of the higher-order program process 102 and the information file subject to the file access so far as ((Pn=Fn).(P1>=F1))ν(Fn=NULL) is true, wherein symbols '.' and 'ν' indicate AND and OR respectively, and NULL indicates that no character string exists.

Information processing device

Systems, methods, and media for high dynamic range imaging are provided, the systems comprising: an image sensor; and a hardware processor configured to: cause the image sensor to capture first image data having a first exposure time, second image data having a second exposure time, and third image data having a third exposure time that is substantially equal to the sum of the first exposure time and the second exposure time; generate combined image data using the first image data and the second image data.

Methods, systems, and media for high dynamic range imaging

PROBLEM TO BE SOLVED: To provide an imaging apparatus with image sensors segmented in plural areas, which is capable of controlling exposure according to a camera shake amountSOLUTION: The imaging apparatus as an aspect of the disclosure includes: an image sensor section that generates a pixel data of pixels constituting a frame by means of photoelectric conversion; a calculation section that calculates a camera shake amount in each area of the frame which is segmented into plural areas; and a control section that controls the exposure time of the image sensors for each of the areas according to the camera shake amount calculated on each of the areas This disclosure is applicable to, for example, electronic devices such as cameras which include area ADC-type image sensorsSELECTED DRAWING: Figure 1

Daisuke Iso

Papers

Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum

Fibonacci Exposure Bracketing for High Dynamic Range Imaging

Information processing device

Methods, systems, and media for high dynamic range imaging

Imaging apparatus, imaging method, program and reproducing apparatuses