2D-3D CNN Based Architectures for Spectral Reconstruction from RGB Images

TLDR: This work proposes a 2D convolution neural network and a 3D convolved neural network based approaches for hyperspectral image reconstruction from RGB images that achieves very good performance in terms of MRAE and RMSE.

Abstract: Hyperspectral cameras are used to preserve fine spectral details of scenes that are not captured by traditional RGB cameras that comprehensively quantizes radiance in RGB images. Spectral details provide additional information that improves the performance of numerous image based analytic applications, but due to high hyperspectral hardware cost and associated physical constraints, hyperspectral images are not easily available for further processing. Motivated by the performance of deep learning for various computer vision applications, we propose a 2D convolution neural network and a 3D convolution neural network based approaches for hyperspectral image reconstruction from RGB images. A 2D-CNN model primarily focuses on extracting spectral data by considering only spatial correlation of the channels in the image, while in 3D-CNN model the inter-channel co-relation is also exploited to refine the extraction of spectral data. Our 3D-CNN based architecture achieves very good performance in terms of MRAE and RMSE. In contrast to 3D-CNN, our 2D-CNN based architecture also achieves comparable performance with very less computational complexity.