Showing papers on "Light field published in 2020"

Topological Funneling of Light

Abstract: Dissipation is a general feature of non-Hermitian systems. But rather than being an unavoidable nuisance, non-Hermiticity can be precisely controlled and hence used for sophisticated applications, such as optical sensors with enhanced sensitivity. In our work, we implement a non-Hermitian photonic mesh lattice by tailoring the anisotropy of the nearest-neighbor coupling. The appearance of an interface results in a complete collapse of the entire eigenmode spectrum, leading to an exponential localization of all modes at the interface. As a consequence, any light field within the lattice travels toward this interface, irrespective of its shape and input position. On the basis of this topological phenomenon, called the "non-Hermitian skin effect," we demonstrate a highly efficient funnel for light.

Immersive light field video with a layered mesh representation

Abstract: We present a system for capturing, reconstructing, compressing, and rendering high quality immersive light field video. We accomplish this by leveraging the recently introduced DeepView view interpolation algorithm, replacing its underlying multi-plane image (MPI) scene representation with a collection of spherical shells that are better suited for representing panoramic light field content. We further process this data to reduce the large number of shell layers to a small, fixed number of RGBA+depth layers without significant loss in visual quality. The resulting RGB, alpha, and depth channels in these layers are then compressed using conventional texture atlasing and video compression techniques. The final compressed representation is lightweight and can be rendered on mobile VR/AR platforms or in a web browser. We demonstrate light field video results using data from the 16-camera rig of [Pozo et al. 2019] as well as a new low-cost hemispherical array made from 46 synchronized action sports cameras. From this data we produce 6 degree of freedom volumetric videos with a wide 70 cm viewing baseline, 10 pixels per degree angular resolution, and a wide field of view, at 30 frames per second video frame rates. Advancing over previous work, we show that our system is able to reproduce challenging content such as view-dependent reflections, semi-transparent surfaces, and near-field objects as close as 34 cm to the surface of the camera rig.

Resonant phase-matching between a light wave and a free-electron wavefunction

Abstract: Quantum light–matter interactions of bound electron systems have been studied extensively. By contrast, quantum interactions of free electrons with light have only become accessible in recent years, following the discovery of photon-induced near-field electron microscopy (PINEM). So far, the fundamental free electron–light interaction in all PINEM experiments has remained weak due to its localized near-field nature, which imposes an energy–momentum mismatch between electrons and light. Here, we demonstrate a strong interaction between free-electron waves and light waves, resulting from precise energy–momentum phase-matching with the extended propagating light field. By exchanging hundreds of photons with the field, each electron simultaneously accelerates and decelerates in a coherent manner. Consequently, each electron’s quantum wavefunction evolves into a quantized energy comb, spanning a bandwidth of over 1,700 eV, requiring us to extend the PINEM theory. Our observation of coherent electron phase-matching with a propagating wave is a type of inverse-Cherenkov interaction that occurs with a quantum electron wavefunction, demonstrating how the extended nature of the electron wavefunction can alter stimulated electron–light interactions. Energy–momentum phase-matching enables strong interactions between free electrons and light waves. As a result, the wavefunction of the electron exhibits a comb structure, which was observed using photon-induced near-field electron microscopy.

Optical trapping gets structure: Structured light for advanced optical manipulation

Abstract: The pace of innovations in the field of optical trapping has ramped up in the past couple of years. The implementation of structured light, leading to groundbreaking inventions such as high-resolution microscopy or optical communication, has unveiled the unexplored potential for optical trapping. Advancing from a single Gaussian light field as trapping potential, optical tweezers have gotten more and more structure; innovative trapping landscapes have been developed, starting from multiple traps realized by holographic optical tweezers, via complex scalar light fields sculpted in amplitude and phase, up to polarization-structured and highly confined vectorial beams. In this article, we provide a timely overview on recent advances in advanced optical trapping and discuss future perspectives given by the combination of optical manipulation with the emerging field of structured light.

Light Field Saliency Detection With Deep Convolutional Networks

Abstract: Light field imaging presents an attractive alternative to RGB imaging because of the recording of the direction of the incoming light. The detection of salient regions in a light field image benefits from the additional modeling of angular patterns. For RGB imaging, methods using CNNs have achieved excellent results on a range of tasks, including saliency detection. However, it is not trivial to use CNN-based methods for saliency detection on light field images because these methods are not specifically designed for processing light field inputs. In addition, current light field datasets are not sufficiently large to train CNNs. To overcome these issues, we present a new Lytro Illum dataset, which contains 640 light fields and their corresponding ground-truth saliency maps. Compared to current publicly available light field saliency datasets [1] , [2] , our new dataset is larger, of higher quality, contains more variation and more types of light field inputs. This makes our dataset suitable for training deeper networks and benchmarking. Furthermore, we propose a novel end-to-end CNN-based framework for light field saliency detection. Specifically, we propose three novel MAC (Model Angular Changes) blocks to process light field micro-lens images. We systematically study the impact of different architecture variants and compare light field saliency with regular 2D saliency. Our extensive comparisons indicate that our novel network significantly outperforms state-of-the-art methods on the proposed dataset and has desired generalization abilities on other existing datasets.

LFNet: Light Field Fusion Network for Salient Object Detection

Abstract: In this work, we propose a novel light field fusion network-LFNet, a CNNs-based light field saliency model using 4D light field data containing abundant spatial and contextual information. The proposed method can reliably locate and identify salient objects even in a complex scene. Our LFNet contains a light field refinement module (LFRM) and a light field integration module (LFIM) which can fully refine and integrate focusness, depths and objectness cues from light field image. The LFRM learns the light field residual between light field and RGB images for refining features with useful light field cues, and then the LFIM weights each refined light field feature and learns spatial correlation between them to predict saliency maps. Our method can take full advantage of light field information and achieve excellent performance especially in complex scenes, e.g., similar foreground and background, multiple or transparent objects and low-contrast environment. Experiments show our method outperforms the state-of-the-art 2D, 3D and 4D methods across three light field datasets.

Ranging and light field imaging with transparent photodetectors

Abstract: The core of any optical imaging system is a photodetector. Whether it is film or a semiconductor chip in a camera, or indeed the retina in an eye, conventional photodetectors are designed to absorb most of the incident light and record a projected two-dimensional (2D) distribution of light from a scene. The intensity distribution of light from 3D objects, however, can be described by a 4D light field, so optical imaging systems that can acquire higher dimensions of optical information are highly desirable1–3. Here, we report a proof-of-concept light field imaging scheme using transparent graphene photodetector stacks. On a transparent substrate we fabricate a photodetector using graphene as the light-sensing layer, the conducting channel layer, the gate layer and interconnects, enabling sensitive light detection and high transparency at the same time. This technology opens up the possibility of developing sensor arrays that can be stacked along the light path, enabling entirely new configurations of optical imaging devices. We experimentally demonstrate depth ranging using a double stack of transparent detectors and develop a method for computational reconstruction of a 4D light field from a single exposure that can be applied following the successful fabrication of dense 2D transparent sensor arrays. A highly transparent photodetector using graphene as the light-sensing layer, conducting channel layer, gate layer and interconnects enables new approaches for light field photodetection and imaging involving simultaneous detection across multiple focal planes.

A Metric for Light Field Reconstruction, Compression, and Display Quality Evaluation

Abstract: Owning to the recorded light ray distributions, light field contains much richer information and provides possibilities of some enlightening applications, and it has becoming more and more popular. To facilitate the relevant applications, many light field processing techniques have been proposed recently. These operations also bring the loss of visual quality, and thus there is need of a light field quality metric to quantify the visual quality loss. To reduce the processing complexity and resource consumption, light fields are generally sparsely sampled, compressed, and finally reconstructed and displayed to the users. We consider the distortions introduced in this typical light field processing chain, and propose a full-reference light field quality metric. Specifically, we measure the light field quality from three aspects: global spatial quality based on view structure matching, local spatial quality based on near-edge mean square error, and angular quality based on multi-view quality analysis. These three aspects have captured the most common distortions introduced in light field processing, including global distortions like blur and blocking, local geometric distortions like ghosting and stretching, and angular distortions like flickering and sampling. Experimental results show that the proposed method can estimate light field quality accurately, and it outperforms the state-of-the-art quality metrics which may be effective for light field.

Modal analysis of structured light with spatial light modulators: a practical tutorial.

Abstract: A quantitative analysis of optical fields is essential, particularly when the light is structured in some desired manner, or when there is perhaps an undesired structure that must be corrected for. A ubiquitous procedure in the optical community is that of optical mode projections—a modal analysis of light—for the unveiling of amplitude and phase information of a light field. When correctly performed, all the salient features of the field can be deduced with high fidelity, including its orbital angular momentum, vectorial properties, wavefront, and Poynting vector. Here, we present a practical tutorial on how to perform an efficient and effective optical modal decomposition, with emphasis on holographic approaches using spatial light modulators, highlighting the care required at each step of the process.

Spatial-Angular Interaction for Light Field Image Super-Resolution

Abstract: Light field (LF) cameras record both intensity and directions of light rays, and capture scenes from a number of viewpoints. Both information within each perspective (i.e., spatial information) and among different perspectives (i.e., angular information) is beneficial to image super-resolution (SR). In this paper, we propose a spatial-angular interactive network (namely, LF-InterNet) for LF image SR. Specifically, spatial and angular features are first separately extracted from input LFs, and then repetitively interacted to progressively incorporate spatial and angular information. Finally, the interacted features are fused to super-resolve each sub-aperture image. Experimental results demonstrate the superiority of LF-InterNet over the state-of-the-art methods, i.e., our method can achieve high PSNR and SSIM scores with low computational cost, and recover faithful details in the reconstructed images.

Efficient full-path optical calculation of scalar and vector diffraction using the Bluestein method

Abstract: Efficient calculation of the light diffraction in free space is of great significance for tracing electromagnetic field propagation and predicting the performance of optical systems such as microscopy, photolithography, and manipulation. However, existing calculation methods suffer from low computational efficiency and poor flexibility. Here, we present a fast and flexible calculation method for computing scalar and vector diffraction in the corresponding optical regimes using the Bluestein method. The computation time can be substantially reduced to the sub-second level, which is 105 faster than that achieved by the direct integration approach (~hours level) and 102 faster than that achieved by the fast Fourier transform method (~minutes level). The high efficiency facilitates the ultrafast evaluation of light propagation in diverse optical systems. Furthermore, the region of interest and the sampling numbers can be arbitrarily chosen, endowing the proposed method with superior flexibility. Based on these results, full-path calculation of a complex optical system is readily demonstrated and verified by experimental results, laying a foundation for real-time light field analysis for realistic optical implementation such as imaging, laser processing, and optical manipulation. A fast and flexible procedure for evaluating the propagation of light in optical systems is achieved by calculating the diffraction of the light using a computational process called the Bluestein method. It yields information along the entire optical path length on both ‘scalar’ variations in the general magnitude of light waves and ‘vector’ – directional – variations. Researchers led by Jiawen Li and Dong Wu at the University of Science and Technology of China, developed the process and demonstrated that it can deliver results between 100 and 100,000 times faster than two existing alternative methods. Understanding the influence of diffraction in prevailing conditions is important for predicting the fine behaviuor of light waves. The new procedure should lead to greatly improved real-time analyses that will assist in microscopy, laser-based fabrication and optical manipulation technologies.

X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation

Abstract: We suggest to represent an X-Field -a set of 2D images taken across different view, time or illumination conditions, i.e., video, light field, reflectance fields or combinations thereof-by learning a neural network (NN) to map their view, time or light coordinates to 2D images. Executing this NN at new coordinates results in joint view, time or light interpolation. The key idea to make this workable is a NN that already knows the "basic tricks" of graphics (lighting, 3D projection, occlusion) in a hard-coded and differentiable form. The NN represents the input to that rendering as an implicit map, that for any view, time, or light coordinate and for any pixel can quantify how it will move if view, time or light coordinates change (Jacobian of pixel position with respect to view, time, illumination, etc.). Our X-Field representation is trained for one scene within minutes, leading to a compact set of trainable parameters and hence real-time navigation in view, time and illumination.

Tunable photo-responsive elastic metamaterials

Abstract: The metamaterial paradigm has allowed an unprecedented space-time control of various physical fields, including elastic and acoustic waves. Despite the wide variety of metamaterial configurations proposed so far, most of the existing solutions display a frequency response that cannot be tuned, once the structures are fabricated. Few exceptions include systems controlled by electric or magnetic fields, temperature, radio waves and mechanical stimuli, which may often be unpractical for real-world implementations. To overcome this limitation, we introduce here a polymeric 3D-printed elastic metamaterial whose transmission spectrum can be deterministically tuned by a light field. We demonstrate the reversible doubling of the width of an existing frequency band gap upon selective laser illumination. This feature is exploited to provide an elastic-switch functionality with a one-minute lag time, over one hundred cycles. In perspective, light-responsive components can bring substantial improvements to active devices for elastic wave control, such as beam-splitters, switches and filters. Here, the authors present a light-responsive elastic metamaterial whose transmission spectrum can be tuned by light stimuli. More specifically, we demonstrate that an appropriate laser illumination is effective in reversibly widening an existing frequency band gap, doubling its initial value.

Temporal modes in quantum optics: then and now

Abstract: We review the concepts of temporal modes (TMs) in quantum optics, highlighting Roy Glauber's crucial and historic contributions to their development, and their growing importance in quantum information science. TMs are orthogonal sets of wave packets that can be used to represent a multimode light field. They are temporal counterparts to transverse spatial modes of light and play analogous roles - decomposing multimode light into the most natural basis for isolating statistically independent degrees of freedom. We discuss how TMs were developed to describe compactly various processes: superfluorescence, stimulated Raman scattering, spontaneous parametric down conversion, and spontaneous four-wave mixing. TMs can be manipulated, converted, demultiplexed, and detected using nonlinear optical processes such as three-wave mixing and quantum optical memories. As such, they play an increasingly important role in constructing quantum information networks.

Learning Light Field Angular Super-Resolution via a Geometry-Aware Network

Abstract: The acquisition of light field images with high angular resolution is costly. Although many methods have been proposed to improve the angular resolution of a sparsely-sampled light field, they always focus on the light field with a small baseline, which is captured by a consumer light field camera. By making full use of the intrinsic \textit{geometry} information of light fields, in this paper we propose an end-to-end learning-based approach aiming at angularly super-resolving a sparsely-sampled light field with a large baseline. Our model consists of two learnable modules and a physically-based module. Specifically, it includes a depth estimation module for explicitly modeling the scene geometry, a physically-based warping for novel views synthesis, and a light field blending module specifically designed for light field reconstruction. Moreover, we introduce a novel loss function to promote the preservation of the light field parallax structure. Experimental results over various light field datasets including large baseline light field images demonstrate the significant superiority of our method when compared with state-of-the-art ones, i.e., our method improves the PSNR of the second best method up to 2 dB in average, while saves the execution time 48$\times$. In addition, our method preserves the light field parallax structure better.

Fast Depth Estimation for Light Field Cameras

Abstract: Fast depth estimation for light field images is an important task for multiple applications such as image-based rendering and refocusing. Most previous approaches to light field depth estimation involve high computational costs. Therefore, in this study, we propose a fast depth estimation method based on multi-view stereo matching for light field images. Similar to other conventional methods, our method consists of initial depth estimation and refinement. For the initial estimation, we use a one-bit feature for each pixel and calculate matching costs by summing all combinations of viewpoints with a fast algorithm. To reduce computational time, we introduce an offline viewpoint selection strategy and cost volume interpolation. Our refinement process solves the minimization problem in which the objective function consists of $\ell _{1}$ data and smoothness terms. Although this problem can be solved via a graph cuts algorithm, it is computationally expensive; therefore, we propose an approximate solver based on a fast-weighted median filter. Experiments on synthetic and real-world data show that our method achieves competitive accuracy with the shortest computational time of all methods.

Unsupervised Monocular Depth Estimation From Light Field Image

Abstract: Learning based depth estimation from light field has made significant progresses in recent years. However, most existing approaches are under the supervised framework, which requires vast quantities of ground-truth depth data for training. Furthermore, accurate depth maps of light field are hardly available except for a few synthetic datasets. In this paper, we exploit the multi-orientation epipolar geometry of light field and propose an unsupervised monocular depth estimation network. It predicts depth from the central view of light field without any ground-truth information. Inspired by the inherent depth cues and geometry constraints of light field, we then introduce three novel unsupervised loss functions: photometric loss, defocus loss and symmetry loss. We have evaluated our method on a public 4D light field synthetic dataset. As the first unsupervised method published in the 4D Light Field Benchmark website, our method can achieve satisfactory performance in most error metrics. Comparison experiments with two state-of-the-art unsupervised methods demonstrate the superiority of our method. We also prove the effectiveness and generality of our method on real-world light-field images.

Learning Light Field Angular Super-Resolution via a Geometry-Aware Network

Abstract: The acquisition of light field images with high angular resolution is costly. Although many methods have been proposed to improve the angular resolution of a sparsely-sampled light field, they always focus on the light field with a small baseline, which is captured by a consumer light field camera. By making full use of the intrinsic \textit{geometry} information of light fields, in this paper we propose an end-to-end learning-based approach aiming at angularly super-resolving a sparsely-sampled light field with a large baseline. Our model consists of two learnable modules and a physically-based module. Specifically, it includes a depth estimation module for explicitly modeling the scene geometry, a physically-based warping for novel views synthesis, and a light field blending module specifically designed for light field reconstruction. Moreover, we introduce a novel loss function to promote the preservation of the light field parallax structure. Experimental results over various light field datasets including large baseline light field images demonstrate the significant superiority of our method when compared with state-of-the-art ones, i.e., our method improves the PSNR of the second best method up to 2 dB in average, while saves the execution time 48$\times$. In addition, our method preserves the light field parallax structure better.

Turbulence aberration correction for vector vortex beams using deep neural networks on experimental data.

Abstract: The vector vortex beams (VVB) possessing non-separable states of light, in which polarization and orbital angular momentum (OAM) are coupled, have attracted more and more attentions in science and technology, due to the unique nature of the light field. However, atmospheric transmission distortion is a recurring challenge hampering the practical application, such as communication and imaging. In this work, we built a deep learning based adaptive optics system to compensate the turbulence aberrations of the vector vortex mode in terms of phase distribution and mode purity. A turbulence aberration correction convolutional neural network (TACCNN) model, which can learn the mapping relationship of intensity profile of the distorted vector vortex modes and the turbulence phase generated by first 20 Zernike modes, is well designed. After supervised learning plentiful experimental samples, the TACCNN model compensates turbulence aberration for VVB quickly and accurately. For the first time, experimental results show that through correction, the mode purity of the distorted VVB improves from 19% to 70% under the turbulence strength of D/r0 = 5.28 with correction time 100 ms. Furthermore, both spatial modes and the light intensity distribution can be well compensated in different atmospheric turbulence.

Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion.

Abstract: A densely-sampled light field (LF) is highly desirable in various applications. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.

Light Field Salient Object Detection: A Review and Benchmark.

Abstract: Salient object detection (SOD) is a long-standing research topic in computer vision and has drawn an increasing amount of research interest in the past decade. This paper provides the first comprehensive review and benchmark for light field SOD, which has long been lacking in the saliency community. Firstly, we introduce preliminary knowledge on light fields, including theory and data forms, and then review existing studies on light field SOD, covering ten traditional models, seven deep learning-based models, one comparative study, and one brief review. Existing datasets for light field SOD are also summarized with detailed information and statistical analyses. Secondly, we benchmark seven representative light field SOD models together with several cutting-edge RGB-D SOD models on four widely used light field datasets, from which insightful discussions and analyses, including a comparison between light field SOD and RGB-D SOD models, are achieved. Besides, due to the inconsistency of datasets in their current forms, we further generate complete data and supplement focal stacks, depth maps and multi-view images for the inconsistent datasets, making them consistent and unified. Our supplemental data makes a universal benchmark possible. Lastly, because light field SOD is quite a special problem attributed to its diverse data representations and high dependency on acquisition hardware, making it differ greatly from other saliency detection tasks, we provide nine hints into the challenges and future directions, and outline several open issues. We hope our review and benchmarking could serve as a catalyst to advance research in this field. All the materials including collected models, datasets, benchmarking results, and supplemented light field datasets will be publicly available on our project site this https URL.

Light Field Super-Resolution Using a Low-Rank Prior and Deep Convolutional Neural Networks

Abstract: Light field imaging has recently known a regain of interest due to the availability of practical light field capturing systems that offer a wide range of applications in the field of computer vision. However, capturing high-resolution light fields remains technologically challenging since the increase in angular resolution is often accompanied by a significant reduction in spatial resolution. This paper describes a learning-based spatial light field super-resolution method that allows the restoration of the entire light field with consistency across all angular views. The algorithm first uses optical flow to align the light field and then reduces its angular dimension using low-rank approximation. We then consider the linearly independent columns of the resulting low-rank model as an embedding, which is restored using a deep convolutional neural network (DCNN). The super-resolved embedding is then used to reconstruct the remaining views. The original disparities are restored using inverse warping where missing pixels are approximated using a novel light field inpainting algorithm. Experimental results show that the proposed method outperforms existing light field super-resolution algorithms, achieving PSNR gains of 0.23 dB over the second best performing method. The performance is shown to be further improved using iterative back-projection as a post-processing step.

Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network

Abstract: This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.

Single Molecule Light Field Microscopy

Abstract: We introduce single molecule light field microscopy (SMLFM), a new class of three-dimensional (3D) single molecule localization microscopy. By segmenting the back focal plane of a microscope objective with an array of microlenses to generate multiple 2D perspective views, the same single fluorophore can be imaged from different angles. These views, in combination with a bespoke fitting algorithm, enable the 3D positions of single fluorophores to be determined from parallax. SMLFM achieves up to 20 nm localization precision throughout an extended 6µm depth of field. The capabilities of SMLFM are showcased by imaging membranes of fixed eukaryotic cells and DNA nanostructures below the optical diffraction limit.

Learning Implicit Surface Light Fields

Abstract: Implicit representations of 3D objects have recently achieved impressive results on learning-based 3D reconstruction tasks. While existing works use simple texture models to represent object appearance, photo-realistic image synthesis requires reasoning about the complex interplay of light, geometry and surface properties. In this work, we propose a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field. In contrast to existing representations, our implicit model represents surface light fields in a continuous fashion and independent of the geometry. Moreover, we condition the surface light field with respect to the location and color of a small light source. Compared to traditional surface light field models, this allows us to manipulate the light source and relight the object using environment maps. We further demonstrate the capabilities of our model to predict the visual appearance of an unseen object from a single real RGB image and corresponding 3D shape information. As evidenced by our experiments, our model is able to infer rich visual appearance including shadows and specular reflections. Finally, we show that the proposed representation can be embedded into a variational auto-encoder for generating novel appearances that conform to the specified illumination conditions.

High-fidelity View Synthesis for Light Field Imaging With Extended Pseudo 4DCNN

Abstract: Multi-view properties of light field (LF) imaging enable exciting applications such as auto-refocusing, depth estimation and 3D reconstruction. However, limited angular resolution has become the main bottleneck of microlens-based plenoptic cameras towards more practical vision applications. Existing view synthesis methods mainly break the task into two steps, i.e. depth estimating and view warping, which are usually inefficient and produce artifacts over depth ambiguities. We have proposed an end-to-end deep learning framework named Pseudo 4DCNN to solve these problems in a conference paper. Rethinking on the overall paradigm, we further extend pseudo 4DCNN and propose a novel loss function which is applicable for all tasks of light field reconstruction i.e. EPI Structure Preserving (ESP) loss function. This loss function is proposed to attenuate the blurry edges and artifacts caused by averaging effect of ${L_2}$ norm based loss function. Furthermore, the extended Pseudo 4DCNN is compared with recent state-of-the-art (SOTA) approaches on more publicly available light field databases, as well as self-captured light field biometrics and microscopy datasets. Experimental results demonstrate that the proposed framework can achieve better performances than vanilla Pseudo 4DCNN and other SOTA methods, especially in the terms of visual quality under occlusions. The source codes and self-collected datasets for reproducibility will be available online soon.

Light Field Super-Resolution By Jointly Exploiting Internal and External Similarities

Abstract: Light field images taken by plenoptic cameras often have a tradeoff between spatial and angular resolutions. In this paper, we propose a novel spatial super-resolution approach for light field images by jointly exploiting internal and external similarities. The internal similarity refers to the correlations across the angular dimensions of the 4D light field itself, while the external similarity refers to the cross-scale correlations learned from an external light field dataset. Specifically, we advance the classic projection-based method that exploits the internal similarity by introducing the intensity consistency checking criterion and a back-projection refinement, while the external correlation is learned by a CNN-based method which aggregates all warped high-resolution sub-aperture images upsampled from the low-resolution input using a single image super-resolution method. By analyzing the error distributions of the above two methods and investigating the upperbound of combining them, we find that the internal and external similarities are complementary to each other. Accordingly, we further propose a pixel-wise adaptive fusion network to take advantage of both their merits by learning a weighting matrix. Experimental results on both synthetic and real-world light field datasets validate the superior performance of the proposed approach over the state-of-the-arts.

Baseline and Triangulation Geometry in a Standard Plenoptic Camera

Abstract: In this paper, we demonstrate light field triangulation to determine depth distances and baselines in a plenoptic camera. Advances in micro lenses and image sensors have enabled plenoptic cameras to capture a scene from different viewpoints with sufficient spatial resolution. While object distances can be inferred from disparities in a stereo viewpoint pair using triangulation, this concept remains ambiguous when applied in the case of plenoptic cameras. We present a geometrical light field model allowing the triangulation to be applied to a plenoptic camera in order to predict object distances or specify baselines as desired. It is shown that distance estimates from our novel method match those of real objects placed in front of the camera. Additional benchmark tests with an optical design software further validate the model's accuracy with deviations of less than +-0.33 % for several main lens types and focus settings. A variety of applications in the automotive and robotics field can benefit from this estimation model.

Variable transformation of singular cylindrical vector beams using anisotropic crystals.

Abstract: We demonstrated and investigated, both theoretically and experimentally, the transformation of cylindrical vector beams with an embedded phase singularity under the condition of focusing perpendicularly to the axis of the anisotropic calcite crystal. Theoretical and numerical analysis, performed on the basis of decomposing the light field into a set of plane waves for an anisotropic medium, allowed us to study the dependence of the structural transformation of the initial laser beam on the polarisation and phase state in detail. The proposed approach allows one to perform the visual recognition of cylindrically-polarised vector beams of various orders and can be used for the demultiplexing of information channels in the case of polarisation-division multiplexing. The experimentally-obtained results agree with the theoretical findings and demonstrate the reliability of the approach.

Stationary optomechanical entanglement between a mechanical oscillator and its measurement apparatus

Abstract: We provide an argument to infer stationary entanglement between light and a mechanical oscillator based on continuous measurement of light only. We propose an experimentally realizable scheme involving an optomechanical cavity driven by a resonant, continuous-wave field operating in the non-sideband-resolved regime. This corresponds to the conventional configuration of an optomechanical position or force sensor. We show analytically that entanglement between the mechanical oscillator and the output field of the optomechanical cavity can be inferred from the measurement of squeezing in (generalized) Einstein-Podolski-Rosen quadratures of suitable temporal modes of the stationary light field. Squeezing can reach levels of up to 50% of noise reduction below shot noise in the limit of large quantum cooperativity. Remarkably, entanglement persists even in the opposite limit of small cooperativity. Viewing the optomechanical device as a position sensor, entanglement between mechanics and light is an instance of object-apparatus entanglement predicted by quantum measurement theory.