Showing papers on "Light field published in 2022"
TL;DR: In this article , a nanophotonic light-field camera incorporating a spin-multiplexed bifocal metalens array capable of capturing high-resolution light field images over a record depth-of-field ranging from centimeter to kilometer scale, simultaneously enabling macro and telephoto modes in a snapshot imaging.
Abstract: A unique bifocal compound eye visual system found in the now extinct trilobite, Dalmanitina socialis, may enable them to be sensitive to the light-field information and simultaneously perceive both close and distant objects in the environment. Here, inspired by the optical structure of their eyes, we demonstrate a nanophotonic light-field camera incorporating a spin-multiplexed bifocal metalens array capable of capturing high-resolution light-field images over a record depth-of-field ranging from centimeter to kilometer scale, simultaneously enabling macro and telephoto modes in a snapshot imaging. By leveraging a multi-scale convolutional neural network-based reconstruction algorithm, optical aberrations induced by the metalens are eliminated, thereby significantly relaxing the design and performance limitations on metasurface optics. The elegant integration of nanophotonic technology with computational photography achieved here is expected to aid development of future high-performance imaging systems.
39 citations
TL;DR: This paper proposes a novel formulation built upon Transformers, by treating LFSR as a sequence-to-sequence reconstruction task, and proposes a detail-preserving Transformer (termed as DPT), by leveraging gradient maps of light field to guide the sequence learning.
Abstract: Recently, numerous algorithms have been developed to tackle the problem of light field super-resolution (LFSR), i.e., super-resolving low-resolution light fields to gain high-resolution views. Despite delivering encouraging results, these approaches are all convolution-based, and are naturally weak in global relation modeling of sub-aperture images necessarily to characterize the inherent structure of light fields. In this paper, we put forth a novel formulation built upon Transformers, by treating LFSR as a sequence-to-sequence reconstruction task. In particular, our model regards sub-aperture images of each vertical or horizontal angular view as a sequence, and establishes long-range geometric dependencies within each sequence via a spatial-angular locally-enhanced self-attention layer, which maintains the locality of each sub-aperture image as well. Additionally, to better recover image details, we propose a detail-preserving Transformer (termed as DPT), by leveraging gradient maps of light field to guide the sequence learning. DPT consists of two branches, with each associated with a Transformer for learning from an original or gradient image sequence. The two branches are finally fused to obtain comprehensive feature representations for reconstruction. Evaluations are conducted on a number of light field datasets, including real-world scenes and synthetic data. The proposed method achieves superior performance comparing with other state-of-the-art schemes. Our code is publicly available at: https://github.com/BITszwang/DPT.
38 citations
TL;DR: The UrbanLF dataset as mentioned in this paper contains 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes, which is the largest and most diverse light field dataset for semantic segmentation.
Abstract: As one of the fundamental technologies for scene understanding, semantic segmentation has been widely explored in the last few years. Light field cameras encode the geometric information by simultaneously recording the spatial information and angular information of light rays, which provides us with a new way to solve this issue. In this paper, we propose a high-quality and challenging urban scene dataset, containing 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes. To the best of our knowledge, it is the largest and the most diverse light field dataset for semantic segmentation. We further design two new semantic segmentation baselines tailored for light field and compare them with state-of-the-art RGB, video and RGB-D-based methods using the proposed dataset. The outperforming results of our baselines demonstrate the advantages of the geometric information in light field for this task. We also provide evaluations of super-resolution and depth estimation methods, showing that the proposed dataset presents new challenges and supports detailed comparisons among different methods. We expect this work inspires new research direction and stimulates scientific progress in related fields. The complete dataset is available at https://github.com/HAWKEYE-Group/UrbanLF .
32 citations
22 citations
TL;DR: Jiang et al. as discussed by the authors proposed a coarse-to-fine learning-based method to reconstruct a densely-sampled light field (LF) from a sparsely sampled one, in which a coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampling LF and leverages it to independently synthesize novel SAIs, in order to fuse the information from different input SAIs.
Abstract: A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement. The code is available at https://github.com/jingjin25/LFASR-FS-GAF .
17 citations
TL;DR: In this article , it was shown that the polarisation inhomogeneity that defines vectorial structured light is immune to all such perturbations, provided they are unitary, and that the robustness of vector vortex beams to tilted lenses and atmospheric turbulence, both highly asymmetric aberrations, remains unaltered from the near-field to far-field.
Abstract: Optical aberrations have been studied for centuries, placing fundamental limits on the achievable resolution in focusing and imaging. In the context of structured light, the spatial pattern is distorted in amplitude and phase, often arising from optical imperfections, element misalignment, or even from dynamic processes due to propagation through perturbing media such as living tissue, free-space, underwater and optical fibre. Here we show that the polarisation inhomogeneity that defines vectorial structured light is immune to all such perturbations, provided they are unitary. By way of example, we study the robustness of vector vortex beams to tilted lenses and atmospheric turbulence, both highly asymmetric aberrations, demonstrating that the inhomogeneous nature of the polarisation remains unaltered from the near-field to far-field, even as the structure itself changes. The unitary nature of the channel allows us to undo this change through a simple lossless operation, tailoring light that appears robust in all its spatial structure regardless of the medium. Our insight highlights the overlooked role of measurement in describing classical vectorial light fields, in doing so resolving prior contradictory reports on the robustness of vector beams in complex media. This paves the way to the versatile application of vectorial structured light, even through non-ideal optical systems, crucial in applications such as imaging deep into tissue and optical communication across noisy channels.
16 citations
TL;DR: In this paper , a review of vectorial metasurface holography can be found, from the basic concept to the practical implementation, and vectorial multiplexing with other degrees of freedom, enriching and broadening its applications in both civil and military field.
Abstract: Tailoring light properties using metasurfaces made of optically thin and subwavelength structure arrays has led to a variety of innovative optical components with intriguing functionalities. Transmitted/reflected light field distribution with exquisite nanoscale resolution achievable with metasurfaces has been utilized to encode holographic complex amplitude, leading to arbitrary holographic intensity profile in the plane of interest. Vectorial metasurface holography, which not only controls the intensity profile, but also modifies the polarization distributions of the light field, has recently attracted enormous attention due to their promising applications in photonics and optics. Here, we review the recent progresses of the vectorial metasurface holography, from the basic concept to the practical implementation. Moreover, vectorial metasurfaces can also be multiplexed with other degrees of freedom, such as wavelength and nonlinearity, enriching and broadening its applications in both civil and military field.
15 citations
TL;DR: In this paper , the authors demonstrate a near-infrared spectropolarimeter based on an electrically-tunable liquid crystal metasurface, which acts as an encoder of the vectorial light field and is tailored to support high-quality-factor guided-mode resonances with diverse and anisotropic spectral features.
Abstract: Abstract While conventional photodetectors can only measure light intensity, the vectorial light field contains much richer information, including polarization and spectrum, that are essential for numerous applications ranging from imaging to telecommunication. However, the simultaneous measurement of multi-dimensional light field information typically requires the multiplexing of dispersive or polarization-selective elements, leading to excessive system complexity. Here, we demonstrate a near-infrared spectropolarimeter based on an electrically-tunable liquid crystal metasurface. The tunable metasurface, which acts as an encoder of the vectorial light field, is tailored to support high-quality-factor guided-mode resonances with diverse and anisotropic spectral features, thus allowing the full Stokes parameters and the spectrum of the incident light to be computationally reconstructed with high fidelity. The concept of using a tunable metasurface for multi-dimensional light field encoding may open up new horizons for developing vectorial light field sensors with minimized size, weight, cost, and complexity.
13 citations
01 Jun 2022
TL;DR: NeuRay as mentioned in this paper predicts the visibility of 3D points to input views within the NeuRay representation, which enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality.
Abstract: We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms perscene optimization methods after finetuning. Project page:https://liuyuan-pal.github.io/NeuRay/
11 citations
TL;DR: In this article , a photoelectron-based scheme was proposed to distinguish the orbital angular momentum (OAM) of focused intense femtosecond optical vortices without the modification of light helical phase.
Abstract: With the rapid development of femtosecond lasers, the generation and application of optical vortices have been extended to the regime of intense-light-matter interaction. The characterization of the orbital angular momentum (OAM) of intense vortex pulses is very critical. Here, we propose and demonstrate a novel photoelectron-based scheme that can in situ distinguish the OAM of the focused intense femtosecond optical vortices without the modification of light helical phase. We employ two-color co-rotating intense circular fields in the strong-field photoionization experiment, in which one color light field is a plane wave serving as the probing pulses and the other one is the vortex pulses whose OAM needs to be characterized. We show that by controlling the spatial profile of the probing pulses, the OAM of the vortex pulses can be clearly identified by measuring the corresponding photoelectron momentum distributions or angle-resolved yields. This work provides a novel in situ detection scenario for the light pulse vorticity and has implications for the studies of ultrafast and intense complex light fields with optical OAM.
10 citations
TL;DR: In this article , a light field image quality evaluation index that attempts to extract less information from the focus stack to accurately evaluate the entire light field quality was proposed, where the gradient and phase congruency operators were used in the extraction framework.
Abstract: The large amount of complex scene information recorded by light field imaging has the potential for immersive media applications. Compression and reconstruction algorithms are crucial for the transmission, storage, and display of such massive data. Most of the existing quality evaluation indexes do not effectively account for light field characteristics. To accurately evaluate the distortions caused by compression and reconstruction algorithms, it is necessary to construct an image evaluation index that reflects the angular-spatial characteristics of the light field. This work proposes a full-reference light field image quality evaluation index that attempts to extract less information from the focus stack to accurately evaluate the entire light field quality. The proposed framework includes three specific steps. First, we construct a key refocused image extraction framework by the maximal spatial information contrast and the minimal angular information variation. Specifically, the gradient and phase congruency operators are used in the extraction framework. Second, a novel light field quality evaluation index is built based on the angular-spatial characteristics of the key refocused images. In detail, the features used in the key refocused image extraction framework and the chrominance feature are combined to construct the union feature. Third, the similarity of the union feature is pooled by the relevant visual saliency map to obtain the predicted score. Finally, the overall quality of the light field is measured by applying the proposed index to the key refocused images. The high efficiency and precision of the proposed method are shown by extensive comparison experiments.
01 Jun 2022
TL;DR: In this article , a two-stage transformer-based model is proposed to combine the strengths and mitigates the limitations of these two directions, which learns to represent view-dependent effects accurately.
Abstract: Classical light field rendering for novel view synthesis can accurately reproduce view-dependent effects such as reflection, refraction, and translucency, but requires a dense view sampling of the scene. Methods based on geometric reconstruction need only sparse views, but cannot accurately model non-Lambertian effects. We introduce a model that combines the strengths and mitigates the limitations of these two directions. By operating on a four-dimensional representation of the light field, our model learns to represent view-dependent effects accurately. By enforcing geometric constraints during training and inference, the scene geometry is implicitly learned from a sparse set of views. Concretely, we introduce a two-stage transformer-based model that first aggregates features along epipolar lines, then aggregates features along reference views to produce the color of a target ray. Our model outperforms the state-of-the-art on multiple forward-facing and 360° datasets, with larger margins on scenes with severe view-dependent variations. Code and results can be found at light-field-neural-rendering.github. io.
TL;DR: In this paper , the authors proposed a hologram that imitates defocus blur of incoherent light by engineering diffracted pattern of coherent light with adopting multi-plane holography, thereby offering real world-like defocus- blur and photorealistic reconstruction.
Abstract: Abstract Holography is one of the most prominent approaches to realize true-to-life reconstructions of objects. However, owing to the limited resolution of spatial light modulators compared to static holograms, reconstructed objects exhibit various coherent properties, such as content-dependent defocus blur and interference-induced noise. The coherent properties severely distort depth perception, the core of holographic displays to realize 3D scenes beyond 2D displays. Here, we propose a hologram that imitates defocus blur of incoherent light by engineering diffracted pattern of coherent light with adopting multi-plane holography, thereby offering real world-like defocus blur and photorealistic reconstruction. The proposed hologram is synthesized by optimizing a wave field to reconstruct numerous varifocal images after propagating the corresponding focal distances where the varifocal images are rendered using a physically-based renderer. Moreover, to reduce the computational costs associated with rendering and optimizing, we also demonstrate a network-based synthetic method that requires only an RGB-D image.
TL;DR: In this article, it was shown theoretically and numerically that an n-th order vector light field containing the central V-point singularity of indefinite linear polarization with polarization singularity index n, with a "flower" 2(n − 1)-petal polarization pattern centered on it, produces an intensity pattern with 2 n − 1) local maxima at the tight focus.
Abstract: We show theoretically and numerically that an n-th order vector light field containing the central V-point singularity of indefinite linear polarization with polarization singularity index n, with a 'flower' 2(n – 1)-petal polarization pattern centered on it, produces an intensity pattern with 2(n – 1) local maxima at the tight focus. Meanwhile, a vector light field with polarization singularity index –n, leading to a 'web' of polarization singularities composed of 2(n + 1) cells, is tightly focused into an intensity pattern with 2(n + 1) intensity maxima. At the intensity nulls at the focus, either 2(n – 1) or 2(n + 1) V-points with alternating + 1 or –1 indices are produced. In addition, we study more general vector fields of the order (n, m) and analytically derive their Poincare-Hopf indices for many values of n and m. Application areas of such light fields with polarization singularities are laser information technologies, laser material processing, microscopy and optical trapping.
01 Jun 2022
TL;DR: In this paper , a ray-space embedding network is proposed to map 4D ray space into an intermediate, interpolable latent space, which is then used to model complicated view dependence.
Abstract: Neural radiance fields (NeRFs) produce state-of-the-art view synthesis results, but are slow to render, requiring hundreds of network evaluations per pixel to approximate a volume rendering integral. Baking NeRFs into explicit data structures enables efficient rendering, but results in large memory footprints and, in some cases, quality reduction. Additionally, volumetric representations for view synthesis often struggle to represent challenging view dependent effects such as distorted reflections and refractions. We present a novel neural light field representation that, in contrast to prior work, is fast, memory efficient, and excels at modeling complicated view dependence. Our method supports rendering with a single network evaluation per pixel for small baseline light fields and with only a few evaluations per pixel for light fields with larger baselines. At the core of our approach is a ray-space embedding network that maps 4D ray-space into an intermediate, interpolable latent space. Our method achieves state-of-the-art quality on dense forward-facing datasets such as the Stanford Light Field dataset. In addition, for forward-facing scenes with sparser inputs we achieve results that are competitive with NeRF-based approaches while providing a better speed/quality/memory trade-off with far fewer network evaluations.
TL;DR: In this paper , a photoelectron-based scheme was proposed to distinguish the orbital angular momentum (OAM) of focused intense femtosecond optical vortices without the modification of light helical phase.
Abstract: With the rapid development of femtosecond lasers, the generation and application of optical vortices have been extended to the regime of intense-light-matter interaction. The characterization of the orbital angular momentum (OAM) of intense vortex pulses is very critical. Here, we propose and demonstrate a novel photoelectron-based scheme that can in situ distinguish the OAM of the focused intense femtosecond optical vortices without the modification of light helical phase. We employ two-color co-rotating intense circular fields in the strong-field photoionization experiment, in which one color light field is a plane wave serving as the probing pulses and the other one is the vortex pulses whose OAM needs to be characterized. We show that by controlling the spatial profile of the probing pulses, the OAM of the vortex pulses can be clearly identified by measuring the corresponding photoelectron momentum distributions or angle-resolved yields. This work provides a novel in situ detection scenario for the light pulse vorticity and has implications for the studies of ultrafast and intense complex light fields with optical OAM.
TL;DR: In this article , a learning-based approach applied to 3D epipolar image (EPI) is proposed to reconstruct high-resolution light field (LF) imaging, which captures both spatial and angular information of a scene.
TL;DR: In this article, a learning-based approach applied to 3D epipolar image (EPI) is proposed to reconstruct high-resolution light field (LF) imaging, which captures both spatial and angular information of a scene.
TL;DR: Recently, Fu et al. as mentioned in this paper provided a comprehensive review and benchmark for light field saliency detection, which has long been lacking in the saliency community, including theory and data forms, and reviewed existing studies on light field SOD, covering ten traditional models, seven deep learning-based models, one comparative study and one brief review.
Abstract: Salient object detection (SOD) is a long-standing research topic in computer vision and has drawn an increasing amount of research interest in the past decade. This paper provides the first comprehensive review and benchmark for light field SOD, which has long been lacking in the saliency community. Firstly, we introduce preliminary knowledge on light fields, including theory and data forms, and then review existing studies on light field SOD, covering ten traditional models, seven deep learning-based models, one comparative study, and one brief review. Existing datasets for light field SOD are also summarized with detailed information and statistical analyses. Secondly, we benchmark nine representative light field SOD models together with several cutting-edge RGB-D SOD models on four widely used light field datasets, from which insightful discussions and analyses, including a comparison between light field SOD and RGB-D SOD models, are achieved. Besides, due to the inconsistency of datasets in their current forms, we further generate complete data and supplement focal stacks, depth maps and multi-view images for the inconsistent datasets, making them consistent and unified. Our supplemental data makes a universal benchmark possible. Lastly, because light field SOD is quite a special problem attributed to its diverse data representations and high dependency on acquisition hardware, making it differ greatly from other saliency detection tasks, we provide nine hints into the challenges and future directions, and outline several open issues. We hope our review and benchmarking could help advance research in this field. All the materials including collected models, datasets, benchmarking results, and supplemented light field datasets will be publicly available on our project site https://github.com/kerenfu/LFSOD-Survey.
TL;DR: In this paper , the structural consistency of light field data is considered and a color-guided refinement algorithm is proposed to reduce the bitrate of the disparity maps and the content-similarity-based arrangement of the SAI-transformed pseudo-sequences.
Abstract: Light field imaging can simultaneously record the position and direction information of light rays; thus, digital refocusing and full depth-of-field extension — functions that are inaccessible for conventional images — can be achieved using the structural consistency of light field data. To meet the challenges of limited bandwidth and storage, such vast numbers of light field data must be compressed to a low bitrate. However, current compression solutions ignore the intrinsic consistency of light fields in pursuit of a low bitrate, thereby leading to the loss of light field capabilities. To solve this issue, this work focuses on structural consistency to achieve efficient light field compression with a low bitrate. The proposed light field compression method encodes the sparsely selected sub-aperture images (SAIs) and the disparity maps corresponding to the unselected SAIs. From the perspective of geometry consistency, the consistency of the initially estimated disparity maps is improved by using a color-guided refinement algorithm, thereby reducing the bitrate of the disparity maps. From the perspective of content consistency, the consistency of the SAI-transformed pseudo sequence is improved by the proposed content-similarity-based arrangement algorithm along with a specific prediction structure; thereby, the bitrate of the sparsely selected SAIs is reduced. The experimental results show that the proposed compression method can reduce the total bitrate while preserving good structural consistency.
TL;DR: In this paper , the authors analyzed the dominant effects on the collective atomic state and the light field, and derived a unified approach that can account for atomic entanglement induced both by measurements on the light fields, and by ignoring the state of light field altogether.
Abstract: The interaction between an atomic ensemble and a light mode in a high-finesse optical cavity can easily reach the strong-coupling regime, where quantum effects dominate. In this regime, the interaction can be used to generate both atom-light and atom-atom entanglement. We analyze the dominant effects on the collective atomic state and the light field, and derive a unified approach that can account for atomic entanglement induced both by measurements on the light field, and by ignoring the state of the light field altogether. We present analytical expressions for the entanglement induced by the interaction, and determine the conditions that maximize the entanglement-induced gain over the standard quantum limit in quantum sensors and atomic clocks.6 MoreReceived 28 June 2021Accepted 25 March 2022Corrected 29 July 2022DOI:https://doi.org/10.1103/PRXQuantum.3.020308Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasCavity quantum electrodynamicsEntanglement productionQuantum metrologyAtomic, Molecular & Optical
TL;DR: The proposed light field image encoding method outperforms other state-of-the-art methods in terms of compression efficiency while providing random access to both views and regions of interest.
Abstract: In light field image compression, facilitating random access to individual views plays a significant role in decoding views quickly, reducing memory footprint, and decreasing the bandwidth requirement for transmission. Highly efficient light field image compression methods mainly use inter view prediction. Therefore, they typically do not provide random access to individual views. On the other hand, methods that provide full random access usually reduce compression efficiency. To address this trade-off, a light field image encoding method that favors random access is proposed in this paper. Light field image views are grouped into independent (3× 3) views, which are called Macro View Images (MVIs). To encode MVIs, the central view is used as a reference to compress its adjacent neighboring views using a hierarchical reference structure. To encode the central view of each MVI, the most central view along with the center of a maximum of three MVIs, are used as reference images for the disparity estimation. In addition, the proposed method allows the use of parallel processing to reduce the maximum encoding/decoding time-complexity in multi-core processors. Tile partitioning can also be used to randomly access different regions of the light field images. The simulation results show that the proposed method outperforms other state-of-the-art methods in terms of compression efficiency while providing random access to both views and regions of interest.
TL;DR: Zhang et al. as mentioned in this paper proposed a light-field image watermarking based on geranion theory and polar harmonic Fourier moments (PHFMs) for protecting the copyright ownership of light field images.
TL;DR: In this paper , the authors put forward and demonstrate the RSS VLP system utilizing data preprocessing and convolutional neural network (CNN) to mitigate light deficient regions in VLP systems, and the results illustrate that the proposed scheme outperforms the other schemes by not only improving the positioning accuracy, but also the error distribution uniformity.
Abstract: New systems and technologies, such as Internet-of-Things (IOT) may require high reliability and high accuracy indoor positioning and tracking of persons and devices in indoor areas. Among different visible-light-positioning (VLP) schemes, received-signal-strength (RSS) scheme is relatively easy to implement. RSS VLP scheme can provide high accuracy positioning if the optical channels between the Txs and Rxs, as well as the received optical powers of different LEDs are accurately known. Unfortunately, these conditions are not easy to achieve in practice. Due to the limited field-of-view (FOV) of the LED lamps, light deficient regions will happen. This light deficient region could be large and significantly affect the positioning accuracy when performing 3-dimentional (3-D) VLP since at these light deficient regions, very weak or even no optical signal is received. In this work, we put forward and demonstrate the RSS VLP system utilizing data pre-processing and convolutional neural network (CNN) to mitigate light deficient regions in VLP system. Traditional ANN model and linear regression (LR) model are also compared with the CNN model, and the results illustrate that the proposed scheme outperforms the other schemes by not only improving the positioning accuracy, but also the error distribution uniformity.
TL;DR: LFDE-OccUnNet as mentioned in this paper proposes an unsupervised learning-based method, which does not require ground-truth depth as supervision during training, and adopts a multi-scale network with a weighted smoothness loss to handle the textureless areas.
Abstract: Depth estimation is a fundamental issue in 4-D light field processing and analysis. Although recent supervised learning-based light field depth estimation methods have significantly improved the accuracy and efficiency of traditional optimization-based ones, these methods rely on the training over light field data with ground-truth depth maps which are challenging to obtain or even unavailable for real-world light field data. Besides, due to the inevitable gap (or domain difference) between real-world and synthetic data, they may suffer from serious performance degradation when generalizing the models trained with synthetic data to real-world data. By contrast, we propose an unsupervised learning-based method, which does not require ground-truth depth as supervision during training. Specifically, based on the basic knowledge of the unique geometry structure of light field data, we present an occlusion-aware strategy to improve the accuracy on occlusion areas, in which we explore the angular coherence among subsets of the light field views to estimate initial depth maps, and utilize a constrained unsupervised loss to learn their corresponding reliability for final depth prediction. Additionally, we adopt a multi-scale network with a weighted smoothness loss to handle the textureless areas. Experimental results on synthetic data show that our method can significantly shrink the performance gap between the previous unsupervised method and supervised ones, and produce depth maps with comparable accuracy to traditional methods with obviously reduced computational cost. Moreover, experiments on real-world datasets show that our method can avoid the domain shift problem presented in supervised methods, demonstrating the great potential of our method. The code will be publicly available at https://github.com/jingjin25/LFDE-OccUnNet.
22 Feb 2022
TL;DR: A novel end-to-end spatial-angular-decorrelated network (SADN) for high-efficiency light field image compression that decouples the angular and spatial information by dilation convolution and stride convolution in spatial- angular interaction, and performs feature fusion to compress spatial and angular information jointly.
Abstract: Light field image becomes one of the most promising media types for immersive video applications. In this paper, we propose a novel end-to-end spatial-angular-decorrelated network (SADN) for high-efficiency light field image compression. Different from the existing methods that exploit either spatial or angular consistency in the light field image, SADN decouples the angular and spatial information by dilation convolution and stride convolution in spatial-angular interaction, and performs feature fusion to compress spatial and angular information jointly. To train a stable and robust algorithm, a large-scale dataset consisting of 7549 light field images is proposed and built. The proposed method provides 2.137 times and 2.849 times higher compression efficiency relative to H.266/VVC and H.265/HEVC inter coding, respectively. It also outperforms the end-to-end image compression networks by an average of 79.6% bitrate saving with much higher subjective quality and light field consistency.
TL;DR: Liu et al. as mentioned in this paper proposed to learn an epipolar shift compensation for light field image super-resolution that allows the restored light field images to be angular coherent with the enhancement of spatial resolution.
TL;DR: In this article , the authors take into account realistic experimental conditions and analyze the resulting correlation function through theory and simulation, and provide an expression to evaluate the pixel-limited resolution of refocused images, as well as a strategy for eliminating artifacts introduced by the finite size of the optical elements.
Abstract: Diffraction-limited light-field imaging has been recently achieved by exploiting light spatial correlations measured on two high-resolution detectors. As in conventional light-field imaging, the typical operations of refocusing and 3D reconstruction are based on ray tracing in a geometrical optics context, and are thus well defined in the ideal case, both conceptually and theoretically. However, some properties of the measured correlation function are influenced by experimental features such as the finite size of apertures, detectors, and pixels. In this work, we take into account realistic experimental conditions and analyze the resulting correlation function through theory and simulation. We also provide an expression to evaluate the pixel-limited resolution of the refocused images, as well as a strategy for eliminating artifacts introduced by the finite size of the optical elements.
TL;DR: In this paper , it was shown that light can bring itself to a complete standstill via selfinteraction mediated by the resonant nonlinearity in a fully homogeneous medium.
Abstract: Here, we show that light can bring itself to a complete standstill (self-stop) via self-interaction mediated by the resonant nonlinearity in a fully homogeneous medium. An intense few-cycle pulse, entering the medium, may reshape to form a strongly coupled light-matter bundle, in which the energy is transferred from light to the medium and back periodically on the single-cycle scale. Such oscillating structure can decelerate, alter its propagation direction, and even completely stop, depending on the state of its internal degrees of freedom. This phenomenon is expected to occur in the few-cycle strong-field regime when the Rabi oscillation frequency becomes comparable with the frequency of the incoming light.