Showing papers by "Wolfgang Heidrich published in 2023"

Curriculum Learning for ab initio Deep Learned Refractive Optics

Abstract: Deep lens optimization has recently emerged as a new paradigm for designing computational imaging systems, however it has been limited to either simple optical systems consisting of a single DOE or metalens, or the fine-tuning of compound lenses from good initial designs. Here we present a deep lens design method based on curriculum learning, which is able to learn optical designs of compound lenses ab initio from randomly initialized surfaces, therefore overcoming the need for a good initial design. We demonstrate this approach with the fully-automatic design of an extended depth-of-field computational camera in a cellphone-style form factor, highly aspherical surfaces, and a short back focal length.

Aberration-Aware Depth-from-Focus

Abstract: Computer vision methods for depth estimation usually use simple camera models with idealized optics. For modern machine learning approaches, this creates an issue when attempting to train deep networks with simulated data, especially for focus-sensitive tasks like Depth-from-Focus. In this work, we investigate the domain gap caused by off-axis aberrations that will affect the decision of the best-focused frame in a focal stack. We then explore bridging this domain gap through aberration-aware training (AAT). Our approach involves a lightweight network that models lens aberrations at different positions and focus distances, which is then integrated into the conventional network training pipeline. We evaluate the generality of pretrained models on both synthetic and real-world data. Our experimental results demonstrate that the proposed AAT scheme can improve depth estimation accuracy without fine-tuning the model or modifying the network architecture.

BUSIFusion: Blind Unsupervised Single Image Fusion of Hyperspectral and RGB Images

Abstract: Hyperspectral images (HSIs) provide rich spectral information that has been widely used in numerous computer vision tasks. However, their low spatial resolution often prevents their use in applications such as image segmentation and recognition. Fusing low-resolution HSIs with high-resolution RGB images to reconstruct high-resolution HSIs has attracted great research attention recently. In this paper, we propose an unsupervised blind fusion network that operates on a single HSI and RGB image pair and requires neither known degradation models nor any training data. Our method takes full advantage of an unrolling network and coordinate encoding to provide a state-of-the-art HSI reconstruction. It can also estimate the degradation parameters relatively accurately through the neural representation and implicit regularization of the degradation model. The experimental results demonstrate the effectiveness of our method both in simulations and in our real experiments. The proposed method outperforms other state-of-the-art nonblind and blind fusion methods on two popular HSI datasets. Our related code and data is available at https://github.com/CPREgroup/Real-Spec-RGB-Fusion.

MoiréTag: Angular Measurement and Tracking with a Passive Marker

Abstract: Stable, low-cost, and precise visual measurement of directional information has many applications in domains such as virtual and augmented reality, visual odometry, or industrial computer vision. Conventional approaches like checkerboard patterns require careful pre-calibration, and can therefore not be operated in snapshot mode. Other optical methods like autocollimators offer very high precision but require controlled environments and are hard to take outside the lab. Non-optical methods like IMUs are low cost and widely available, but suffer from high drift errors. To overcome these challenges, we propose a novel snapshot method for angular measurement and tracking with Moiré patterns that are generated by binary structures printed on both sides of a glass plate. The Moiré effect amplifies minute angular shifts and translates them into spatial phase shifts that can be readily measured with a camera, effectively implementing an optical Vernier scale. We further extend this principle from a simple phase shift to a chirp model, which allows for full 6D tracking as well as estimation of camera intrinsics like the field of view. Simulation and experimental results show that the proposed non-contact object tracking framework is computationally efficient and the average angular accuracy of 0.17° outperforms the state-of-the-arts.

Large Numerical Aperture Metalens with High Modulation Transfer Function

Abstract: Large numerical aperture (NA) lenses with high modulation transfer functions (MTFs) promise high image resolution for advanced optical imaging. However, it is challenging to achieve a high MTF using traditional large-NA lenses, which are fundamentally limited by the amplitude mismatch. In contrast, metasurfaces are promising for realizing amplitude and phase matching for ideal lenses. However, current metalenses are mostly based on a phase-only (PO) profile because the strong coupling among the metaatoms in large-NA lenses makes perfect amplitude matching quite challenging to realize. Here, we derive a phase-and-amplitude (PA) profile that approaches the theoretical MTF limit for large-NA lenses and use interferometric unit cells combined with a segmented sampling approach to achieve the desired amplitude and phase control. For the first time, we show that the amplitude does not require a perfect match; realizing the trend of the required amplitude is sufficient to significantly increase the MTF of a large-NA lens. We demonstrated a 0.9 NA cylindrical metalens at 940 nm with a Struve ratio (SR), which describes how close the MTF is to the upper limit, increasing from 0.68 to 0.90 compared with the PO metalens. Experimentally, we achieved an SR of 0.77 for the 0.9 NA lens, which is even 0.09 higher than the simulated SR of the PO metalens. Our investigation provides new insights for large-NA lenses and has potential applications in high-image-resolution optical systems.

Scattering-aware Holographic PIV with Physics-based Motion Priors

Abstract: —Particle imaging velocimetry is a classical method in 2D fluid imaging. While 3D extensions exist, they are limited by practical restrictions of multi-camera systems. Holographic particle imaging velocimetry has emerged as a solution for a simple and compact 3D imaging system. However, with dense particle seeding, scattering effects become apparent, and the reconstruction quality suffers, especially in the axial direction. To address these challenges, we propose a simple in-line HPIV approach with a plane-to-plane propagation model to account for the scattering effect. Instead of independently reconstructing particle volume and flow velocity, we present a joint optimization problem for particle and flow reconstruction. This optimization problem combines the a differentiable formulation of the holographic image formation with physical motion priors (incompressible flow and particle motion consistency) to improve the reconstruction quality. We solve this joint optimization problem using an extendable automatic differentiation and alternating optimization framework, and we evaluate the proposed method in synthetic and real experiments. The results demonstrate improved reconstruction quality for both particle density and flow velocity fields. With the plane-to-plane propagation model and physics prior, we push HPIV a step further regarding particle density, tank depth, and reconstruction accuracy.

Image Quality Is Not All You Want: Task-Driven Lens Design for Image Classification

Abstract: In computer vision, it has long been taken for granted that high-quality images obtained through well-designed camera lenses would lead to superior results. However, we find that this common perception is not a"one-size-fits-all"solution for diverse computer vision tasks. We demonstrate that task-driven and deep-learned simple optics can actually deliver better visual task performance. The Task-Driven lens design approach, which relies solely on a well-trained network model for supervision, is proven to be capable of designing lenses from scratch. Experimental results demonstrate the designed image classification lens (``TaskLens'') exhibits higher accuracy compared to conventional imaging-driven lenses, even with fewer lens elements. Furthermore, we show that our TaskLens is compatible with various network models while maintaining enhanced classification accuracy. We propose that TaskLens holds significant potential, particularly when physical dimensions and cost are severely constrained.

Extended Depth-of-Field Projector using Learned Diffractive Optics

Abstract: Projector Depth-of-Field (DOF) refers to the projection range of projector images in focus. It is a crucial property of projectors in spatial augmented reality (SAR) applications since wide projector DOF can increase the effective projection area on the projection surfaces with large depth variances and thus reduce the number of projectors required. Existing state-of-the-art methods attempt to create all-in-focus displays by adopting either a deep deblurring network or light modulation. Unlike previous work that considers the optimization of the deblurring model and physic modulation separately, in this paper, we propose an end-to-end joint optimization method to learn a diffractive optical element (DOE) placed in front of a projector lens and a compensation network for deblurring. Using the desired image and the captured projection result image, the compensation network can directly output the compensated image for display. We evaluate the proposed method in physical simulation and with a real experimental prototype, showing that the proposed method can extend the projector DOF by a minor modification to the projector and thus superior to the normal projection with a shallow DOF. The compensation method is also compared with the state-of-the-art methods and shows the advance in radiometric compensation in terms of computational efficiency and image quality.