The size of the pupil has a large effect on visual function, and pupil size depends mainly on the adapting luminance, modulated by other factors. Over the last century, a number of formulas have been proposed to describe this dependence. Here we review seven published formulas and develop a new unified formula that incorporates the effects of luminance, size of the adapting field, age of the observer, and whether one or both eyes are adapted. We provide interactive demonstrations and software implementations of the unified formula.

A Unified Formula For Light-adapted Pupil Size

With rapid advances in high-speed communication and computation, augmented reality (AR) and virtual reality (VR) are emerging as next-generation display platforms for deeper human-digital interactions. Nonetheless, to simultaneously match the exceptional performance of human vision and keep the near-eye display module compact and lightweight imposes unprecedented challenges on optical engineering. Fortunately, recent progress in holographic optical elements (HOEs) and lithography-enabled devices provide innovative ways to tackle these obstacles in AR and VR that are otherwise difficult with traditional optics. In this review, we begin with introducing the basic structures of AR and VR headsets, and then describing the operation principles of various HOEs and lithography-enabled devices. Their properties are analyzed in detail, including strong selectivity on wavelength and incident angle, and multiplexing ability of volume HOEs, polarization dependency and active switching of liquid crystal HOEs, device fabrication, and properties of micro-LEDs (light-emitting diodes), and large design freedoms of metasurfaces. Afterwards, we discuss how these devices help enhance the AR and VR performance, with detailed description and analysis of some state-of-the-art architectures. Finally, we cast a perspective on potential developments and research directions of these photonic devices for future AR and VR displays.

/pdf/augmented-reality-and-virtual-reality-displays-emerging-4j54gs3u78.pdf

Augmented reality and virtual reality displays: emerging technologies and future perspectives.

Nowadays, digital 3D models are in widespread and ubiquitous use, and each specific application dealing with 3D geometry has its own quality requirements that restrict the class of acceptable and supported models. This article analyzes typical defects that make a 3D model unsuitable for key application contexts, and surveys existing algorithms that process, repair, and improve its structure, geometry, and topology to make it appropriate to case-by-case requirements.The analysis is focused on polygon meshes, which constitute by far the most common 3D object representation. In particular, this article provides a structured overview of mesh repairing techniques from the point of view of the application context. Different types of mesh defects are classified according to the upstream application that produced the mesh, whereas mesh quality requirements are grouped by representative sets of downstream applications where the mesh is to be used. The numerous mesh repair methods that have been proposed during the last two decades are analyzed and classified in terms of their capabilities, properties, and guarantees. Based on these classifications, guidelines can be derived to support the identification of repairing algorithms best-suited to bridge the compatibility gap between the quality provided by the upstream process and the quality required by the downstream applications in a given geometry processing scenario.

http://pers.ge.imati.cnr.it/attene/PersonalPage/pdf/survey_meshrepair.pdf

Polygon mesh repairing: An application perspective

The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference1. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion2,3. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical4. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions5 and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design6,7, optical and acoustic tweezer-based microscopic manipulation8–10, holographic microscopy11 and single-exposure volumetric 3D printing12,13. A deep-learning-based approach using a convolutional neural network is used to synthesize photorealistic colour three-dimensional holograms from a single RGB-depth image in real time, and termed tensor holography.

Towards real-time photorealistic 3D holography with deep neural networks

3D reconstruction methods for digital preservation of cultural heritage

A virtual reality headset displays a three-dimensional (3D) virtual scene and includes a varifocal element to dynamically adjust a focal length of an optics block included in the virtual reality headset based on a location in the virtual scene where the user is looking. The headset tracks a user's eyes to approximate gaze lines and determines a plane of focus for a frame of the virtual scene as the intersection of the gaze lines. The varifocal element adjusts the focal length of the optics block so the optics block is focused at the plane of focus, which keeps the user's eyes in a zone of comfort as vergence and accommodation change. Based on the plane of focus, the virtual reality headset may provide depth cues, such as depth of field blur, to planes in the virtual scene deeper in the user's field of view than the plane of focus.

Focus adjusting virtual reality headset

Due to higher resolutions and refresh rates, as well as more photorealistic effects, real-time rendering has become increasingly challenging for video games and emerging virtual reality headsets. To meet this demand, modern graphics hardware and game engines often reduce the computational cost by rendering at a lower resolution and then upsampling to the native resolution. Following the recent advances in image and video superresolution in computer vision, we propose a machine learning approach that is specifically tailored for high-quality upsampling of rendered content in real-time applications. The main insight of our work is that in rendered content, the image pixels are point-sampled, but precise temporal dynamics are available. Our method combines this specific information that is typically available in modern renderers (i.e., depth and dense motion vectors) with a novel temporal network design that takes into account such specifics and is aimed at maximizing video quality while delivering real-time performance. By training on a large synthetic dataset rendered from multiple 3D scenes with recorded camera motion, we demonstrate high fidelity and temporally stable results in real-time, even in the highly challenging 4 × 4 upsampling scenario, significantly outperforming existing superresolution and temporal antialiasing work.

Neural supersampling for real-time rendering

A scene presented by a headset is adjusted to correct for distortion from optical errors of an optics block in the headset. To correct for the distortion, the scene is pre-distorted when presented based on previously modeled distortion of the optics block, so distortion from the optics block corrects the pre-distortion. To model the distortion, the headset displays calibration image including features and images of the calibration image are captured from multiple positions. Differences between locations of features in the calibration images and locations of corresponding features in captured images of the calibration image are identified and a distortion correction is determined based on the differences.

Autofocus virtual reality headset

Holographic optical elements (HOEs) have a wide range of applications, including their emerging use in virtual and augmented reality displays, but their design and fabrication have remained largely limited to configurations using simple wavefronts. In this paper, we present a pipeline for the design, optimization, and fabrication of complex, customized HOEs that enhances their imaging performance and enables new applications. In particular, we propose an optimization method for grating vector fields that accounts for the unique selectivity properties of HOEs. We further show how our pipeline can be applied to two distinct HOE fabrication methods. The first uses a pair of freeform refractive elements to manufacture HOEs with high optical quality and precision. The second uses a holographic printer with two wavefront-modulating arms, enabling rapid prototyping. We propose a unified wavefront decomposition framework suitable for both fabrication approaches. To demonstrate the versatility of these methods, we fabricate and characterize a series of specialized HOEs, including an aspheric lens, a head-up display lens, a lens array, and, for the first time, a full-color caustic projection element.

/pdf/design-and-fabrication-of-freeform-holographic-optical-4l0m93w9zr.pdf

Design and fabrication of freeform holographic optical elements

Addressing vergence-accommodation conflict in head-mounted displays (HMDs) requires resolving two interrelated problems. First, the hardware must support viewing sharp imagery over the full accommodation range of the user. Second, HMDs should accurately reproduce retinal defocus blur to correctly drive accommodation. A multitude of accommodation-supporting HMDs have been proposed, with three architectures receiving particular attention: varifocal, multifocal, and light field displays. These designs all extend depth of focus, but rely on computationally expensive rendering and optimization algorithms to reproduce accurate defocus blur (often limiting content complexity and interactive applications). To date, no unified framework has been proposed to support driving these emerging HMDs using commodity content. In this paper, we introduce DeepFocus, a generic, end-to-end convolutional neural network designed to efficiently solve the full range of computational tasks for accommodation-supporting HMDs. This network is demonstrated to accurately synthesize defocus blur, focal stacks, multilayer decompositions, and multiview imagery using only commonly available RGB-D images, enabling real-time, near-correct depictions of retinal blur with a broad set of accommodation-supporting HMDs.

Douglas Robert Lanman

Papers

Focus adjusting virtual reality headset

Neural supersampling for real-time rendering

Autofocus virtual reality headset

Design and fabrication of freeform holographic optical elements

DeepFocus: learned image synthesis for computational displays