THIS is a thoroughly good book which deserves to run through many editions. It is not (as its title might suggest) a book on mathematical logic or philosophy: it deals not with the nature of mathematics but with its content. Its purpose is to show, not by general disquisitions but by concrete examples, drawn from almost every branch of pure mathematics, how mathematicians think and what they do. What is Mathematics? An Elementary Approach to Ideas and Methods. By Richard Courant and Herbert Robbins. Pp. xix + 521. (London, New York and Toronto: Oxford University Press, 1941.) 25s. net.

What is Mathematics

This paper presents a novel immersive system called MR360 that provides interactive mixed reality (MR) experiences using a conventional low dynamic range (LDR) 360° panoramic video (360-video) shown in head mounted displays (HMDs). MR360 seamlessly composites 3D virtual objects into a live 360-video using the input panoramic video as the lighting source to illuminate the virtual objects. Image based lighting (IBL) is perceptually optimized to provide fast and believable results using the LDR 360-video as the lighting source. Regions of most salient lights in the input panoramic video are detected to optimize the number of lights used to cast perceptible shadows. Then, the areas of the detected lights adjust the penumbra of the shadow to provide realistic soft shadows. Finally, our real-time differential rendering synthesizes illumination of the virtual 3D objects into the 360-video. MR360 provides the illusion of interacting with objects in a video, which are actually 3D virtual objects seamlessly composited into the background of the 360-video. MR360 was implemented in a commercial game engine and tested using various 360-videos. Since our MR360 pipeline does not require any pre-computation, it can synthesize an interactive MR scene using a live 360-video stream while providing realistic high performance rendering suitable for HMDs.

MR360: Mixed Reality Rendering for 360° Panoramic Videos

Free-viewpoint television (FTV) and virtual navigation appear to be hot research topics. In this paper, the authors study the practical development of free-viewpoint television systems that provide the functionality of virtual horizontal navigation around real scenes. The considerations are focused on practical systems that use purely optical depth estimation and might be employed in the next few years. The architectures of such systems are discussed in detail, including acquisition, preprocessing, depth estimation, compression, and presentation. In particular, the optimization of camera locations is discussed, and it is shown that video acquisition using camera pairs is advantageous for scenes with a substantial amount of occlusions. The theoretical considerations are supported by experimental results obtained for standard test multiview video sequences. Furthermore, the paper describes FTV video acquisition systems that consist of modules with pairs of cameras. The modules are sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor. The hardware requirements, video processing algorithms, and experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new three-dimensional HEVC technology. The paper also describes new test video sequences that are obtained from the camera pairs sparsely distributed around scenes.

A Free-Viewpoint Television System for Horizontal Virtual Navigation

In this paper we consider immersive visual media that are currently researched within scientific community. These include the well-established technologies, like 360-degree panoramas, as well as those being intensively developed, like free viewpoint video and point-cloud based systems. By the use of characteristic examples, we define the features of the immersive visual media that distinguish them from the classical 2D video. Also, we present the representation technologies that are currently considered by the scientific community, especially in the context of standardization of the immersive visual media in the MPEG-I project recently launched by ISO/IEC.

Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond

This work was supported by Ministerio de Economia, Industria y Competitividad (Spain) under projects FIS2017-82919-R (MINECO/AEI/FEDER, UE) and FIS2015-66570-P (MINECO/FEDER), and by Generalitat Valenciana (Spain) under project PROMETEO II/2015/015.

/pdf/roadmap-on-holography-289uo7kvcp.pdf

Roadmap on holography

Concave micro-mirror arrays fabricated as holographic optical elements are used in projector-based light field displays due to their see-through characteristics. The optical axes of each micro-mirror in the array are usually made parallel to each other, which simplifies the fabrication, integral image rendering, and calibration process. However, this demands that the beam from the projector be collimated and made parallel to the optical axis of each elemental micro-mirror. This requires additional collimation optics, which puts serious limitations on the size of the display. In this Letter, we propose a solution to the above issue by introducing a new method to fabricate holographic concave micro-mirror array sheets and explain how they work in detail. 3D light field reconstructions of the size 20  cm×10  cm and 6 cm in depth are achieved using a conventional projector without any collimation optics.

/pdf/digitally-designed-holographic-optical-element-for-light-44xadlhmw3.pdf

Digitally designed holographic optical element for light field displays.

ISO/IEC MPEG and ITU-T VCEG have recently jointly issued
a new multiview video compression standard, called 3D-HEVC,
which reaches unpreceded compression performances for linear,dense camera arrangements. In view of supporting future highquality,auto-stereoscopic 3D displays and Free Navigation virtual/augmented reality applications with sparse, arbitrarily arranged camera setups, innovative depth estimation and virtual view synthesis techniques with global optimizations over all camera views should be developed. Preliminary studies in response to the MPEG-FTV (Free viewpoint TV) Call for Evidence suggest these
targets are within reach, with at least 6% bitrate gains over 3DHEVC
technology.

/pdf/new-visual-coding-exploration-in-mpeg-super-multiview-and-2vn1d6jbym.pdf

New visual coding exploration in MPEG: Super-multiview and free navigation in free viewpoint TV

In this paper, we propose a novel, fully automatic method to obtain accurate view synthesis for soccer games. Existing methods often make assumptions about the scene. This usually requires manual input and introduces artifacts in situations not handled by those assumptions. Our method does not make assumptions about the scene; it solely relies on feature detection and utilizes the structures visible in a 3D light field to limit the search range of traditional view synthesis methods. A visual comparison between a standard plane sweep, a depth-aware plane sweep and our method is provided, showing that our method provides more accurate results in most cases.

Multi-camera epipolar plane image feature detection for robust view synthesis

This paper presents the augmentation of immersive omnidirectional video with realistically lit objects. Recent years have known a proliferation of real-time capturing and rendering methods of omnidirectional video. Together with these technologies, rendering devices such as Oculus Rift have increased the immersive experience of users. We demonstrate the use of structure from motion on omnidirectional video to reconstruct the trajectory of the camera. The position of the car is then linked to an appropriate \(360^{\circ }\) environment map. State-of-the-art augmented reality applications have often lacked realistic appearance and lighting. Our system is capable of evaluating the rendering equation in real-time, by using the captured omnidirectional video as a lighting environment. We demonstrate an application in which a computer generated vehicle can be controlled through an urban environment.

/pdf/interactive-augmented-omnidirectional-video-with-realistic-19iixhrgza.pdf

Interactive Augmented Omnidirectional Video with Realistic Lighting

In this paper, we present a novel optical tracking approach to accurately estimate the pose of a camera in large scene augmented reality (AR). Traditionally, larger scenes are provided with multiple markers with their own identifier and coordinate system. However, when any part of a single marker is occluded, the marker cannot be identified. Our system uses a seamless structure of dots where the world position of each dot is represented by its spatial relation to neighboring dots. By using only the dots as features, our marker can be robustly identified. We use projective invariants to estimate the global position of the features and exploit temporal coherence using optical flow. With this design, our system is more robust against occlusions. It can also give the user more freedom of movement allowing them to explore objects up close and from a distance.

/pdf/robust-global-tracking-using-a-seamless-structured-pattern-36m97lfwqk.pdf

Lode Jorissen

Papers

Digitally designed holographic optical element for light field displays.

New visual coding exploration in MPEG: Super-multiview and free navigation in free viewpoint TV

Multi-camera epipolar plane image feature detection for robust view synthesis

Interactive Augmented Omnidirectional Video with Realistic Lighting

Robust Global Tracking Using a Seamless Structured Pattern of Dots