Recent breakthroughs in consumer level virtual reality (VR) headsets are creating a growing user-base in demand for immersive, full 3D VR experiences. While monoscopic 360-videos are perhaps the most prevalent type of content for VR headsets, they lack 3D information and thus cannot be viewed with full 6 degree-of-freedom (DOF). We present an approach that addresses this limitation via a novel warping algorithm that can synthesize new views both with rotational and translational motion of the viewpoint. This enables the ability to perform VR playback of input monoscopic 360-videos files in full stereo with full 6-DOF of head motion. Our method synthesizes novel views for each eye in accordance with the 6-DOF motion of the headset. Our solution tailors standard structure-from-motion and dense reconstruction algorithms to work accurately for 360-videos and is optimized for GPUs to achieve VR frame rates (>120 fps). We demonstrate the effectiveness our approach on a variety of videos with interesting content.

6-DOF VR videos with a single 360-camera

An autonomous computer based method and system is described for personalized production of videos such as team sport videos such as basketball videos from multi- sensored data under limited display resolution. Embodiments of the present invention relate to the selection of a view to display from among the multiple video streams captured by the camera network. Technical solutions are provided to provide perceptual comfort as well as an efficient integration of contextual information, which is implemented, for example, by smoothing generated viewpoint/camera sequences to alleviate flickering visual artefacts and discontinuous story-telling artefacts. A design and implementation of the viewpoint selection process is disclosed that has been verified by experiments, which shows that the method and system of the present invention efficiently distribute the processing load across cameras, and effectively selects viewpoints that cover the team action at hand while avoiding major perceptual artefacts.

Systems and methods for the autonomous production of videos from multi-sensored data

Octrees have been extensively used for 3D object representation. The authors have previously developed a scheme, that exploits the volume intersection approach, to generate the octree of an object from three orthogonal views. In this paper, the scheme is extended to generate a “generalized octree” of an object from three known non-coplanar views. The block associated with each node in a generalized octree is a parallelepiped, instead of being a cube. To identify an observed object, the generalized octree of the object is projected onto the corresponding principal planes and matched against a number of models, each of which is represented by a set of the quadtrees from three principal views. To speed up the processing, reconstruction of the structure of the object as well as computation of its moment of inertia matrix (comprising all the second-order moments) are performed in a generalized coordinate system specified by the three viewing directions. The coordinate transformation is applied only to the moment of inertia matrix. The principal axes are obtained from the transformed moment of inertia matrix by computing its eigenvectors. A “coarse” matching is performed by matching the quadtrees of the unknown object to those of the models, so as to extract a smaller set of models with strong similarity to the unknown object. The octrees of these models are then matched to that of the object using a “fine” matching process. The results of experiments conducted demonstrate that this approach makes both storage and computation highly efficient.

Identification of 3d objects from multiple silhouettes using quadtrees/octrees.

This paper presents a novel method for virtual view synthesis that allows viewers to virtually fly through real soccer scenes, which are captured by multiple cameras in a stadium. The proposed method generates images of arbitrary viewpoints by view interpolation of real camera images near the chosen viewpoints. In this method, cameras do not need to be strongly calibrated since projective geometry between cameras is employed for the interpolation. For avoiding the complex and unreliable process of 3-D recovery, object scenes are segmented into several regions according to the geometric property of the scene. Dense correspondence between real views, which is necessary for intermediate view generation, is automatically obtained by applying projective geometry to each region. By superimposing intermediate images for all regions, virtual views for the entire soccer scene are generated. The efforts for camera calibration are reduced and correspondence matching requires no manual operation; hence, the proposed method can be easily applied to dynamic events in a large space. An application for fly-through observations of soccer match replays is introduced along with the algorithm of view synthesis and experimental results. This is a new approach for providing arbitrary views of an entire dynamic event.

/pdf/virtual-viewpoint-replay-for-a-soccer-match-by-view-2bqskwphya.pdf

Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras

Distributed collaboration has been enhanced in recent years by sophisticated new video conferencing setups like HP Halo and Cisco Telepresence, improving the user experience of distributed meeting situations over traditional video conferencing. The experience created can be described as one of "blending" distributed physical locations into one shared space. Inspired by this trend, we have been exploring the systematic creation of blended spaces for distributed collaboration through the design of appropriate shared spatial geometries. We present early iterations of our design work: the Blended Interaction Space One prototype, BISi, and the lessons learned from its creation.

/pdf/bisi-a-blended-interaction-space-44t6iiggsi.pdf

BISi: a blended interaction space

We propose a method for arbitrary view synthesis from uncalibrated multiple camera system, targeting large spaces such as soccer stadiums. In Projective Grid Space (PGS), which is a three-dimensional space defined by epipolar geometry between two basis cameras in the camera system, we reconstruct three-dimensional shape models from silhouette images. Using the three-dimensional shape models reconstructed in the PGS, we obtain a dense map of the point correspondence between reference images. The obtained correspondence can synthesize the image of arbitrary view between the reference images. We also propose a method for merging the synthesized images with the virtual background scene in the PGS. We apply the proposed methods to image sequences taken by a multiple camera system, which installed in a large concert hall. The synthesized image sequences of virtual camera have enough quality to demonstrate effectiveness of the proposed method.

/pdf/arbitrary-viewpoint-video-synthesis-from-multiple-8bafsfyvaj.pdf

Arbitrary viewpoint video synthesis from multiple uncalibrated cameras

In this paper, we propose a method for arbitrary view generation from multiple view images taken with uncalibrated camera system. In PGS, that is the 3D space which is defined by epipolar geometry between the two basis cameras in the multiple cameras, we reconstruct 3D shape model from the silhouette images of the multiple cameras. For het shape reconstruction in the PGS, the multiple cameras do not have to be fully calibrated, but the fundamental matrices of every camera to the two basis cameras must be collected. By using the 3D model reconstructed in the PGS, we can obtain the point correspondence between arbitrary pair of images which can generate the image of arbitrary view between the pair of images.

Arbitrary-view image generation from multiple silhouette images in projective grid space

Arbitrary viewpoint video synthesis from uncalibrated multiple cameras

In this paper, we present a method to synthesize high-quality virtual viewpoint image targeting the detailed texture objects. About 30 images are taken from multiple uncalibrated cameras around the object, and the Visual Hull model is reconstructed with Shape from Silhouette method. To deform 3D surface model that is converted from Visual Hull Model using the information such as image texture and object silhouette, the difference between the real object and the reconstructed model is evaluated as a cost function of optimization problem. Our deforming model algorithm is based on single vertex iterative shifting. The vertex of surface triangle mesh is moved to the selected candidate point that maximizes the cost function. The cost function is consisted by four constraint criteria, texture correlation, smoothness, object silhouette, and mesh shape regularity. In addition to the cost function, such as judging mesh direction and combining / dividing meshes are applied for refined 3D models to avoid mesh folding and mesh size unevenness. The refined model provides a quite accurate dense corresponding relationship between the input images, so that high quality image can be synthesized at virtual viewpoint. We also demonstrate the proposed method by showing virtual viewpoint images to applying the real image that are taken from multiple uncalibrated cameras.

/pdf/improving-quality-of-free-viewpoint-image-by-mesh-based-3d-39kw5twvi2.pdf

Improving quality of free-viewpoint image by mesh based 3D shape deformation

In this paper, we present a method to improve the quality of synthesized new view images for Image Based Rendering approach from uncalibrated multiple views targeting the detailed texture objects. To deform 3D model using the information such as image texture and object silhouette, the difference between the real object and the reconstructed model is evaluated as a function of optimization problem. The refined model provides a quite accurate dense corresponding relation between the images, so that high quality image can be synthesized at arbitrary viewpoint. We also demonstrate the proposed method by showing new view images rendered by the proposed method.

/pdf/mesh-based-3d-shape-deformation-for-image-based-rendering-4mm8ybug24.pdf

Satoshi Yaguchi

Papers

Arbitrary viewpoint video synthesis from multiple uncalibrated cameras

Arbitrary-view image generation from multiple silhouette images in projective grid space

Arbitrary viewpoint video synthesis from uncalibrated multiple cameras

Improving quality of free-viewpoint image by mesh based 3D shape deformation

Mesh based 3D shape deformation for image based rendering from uncalibrated multiple views