Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.

/pdf/real-time-video-super-resolution-with-spatio-temporal-1qw8m7lifm.pdf

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Motion blur is crucial for high-quality rendering, but is also very expensive. Our first contribution is a frequency analysis of motion-blurred scenes, including moving objects, specular reflections, and shadows. We show that motion induces a shear in the frequency domain, and that the spectrum of moving scenes can be approximated by a wedge. This allows us to compute adaptive space-time sampling rates, to accelerate rendering. For uniform velocities and standard axis-aligned reconstruction, we show that the product of spatial and temporal bandlimits or sampling rates is constant, independent of velocity. Our second contribution is a novel sheared reconstruction filter that is aligned to the first-order direction of motion and enables even lower sampling rates. We present a rendering algorithm that computes a sheared reconstruction filter per pixel, without any intermediate Fourier representation. This often permits synthesis of motion-blurred images with far fewer rendering samples than standard techniques require.

Frequency analysis and sheared reconstruction for rendering motion blur

Foveated rendering coupled with eye-tracking has the potential to dramatically accelerate interactive 3D graphics with minimal loss of perceptual detail. In this paper, we parameterize foveated rendering by embedding polynomial kernel functions in the classic log-polar mapping. Our GPU-driven technique uses closed-form, parameterized foveation that mimics the distribution of photoreceptors in the human retina. We present a simple two-pass kernel foveated rendering (KFR) pipeline that maps well onto modern GPUs. In the first pass, we compute the kernel log-polar transformation and render to a reduced-resolution buffer. In the second pass, we carry out the inverse-log-polar transformation with anti-aliasing to map the reduced-resolution rendering to the full-resolution screen. We have carried out pilot and formal user studies to empirically identify the KFR parameters. We observe a 2.8X -- 3.2X speedup in rendering on 4K UHD (2160p) displays with minimal perceptual loss of detail. The relevance of eye-tracking-guided kernel foveated rendering can only increase as the anticipated rise of display resolution makes it ever more difficult to resolve the mutually conflicting goals of interactive rendering and perceptual realism.

https://dl.acm.org/doi/pdf/10.1145/3203199

Kernel Foveated Rendering

A display system renders a motion parallax view of object images based upon multiple observers. Also, a headset renders stereoscopic images that augment either physical objects viewed through the headset, or virtual objects projected by a stereoscopic display separate from the headset, or both. The headset includes a system for locating both physical objects and object images within a stereographic projection. Also, a proprioceptive user interface defines interface locations relative to body parts of an observer, the body parts include the head and the shoulders of the observer. The observer may enable a process associated with the interface location by indicating an intersection with the interface location with a physical object, such as a wand or a finger of the observer.

Augmented reality based user interfacing

There is provided a display control device including a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, and an operation acquisition part configured to acquire a user operation. When the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.

Display control device, display control method, and recording medium

The disclosure provides an approach for generating virtual terrains. A terrain editing application is configured to receive assets of various types, including a blank canvas, two-dimensional (2D) sketches, real-world elevation maps, authored heightfields, etc. The assets specify characteristics of a terrain, and provide starting points for creating the virtual terrain. The editing application further provides a set of tools allowing a user to modify the virtual terrain. In one embodiment, the set of tools may include a copy-and-paste tool, a peak creation tool, a ridge creation tool, a ridge tracing tool, and a resynthesis tool. The editing application generates a new layer for each edit, as well as 2D and three-dimensional (3D) previews of the edited terrain. The editing application also provides a user-adjustable frequency decomposition of each layer. The editing application combines layers using Laplacian blending to produce the final virtual terrain.

Example based editing of virtual terrain maps

Systems and methods for arranging scenes of animated content are presented herein. Scenes may be arranged to simulate three-dimensionality in the scenes by shifting objects in the scenes relative to each other and/or changing other properties of one or more of the objects. A shift and/or other property change may be based on a position and/or orientation of a display of a computing platform presenting the scenes relative to a user's perspective of the display.

Systems and methods for arranging scenes of animated content to stimulate three-dimensionality

A method of rendering content items on a display via an electronic device involves mapping linked content items to a three-dimensional object defined by layout data. The layout data is then transmitted to an electronic device for display.

Visualisation and navigation of transmedia content data

Systems, methods, and articles of manufacture are disclosed that enable the compression of depth data and real-time reconstruction of high-quality light fields. In one aspect, spatial compression and decompression of depth images is divided into the following stages: generating a quadtree data structure for each depth image captured by a light field probe and difference mask associated with the depth image, with each node of the quadtree approximating a corresponding portion of the depth image data using an approximating function; generating, from the quadtree for each depth image, a runtime packed form that is more lightweight and has a desired maximum error; and assembling multiple such runtime packed forms into per-probe stream(s); and decoding at runtime the assembled per-probe stream(s). Further, a block compression format is disclosed for approximating depth data by augmenting the block compression format 3DC+(BC4) with a line and two pairs of endpoints.

Depth codec for real-time, high-quality light field reconstruction

A method is disclosed for reducing distortions introduced by deformation of a surface with an existing parameterization. In an exemplary embodiment, the method comprises receiving a rest pose mesh comprising a plurality of faces, a rigidity map corresponding to the rest pose mesh, and a deformed pose mesh; using the rigidity map to generate a simulation grid on the rest pose mesh, the simulation grid comprising a plurality of cells; defining a set of constraints on the simulation grid, the constraints being derived at least in part from the rigidity map; running a simulation using the simulation grid and the set of constraints to obtain a warped grid; and texture mapping the deformed pose mesh based on data from the warped grid.

Kenneth J. Mitchell

Papers

Example based editing of virtual terrain maps

Systems and methods for arranging scenes of animated content to stimulate three-dimensionality

Visualisation and navigation of transmedia content data

Depth codec for real-time, high-quality light field reconstruction

Simulation and skinning of heterogeneous texture detail deformation