A Comprehensive Review of Redirected Walking Techniques: Taxonomy, Methods, and Future Directions

Rapidly developing Redirected Walking (ROW) technologies have enabled VR applications to immerse users in large virtual environments (VE) while actually walking in relatively small physical environments (PE). When an unavoidable collision emerges in a PE, the ROW controller suspends the user's immersive experience and resets the user to a new direction in PE. Existing ROW methods mainly aim to reduce the number of resets. However, from the perspective of the user experience, when users are about to reach a point of interest (POI) in a VE, reset interruptions are more likely to have an impact on user experience. In this paper, we propose a new ROW method, aiming to keep resets occurring at a longer distance from the virtual target, as well as to reduce the number of resets. Simulation experiments and real user studies demonstrate that our method outperforms state-of-the-art ROW methods in the number of resets and dramatically increases the distance between the reset locations and the virtual targets.

Making Resets away from Targets: POI aware Redirected Walking

We present a novel framework to regularize Neural Radiance Field (NeRF) in a few-shot setting with a geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint to warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths to facilitate learning of NeRF. By encouraging such geometry-aware consistency at a feature-level instead of using pixel-level reconstruction loss, we regularize the NeRF at semantic and structural levels while allowing for modeling view dependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achieves competitive results compared to state-of-the-art few-shot NeRF models. Project page is available at https://ku-cvlab.github.io/GeCoNeRF/.

GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency

Talking head generation is to generate video based on a given source identity and target motion. However, current methods face several challenges that limit the quality and controllability of the generated videos. First, the generated face often has unexpected deformation and severe distortions. Second, the driving image does not explicitly disentangle movement-relevant information, such as poses and expressions, which restricts the manipulation of different attributes during generation. Third, the generated videos tend to have flickering artifacts due to the inconsistency of the extracted landmarks between adjacent frames. In this paper, we propose a novel model that produces high-fidelity talking head videos with free control over head pose and expression. Our method leverages both self-supervised learned landmarks and 3D face model-based landmarks to model the motion. We also introduce a novel motion-aware multi-scale feature alignment module to effectively transfer the motion without face distortion. Furthermore, we enhance the smoothness of the synthesized talking head videos with a feature context adaptation and propagation module. We evaluate our model on challenging datasets and demonstrate its state-of-the-art performance. More information is available at https://yuegao.me/PECHead.

High-Fidelity and Freely Controllable Talking Head Video Generation

Virtual reality interactions that require a specific relationship between the virtual and physical coordinate systems, such as passive haptic interactions, are not possible with locomotion techniques using redirected walking. To address this limitation, recent research has introduced environmental alignment, which is the notion of using redirected walking techniques to align the virtual and physical coordinate systems such that these interactions are possible. However, the previous research has only implemented environmental alignment in a reactive manner, and the authors posited that better results could be achieved if a predictive algorithm was instead used. In this work, we introduce a novel way to model the environmental alignment problem as a version of the inverse kinematics problem which can be incorporated into several existing predictive algorithms, as well as a simple proof-of-concept implementation. An exploratory human subject study (N=17) was conducted to evaluate this implementation’s usability as a tool for authoring planned path redirected walking scenarios that incorporate physical interactivity. To our knowledge, this is the first study to evaluate redirected walking experience design tools and provides a possible framework for future studies. Our qualitative analysis of the results generated both guidance for integrating automatic solvers and broad recommendations for designing redirected path authoring tools.

Inverse Kinematics Assistance for the Creation of Redirected Walking Paths

Redirected Walking (RDW) algorithms aim to impose several types of gains on users immersed in Virtual Reality and distort their walking paths in the real world, thus enabling them to explore a larger space. Since collision with physical boundaries is inevitable, a reset strategy needs to be provided to allow users to reset when they hit the boundary. However, most reset strategies are based on simple heuristics by choosing a seemingly suitable solution, which may not perform well in practice. In this paper, we propose a novel optimization-based reset algorithm adaptive to different RDW algorithms. Inspired by the approach of finite element analysis, our algorithm splits the boundary of the physical world by a set of endpoints. Each endpoint is assigned a reset vector to represent the optimized reset direction when hitting the boundary. The reset vectors on the edge will be determined by the interpolation between two neighbouring endpoints. We conduct simulation-based experiments for three RDW algorithms with commonly used reset algorithms to compare with. The results demonstrate that the proposed algorithm significantly reduces the number of resets.

Adaptive Optimization Algorithm for Resetting Techniques in Obstacle-ridden Environments

In this work, we proposed a new out-of-place resetting strategy that guides users to optimal physical locations with the most potential for free movement and a smaller amount of resetting required for their further movements. For this purpose, we calculate a heat map of the walking area according to the average walking distance using a simulation of the used RDW algorithm. Based on this heat map we identify the most suitable position for a one-step reset within a predefined searching range and use the one as the reset point. The results show that our method increases the average moving distance within one cycle of resetting. Furthermore, our resetting method can be applied to any physical area with obstacles. That means that RDW methods that were not suitable for such environments (e.g. Steer to Center) combined with our resetting can also be extended to such complex walking areas. In addition, we also present a resetting user interface to instruct users to move the nearby point, by using light spots to bring user a sense of relative displacement while the virtual scenario is still.

One-step out-of-place resetting for redirected walking in VR

Neural Radiance Fields (NeRF) achieve photo-realistic view synthesis with densely captured input images. However, the geometry of NeRF is extremely under-constrained given sparse views, resulting in significant degradation of novel view synthesis quality. Inspired by self-supervised depth estimation methods, we propose StructNeRF, a solution to novel view synthesis for indoor scenes with sparse inputs. StructNeRF leverages the structural hints naturally embedded in multi-view inputs to handle the unconstrained geometry issue in NeRF. Specifically, it tackles the texture and non-texture regions respectively: a patch-based multi-view consistent photometric loss is proposed to constrain the geometry of textured regions; for non-textured ones, we explicitly restrict them to be 3D consistent planes. Through the dense self-supervised depth constraints, our method improves both the geometry and the view synthesis performance of NeRF without any additional training on external data. Extensive experiments on several real-world datasets demonstrate that StructNeRF surpasses state-of-the-art methods for indoor scenes with sparse inputs both quantitatively and qualitatively.

StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints

Redirected walking (RDW) aims to reduce the collisions in the physical space for VR applications. However, most of the previous RDW methods do not consider future possibilities of collisions after imperceptibly redirecting users. In this paper, we combine the subtle RDW methods and reset strategy in our method design and propose a novel solution for RDW that can make better use of physical space and trigger fewer resets. The key idea of our method is to discretize the representation of possible user positions and orientations by a series of standard poses and rate them based on the possibilities of hitting obstacles of their reachable poses. A transfer path algorithm is proposed to measure the accessibility among standard poses and is used to support the calculation of the scores of standard poses. Using our method, the user can be redirected imperceptibly to the optimal pose with the best score among all the reachable poses from the user’s current pose during walking. Experiments demonstrate that our method outperforms state-of-the-art methods in various environment sizes and obstacle layouts.

Optimal Pose Guided Redirected Walking with Pose Score Precomputation

Image matting is widely studied for accurate foreground extraction. Most algorithms, including deep-learning based solutions, require a carefully edited trimap. Recent works attempt to combine the segmentation stage and matting stage in one CNN model, but errors occurring at the segmentation stage lead to unsatisfactory matte. We propose a user-guided approach for practical human matting. More precisely, we provide a good automatic initial matting and a natural way of interaction that reduces the workload of drawing trimaps and allows users to guide the matting in ambiguous situation. We also combine the segmentation and matting stage in an end-to-end CNN architecture and introduce a residual-learning module to support convenient stroke-based interaction. The proposed model learns to propagate the input trimap and modify the deep image features, which can efficiently correct the segmentation errors. Our model supports arbitrary forms of trimaps from carefully edited to totally unknown maps. Our model also allows users to choose from different foreground estimations according to their preference. We collected a large human matting dataset consisting of 12K real-world human images with complex background and human-object relations. The proposed model is trained on the new dataset with a novel trimap generation strategy that enables the model to tackle different test situations and highly improves the interaction efficiency. Our method outperforms other state-of-the-art automatic methods and achieve competitive accuracy when high-quality trimaps are provided. Experiments indicate that our interactive matting strategy is superior to separately estimating the trimap and alpha matte using two models.

Song-Hai Zhang

Papers

Adaptive Optimization Algorithm for Resetting Techniques in Obstacle-ridden Environments

One-step out-of-place resetting for redirected walking in VR

StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints

Optimal Pose Guided Redirected Walking with Pose Score Precomputation

User-Guided Deep Human Image Matting Using Arbitrary Trimaps