“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

Although there is a vast clinical literature on phantom limbs, there have been no experimental studies on the effects of visual input on phantom sensations. We introduce an inexpensive new device--a 'virtual reality box'--to resurrect the phantom visually to study inter-sensory effects. A mirror is placed vertically on the table so that the mirror reflection of the patient's intact had is 'superimposed' on the felt position of the phantom. We used this procedure on ten patients and found the following results. 1. In six patients, when the normal hand was moved, so that the phantom was perceived to move in the mirror, it was also felt to move; i.e. kinesthetic sensations emerged in the phantom. In D.S. this effect occurred even though he had never experienced any movements in the phantom for ten years before we tested him. He found the return of sensations very enjoyable. 2. Repeated practice led to a permanent 'disappearance' of the phantom arm in patient D.S. and the hand became telescoped into the stump near the shoulder. 3. Using an optical trick, impossible postures--e.g. extreme hyperextension of the fingers--could be induced visually in the phantom. In one case this was felt as a transient 'painful tug' in the phantom. 4. Five patients experienced involuntary painful 'clenching spasms' in the phantom hand and in four of them the spasms were relieved when the mirror was used to facilitate 'opening' of the phantom hand; opening was not possible without the mirror. 5. In three patients, touching the normal hand evoked precisely localized touch sensations in the phantom. Interestingly, the referral was especially pronounced when the patients actually 'saw' their phantom being touched in the mirror. Indeed, in a fourth patient (R.L.) the referral occurred only if he saw his phantom being touched: a curious form of synaesthesia. These experiments lend themselves readily to imaging studies using PET and fMRI. Taken collectively, they suggest that there is a considerable amount of latent plasticity even in the adult human brain. For example, precisely organized new pathways, bridging the two cerebral hemispheres, can emerge in less than three weeks. Furthermore, there must be a great deal of back and forth interaction between vision and touch, so that the strictly modular, hierarchical model of the brain that is currently in vogue needs to be replaced with a more dynamic, interactive model, in which 're-entrant' signalling plays the main role.

Synaesthesia in phantom Limbs induced with mirrors

Due to recent improvements in virtual reality (VR) technology, the number of novel applications for entertainment, education, and rehabilitation has increased. The primary goal of these applications is to enhance the sense of belief that the user is “present” in the virtual environment. By tracking the user’s skeleton in real-time, it is possible to synchronize the avatar’s motions with the user’s motions. Although current common devices implement body tracking to a certain degree, most approaches are limited by either high latency or insufficient accuracy. Due to the lack of positional and rotation data, the current VR applications typically do not represent the user’s motions. In this paper, we present an accurate, low-latency body tracking approach for VR-based applications using Vive Trackers. Using a HTC Vive headset and Vive Trackers, we have been able to create an immersive VR experience, by animating the motions of the avatar as smoothly, rapidly and as accurately as possible. An evaluation showed our solution is capable of tracking both joint rotation and position with reasonable accuracy and a very low end-to-latency of $$6.71 \pm 0.80\hbox { ms}$$
 . Due to this merely imperceptible delay and precise tracking, our solution can show the movements of the user in real-time in order to create deeper immersion.

Real-time body tracking in virtual reality using a Vive tracker

Over the last few decades, Mixed Reality (MR) interfaces have received great attention from academia and industry. Although a considerable amount of research has already been done to support collaboration between users in MR, there is still no systematic review to determine the current state of collaborative MR applications. In this paper, collaborative MR studies published from 2013 to 2018 were reviewed. A total of 259 papers have been categorised based on their application areas, types of display devices used, collaboration setups, and user interaction and experience aspects. The primary contribution of this paper is to present a high-level overview of collaborative MR influence across several research disciplines. The achievements from each application area are summarised. In addition, remarkable papers in their respective areas are highlighted. Among other things, our study finds that there are three complementary factors to support and enhance collaboration in MR environments: (i) annotation techniques, which provide non-verbal communication cues to users, (ii) cooperative object manipulation techniques, which divide complex 3D object manipulation process into simpler tasks between different users, and (iii) user perception and cognition studies, which aim to lessen cognitive workload for task understanding and completion, and to increase users’ perceptual awareness and presence. Finally, this paper identifies research gaps and future directions that can be useful for researchers who want to explore ways on how to foster collaboration between users and to develop collaborative applications in MR.

A systematic review of the current state of collaborative mixed reality technologies: 2013–2018

Due to recent advances in virtual reality (VR) technology, the development of immersive VR applications that track body motions and visualize a full-body avatar is attracting increasing research interest. This paper reviews related research to gather and to critically analyze recent improvements regarding the potential of full-body motion reconstruction in VR applications. We conducted a systematic literature search, matching VR and full-body tracking related keywords on IEEE Xplore, PubMed, ACM, and Scopus. Fifty-three publications were included and assigned in three groups: studies using markerless and marker-based motion tracking systems as well as systems using inertial measurement units. All analyzed research publications track the motions of the user wearing a head-mounted display and visualize a full-body avatar. The analysis confirmed that a full-body avatar can enhance the sense of embodiment and can improve the immersion within the VR. The results indicated that the Kinect device is still the most frequently used sensor (27 out of 53). Furthermore, there is a trend to track the movements of multiple users simultaneously. Many studies that enable multiplayer mode in VR use marker-based systems (7 out of 17) because they are much more robust and can accurately track full-body movements of multiple users in real-time.

A Survey of Full-Body Motion Reconstruction in Immersive Virtual Reality Applications

Rehabilitation for stroke afflicted patients, through exercises tailored for individual needs, aims at relearning basic motor skills, especially in the extremities. Rehabilitation through Augmented Reality (AR) based games engage and motivate patients to perform exercises which, otherwise, maybe boring and monotonic. Also, mirror therapy allows users to observe one's own movements in the game providing them with good visual feedback. This paper presents an augmented reality based system for rehabilitation by playing four interactive, cognitive and fun Exergames (exercise and gaming). The system uses low-cost RGB-D cameras such as Microsoft Kinect V2 to capture and generate 3D model of the person by extracting him/her from the entire captured data and immersing it in different interactive virtual environments. Animation based limb movement enhancement along with cognitive aspects incorporated in the game can help in positive reinforcement, progressive challenges and motion improvement. Recording module of the toolkit allows future reference and facilitates feedback from the physician. 10 able-bodied users, 2 psychological experts and 2 Physical Medicine and Rehabilitation physicians evaluated the user experience and usability aspects of the exergames. Results obtained shows the games to be fun and realistic, and at the same time engaging and motivating for performing exercises.

Augmented reality-based exergames for rehabilitation

Training through Multi-modal Collaborative Virtual Laboratory (MMCVL) aims at developing or improving skills of students without the need of them being physically present. MMCVL addresses the problems faced by the current education system such as lack of resources and safe use of expensive laboratory equipment. It also provides a means for easy collaboration between students and teachers from different universities and schools. Multimedia data in terms of 3D mesh, skeleton, audio and virtual object physics interaction are incorporated in MMCVL to provide a 3D Tele-Immersion environment. The system uses low-cost RGB-D cameras such as Microsoft Kinect V2 to capture and generate 3D model of the person by extracting him/her from the entire captured data and immersing it in different interactive virtual environments. Different lab environments can be virtually created to provide for and replace the real laboratories. Object interactions allow for users to interact with the virtual objects present in the scene similar to real lab setup. Recording module of the toolkit allows future reference for training and testing of the student. The system also facilitates virtual feedback by giving the correct measurements, video demonstrations as well as audio communication with the collaborating teacher. Two different Chemistry experiments were performed to evaluate the user experience and usability aspects of MMCVL. Results obtained from the user study analysis prove the system to be fun and realistic, and at the same time engaging and motivating for learning laboratory experiments.

Experiences with Multi-modal Collaborative Virtual Laboratory (MMCVL)

With the ever-increasing amount of data, the central challenge in multimodal learning involves limitations of labelled samples. For the task of classification, techniques such as meta-learning, zero-shot learning, and few-shot learning showcase the ability to learn information about novel classes based on prior knowledge. Recent techniques try to learn a cross-modal mapping between the semantic space and the image space. However, they tend to ignore the local and global semantic knowledge. To overcome this problem, we propose a Multimodal Variational Auto-Encoder (M-VAE) which can learn the shared latent space of image features and the semantic space. In our approach we concatenate multimodal data to a single embedding before passing it to the VAE for learning the latent space. We propose the use of a multi-modal loss during the reconstruction of the feature embedding through the decoder. Our approach is capable to correlating modalities and exploit the local and global semantic knowledge for novel sample predictions. Our experimental results using a MLP classifier on four benchmark datasets show that our proposed model outperforms the current state-of-the-art approaches for generalized zero-shot learning.

Generalized Zero-Shot Learning Using Multimodal Variational Auto-Encoder With Semantic Concepts

Applications involving 3D scanning and reconstruction & 3D Tele-immersion provide an immersive experience by capturing a scene using multiple RGB-D cameras, such as Kinect. Prior knowledge of intrinsic calibration of each of the cameras, and extrinsic calibration between cameras, is essential to reconstruct the captured data. The intrinsic calibration for a given camera rarely ever changes, so only needs to be estimated once. However, the extrinsic calibration between cameras can change, even with a small nudge to the camera. Calibration accuracy depends on sensor noise, features used, sampling method, etc., resulting in the need for iterative calibration to achieve good calibration.In this paper, we introduce a skeleton based approach to calibrate multiple RGB-D Kinect cameras in a closed setup, automatically without any intervention, within a few seconds. The method uses only the person present in the scene to calibrate, removing the need for manually inserting, detecting and extracting other objects like plane, checker-board, sphere, etc. 3D joints of the extracted skeleton are used as correspondence points between cameras, after undergoing accuracy and orientation checks. Temporal, spatial, and motion constraints are applied during the point selection strategy. Our calibration error checking is inexpensive in terms of computational cost and time and hence is continuously run in the background. Automatic re-calibration of the cameras can be performed when the calibration error goes beyond a threshold due to any possible camera movement. Evaluations show that the method can provide fast, accurate and continuous calibration, as long as a human is moving around in the captured scene.

https://dl.acm.org/doi/pdf/10.1145/3204949.3204969

Skeleton-based continuous extrinsic calibration of multiple RGB-D kinect cameras

Cybersickness prediction is one of the significant research challenges for real-time cybersickness reduction. Researchers have proposed different approaches for predicting cybersickness from bio-physiological data (e.g., heart rate, breathing rate, electroencephalogram). However, collecting bio-physiological data often requires external sensors, limiting locomotion and 3D-object manipulation during the virtual reality (VR) experience. Limited research has been done to predict cybersickness from the data readily available from the integrated sensors in head-mounted displays (HMDs) (e.g., head-tracking, eye-tracking, motion features), allowing free locomotion and 3D-object manipulation. This research proposes a novel deep fusion network to predict cybersickness severity from heterogeneous data readily available from the integrated HMD sensors. We extracted 1755 stereoscopic videos, eye-tracking, and head-tracking data along with the corresponding self-reported cybersickness severity collected from 30 participants during their VR gameplay. We applied several deep fusion approaches with the heterogeneous data collected from the participants. Our results suggest that cybersickness can be predicted with an accuracy of 87.77% and a root-mean-square error of 0.51 when using only eye-tracking and head-tracking data. We concluded that eye-tracking and head-tracking data are well suited for a standalone cybersickness prediction framework.

Kevin Desai

Papers

Augmented reality-based exergames for rehabilitation

Experiences with Multi-modal Collaborative Virtual Laboratory (MMCVL)

Generalized Zero-Shot Learning Using Multimodal Variational Auto-Encoder With Semantic Concepts

Skeleton-based continuous extrinsic calibration of multiple RGB-D kinect cameras

Cybersickness Prediction from Integrated HMD’s Sensors: A Multimodal Deep Fusion Approach using Eye-tracking and Head-tracking Data