We present an end-to-end system for augmented and virtual reality telepresence, called Holoportation. Our system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture and objects, using a set of new depth cameras. These 3D models can also be transmitted in real-time to remote users. This allows users wearing virtual or augmented reality displays to see, hear and interact with remote participants in 3D, almost as if they were present in the same physical space. From an audio-visual perspective, communicating and interacting with remote users edges closer to face-to-face communication. This paper describes the Holoportation technical system in full, its key interactive capabilities, the application scenarios it enables, and an initial qualitative study of using this new communication medium.

Holoportation: Virtual 3D Teleportation in Real-time

Creating realistic animated models of people is a central task in digital content production. Traditionally, highly skilled artists and animators construct shape and appearance models for digital character. They then define the character's motion at each time frame or specific key-frames in a motion sequence to create a digital performance. Increasingly, producers are using motion capture technology to record animations from an actor's performance. This technology reduces animation production time and captures natural movements to create a more believable production. However, motion capture requires the use of specialist suits and markers and only records skeletal motion. It lacks the detailed secondary surface dynamics of cloth and hair that provide the visual realism of a live performance. Over the last decade, we have investigated studio capture technology with the objective of creating models of real people that accurately reflect the time-varying shape and appearance of the whole body with clothing. Surface capture is a fully automated system for capturing a human's shape and appearance as well as motion from multiple video cameras to create highly realistic animated content from an actor's performance in full wardrobe. Our system solves two key problems in performance capture: scene capture from a limited number of camera views and efficient scene representation for visualization

/pdf/surface-capture-for-performance-based-animation-20kfxcx8s8.pdf

Surface Capture for Performance-Based Animation

This review paper discusses the role of haptics within virtual medical training applications, particularly, where it can be used to aid a practitioner to learn and practice a task. The review summarizes aspects to be considered in the deployment of haptics technologies in medical training. First, both force/torque and tactile feedback hardware solutions that are currently produced commercially and in academia are reviewed, followed by the available haptics-related software and then an in-depth analysis of medical training simulations that include haptic feedback. The review is summarized with scrutiny of emerging technologies and discusses future directions in the field.

The Role of Haptics in Medical Training Simulators: A Survey of the State of the Art

As both CPUs and GPUs become employed in a wide range of applications, it has been acknowledged that both of these Processing Units (PUs) have their unique features and strengths and hence, CPU-GPU collaboration is inevitable to achieve high-performance computing. This has motivated a significant amount of research on heterogeneous computing techniques, along with the design of CPU-GPU fused chips and petascale heterogeneous supercomputers. In this article, we survey Heterogeneous Computing Techniques (HCTs) such as workload partitioning that enable utilizing both CPUs and GPUs to improve performance and/or energy efficiency. We review heterogeneous computing approaches at runtime, algorithm, programming, compiler, and application levels. Further, we review both discrete and fused CPU-GPU systems and discuss benchmark suites designed for evaluating Heterogeneous Computing Systems (HCSs). We believe that this article will provide insights into the workings and scope of applications of HCTs to researchers and motivate them to further harness the computational powers of CPUs and GPUs to achieve the goal of exascale performance.

https://dl.acm.org/doi/pdf/10.1145/2788396

A Survey of CPU-GPU Heterogeneous Computing Techniques

This paper describes a tactile display which provides unrestricted tactile feedback in air without any mechanical contact. It controls ultrasound and produces a stress field in a 3D space. The principle is based on a nonlinear phenomenon of ultrasound: Acoustic radiation pressure. The fabricated prototype consists of 324 airborne ultrasound transducers, and the phase and intensity of each transducer are controlled individually to generate a focal point. The DC output force at the focal point is 16 mN and the diameter of the focal point is 20 mm. The prototype produces vibrations up to 1 kHz. An interaction system including the prototype is also introduced, which enables users to see and touch virtual objects.

Noncontact Tactile Display Based on Radiation Pressure of Airborne Ultrasound

SOFA is an Open Source framework primarily targeted at real-time simulation, with an emphasis on medical simulation. It is mostly intended for the research community to help develop newer algorithms, but can also be used as an efficient prototyping tool. Based on an advanced software architecture, it allows to: * create complex and evolving simulations by combining new algorithms with algorithms already included in SOFA * modify most parameters of the simulation - deformable behavior, surface representation, solver, constraints, collision algorithm, etc. - by simply editing an XML file * build complex models from simpler ones using a scene-graph description * efficiently simulate the dynamics of interacting objects using abstract equation solvers * reuse and easily compare a variety of available methods

https://hal.inria.fr/inria-00319416/document

SOFA--an open source framework for medical simulation.

Simulation Open Framework Architecture (SOFA) is an open-source C++ library primarily targeted at interactive computational medical simulation. SOFA facilitates collaborations between specialists from various domains, by decomposing complex simulators into components designed independently and organized in a scenegraph data structure. Each component encapsulates one of the aspects of a simulation, such as the degrees of freedom, the forces and constraints, the differential equations, the main loop algorithms, the linear solvers, the collision detection algorithms or the interaction devices. The simulated objects can be represented using several models, each of them optimized for a different task such as the computation of internal forces, collision detection, haptics or visual display. These models are synchronized during the simulation using a mapping mechanism. CPU and GPU implementations can be transparently combined to exploit the computational power of modern hardware architectures. Thanks to this flexible yet efficient architecture, SOFA can be used as a test-bed to compare models and algorithms, or as a basis for the development of complex, high-performance simulators.

/pdf/sofa-a-multi-model-framework-for-interactive-physical-131uqsiu67.pdf

SOFA: A Multi-Model Framework for Interactive Physical Simulation

This article describes a series of contributions in the field of real-time simulation of soft tissue biomechanics These contributions address various requirements for interactive simulation of complex surgical procedures In particular, this article presents results in the areas of soft tissue deformation, contact modelling, simulation of cutting, and haptic rendering, which are all relevant to a variety of medical interventions The contributions described in this article share a common underlying model of deformation and rely on GPU implementations to significantly improve computation times This consistency in the modelling technique and computational approach ensures coherent results as well as efficient, robust and flexible solutions

/pdf/gpu-based-real-time-soft-tissue-deformation-with-cutting-and-45jh0xgq9j.pdf

GPU-based real-time soft tissue deformation with cutting and haptic feedback.

Real-time simulation of contact and cutting of heterogeneous soft-tissues

A service discovery framework provides a collection of protocols for developing dynamic client/server applications, allowing clients to find and use services without any previous knowledge of the locations or characteristics of the services. There are currently many service discovery, technologies available or in development, including Jini, UPnP, SLP, Salutation, Bluetooth SDP, and Ninja. These have similar high-level goals, but quite different architectures. Each software or hardware product utilizing service discovery will typically use only one of these protocols, meaning that clients and services using different technologies will not be able to cooperate. Since it is likely that several protocols will be widely used, there is a need for interoperability frameworks that allow clients and services written using different service discovery technologies to cooperate. This paper presents a Jini/UPnP interoperability framework that allows Jini clients to use UPnP services and UPnP clients to use Jini services, without modification to service or client implementations. As service specifications are typically developed independently for each protocol, a fully automatic interoperability solution is not currently practical, so we introduce service-specific proxies to bridge Jini and UPnP. Our goal is to reduce the amount of effort required to support new service types and our framework includes a substantial amount of support for rapid proxy development. A modest development effort is required to support each new service type, and our initial (and highly unscientific) measurements reveal that the level of effort is typically on the order of one day by a member of our team.

/pdf/jini-meets-upnp-an-architecture-for-jini-upnp-247xdw5rct.pdf

Jérémie Allard

Papers

SOFA--an open source framework for medical simulation.

SOFA: A Multi-Model Framework for Interactive Physical Simulation

GPU-based real-time soft tissue deformation with cutting and haptic feedback.

Real-time simulation of contact and cutting of heterogeneous soft-tissues

Jini meets UPnP: an architecture for Jini/UPnP interoperability