Journal ArticleDOI
Gesture controllers
Sergey Levine,Philipp Krähenbühl,Sebastian Thrun,Vladlen Koltun +3 more
- pp 124
TLDR
The modularity of the proposed method allows customization of a character's gesture repertoire, animation of non-human characters, and the use of additional inputs such as speech recognition or direct user control.Abstract:
We introduce gesture controllers, a method for animating the body language of avatars engaged in live spoken conversation. A gesture controller is an optimal-policy controller that schedules gesture animations in real time based on acoustic features in the user's speech. The controller consists of an inference layer, which infers a distribution over a set of hidden states from the speech signal, and a control layer, which selects the optimal motion based on the inferred state distribution. The inference layer, consisting of a specialized conditional random field, learns the hidden structure in body language style and associates it with acoustic features in speech. The control layer uses reinforcement learning to construct an optimal policy for selecting motion clips from a distribution over the learned hidden states. The modularity of the proposed method allows customization of a character's gesture repertoire, animation of non-human characters, and the use of additional inputs such as speech recognition or direct user control.read more
Citations
More filters
Proceedings ArticleDOI
Learning Individual Styles of Conversational Gesture
TL;DR: A method for cross-modal translation from "in-the-wild" monologue speech of a single speaker to their conversational gesture motion is presented and significantly outperforms baseline methods in a quantitative comparison.
Proceedings ArticleDOI
Virtual character performance from speech
TL;DR: A method for generating a 3D virtual character performance from the audio signal by inferring the acoustic and semantic properties of the utterance by utilizing semantics in addition to prosody to generate virtual character performances that are more appropriate than methods that use only prosody.
Proceedings ArticleDOI
Teaching Language and Culture with a Virtual Reality Game
TL;DR: Crystallize, a 3D video game for learning Japanese, is adapted so that it can be played in virtual reality with the Oculus Rift, and it is shown that the virtual reality design trained players how and when to bow, and that it increased participants' sense of involvement in Japanese culture.
Journal ArticleDOI
Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows
TL;DR: This paper proposes a new generative model for generating state‐of‐the‐art realistic speech‐driven gesticulation, called MoGlow, and demonstrates the ability to exert directorial control over the output style, such as gesture level, speed, symmetry and spacial extent.
Proceedings ArticleDOI
Synchronized gesture and speech production for humanoid robots
TL;DR: A model that is capable of synchronizing expressive gestures with speech, implemented on a Honda humanoid robot, and demonstrating the ability of observers to differentiate varying levels of expressiveness, excitement and speech synchronization is presented.
References
More filters
Journal ArticleDOI
A tutorial on hidden Markov models and selected applications in speech recognition
TL;DR: In this paper, the authors provide an overview of the basic theory of hidden Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and give practical details on methods of implementation of the theory along with a description of selected applications of HMMs to distinct problems in speech recognition.
Proceedings Article
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
TL;DR: This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Book
Dynamic Programming and Optimal Control
TL;DR: The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization.
Book
Hand and Mind: What Gestures Reveal about Thought
TL;DR: McNeill et al. as mentioned in this paper argue that gestures do not simply form a part of what is said and meant but have an impact on thought itself, and that gestures are global, synthetic, idiosyncratic, and imagistic.