https://www.jstage.jst.go.jp/article/jsprs1975/37/6/37_6_39/_pdf

1998 IEEE International Conference on SMCに参加して

This review summarises the status of silent speech interface (SSI) research. SSIs rely on non-acoustic biosignals generated by the human body during speech production to enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the first case and present latest SSI research aimed at providing new alternative and augmentative communication methods for persons with severe speech disorders. SSIs can employ a variety of biosignals to enable silent communication, such as electrophysiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking of articulator movements using imaging techniques. Depending on the disorder, some sensing techniques may be better suited than others to capture speech-related information. For instance, EMG and imaging techniques are well suited for laryngectomised patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for severely paralysed individuals. From the biosignals, SSIs decode the intended message, using automatic speech recognition or speech synthesis algorithms. Despite considerable advances in recent years, most present-day SSIs have only been validated in laboratory settings for healthy users. Thus, as discussed in this paper, a number of challenges remain to be addressed in future research before SSIs can be promoted to real-world applications. If these issues can be addressed successfully, future SSIs will improve the lives of persons with severe speech impairments by restoring their communication capabilities.

Silent Speech Interfaces for Speech Restoration: A Review

In this letter, we propose a high-quality emotional speech synthesis system, using emotional vector space, i.e., the weighted sum of global style tokens (GSTs). Our previous research verified the feasibility of GST-based emotional speech synthesis in an end-to-end text-to-speech synthesis framework. However, selecting appropriate reference audio (RA) signals to extract emotion embedding vectors to the specific types of target emotions remains problematic. To ameliorate the selection problem, we propose an effective way of generating emotion embedding vectors by utilizing the trained GSTs. By assuming that the trained GSTs represent an emotional vector space, we first investigate the distribution of all the training samples depending on the type of each emotion. We then regard the centroid of the distribution as an emotion-specific weighting value, which effectively controls the expressiveness of synthesized speech, even without using the RA for guidance, as it did before. Finally, we confirm that the proposed controlled weight-based method is superior to the conventional emotion label-based methods in terms of perceptual quality and emotion classification accuracy.

An Effective Style Token Weight Control Technique for End-to-End Emotional Speech Synthesis

State-of-the-art silent speech interface systems apply vocoders to generate the speech signal directly from articulatory data Most of these approaches concentrate on estimating just the spectral features of the vocoder, and use the original F0, a constant F0 or white noise as excitation This solution is based on the assumption that the F0 curve is unpredictable from articulatory data that does not contain direct measurements of the vocal fold vibration Here, we experimented with deep neural networks to perform articulatory-to-acoustic conversion from ultrasound images, with an emphasis on estimating the voicing feature and the F0 curve from the ultrasound input Contrary to the common belief that F0 is unpredictable, we attained a correlation rate of 074 between the original and the predicted F0 curve What is more, the listening tests revealed that our subjects could not distinguish the sentences synthesized using the DNN-estimated and the original F0 curve, and ranked them as having the same quality

F0 Estimation for DNN-Based Ultrasound Silent Speech Interfaces

This paper gives an overview of facial animation techniques. While facial animation is currently used in the entertainment and advertisement industry, it becomes part of dialog systems in commercial applications. Animation techniques based on 3D models and image-based rendering are presented and evaluated by the following characteristics: automatism, realism and flexibility. While animation techniques based on 3D models provide for high flexibility and automatism, they often lack realism. Image-based techniques achieve photo realism, but lack in automatism and flexibility.

Talking Faces - Technologies and Applications.

This paper presents an articulatory-to-acoustic conversion method using electromagnetic midsagittal articulography (EMA) measurements as input features. Neural networks, including feed-forward deep neural networks (DNNs) and recurrent neural networks (RNNs) with long short-term term memory (LSTM) cells, are adopted to map EMA features towards not only spectral features (i.e. mel-cepstra) but also excitation features (i.e. power, U/V flag and F0). Then speech waveforms are reconstructed using the predicted spectral and excitation features. A cascaded prediction strategy is proposed to utilize the predicted spectral features as auxiliary input to boost the prediction of excitation features. Experimental results show that LSTM-RNN models can achieve better objective and subjective performance in articulatory-to-spectral conversion than DNNs and Gaussian mixture models (GMMs). The strategy of cascaded prediction can increase the accuracy of excitation feature prediction and the neural network-based methods also outperform the GMM-based approach when predicting power features.

Articulatory-to-Acoustic Conversion with Cascaded Prediction of Spectral and Excitation Features Using Neural Networks.

Articulatory-to-acoustic conversion using BLSTM-RNNs with augmented input representation

This letter proposes an improved statistical parametric speech synthesis (SPSS) method which utilizes auxiliary information for acoustic modeling under generalized distillation framework. In conventional SPSS, acoustic models are trained using context features as input and acoustic features as output. In our proposed method, two acoustic models, so-called teacher and student, are involved. Both of them are recurrent neural networks (RNN) with bidirectional long short-term memory (BLSTM) units. The teacher, which aims to provide the student with additional knowledge, employs auxiliary features (e.g., articulatory features or spectra of short-time Fourier transform) in addition to the conventional input and output of acoustic models. The student, which serves as the final acoustic model for synthesis, adopts a multitask learning architecture which uses the outcome of the teacher as the target of its secondary task. Experimental results show that this method can achieve better accuracy of acoustic feature prediction and produce more natural synthetic speech than conventional BLSTM-RNN-based acoustic modeling with single-task or multitask learning.

Statistical Parametric Speech Synthesis Using Generalized Distillation Framework

This paper proposes methods of using restricted Boltzmann machines (RBM) to generate the sequence of lip images for visual speech synthesis. The aim of our proposed methods is to alleviate the over-smoothing effect of the conventional hidden Markov model (HMM) based statistical approach for lip synthesis. Two model structures using RBMs to model and generate lip movements are investigated in this paper. First, RBMs are adopted to replace Gaussian distributions as the density functions of HMM states. Second, a deep belief network (DBN) is constructed by stacking up multiple RBMs to model the joint distribution between the lip image of each frame and its corresponding context features. Experimental results show that our proposed methods can improve the quality of generated lip images significantly. The method of using DBN model structure and raw pixel features achieves the best performance in our experiments.

Zheng-Chen Liu

Papers

Articulatory-to-Acoustic Conversion with Cascaded Prediction of Spectral and Excitation Features Using Neural Networks.

Articulatory-to-acoustic conversion using BLSTM-RNNs with augmented input representation

Statistical Parametric Speech Synthesis Using Generalized Distillation Framework

LIP movement generation using restricted Boltzmann machines for visual speech synthesis