In this paper, we present a novel technique of constructing phonetic decision trees (PDTs) for acoustic modeling in conversational speech recognition. We use random forests (RFs) to train a set of PDTs for each phone state unit and obtain multiple acoustic models accordingly. We investigate several methods of combining acoustic scores from the multiple models, including maximum-likelihood estimation of the weights of different acoustic models from training data, as well as using confidence score of -value or relative entropy to obtain the weights dynamically from online data. Since computing acoustic scores from the multiple models slows down decoding search, we propose clustering methods to compact the RF-generated acoustic models. The conventional concept of PDT-based state tying is extended to RF-based state tying. On each RF tied state, we cluster the Gaussian density functions (GDFs) from multiple acoustic models into classes and compute a prototype for each class to represent the original GDFs. In this way, the number of GDFs in each RF tied state is decreased greatly, which significantly reduces the time for computing acoustic scores. Experimental results on a telemedicine automatic captioning task demonstrate that the proposed RF-PDT technique leads to significant improvements in word recognition accuracy.

Random Forests of Phonetic Decision Trees for Acoustic Modeling in Conversational Speech Recognition

With a growing number of online videos, many producers feel the need to use video captions in order to expand content accessibility and face two main issues: production and alignment of the textual transcript. Both activities are expensive either for the high labor of human resources or for the employment of dedicated software. In this paper, we focus on caption alignment and we propose a novel, automatic, simple and low-cost mechanism that does not require human transcriptions or special dedicated software to align captions. Our mechanism uses a unique audio markup and intelligently introduces copies of it into the audio stream before giving it to an off-the-shelf automatic speech recognition (ASR) application; then it transforms the plain transcript produced by the ASR application into a timecoded transcript, which allows video players to know when to display every single caption while playing out the video. The experimental study evaluation shows that our proposal is effective in producing timecoded transcripts and therefore it can be helpful to expand video content accessibility.

An automatic caption alignment mechanism for off-the-shelf speech recognition technologies

We propose an RF-PDT+CD-DNN approach to generate an ensemble of context-dependent pre-trained deep neural networks (CD-DNNs) using random forests of phonetic decision trees (RF-PDTs) and constructing a CD-DNN-HMM-based ensemble acoustic model (EAM). We present evaluation results on the TIMIT dataset and a telemedicine automatic captioning dataset and demonstrate that the proposed RF-PDT+CD-DNN based EAM significantly outperforms the CD-DNN based single acoustic model (SAM) in phone and word recognition accuracies.

Building an ensemble of CD-DNN-HMM acoustic model using random forests of phonetic decision trees

This paper presents a new method of constructing phonetic decision trees (PDTs) for acoustic model state tying based on implicitly induced prior knowledge. Our hypothesis is that knowledge on pronunciation variation in spontaneous, conversational speech contained in a relatively large corpus can be used for building domain-specific or speaker-dependent PDTs. In view of tree-structure adaptation, this method leads to transformation of tree topology in contrast to keeping fixed tree structure as in traditional methods of speaker adaptation. A Bayesian learning framework is proposed to incorporate prior knowledge on decision rules in a greedy search of new decision trees, where the prior is generated by a decision tree growing process on a large data set. Experimental results on the telemedicine automatic captioning task demonstrate that the proposed approach results in consistent improvement in model quality and recognition accuracy.

Knowledge-Based Adaptive Decision Tree State Tying for Conversational Speech Recognition

We propose a maximum likelihood method for selecting template representatives, and in order to include more information in the selected template representatives, we further propose to create compressed template representatives by Gaussian mixture model (GMM) merging algorithm. A Kullback-Leibler (KL) divergence based local distance is proposed for Dynamic Time Warping (DTW) in template matching. Experimental results on the tasks of TIMIT phone recognition and large vocabulary continuous speech recognition demonstrated that the proposed template selection method significantly improved the recognition accuracy over the HMM baseline while only 5% or 10% templates were selected from the total templates, and the template compression method has provided further recognition accuracy gains over the template selection method.

New Methods for Template Selection and Compression in Continuous Speech Recognition.

We are developing an automatic captioning system for teleconsultation video teleconferencing (TC-VTC) in telemedicine, based on large vocabulary conversational speech recognition. In TC-VTC, doctors' speech contains a large number of infrequently used medical terms in spontaneous styles. Due to insufficiency of data, we adopted mixture language modeling, with models trained from several datasets of medical and nonmedical domains. This paper proposes novel modeling and estimation methods for the mixture language model (LM). Component LMs are trained from individual datasets, with class n-gram LMs trained from in-domain datasets and word n-gram LMs trained from out-of-domain datasets, and they are interpolated into a mixture LM. For class LMs, semantic categories are used for class definition on medical terms, names, and digits. The interpolation weights of a mixture LM are estimated by a greedy algorithm of forward weight adjustment (FWA). The proposed mixing of in-domain class LMs and out-of-domain word LMs, the semantic definitions of word classes, as well as the weight-estimation algorithm of FWA are effective on the TC-VTC task. As compared with using mixtures of word LMs with weights estimated by the conventional expectation-maximization algorithm, the proposed methods led to a 21% reduction of perplexity on test sets of five doctors, which translated into improvements of captioning accuracy

L. Schopp

Papers

A Novel Method of Language Modeling for Automatic Captioning in TC Video Teleconferencing