BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization

TLDR: Results indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new speakers using less than ten minutes of audio, achieving the same naturalness as produced for the speakers used to train the base model.

Abstract: We present BOFFIN TTS (Bayesian Optimization For FIne-tuning Neural Text To Speech), a novel approach for few-shot speaker adaptation. Here, the task is to fine-tune a pre-trained TTS model to mimic a new speaker using a small corpus of target utterances. We demonstrate that there does not exist a one-size-fits-all adaptation strategy, with convincing synthesis requiring a corpus-specific configuration of the hyper-parameters that control fine-tuning. By using Bayesian optimization to efficiently optimize these hyper-parameter values for a target speaker, we are able to perform adaptation with an average 30% improvement in speaker similarity over standard techniques. Results indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new speakers using less than ten minutes of audio, achieving the same naturalness as produced for the speakers used to train the base model.

Figures

Table 1: Comparing the mean naturalness scores achieved by BOFFIN TTS on target speakers (adapt-synth), by the base multi-speaker model on base speakers(base-synth), and by true audio for both target (adapt-truth) and base-model speakers (base-truth). We present each listener with samples across multiple base and adapted speakers and ask for a 5 point score from ‘completely unnatural’ to ‘completely natural’. We print mean responses alongside 95% confidence bounds.

Fig. 2: (a, b, c): Loss of the current best hyper-parameter configuration found by each system as we adapt to three randomly selected speakers from each corpora. We plot means and standard error for BOFFIN TTS and RS based on 5 runs with different random seeds, alongside the loss achieved by the base-line adaptation system. (d): Hyper-parameter values chosen by BOFFIN TTS for multiple target speakers across three different data-sets. Each point represents a single speaker. We plot the six hyper-parameters whose optimal values show the largest variation across speakers.

Fig. 1: Multi-speaker acoustic model architecture.

Fig. 3: MUSHRA tests for speaker similarity and naturalness. For similarity, we presented the same utterance synthesized by each system alongside a reference recording of the target speaker on another utterance and requested a rating of each system between ‘definitely a different person’ (0) and ‘definitely the same person’ (100). For naturalness, we repeat without a reference recording and instead asked for ratings between ‘completely unnatural’ and ‘completely natural’

Frequently asked questions

Q1. What are the contributions in "Boffin tts: few-shot speaker adaptation by bayesian optimization" ?

The authors present BOFFIN TTS ( Bayesian Optimization For FInetuning Neural Text To Speech ), a novel approach for few-shot speaker adaptation. The authors demonstrate that there does not exist a one-size-fits-all adaptation strategy, with convincing synthesis requiring a corpus-specific configuration of the hyperparameters that control fine-tuning. By using Bayesian optimization to efficiently optimize these hyper-parameter values for a target speaker, the authors are able to perform adaptation with an average 30 % improvement in speaker similarity over standard techniques.