We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

High Fidelity Neural Audio Compression

Preface. Acronyms. Notation. Introduction. Signal Processing Techniques. Stochastic Processes and Models. Linear Prediction. Scalar Quantization. Pulse Code Modulation and Its Variants. Vector Quantization. Scalar Quantization of Linear Prediction Coefficient. Linear Prediction Coding. Regular-Pulse Excitation Coders. Code-Excited Linear Prediction. The Federal Standard Version of CELP. Vector Sum Excited Linear Prediction. Low-Delay CELP. Vector Quantization of Linear Prediction Coefficient. Algebraic CELP. Mixed Excitation Linear Prediction. Source-Controlled Variable Bit-Rate CELP. Speech Quality Assessment. Appendix A: Minimum-Phase Property of the Forward Prediction-Error Filter. Appendix B: Some Properties of Line Spectral Frequency. Appendix C: Research Directions in Speech Coding. Appendix D: Linear Combiner for Pattern Classification. Appendix E: CELP: Optimal Long-Term Predictor to Minimize the Weighted Difference. Appendix F: Review of Linear Algebra: Orthogonality, Basis, Linear Independence, and the Gram-Schmidt Algorithm. Bibliography. Index.

Speech coding algorithms : foundation and evolution of standardized coders

In this paper, we present a new speech bandwidth extension method (BWE) using recurrent temporal restricted Boltzmann machine (RTRBM). The conventional Gaussian mixture model (GMM)-based and deep neural networks (DNNs)-based BWE methods perform stably and effectively. However, the mapping function of GMM-based methods is a piecewise linear transformation, which is insufficient to model the complex nonlinear mapping relationship between the spectral envelope features of low frequency (LF) and high frequency (HF). In the conventional DNNs methods, temporal correlations across speech frames are ignored, resulting in spectral detail loss of the reconstructed speech by BWE. To counter these issues, a multilayer DNN which is composed of two RTRBMs and a feedforward neural network (NN) is employed to obtain time information and model deep nonlinear relationships between the spectral envelope features of LF and HF. The proposed method takes advantage of the strong ability of RTRBM in discovering the temporal correlation in the high-order space and modeling deep nonlinear relationships between input and output. Both the objective and subjective evaluations indicate that our proposed method outperforms the conventional GMM-based methods and other NN-based methods.

Speech Bandwidth Extension Using Recurrent Temporal Restricted Boltzmann Machines

This research addresses an issue of wide band (WB) speech transmission (having cut-off frequency $$\hbox {f}_{\mathrm{c}}=8$$fc=8 kHz) over standard narrow band (NB) communication link (supporting bandwidth of 300---3,400 Hz). A long transition time for technological up-gradation from NB to WB systems eventually lead to development of backward compatible techniques such as artificial bandwidth extension (ABE) which is capable of providing bandwidth of 50---7,000 Hz, in turn contributing toll quality recovered speech at receiving end. This paper investigates a novel approach to compute high band (HB) features using linear predictive coding (LPC) technique at transmitter from given input WB speech corpus. These encoded features are embedded into bitstream of proposed GSM Full Rate 06.10 NB speech coder using joint source coding and data hiding technique and then transmitted to receiver. At receiver, these HB features are extracted to reproduce HB recovered speech using watermark extraction algorithm and for the same different extension of excitation techniques have been adopted and implemented. An e-test bench is created to implement this proposed ABE coder in MATLAB and series of simulations are carried out using Subjective (mean opinion score--MOS) and Objective (perceptual evaluation of speech quality--PESQ) analysis. Obtained results for both analyses advocate performance improvement of proposed ABE coder over legacy GSM 06.10 FR NB coder for various extension of excitation techniques.

A novel approach for artificial bandwidth extension of speech signals by LPC technique over proposed GSM FR NB coder using high band feature extraction and various extension of excitation methods

In this paper, we propose a subband-based ensemble of sequential deep neural networks (DNNs) for bandwidth extension (BWE). First, the narrow-band spectra are folded into the high-band (HB) region to generate the high-band spectra, and then the energy levels of the HB spectra are adjusted using the DNN-based on the log-power spectra feature. For this, we basically build the multiple DNNs, which is responsible for each subband of the HB and the DNN ensemble is sequentially connected from lower to higher subbands. This sequential structure for the DNN ensemble carries out the denoising and HB regression to better estimate the HB energy levels. In addition, we use the voiced/unvoiced (V/UV) classification to differently apply the DNN ensemble depending on either V/UV sounds. To demonstrate the performance of the proposed BWE algorithm, we compare it with a speech production model-based BWE system and a DNN-based BWE system in which the log-power spectra in the HB are estimated directly. The experimental results show that the proposed approach provides better speech quality than conventional approaches.

Sequential Deep Neural Networks Ensemble for Speech Bandwidth Extension

In the recent scenario of wired and wireless communication systems, amongst many reasons for the overall degradation of recovered speech quality at receiver, one of the major reasons to be considered is utilization of Narrow Band (NB) end devices and NB transmission medium supporting bandwidth of 300 Hz-3400Hz. The inherent drawback of such NB speech signal is it sounds muffled and thin because of absence of High Band (HB) spectral components. With state of the art development of communication technologies and with increased availability of end terminals capable of transmitting and receiving Wide Band (WB) signals (having bandwidth from 50 Hz to 7000 Hz), end users prefer to listen to WB speech. In order to offer a fully WB communication over wired and wireless media, both end devices and network need to be made WB compatible. A long transition period has been elapsed for upgrading existing NB systems (both end terminals and network) to fully WB compatible systems. In-between new methods have been developed to artificially extend the bandwidth of NB telephonic speech at receiver for improving the quality of recovered speech. A major task of Artificial Bandwidth Extension (ABE) is to reconstruct missing WB spectral components at receiver with the use of available NB speech. This paper discusses motivation for developing ABE algorithm along with exhaustive comparative studies of implementing it with various approaches. Issues and limitations related to real time implementation of ABE algorithm are also addressed. Alternative approaches like usage of ABE with side information transmission along with coded NB speech are also demonstrated. Main objective of ABE with side information is to extract WB spectral components from WB input speech and to embed these derived spectral components into coded NB speech signal and finally transmit them onto a NB channel. Reverse procedure can be carried out at receiver to artificially produce WB speech. Here, it is to be noted that transmission channel is NB whereas end terminals are made WB compatible so this method provides alternative solution to coexisting state of the art WB coders (which require WB channels) while offering comparable speech quality and giving natural sounding in terms of intelligibility and naturalness.

Artificial Bandwidth Extension of Speech & Its Applications in Wireless Communication Systems: A Review

Today, the primary constrain in wireless communication system is limited bandwidth and power. Wireless systems involved in transmission of speech envisage that efficient and effective methods be developed (bandwidth usage & power) to transmit and receive the same while maintaining quality-of-speech, especially at the receiving end. Speech coding is a technique, since the era of digitization (digital) and computerization (computational and processing horsepower--DSP) that has been a material-of-research for quite some time amongst the scientific and academic community. Amongst all elements of the communication system (transmitter, channel and receiver), transmission channel (carrier of information/data, also called the medium) is the most critical and plays a key role in the transmission and reception of information/data.

This paper proposes some modifications in the selection process of grid positions in Regular Pulse Excitation section of 13 kbps ETSI GSM 06.10 Full Rate Speech coder so that there is an overall 1.8 kbps (36 bits / each 20 ms frame) reduction in bit-rate which can be utilized for either improving error detection and correction at channel coding or for hidden data embedding and transmission over wireless link. Both Standard GSM FR and proposed GSM FR are implemented in MATLAB. Here, Subjective and Objective analysis are carried out on a proposed system to evaluate its performance and the results obtained are then compared with the results of GSM 06.10 Full Rate coder using set of tables and graphs. As can be observed from obtained results that both PESQ and MOS scores are quite comparable for each wave files and marginal degradation of both can be witnessed with respect to decrease in codec bitrates.

Proposed modifications in ETSI GSM 06.10 full rate speech codec and its overall evaluation of performance using MATLAB

This paper investigates application of Code Excited Linear Prediction algorithm on Adaptive Multi Rate Wideband coder. The proposed coder can adaptively change its bit-rate based on C/I ratio depending on channel conditions. The coder has nine bit-rates from 6.6 kbps to 23.85 kbps. An e-test bench using MATLAB is created to implement proposed coder and series of simulations are carried out to judge the performance of implemented coder using Subjective and Objective analysis. Simulation results clearly advocate that it is possible to produce variable bit rate (by tuning to channel conditions) in CELP coder by affecting coefficients of coder while still maintaining comparable speech quality with reference to AMR WB coder standardized by 3GPP and ITU-T [5]. It is also evident from the simulation results that Signal to Noise Ratio (SNR), Segmented SNR, Perceptual Evaluation of Speech Quality (PESQ) and Mean Opinion Score (MOS) increases with increase in bit rates of proposed coder and Absolute Error (Abs Err), Mean Square Error (MSE), Root Mean Square Error (RMSE) reduces with increase in bitrates.

Adaptive Multi-rate Wideband Speech Codec Based on CELP Algorithm: Architectural Study, Implementation & Performance Analysis

Today, the primary constrain in wireless communication system is limited bandwidth and power. Wireless systems involved in transmission of speech envisage that efficient and effective methods need to be developed for maintaining quality-of-speech, especially at the receiving end, with maximum saving of bandwidth and power. Amongst all elements of the communication system (transmitter, channel and receiver), transmission channel (carrier of information/data, also called the medium) is the most critical and plays a key role in the transmission and reception of information/data. Channel conditions decide the quality of speech at receiver. Modeling a channel is a complex task. Many techniques are adopted to mitigate the effect of the channel. AMR (Adaptive Multi Rate) is one such technique that counteracts the deleterious effect of the channel on speech. This technique employs variable bit rate that dynamically switches to specific modes of operation (switching bit rates--called modes of operation) depending upon the channel conditions.

In this paper, the application of Code Excited Linear Prediction (CELP) source coder on speech followed by AMR codec is investigated and studied. An e-test bench using MATLAB is created to implement the CELP based AMR Codec scheme, and the same studied and investigated through a series of simulation. Here, both subjective and objective evaluations are carried out. Objective evaluations are categorized into waveform based, spectral based and perceptual based analysis. The results of the simulations are recorded and compared in various graphs and tables, which include calculation of various parameters like Absolute Error (ABS), Mean Square Error (MSE), Root Mean Square Error (RMSE), Signal to Noise Ratio (SNR), segmental SNR (segSNR) (Y. Hu and P. Loizou in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 1, pp. 153---156, 2006a; Proc. Interspeech, pp. 1447---1450, 2006b), Weighted-Slope Spectral distance (WSS) (Y. Hu and P. Loizou in Speech Commun. 49, 588---601, 2007), Perceptual Evaluation of Speech Quality (PESQ) (ITU-T rec. P.862, 2000), Log-Likelihood Ratio (LLR), Itakura-Saito Distance measure (ISD), Cepstrum Distance Measures (CEP) (V. Turbin and N. Faucheur in Proc. Online Workshop Meas. Speech Audio Quality Netw., pp. 81---84, 2005), Frequency Weighted Segmental SNR (fwSNRseg), Predicted rating of overall Quality (Covl), Rating of Speech Distortion (Csig), Rating of Background Distortion (Cbak) (ITU-T rec. P.835, 2003) and MeanOpinion Score (MOS). Simulation results clearly advocate that, it is possible to producevariable bitrates (tuning to channel conditions) in CELP coder by affecting coefficients of the coder while still maintaining a good quality of speech. Further, higher the bit-rate used, the better is the quality of speech (which can be verified from the results obtained with PESQ and MOS analysis) and at the same time offered simulation delay time also increases.

Yogeshwar Kosta

Papers

Artificial Bandwidth Extension of Speech & Its Applications in Wireless Communication Systems: A Review

A novel approach for artificial bandwidth extension of speech signals by LPC technique over proposed GSM FR NB coder using high band feature extraction and various extension of excitation methods

Proposed modifications in ETSI GSM 06.10 full rate speech codec and its overall evaluation of performance using MATLAB

Adaptive Multi-rate Wideband Speech Codec Based on CELP Algorithm: Architectural Study, Implementation & Performance Analysis

Overall performance evaluation of adaptive multi rate 06.90 speech codec based on code excited linear prediction algorithm using MATLAB