116 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 5, NO. 2, MARCH 1997
Bias Propagation in the Autocorrelation
Method of Linear Prediction
Jan S. Erkelens and Piet M. T. Broersen
Abstract— A time-domain analysis of the autocorrelation
method for autoregressive estimation is given. It is shown that a
small bias in a reflection coefficient close to one in absolute value
is propagated and prohibits an accurate estimation of further
reflection coefficients. Tapered data windows largely reduce this
effect, but increase the variance of the models.
I. INTRODUCTION
M
ANY low bit-rate speech coders use the autocorrelation
method (ACM) to find a linear prediction model of the
speech signal. It is known that a tapered data window improves
the performance of the autocorrelation method in speech
analysis. Sometimes, the frequency domain interpretation of
the ACM is given as the motivation for the use of a tapered
window, but it will be shown that the reduction of the
edge effects is important in the time-domain. A time-domain
analysis of the autocorrelation method will be presented that
also gives theoretical support to the use of tapered data
windows with the ACM.
For stochastic signals, the behavior of autoregressive es-
timation methods can be expressed in terms of bias and
variance of the model parameters. Estimation methods have
been treated extensively in the literature (e.g., [1]–[11]). Some
studies express the bias in terms of the poles or the parameters
of the autoregressive process [3], [4]. Although the bias is
described accurately, the formulas do not give insight in what
causes the bias and what are the consequences of it for higher
order processes. We show that the edge effects in the ACM
may cause a large bias in the residual variance if a reflection
coefficient is close to one in absolute value. Propagation of
this bias via the residual variance may lead to a large bias in
higher order reflection coefficients.
II. T
HE AUTOCORRELATION METHOD
An autoregressive process of order is described by
a weighted sum of preceding signal values plus an independent
identically distributed (i.i.d.) noise signal
with variance
(1)
Manuscript received December 1, 1995; revised September 24, 1996. This
research was supported by the Technology Foundation (STW) under Grant
DTN11.2436. The associate editor coordinating the review of this manuscript
and approving it for publication was Dr. Douglas D. O’Shaughnessy.
The authors are with the Department of Applied Physics, Delft University of
Technology, 2628 CJ Delft, The Netherlands (e-mail: broersen@tn.tudelft.nl).
Publisher Item Identifier S 1063-6676(97)01893-2.
The coefficients are the autoregressive parameters, called
LPC parameters in speech coding. The autocorrelation method
assumes the signal to be zero outside the interval of observa-
tion of length
and estimates the parameters of a th
order model by minimizing the residual variance
from
minus to plus infinity. This residual variance is given by
(2)
where the
are samples of the windowed signal.
Because of the infinite sum in (2), the parameters can be
found efficiently with the Levinson–Durbin algorithm
(3)
(4)
(5)
where
is the th estimated reflection coefficient, is
the
th parameter of an th order model and is the
residual variance for the
th order model. The autocorrelation
coefficients
of the data are defined by
(6)
The autocorrelation coefficients
are biased estimates
of the theoretical autocorrelation coefficients
of the
process, because
contains only nonzero prod-
ucts. This ensures that the resulting model is always stable.
There are other estimation methods that ensure a stable model
[7]–[10], but the ACM is the most popular one. The bias
in the autocorrelation coefficients is caused by the way the
autocorrelation method handles the edges. In the next section,
we will show that this method yields poor results if some of
the reflection coefficients in the process are close to plus or
minus one.
1063–6676/97$10.00 1997 IEEE
ERKELENS AND BROERSEN: BIAS PROPAGATION 117
III. ANALYSIS OF THE AUTOCORRELATION METHOD
The bias in the sample autocorrelation function (6) also
causes bias in the estimated reflection coefficients. But there
are other sources of bias as well; the bias consists of dif-
ferent parts. For example, the solution for the first reflection
coefficient
is given by
(7)
where
is the normalized autocorrelation coefficient of lag
one of the data. An expression for the bias in the reflection
coefficients can be found by making a second order Taylor
expansion and taking the expectation. For example, for (7) the
result is
var
cov (8)
where
is the true normalized autocorrelation coefficient of
lag one of the process,
is the normalized autocorrelation
coefficient of lag one of the window,
is the expectation
of
.Var and cov are the variance of
and the covariance between and , respectively.
First an analysis of the ACM will be given when a rectangular
window is used. In speech applications, almost always a non-
rectangular window is used. Yet, the analysis for a rectangular
window is of interest, because it explains why the use of a
nonrectangular window is necessary and what properties a
window must posses when used with the ACM.
The first term on the right hand side of (8) shows the
explicit contribution of windowing to the bias in
. For a
rectangular window,
equals and since
equals (the true first reflection coefficient of the process),
this bias contribution is equal to
. For all reflection
coefficients, the explicit bias contribution due to windowing
can be found by multiplying each normalized autocorrelation
coefficient of the process with the corresponding normalized
autocorrelation coefficient of the window, and transforming
the resulting modified normalized autocorrelation function
to reflection coefficients. For a rectangular window,
equals and, therefore, this bias contribution will
be called triangular bias because of the specific form. The
triangular bias is only present in the ACM and not in other
estimation methods, such as the Burg method or the covariance
methods. The second and third terms in (8) are due to the
variances of and the covariances between the autocorrelation
coefficients in the expressions and will be called Taylor bias.
Such contributions of Taylor bias to the overall bias are present
in the parameter estimates of all estimation methods and
contain an implicit contribution due to windowing. Here, only
the explicit triangular bias in the ACM is considered. Firstly, a
qualitative analysis of the effects associated with the triangular
bias will be given by looking at the residual for a first- and
second-order model. Next, we will generalize the analysis and
give an experimental illustration.
From (2) it is seen that the residual variance for a first
order model is given by
(9)
The value
minimizing this equation is the estimate of
the first reflection coefficient. The discontinuities at the frame
edges cause incomplete residual terms in
such as and
. The incomplete term has no influence on
the solution
. The incomplete term causes to
have a small bias that makes it closer to zero in absolute value.
The influence of the incomplete term is small because before
minimization with respect to
, all terms in the residual
variance are of the same order of magnitude. The triangular
bias in
is , the true first reflection coefficient
of the process. Both incomplete terms
and
are an important source of bias in the residual variance .If
is close to one in absolute value, the magnitude of these
incomplete terms after minimization is large in comparison
with the magnitude of the complete terms [after minimization
is reduced with a factor ]. A consequence is that
the influence of the incomplete terms on the second reflection
coefficient is much larger than on the first one.
The residual variance for a second order model is
(10)
The value
minimizing this expression, given the solution
for the first reflection coefficient, is the solution for .
The complete terms in the sum are much smaller than the
incomplete terms if
is close to one in absolute value,
because for the first order the complete terms have already
been minimized with respect to
and is reduced with a
factor
). Therefore, the incomplete terms have a much
larger influence on
than they had on . To put this another
way, the incomplete terms cause
to be considerably larger
than
. The biased residual variance is propagated via
the Levinson–Durbin algorithm (3) to the second reflection
coefficient
, and this causes to be considerably smaller
than the true value
. Higher order reflection coefficients
cannot compensate for this large bias because incomplete
terms are present for all model orders in
and are large
in comparison with the complete terms.
This effect will occur after the occurrence of a reflection
coefficient close to one in absolute value. For example, con-
sider a process of which the
th reflection coefficient has a
distance of
from one, i.e., equals in absolute
value. For reasons of clarity, it is supposed that all lower
order reflection coefficients are equal to zero. The
th
reflection coefficient may have an arbitrary value. For such a
process
equals . The autocorrelation method
118 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 5, NO. 2, MARCH 1997
Fig. 1. Normalized residual variance for different model orders obtained
with the autocorrelation method, with a rectangular window (“0”) and a
Hanning window (“*”). Edge effects in the autocorrelation method may
cause serious bias in the residual variance and reflection coefficients, when a
rectangular window is applied. A tapered window largely reduces this bias.
uses estimates of that are biased with a factor
, and, therefore, the reflection coefficient corresponding
to this biased autocorrelation will be
; a bias of only order . If the biased
value
for is used in (4), will get a value
twice the value it should have had. This leads to a value of
with (3) which is one half the true value and the result
is a large bias of order 1 (as compared to the bias in
,
which is of order
).
This example is of practical interest because the first re-
flection coefficient of sampled bandlimited data can be close
to
1. The first reflection coefficient equals and
this quotient approaches
1 if the continuous signal is sampled
with a very high sampling frequency. In fact, the first estimated
reflection coefficient often is close to
1 in voiced speech
(and the second one close to
1). The conclusion to be drawn
is that the autocorrelation method in its basic form without
windowing is not a useful method for parameter estimation.
A tapered data window smoothly brings the amplitude of the
signal down at the edges. Such windows reduce the bias in the
autocorrelation function. Only the normalized autocorrelation
coefficients of the window up to the LPC analysis order
are of importance. For the rectangular window, these are
equal to 1
; for other windows such as the popular
Hamming and Hanning windows, these are much closer to
one. This means that severe bias propagation will only occur
with these windows if the reflection coefficients of the process
are much closer to one in absolute value than
. The
application of a tapered data window also has a disadvantage:
When a data window is used, effectively, some information is
thrown away. The result is that the variance of the estimated
parameters is increased. We have observed that the increase
in variance as a result of windowing corresponds roughly to
the decrease in effective number of observations as defined in
[11], which makes sense because the theoretical variances of
LPC coefficients are inversely proportional to the number of
observations.
At this point, it is interesting to compare the bias in the
reflection coefficients with that in the log area ratios. An
estimated model has a certain spectrum distortion with respect
to the original process. It has been shown [12] that the
reflection coefficients are more sensitive to quantization than
the log area ratios if their absolute value is close to one. This
means that the bias in the log area ratios is larger than the
bias in the reflection coefficients, if the latter have an absolute
value close to one. If the reflection coefficients are small in
absolute value, they are approximately linearly related to the
log area ratios [12]. In that case, the log area ratios have
bias properties similar to the reflection coefficients, e.g., if
the bias in a reflection coefficient is a factor one half, it also
is approximately a factor one half in the corresponding log
area ratio.
The bias propagation can be illustrated by a simulation
experiment. We have generated 5000 realizations of length
100 samples of a tenth-order autoregressive process with
the reflection coefficient vector in (11), shown at the bottom
of the page.
The first reflection coefficient is equal to
. From
these realizations we have estimated LPC models with a
rectangular window and with a Hanning window. The average
reflection coefficient vectors for these 5000 realizations are,
respectively, for the rectangular window and the Hanning
window as shown in (12) and (13), at the bottom of the page.
In Fig. 1, the product of (
), , for orders
, is shown, which is the normalized residual
variance. Results for the rectangular window are denoted by
“0,” results for the Hanning window are denoted by “*.” From
this figure, and from the average reflection coefficient vectors,
the phenomenon of bias propagation can be clearly seen. The
bias in the first estimated reflection coefficient is small, but
the residual variance for the rectangular window is close to
(11)
(12)
(13)
ERKELENS AND BROERSEN: BIAS PROPAGATION 119
Fig. 2. Original spectrum (solid unmarked line) and spectra obtained with
the autocorrelation method, with a rectangular window (“0”) and a Hanning
window (“*”).
a factor 2 larger than for the Hanning window for this order,
due to the influence of the triangular bias. The bias in the
second estimated reflection coefficient consequently is large:
for the rectangular window the second reflection coefficient
is close to one half that obtained with the Hanning window.
The decrease in residual variance for the rectangular window
is therefore smaller than for the Hanning window.
The spectrum of the process that was used in the simulations
is the unmarked solid line in Fig. 2. The spectra obtained with
the rectangular and Hanning window are marked by “0” and
“*,” respectively. It is clear that the Hanning window greatly
reduces the bias.
We have explained the influence on the reflection coeffi-
cients due to the edge effects in the ACM. It is difficult to relate
this bias in the reflection coefficients quantitatively to the bias
in individual formant frequencies and bandwidths. However,
the bias always tends to make the reflection coefficients smaller
in absolute value, and this means that in general the poles will
stay away further from the unit circle. Therefore, an important
effect of the bias propagation is an increase in the bandwidth
of spectral peaks.
IV. C
ONCLUSIONS
It is shown that the autocorrelation method of autoregressive
estimation is not suitable if reflection coefficients are close
to
or 1. This result is of practical interest because for
sampled continuous signals the first reflection coefficient is
often close to
1 unless the sample frequency is low. The
poor performance of the autocorrelation method is due to
edge effects; incomplete terms in the residual cause a large
bias in the residual variance, this bias is propagated via the
denominator of the Levinson–Durbin recursion and causes
higher order reflection coefficients to be seriously biased as
well. A tapered data window decreases the edge effects, but
increases the variance of estimated models, because effectively
the number of data available for estimation is decreased.
R
EFERENCES
[1] J. D. Markel and A. H. Gray, Linear Prediction of Speech. New York:
Springer-Verlag, 1976.
[2] P. M. T. Broersen and H. E. Wensink, “On finite sample theory for
autoregressive model order selection,” IEEE Trans. Signal Processing,
vol. 41, pp. 194–204, 1993.
[3] D. Tjøstheim and J. Paulsen, “Bias of some commonly used time series
estimates,” Biometrika 70, pp. 389–399, 1983; correction in Biometrika,
71, p. 656.
[4] J. Paulsen and D. Tjøstheim, “On the estimation of residual variance
and order in autoregressive time series,” J. Roy. Stat. Soc. B, vol. 47,
pp. 216–228, 1985.
[5] A. H. Gray and D. Y. Wong, “The Burg algorithm for LPC speech anal-
ysis/synthesis,” IEEE Trans. Signal Processing, vol. 28, pp. 609–615,
1980.
[6] L. R. Rabiner and R. W. Shafer, Digital Processing of Speech Signals.
Prentice-Hall, 1978.
[7] S. M. Kay, Modern Spectral Estimation. Englewood Cliffs, NJ:
Prentice-Hall, 1988.
[8] B. S. Atal, “Predictive coding of speech at low bit rates,” IEEE Trans.
Commun., vol. 30, pp. 600–614, 1982.
[9] J. Makhoul, “Stable and efficient lattice methods for linear prediction,”
in Modern Spectrum Analysis, D. G. Childers, Ed. IEEE Press, 1978,
pp. 256–261.
[10] B. W. Dickinson, “Autoregressive estimation using residual energy
ratios,” IEEE Trans. Inform. Theory, vol. 24, pp. 503–506, 1978.
[11] J. Makhoul, “Linear prediction: A tutorial review,” Modern Spectrum
Analysis, D. G. Childers, Ed. IEEE Press, 1978, pp. 99–118.
[12] V. R. Viswanathan and J. Makhoul, “Quantization properties of trans-
mission parameters in linear predictive systems,” IEEE Trans. Acoust.,
Speech, Signal Processing, vol. 23, pp. 309–312, 1975.
Jan S. Erkelens was born in Maassluis, The
Netherlands, in 1969. He received the M.Sc. degree
in applied physics and the Ph.D. degree in 1996,
both from the Delft University of Technology, The
Netherlands. His Ph.D. thesis focuses on linear-
prediction-based speech coding.
Currently, he holds a post-doctoral posi-
tion at the International Research Centre of
Telecommunications-Transmission and Radar
IRCTR, Delft, where he is studying the micro-
physical properties of clouds by means of radar
measurements.
Piet M. T. Broersen was born in Zijdewind, The
Netherlands, in 1944. He received the M.Sc. degree
in applied physics and the Ph.D. degree in 1976,
both from the Delft University of Technology, The
Netherlands.
He is with the Department of Applied Physics of
the Delft University of Technology. His research
interests are the application of signal processing,
time series analysis, and parameter estimation to
model building.
