This paper presents an analysis of stochastic gradient-based adaptive algorithms with general cost functions. The analysis holds under mild assumptions on the inputs and the cost function. The method of analysis is based on an asymptotic analysis of fixed stepsize adaptive algorithms and gives almost sure results regarding the behavior of the parameter estimates, whereas previous stochastic analyses typically consider mean and mean square behavior. The parameter estimates are shown to enter a small neighborhood about the optimum value and remain there for a finite length of time. Furthermore, almost sure exponential bounds are given for the rate of convergence of the parameter estimates. The asymptotic distribution of the parameter estimates is shown to be Gaussian with mean equal to the optimum value and covariance matrix that depends on the input statistics. Specific adaptive algorithms that fall under the framework of this paper are signed error least mean squre (LMS), dual sign LMS, quantized state LMS, least mean fourth, dead zone algorithms, momentum algorithms, and leaky LMS.

http://ieeexplore.ieee.org/iel4/78/11401/00536676.pdf

Asymptotic Analysis of

A least mean square (LMS) algorithm with clipped data is studied for use when updating the weights of an adaptive filter with correlated Gaussian input. Both stationary and nonstationary environments are considered. Three main contributions are presented. The first, corresponding to the stationary case, is a proof of the convergence of the algorithm in the case of a M-dependent sequence of correlated observation vectors. It is proven that the steady state mean square misalignment of the adaptive filter weights has an upper bound proportional to the algorithm step size mu . The second contribution, also belonging to the stationary case, is the derivation of the expressions of convergence time N/sub c/ and steady state mean square excess estimation error epsilon . It is shown that N/sub c/ is proportional to 1/( mu lambda ), with lambda being the minimum eigenvalue of the input covariance matrix. It is also shown that the product N/sub c/ epsilon is independent of mu . For a given epsilon , the convergence time increases with the eigenvalue spread of the input covariance matrix and the filter length, as well as its input noise power. The range of mu that achieves tolerable values of N/sub c/ and epsilon is determined. The third contribution is concerned with the nonstationary case. It is shown that the mean square excess estimation error is the sum of the two terms with opposite dependencies on mu . An optimum value of mu is derived. >

Analysis and design of a signed regressor LMS algorithm for stationary and nonstationary adaptive filtering with correlated Gaussian data

A tracking analysis of the adaptive filters equipped with the sign algorithm and operating in nonstationary environments is presented. Under the assumption that the nonstationary can be modeled using a random disturbance, it is shown that the long-term time average of the mean-absolute error is bounded, and that there exists an optimal choice of the convergence constant mu which minimizes this quality. Applying the commonly used independence assumption, and under the assumption that the nonstationarity is solely due to the time-varying behavior of the optimal coefficients, it is shown that the distributions of the successive coefficient misalignment vectors converge to a limiting distribution when the adaptive filter is used in the system identification mode. Finally, under the additional assumption that the signals involved are zero mean and Gaussian, a set of nonlinear difference equations is derived that characterizes the mean and mean-squared behavior of the filter coefficients and the mean-squared estimation error during adaptation and tracking. Results of several experiments that show very good correlation with the theoretical analyses are presented. >

Tracking analysis of the sign algorithm in nonstationary environments

This paper presents an analysis of stochastic gradient-based adaptive algorithms with general cost functions. The analysis holds under mild assumptions on the inputs and the cost function. The method of analysis is based on an asymptotic analysis of fixed stepsize adaptive algorithms and gives almost sure results regarding the behavior of the parameter estimates, whereas previous stochastic analyses typically considered mean and mean square behavior. The parameter estimates are shown to enter a small neighborhood about the optimum value and remain there for a finite length of time. Furthermore, almost sure exponential bounds are given for the rate of convergence of the parameter estimates. The asymptotic distribution of the parameter estimates is shown to be Gaussian with mean equal to the optimum value and covariance matrix that depends on the input statistics. Specific adaptive algorithms that fall under the framework of this paper are signed error least mean square (LMS), dual sign LMS, quantized state LMS, least mean fourth, dead zone algorithms, momentum algorithms, and leaky LMS.

Asymptotic analysis of stochastic gradient-based adaptive filtering algorithms with general cost functions

The adaptive filtering sign algorithm is analyzed in the case of nonstationary and correlated data. The excess time average absolute estimation error (EAAE) is used as a performance index of the algorithm. It is proved that the EAAE is the sum of two terms: one proportional to the algorithm step size mu and the other proportional to 1/ mu . The optimum step size that minimizes the EAAE is found equal to square root Q/P, with Q being the power of fluctuations of the optimal filter weights, and P being the power of filter input. This formula is derived under the assumption that increments of optimal filter weights are white. Computer simulations are used to examine the validity of the formula for other models of variation of the optimal filter. Simulation results show a good fit with the obtained optimum step size formula. >

Optimum step size of sign algorithm for nonstationary adaptive filtering

The convergence analysis of a sign algorithm with decreasing gain, when governing the weights of an adaptive filter, is presented. The analysis is done in a noiseless case. A rigorous proof of almost-sure convergence to the optimal filter is attained under a weak-ergodicity assumption that includes the case, important in practice, of correlated observations. >

E. Eweda

Papers

Almost sure convergence of a decreasing gain sign algorithm for adaptive filtering