Exact Expectation Evaluation and Design of Variable Step-Size Adaptive Algorithms

TLDR: It is argued that such a comparison can be misleading because the supposedly optimal step-size sequence sometimes induces divergence in the initial phase of learning, which occurs most often when the input signal is colored and/or heavy tailed.

Abstract: The choice of a fixed step size in adaptive filtering algorithms implies a conflict between the convergence rate and the steady-state performance. In order to address this tradeoff more effectively, variable step-size schemes have been proposed. The efficiency evaluation of such techniques requires comparisons of the resulting step-size values with theoretical optimum values obtained from a statistical analysis of the adaptive algorithm convergence. The analysis generally employs statistical approximations, the most critical being the assumption of independence between the input signal and the filter coefficients. In this letter, it is argued that such a comparison can be misleading because the supposedly optimal step-size sequence sometimes induces divergence in the initial phase of learning. This occurs most often when the input signal is colored and/or heavy tailed. This instability trend can be explained by a convergence analysis that does not employ the independence hypothesis. In addition, the use of this exact analysis implies an optimal step-size sequence that can be significantly different from that obtained with standard analysis methods. This approach can be used to improve the design process of variable step-size adaptive filtering algorithms.