Showing papers by "Ali H. Sayed published in 1996"
TL;DR: It is shown that the celebrated least-mean squares (LMS) adaptive algorithm is H/sup /spl infin// optimal, and it is established that it is a minimax filter, which minimizes the maximum energy gain from the disturbances to the predicted errors.
Abstract: We show that the celebrated least-mean squares (LMS) adaptive algorithm is H/sup /spl infin// optimal. The LMS algorithm has been long regarded as an approximate solution to either a stochastic or a deterministic least-squares problem, and it essentially amounts to updating the weight vector estimates along the direction of the instantaneous gradient of a quadratic cost function. We show that the LMS can be regarded as the exact solution to a minimization problem in its own right. Namely, we establish that it is a minimax filter: it minimizes the maximum energy gain from the disturbances to the predicted errors, whereas the closely related so-called normalized LMS algorithm minimizes the maximum energy gain from the disturbances to the filtered errors. Moreover, since these algorithms are central H/sup /spl infin// filters, they minimize a certain exponential cost function and are thus also risk-sensitive optimal. We discuss the various implications of these results and show how they provide theoretical justification for the widely observed excellent robustness properties of the LMS filter.
239 citations
TL;DR: The approach discussed here allows for interesting generalizations, such as finite memory adaptive filtering with varying sliding patterns, and by considering the appropriate state space models and error Gramians, the Krein-space estimation theory.
Abstract: We have shown that several interesting problems in H/sup /spl infin//-filtering, quadratic game theory, and risk sensitive control and estimation follow as special cases of the Krein-space linear estimation theory developed in Part I. We show that all these problems can be cast into the problem of calculating the stationary point of certain second-order forms, and that by considering the appropriate state space models and error Gramians, we can use the Krein-space estimation theory to calculate the stationary points and study their properties. The approach discussed here allows for interesting generalizations, such as finite memory adaptive filtering with varying sliding patterns.
221 citations
TL;DR: The authors develop a self-contained theory for linear estimation in Krein spaces based on simple concepts such as projections and matrix factorizations and leads to an interesting connection between Krein space projection and the recursive computation of the stationary points of certain second-order (or quadratic) forms.
Abstract: The authors develop a self-contained theory for linear estimation in Krein spaces. The derivation is based on simple concepts such as projections and matrix factorizations and leads to an interesting connection between Krein space projection and the recursive computation of the stationary points of certain second-order (or quadratic) forms. The authors use the innovations process to obtain a general recursive linear estimation algorithm. When specialized to a state-space structure, the algorithm yields a Krein space generalization of the celebrated Kalman filter with applications in several areas such as H/sup /spl infin//-filtering and control, game problems, risk sensitive control, and adaptive filtering.
208 citations
TL;DR: It is shown that an intrinsic feedback structure can be associated with the varied adaptive schemes and extended the so-called transfer function approach to a general time-variant scenario without any approximations.
Abstract: This paper provides a time-domain feedback analysis of gradient-based adaptive schemes. A key emphasis is on the robustness performance of the adaptive filters in the presence of disturbances and modeling uncertainties (along the lines of H/sup /spl infin//-theory and robust filtering). The analysis is carried out in a purely deterministic framework and assumes no prior statistical information or independence conditions. It is shown that an intrinsic feedback structure can be associated with the varied adaptive schemes. The feedback structure is motivated via energy arguments and is shown to consist of two major blocks: a time-variant lossless (i.e., energy preserving) feedforward path and a time-variant feedback path. The configuration is further shown to lend itself to analysis via a so-called small gain theorem, thus leading to stability and robustness conditions that require the contractivity of certain operators. Choices for the step-size parameter in order to guarantee faster rates of convergence are also derived, and simulation results are included to demonstrate the theoretical findings. In addition, the time-domain analysis provided in this paper is shown to extend the so-called transfer function approach to a general time-variant scenario without any approximations.
138 citations
TL;DR: A perturbation analysis is used to indicate the best accuracy that can be expected from a finite-precision algorithm that uses the generator matrix as the input data and shows that the modified Schur algorithm is backward stable for a large class of structured matrices.
Abstract: This paper provides a detailed analysis that shows how to stabilize the {\em generalized} Schur algorithm, which is a fast procedure for the Cholesky factorization of positive-definite structured matrices $R$ that satisfy displacement equations of the form $R-FRF^T=GJG^T$, where $J$ is a $2\times 2$ signature matrix, $F$ is a stable lower-triangular matrix, and $G$ is a generator matrix. In particular, two new schemes for carrying out the required hyperbolic rotations are introduced and special care is taken to ensure that the entries of a Blaschke matrix are computed to high relative accuracy. Also, a condition on the smallest eigenvalue of the matrix, along with several computational enhancements, is introduced in order to avoid possible breakdowns of the algorithm by assuring the positive-definiteness of the successive Schur complements. We use a perturbation analysis to indicate the best accuracy that can be expected from {\em any} finite-precision algorithm that uses the generator matrix as the input data. We then show that the modified Schur algorithm proposed in this work essentially achieves this bound when coupled with a scheme to control the generator growth. The analysis further clarifies when pivoting strategies may be helpful and includes illustrative numerical examples. For all practical purposes, the major conclusion of the analysis is that the modified Schur algorithm is backward stable for a large class of structured matrices.
55 citations
TL;DR: The paper establishes robustness, optimality, and convergence properties of the widely used class of instantaneous-gradient adaptive algorithms and employs the Cauchy-Schwarz inequality for vectors in an Euclidean space.
Abstract: The paper establishes robustness, optimality, and convergence properties of the widely used class of instantaneous-gradient adaptive algorithms. The analysis is carried out in a purely deterministic framework and assumes no a priori statistical information. It employs the Cauchy-Schwarz inequality for vectors in an Euclidean space and derives local and global error-energy bounds that are shown to highlight, as well as explain, relevant aspects of the robust performance of adaptive gradient filters (along the lines of H/sup /spl infin// theory).
54 citations
01 Jan 1996
TL;DR: In this paper, the relation between the solutions of two minimization problems with indefinite quadratic forms is studied and a complete link between both solutions can be established by invoking a fundamental set of inertia conditions.
Abstract: We study the relation between the solutions of two minimization problems with indefinite quadratic forms. We show that a complete link between both solutions can be established by invoking a fundamental set of inertia conditions. While these inertia conditions are automatically satisfied in a standard Hilbert space setting, which is the case of classical least-squares problems in both the deterministic and stochastic frameworks, they nevertheless turn out to mark the differences between the two optimization problems in indefinite metric spaces. Applications to H∞-filtering, robust adaptive filtering, and approximate total-least-squares methods are included.
32 citations
TL;DR: A time-domain feedback analysis of the robustness performance of Gauss-Newton recursive methods that are often used in identification and control is provided, showing that by properly selecting the free parameters, the resulting filters can be made to impose certain bounds on the error quantities, thus resulting in desirable robustness properties.
16 citations
TL;DR: It is shown that a complete link between both solutions can be established by invoking a fundamental set of inertia conditions, which turn out to mark the differences between the two estimation problems in indefinite metric spaces.
Abstract: We study the relation between the solutions of two estimation problems with indefinite quadratic forms. We show that a complete link between both solutions can be established by invoking a fundamental set of inertia conditions. While these inertia conditions are automatically satisfied in a standard Hilbert space setting, they nevertheless turn out to mark the differences between the two estimation problems in indefinite metric spaces. They also include, as special cases, the well-known conditions for the existence of H/sup -/spl infin//-filters and controllers. Given two Hermitian matrices {/spl Pi/, W}, a column vector y, and an arbitrary matrix A of appropriate dimensions, we study the relation between two minimization problems with quadratic cost functions, and also refer to the indefinite-weighted least-squares problem.
15 citations
03 Nov 1996
TL;DR: In this article, a modified leaky-LMS filter is proposed to ensure stability of the estimates w(k) in the presence of bounded noise, without introducing any bias term and with the added cost of only a comparison and a multiplication per iteration when compared to the classical LMS algorithm.
Abstract: We propose a modified leaky-LMS filter that ensures stability of the estimates w(k) in the presence of bounded noise, without introducing any bias term and with the added cost of only a comparison and a multiplication per iteration when compared to the classical LMS algorithm. The new algorithm is further shown to converge for l/sub p/ noise and persistently exciting regressors. It also provides bounded estimates even in finite precision arithmetic. The stability and convergence properties of the new algorithm are established through a deterministic analysis that is based on the Lyapunov theory for the stability of nonlinear difference equations.
9 citations
TL;DR: A Schur-type algorithm is presented for the simultaneous triangular factorization of a given (non-degenerate) structured matrix and its inverse, especially suited for parallel (systolic array) implementations.
Abstract: A Schur-type algorithm is presented for the simultaneous triangular factorization of a given (non-degenerate) structured matrix and its inverse The algorithm takes the displacement generator of a Hermitian, strongly regular matrixR as an input, and computes the displacement generator of the inverse matrixR−1 as an output From these generators we can directly deduce theLD−1L* (lower-diagonal-upper) decomposition ofR, and theUD−1U* (upper-diagonallower) decomposition ofR−1 The computational complexity of the algorithm isO(rn2) operations, wheren andr denote the size and the displacement rank ofR, respectively Moreover, this method is especially suited for parallel (systolic array) implementations: usingn processors the algorithm can be carried out inO(n) steps
03 Nov 1996
TL;DR: The derivation clarifies the combined effects of the step-size parameter and the nature of the nonlinear functional on the convergence and robustness performance of adaptive schemes.
Abstract: We pursue a time-domain feedback analysis of adaptive schemes with nonlinear update relations. We consider commonly used algorithms in blind equalization and study their performance in a purely deterministic framework. The derivation employs insights from system theory and feedback analysis, and it clarifies the combined effects of the step-size parameter and the nature of the nonlinear functional on the convergence and robustness performance of adaptive schemes.
01 Jan 1996
TL;DR: In this paper, a self-contained theory for linear estimation in Krein spaces is developed based on simple concepts such as projections and matrix factorizations and leads to an interesting connection between Krein space projection and the recursive computation of the stationary points of certain second-order (or quadratic) forms.
Abstract: The authors develop a self-contained theory for linear estimation in Krein spaces. The derivation is based on simple concepts such as projections and matrix factorizations and leads to an interesting connection between Krein space projection and the recursive computation of the stationary points of certain second-order (or quadratic) forms. The authors use the innovations process to obtain a general recursive linear estimation algorithm. When specialized to a state-space structure, the algorithm yields a Krein space generalization of the celebrated Kalman filter with applications in several areas such as Hw- filtering and control, game problems, risk sensitive control, and adaptive filtering.
Proceedings Article•
01 Sep 1996
TL;DR: This work pursues a time-domain feedback analysis of adaptive schemes with nonlinear update relations and clarifies the combined effects of the step-size parameters and the nature of the nonlinear functionals on the convergence and robustness performance of the adaptive schemes.
Abstract: We pursue a time-domain feedback analysis of adaptive schemes with nonlinear update relations. We consider commonly used algorithms in blind equalization and neural network training and study their performance in a purely deterministic framework. The derivation employs insights from system theory and feedback analysis, and it clarifies the combined effects of the step-size parameters and the nature of the nonlinear functionals on the convergence and robustness performance of the adaptive schemes.
TL;DR: A bound is established on the step-size parameter in order to guarantee that the training algorithm will behave as a robust filter in the sense of H ∞ -theory, and the training scheme can be interpreted in terms of a feedback interconnection that consists of two major blocks.
01 Jan 1996
TL;DR: In this paper, a time-domain feedback analysis of gradient-based adaptive schemes is provided, where the robustness performance of the adaptive filters in the presence of disturbances and modeling uncertainties is investigated.
Abstract: Absiruct-This paper provides a time-domain feedback analysis of gradient-based adaptive schemes. A key emphasis is on the robustness performance of the adaptive filters in the presence of disturbances and modeling uncertainties (along the lines of H”-theory and robust filtering). The analysis is carried out in a purely deterministic framework and assumes no prior statistical information or independence conditions. It is shown that an intrinsic feedback structure can be associated with the varied adaptive schemes. The feedback structure is motivated via energy arguments and is shown to consist of two major blocks: a time-variant lossless (i.e., energy preserving) feedforward path and a time-variant feedback path. The configuration is further shown to lend itself to analysis via a socalled small gain theorem, thus leading to stability and robustness conditions that require the contractivity of certain operators. Choices for the step-size parameter in order to guarantee faster rates of convergence are also derived, and simulation results are included to demonstrate the theoretical findings. In addition, the time-domain analysis provided in this paper is shown to extend the so-called transfer function approach to a general time-variant scenario without any approximations.
01 Jan 1996
TL;DR: In this article, the authors show that several interesting problems in filtering, quadratic game theory, and risk sensitive control and estimation follow as special cases of the Krein-space linear estimation theory developed in (l).
Abstract: We show that several interesting problems in Rw - filtering, quadratic game theory, and risk sensitive control and estimation follow as special cases of the Krein-space linear esti- mation theory developed in (l). We show that a11 these problems can be cast into the problem of calculating the stationary point of certain second-order forms, and that by considering the ap- propriate state space models and error Gramians, we can use the ~re~n-spa~e estimation theory to calculate the stationary points and study their properties. The approach discussed here allows for interesting generalizations, such as finite memory adaptive filtering with varying sliding patterns.