Sparsity-Cognizant Total Least-Squares for Perturbed Compressive Sampling
01 May 2011-IEEE Transactions on Signal Processing (Institute of Electrical and Electronics Engineers Inc.)-Vol. 59, Iss: 5, pp 2002-2016
TL;DR: Analysis and simulations demonstrate the practical impact of S-TLS in calibrating the mismatch effects of contemporary grid-based approaches to cognitive radio sensing, and robust direction-of-arrival estimation using antenna arrays, as well as formulating and solving (regularized) TLS optimization problems under sparsity constraints.
Abstract: Solving linear regression problems based on the total least-squares (TLS) criterion has well-documented merits in various applications, where perturbations appear both in the data vector as well as in the regression matrix. However, existing TLS approaches do not account for sparsity possibly present in the unknown vector of regression coefficients. On the other hand, sparsity is the key attribute exploited by modern compressive sampling and variable selection approaches to linear regression, which include noise in the data, but do not account for perturbations in the regression matrix. The present paper fills this gap by formulating and solving (regularized) TLS optimization problems under sparsity constraints. Near-optimum and reduced-complexity suboptimum sparse (S-) TLS algorithms are developed to address the perturbed compressive sampling (and the related dictionary learning) challenge, when there is a mismatch between the true and adopted bases over which the unknown vector is sparse. The novel S-TLS schemes also allow for perturbations in the regression matrix of the least-absolute selection and shrinkage selection operator (Lasso), and endow TLS approaches with ability to cope with sparse, under-determined “errors-in-variables” models. Interesting generalizations can further exploit prior knowledge on the perturbations to obtain novel weighted and structured S-TLS solvers. Analysis and simulations demonstrate the practical impact of S-TLS in calibrating the mismatch effects of contemporary grid-based approaches to cognitive radio sensing, and robust direction-of-arrival estimation using antenna arrays.
Citations
More filters
01 Jan 1997
TL;DR: In this paper, a nonlinear fractional programming problem is considered, where the objective function has a finite optimal value and it is assumed that g(x) + β + 0 for all x ∈ S,S is non-empty.
Abstract: In this chapter we deal with the following nonlinear fractional programming problem:
$$P:\mathop{{\max }}\limits_{{x \in s}} q(x) = (f(x) + \alpha )/((x) + \beta )$$
where f, g: R n → R, α, β ∈ R, S ⊆ R n . To simplify things, and without restricting the generality of the problem, it is usually assumed that, g(x) + β + 0 for all x ∈ S,S is non-empty and that the objective function has a finite optimal value.
797 citations
TL;DR: An iterative algorithm is developed based on the off-grid model from a Bayesian perspective while joint sparsity among different snapshots is exploited by assuming a Laplace prior for signals at all snapshots.
Abstract: Direction of arrival (DOA) estimation is a classical problem in signal processing with many practical applications. Its research has recently been advanced owing to the development of methods based on sparse signal reconstruction. While these methods have shown advantages over conventional ones, there are still difficulties in practical situations where true DOAs are not on the discretized sampling grid. To deal with such an off-grid DOA estimation problem, this paper studies an off-grid model that takes into account effects of the off-grid DOAs and has a smaller modeling error. An iterative algorithm is developed based on the off-grid model from a Bayesian perspective while joint sparsity among different snapshots is exploited by assuming a Laplace prior for signals at all snapshots. The new approach applies to both single snapshot and multi-snapshot cases. Numerical simulations show that the proposed algorithm has improved accuracy in terms of mean squared estimation error. The algorithm can maintain high estimation accuracy even under a very coarse sampling grid.
623 citations
Cites background or methods from "Sparsity-Cognizant Total Least-Squa..."
...The off-grid distance (the distance from the true DOA to the nearest grid point) that lies in a bounded interval is assumed to be uniformly distributed (noninformative) rather than Gaussian as in [17]....
[...]
...In the following we re-derive the off-grid model proposed in [17] using linear approximation and further show its relationship with the on-grid one....
[...]
...As argued in [5] the SVD used in OGSBI-SVD can alleviate the sensitivity to the measurement noise in the MMV case that is not used in [17]....
[...]
...An off-grid model for DOA estimation is studied in [17] where the estimated DOAs are no longer constrained in the sampling grid set....
[...]
...In this paper, we studied the off-grid DOA estimation model firstly proposed in [17] for reducing the modeling error due to discretization of a continuous range....
[...]
TL;DR: This survey paper aims to give an account of some of the main contributions in data-driven inverse problems.
Abstract: Recent research in inverse problems seeks to develop a mathematically coherent foundation for combining data-driven models, and in particular those based on deep learning, with domain-specific knowledge contained in physical–analytical models. The focus is on solving ill-posed inverse problems that are at the core of many challenging applications in the natural sciences, medicine and life sciences, as well as in engineering and industrial applications. This survey paper aims to give an account of some of the main contributions in data-driven inverse problems.
473 citations
TL;DR: A review of the state-of-the-art and most recent advances of compressive sensing and related methods as applied to electromagnetics can be found in this article, where a wide set of applicative scenarios comprising the diagnosis and synthesis of antenna arrays, the estimation of directions of arrival, and the solution of inverse scattering and radar imaging problems are reviewed.
Abstract: Several problems arising in electromagnetics can be directly formulated or suitably recast for an effective solution within the compressive sensing (CS) framework. This has motivated a great interest in developing and applying CS methodologies to several conventional and innovative electromagnetic scenarios. This work is aimed at presenting, to the best of the authors knowledge, a review of the state-of-the-art and most recent advances of CS formulations and related methods as applied to electromagnetics. Toward this end, a wide set of applicative scenarios comprising the diagnosis and synthesis of antenna arrays, the estimation of directions of arrival, and the solution of inverse scattering and radar imaging problems are reviewed. Current challenges and trends in the application of CS to the solution of traditional and new electromagnetic problems are also discussed.
318 citations
Cites methods from "Sparsity-Cognizant Total Least-Squa..."
...In [66], the sparse total least square (S-TLS) method has been proposed to solve the following single-snapshot problem:...
[...]
...Despite the single-snapshot data at hand [66], the proposed approach was shown to overcome LASSO strategies [35]....
[...]
TL;DR: In this paper, the problem of line spectrum denoising and estimation with an ensemble of spectrally-sparse signals composed of the same set of continuous-valued frequencies from their partial and noisy observations is studied.
Abstract: Compressed Sensing suggests that the required number of samples for reconstructing a signal can be greatly reduced if it is sparse in a known discrete basis, yet many real-world signals are sparse in a continuous dictionary. One example is the spectrally-sparse signal, which is composed of a small number of spectral atoms with arbitrary frequencies on the unit interval. In this paper we study the problem of line spectrum denoising and estimation with an ensemble of spectrally-sparse signals composed of the same set of continuous-valued frequencies from their partial and noisy observations. Two approaches are developed based on atomic norm minimization and structured covariance estimation, both of which can be solved efficiently via semidefinite programming. The first approach aims to estimate and denoise the set of signals from their partial and noisy observations via atomic norm minimization, and recover the frequencies via examining the dual polynomial of the convex program. We characterize the optimality condition of the proposed algorithm and derive the expected error rate for denoising, demonstrating the benefit of including multiple measurement vectors. The second approach aims to recover the population covariance matrix from the partially observed sample covariance matrix by motivating its low-rank Toeplitz structure without recovering the signal ensemble. Performance guarantee is derived with a finite number of measurement vectors. The frequencies can be recovered via conventional spectrum estimation methods such as MUSIC from the estimated covariance matrix. Finally, numerical examples are provided to validate the favorable performance of the proposed algorithms, with comparisons against several existing approaches.
202 citations
References
More filters
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Abstract: SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.
40,785 citations
"Sparsity-Cognizant Total Least-Squa..." refers methods in this paper
...From a high-level view, the novel scheme comprises anouter iteration loop based on the bisection method [14], and aninner iteration loop that relies on a variant of the branch-and-bound (BB) method [1]....
[...]
Book•
28 Jul 2013TL;DR: In this paper, the authors describe the important ideas in these areas in a common conceptual framework, and the emphasis is on concepts rather than mathematics, with a liberal use of color graphics.
Abstract: During the past decade there has been an explosion in computation and information technology. With it have come vast amounts of data in a variety of fields such as medicine, biology, finance, and marketing. The challenge of understanding these data has led to the development of new tools in the field of statistics, and spawned new areas such as data mining, machine learning, and bioinformatics. Many of these tools have common underpinnings but are often expressed with different terminology. This book describes the important ideas in these areas in a common conceptual framework. While the approach is statistical, the emphasis is on concepts rather than mathematics. Many examples are given, with a liberal use of color graphics. It is a valuable resource for statisticians and anyone interested in data mining in science or industry. The book's coverage is broad, from supervised learning (prediction) to unsupervised learning. The many topics include neural networks, support vector machines, classification trees and boosting---the first comprehensive treatment of this topic in any book.
This major new edition features many topics not covered in the original, including graphical models, random forests, ensemble methods, least angle regression and path algorithms for the lasso, non-negative matrix factorization, and spectral clustering. There is also a chapter on methods for ``wide'' data (p bigger than n), including multiple testing and false discovery rates.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Stanford University. They are prominent researchers in this area: Hastie and Tibshirani developed generalized additive models and wrote a popular book of that title. Hastie co-developed much of the statistical modeling software and environment in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data-mining tools including CART, MARS, projection pursuit and gradient boosting.
19,261 citations
Book•
01 Jan 2001
19,211 citations
"Sparsity-Cognizant Total Least-Squa..." refers background or methods in this paper
...The Lagrangian form of BP is also popular in statistics for fitting sparse linear regression models, using the so-termed least-absolute shrinkage and selection operator (Lasso); see e.g., [ 19 ], [29], and references thereof....
[...]
...(For large , the solution is driven toward the all-zero vector; whereas for small it tends to the LS solution.) This form of BP coincides with the Lasso approach developed for variable selection in linear regression problems [ 19 ], [29]....
[...]
...Lasso problem is known to admit a closed-form solution expressed in terms of a soft thresholding operator (see, e.g., [ 19 ])...
[...]
...From the plethora of available options to solve (17), it is worth mentioning two computationally efficient ones: the least-angle regression (LARS), and the coordinate descent (CD); see e.g., [ 19 ]....
[...]
...Remark 2 (Group Lasso and Matrix S-TLS): When groups of entries are ap rioriknown to be zero or nonzero (as a group), the -norm in (3) must be replaced by the sum of -norms, namely . The resulting group S-TLS estimate can be obtained using the group-Lasso solver [ 19 ]....
[...]
TL;DR: The Elements of Statistical Learning: Data Mining, Inference, and Prediction as discussed by the authors is a popular book for data mining and machine learning, focusing on data mining, inference, and prediction.
Abstract: (2004). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Journal of the American Statistical Association: Vol. 99, No. 466, pp. 567-567.
10,549 citations
TL;DR: Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions.
Abstract: The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), Matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).
Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation denoising, and multiscale edge denoising.
BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.
9,950 citations
Additional excerpts
...…that [7, Sec 3.3.4]: using the solution̂xS−TLS of (6) for a given multiplierλ > 0 and lettingµ := ‖x̂S−TLS‖1, the pertinent constraint isX1(µ) := {x ∈ Rn : ‖x‖1 ≤ µ}; and the equivalent constrained minimization problem is [cf. (6)] x̂S−TLS := arg min x∈X1(µ) f(x) , f(x) := ‖y −Ax‖22 1 + ‖x‖22 ....
[...]