SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.

/pdf/regression-shrinkage-and-selection-via-the-lasso-6mx35txfr5.pdf

Regression Shrinkage and Selection via the Lasso

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

Summary. We propose the elastic net, a new regularization and variable selection method. Real world data and a simulation study show that the elastic net often outperforms the lasso, while enjoying a similar sparsity of representation. In addition, the elastic net encourages a grouping effect, where strongly correlated predictors tend to be in or out of the model together.The elastic net is particularly useful when the number of predictors (p) is much bigger than the number of observations (n). By contrast, the lasso is not a very satisfactory variable selection method in the

/pdf/regularization-and-variable-selection-via-the-elastic-net-4z8q6j5o0i.pdf

Regularization and variable selection via the elastic net

The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), Matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).
Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation denoising, and multiscale edge denoising.
BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.

Atomic Decomposition by Basis Pursuit

Donoho and Johnstone (1994) proposed a method for reconstructing an unknown function f on [0,1] from noisy data d/sub i/=f(t/sub i/)+/spl sigma/z/sub i/, i=0, ..., n-1,t/sub i/=i/n, where the z/sub i/ are independent and identically distributed standard Gaussian random variables. The reconstruction f/spl circ/*/sub n/ is defined in the wavelet domain by translating all the empirical wavelet coefficients of d toward 0 by an amount /spl sigma//spl middot//spl radic/(2log (n)/n). The authors prove two results about this type of estimator. [Smooth]: with high probability f/spl circ/*/sub n/ is at least as smooth as f, in any of a wide variety of smoothness measures. [Adapt]: the estimator comes nearly as close in mean square to f as any measurable estimator can come, uniformly over balls in each of two broad scales of smoothness classes. These two properties are unprecedented in several ways. The present proof of these results develops new facts about abstract statistical inference and its connection with an optimal recovery model. >

De-noising by soft-thresholding

Much recent effort has sought asymptotically minimax methods for recovering infinite dimensional objects-curves, densities, spectral densities, images-from noisy data A now rich and complex body of work develops nearly or exactly minimax estimators for an array of interesting problems Unfortunately, the results have rarely moved into practice, for a variety of reasons-among them being similarity to known methods, computational intractability and lack of spatial adaptivity We discuss a method for curve estimation based on n noisy data: translate the empirical wavelet coefficients towards the origin by an amount √(2 log n) /√n The proposal differs from those in current use, is computationally practical and is spatially adaptive; it thus avoids several of the previous objections Further, the method is nearly minimax both for a wide variety of loss functions-pointwise error, global error measured in L p -norms, pointwise and global error in estimation of derivatives-and for a wide range of smoothness classes, including standard Holder and Sobolev classes, and bounded variation This is a much broader near optimality than anything previously proposed: we draw loose parallels with near optimality in robustness and also with the broad near eigenfunction properties of wavelets themselves Finally, the theory underlying the method is interesting, as it exploits a correspondence between statistical questions and questions of optimal recovery and information-based complexity

https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1995.tb02032.x

Wavelet Shrinkage: Asymptopia?

Density estimation is a commonly used test case for nonparametric estimation methods. We explore the asymptotic properties of estimators based on thresholding of empirical wavelet coefficients. Minimax rates of convergence are studied over a large range of Besov function classes $B_{\sigma pq}$ and for a range of global $L'_p$ error measures, $1 \leq p' p$, some form of nonlinearity is essential, since the minimax linear estimators are suboptimal by polynomial powers of n. A second approach, using an approximation of a Gaussian white-noise model in a Mallows metric, is used to attain exactly optimal rates of convergence for quadratic error $(p' = 2)$.

https://projecteuclid.org/download/pdf_1/euclid.aos/1032894451

Density estimation by wavelet thresholding

1 Wavelets.- 1.1 What can wavelets offer?.- 1.2 General remarks.- 1.3 Data compression.- 1.4 Local adaptivity.- 1.5 Nonlinear smoothing properties.- 1.6 Synopsis.- 2 The Haar basis wavelet system.- 3 The idea of multiresolution analysis.- 3.1 Multiresolution analysis.- 3.2 Wavelet system construction.- 3.3 An example.- 4 Some facts from Fourier analysis.- 5 Basic relations of wavelet theory.- 5.1 When do we have a wavelet expansion?.- 5.2 How to construct mothers from a father.- 5.3 Additional remarks.- 6 Construction of wavelet bases.- 6.1 Construction starting from Riesz bases.- 6.2 Construction starting from m0.- 7 Compactly supported wavelets.- 7.1 Daubechies' construction.- 7.2 Coiflets.- 7.3 Symmlets.- 8 Wavelets and Approximation.- 8.1 Introduction.- 8.2 Sobolev Spaces.- 8.3 Approximation kernels.- 8.4 Approximation theorem in Sobolev spaces.- 8.5 Periodic kernels and projection operators.- 8.6 Moment condition for projection kernels.- 8.7 Moment condition in the wavelet case.- 9 Wavelets and Besov Spaces.- 9.1 Introduction.- 9.2 Besov spaces.- 9.3 Littlewood-Paley decomposition.- 9.4 Approximation theorem in Besov spaces.- 9.5 Wavelets and approximation in Besov spaces.- 10 Statistical estimation using wavelets.- 10.1 Introduction.- 10.2 Linear wavelet density estimation.- 10.3 Soft and hard thresholding.- 10.4 Linear versus nonlinear wavelet density estimation.- 10.5 Asymptotic properties of wavelet thresholding estimates.- 10.6 Some real data examples.- 10.7 Comparison with kernel estimates.- 10.8 Regression estimation.- 10.9 Other statistical models.- 11 Wavelet thresholding and adaptation.- 11.1 Introduction.- 11.2 Different forms of wavelet thresholding.- 11.3 Adaptivity properties of wavelet estimates.- 11.4 Thresholding in sequence space.- 11.5 Adaptive thresholding and Stein's principle.- 11.6 Oracle inequalities.- 11.7 Bibliographic remarks.- 12 Computational aspects and software.- 12.1 Introduction.- 12.2 The cascade algorithm.- 12.3 Discrete wavelet transform.- 12.4 Statistical implementation of the DWT.- 12.5 Translation invariant wavelet estimation.- 12.6 Main wavelet commands in XploRe.- A Tables.- A.1 Wavelet Coefficients.- A.2.- B Software Availability.- C Bernstein and Rosenthal inequalities.- D A Lemma on the Riesz basis.- Author Index.

Wavelets, Approximation, and Statistical Applications

Density estimation in Besov spaces

In this paper, we present an inverse estimation procedure which combines Fourier analysis with wavelet expansion. In the periodic setting, our method can recover a blurred function observed in white noise. The blurring process is achieved through a convolution operator which can either be smooth (polynomial decay of the Fourier transform) or irregular (such as the convolution with a box-car). The proposal is non-linear and does not require any prior knowledge of the smoothness class; it enjoys fast computation and is spatially adaptive. This contrasts with more traditional ltering methods which demand a certain amount of regularisation and often fail to recover non-homogeneous functions. A ne tuning of our method is derived via asymptotic minimax theory which reveals some key dierences with the direct case of Donoho et al. (1995): (a) band-limited wavelet families have nice theoretical and computing features; (b) the high frequency cut o depends on the spectral characteristics of the convolution kernel; (c) thresholds are level dependent in a geometric fashion. We tested our method using simulated lidar data for underwater remote sensing. Both visual and numerical results show an improvement over existing methods. Finally, the theory behind our estimation paradigm gives a complete characterisation of the ’Maxiset’ of the method i.e. the set of functions where the method attains a near-optimal rate of convergence for a variety of L p loss functions.

/pdf/wavelet-deconvolution-in-a-periodic-setting-2kw1pqg0nh.pdf

Gerard Kerkyacharian

Papers

Wavelet Shrinkage: Asymptopia?

Density estimation by wavelet thresholding

Wavelets, Approximation, and Statistical Applications

Density estimation in Besov spaces

Wavelet deconvolution in a periodic setting