SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.

/pdf/regression-shrinkage-and-selection-via-the-lasso-6mx35txfr5.pdf

Regression Shrinkage and Selection via the Lasso

We present a new technique called “t-SNE” that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map. The technique is a variation of Stochastic Neighbor Embedding (Hinton and Roweis, 2002) that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map. t-SNE is better than existing techniques at creating a single map that reveals structure at many different scales. This is particularly important for high-dimensional data that lie on several different, but related, low-dimensional manifolds, such as images of objects from multiple classes seen from multiple viewpoints. For visualizing the structure of very large datasets, we show how t-SNE can use random walks on neighborhood graphs to allow the implicit structure of all of the data to influence the way in which a subset of the data is displayed. We illustrate the performance of t-SNE on a wide variety of datasets and compare it with many other non-parametric visualization techniques, including Sammon mapping, Isomap, and Locally Linear Embedding. The visualizations produced by t-SNE are significantly better than those produced by the other techniques on almost all of the datasets.

/pdf/visualizing-data-using-t-sne-2w1n304iq0.pdf

Visualizing Data using t-SNE

다중혈관 관상동맥 환자에서 y-문합을 이용하여 양쪽 내흉동맥만을 사용한 우회술의 조기 성적

Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

Publisher Summary This chapter provides a simple and general framework into which a number of the minimum entropy deconvolution (MED) procedures inspired by the work of Wiggens can be fit This makes possible an analysis and comparison of these procedures according to the large-sample statistical properties of the coefficient estimates they produce The chapter presents a demonstration of the role of entropy within this framework, vindicating Wiggens intuitive use of the term MED is a technique introduced by Wiggins for deconvolution without making prior assumptions about the delay characteristics of the filter f The eye, using a judgement of simplicity, can identify the correct solution to the deconvolution problem even though correlation/spectrum technologies could not Wiggins used this simple observation as the basis for a formal procedure for deconvolution The chapter discusses a basic theory of the MED technique in an ideal case The MED idea and its variants are consistent ways of solving the deconvolution problem when the data follows the convolutional form The chapter describes a partial order arising naturally from the probabilistic structure of the problem and presents a comparison of the precision of various proposals that have been made under the assumption that the size of the data sample is large It further discusses the limitations of the current analysis and the application of these results in seismology and in other fields

On minimum entropy deconvolution

We attempt to recover an n-dimensional vector observed in white noise, where n is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the lp norm for p small. We obtain a procedure which is asymptotically minimax for l r loss, simultaneously throughout a range of such sparsity classes. The optimal procedure is a data-adaptive thresholding scheme, driven by control of the False Discovery Rate (FDR). FDR control is a relatively recent innovation in simultaneous testing, ensuring that at most a certain fraction of the rejected null hypotheses will correspond to false rejections. In our treatment, the FDR control parameter qn also plays a determining role in asymptotic minimaxity. If q = lim qn ∈ [0,1/2] and also qn > γ/log(n) we get sharp asymptotic minimaxity, simultaneously, over a wide range of sparse parameter spaces and loss functions. On the other hand, q = lim qn ∈ (1/2,1], forces the risk to exceed the minimax risk by a factor growing with q. To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new. Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form 2 � log( potential model size / actual model size ). We exhibit a close connection with FDR-controlling procedures under stringent control of the false discovery rate.

/pdf/adapting-to-unknown-sparsity-by-controlling-the-false-3924cjsinb.pdf

Adapting to unknown sparsity by controlling the false discovery rate

In a recent paper, the authors proposed a new class of low-complexity iterative thresholding algorithms for reconstructing sparse signals from a small set of linear measurements [1]. The new algorithms are broadly referred to as AMP, for approximate message passing. This is the first of two conference papers describing the derivation of these algorithms, connection with the related literature, extensions of the original framework, and new empirical evidence. In particular, the present paper outlines the derivation of AMP from standard sum-product belief propagation, and its extension in several directions. We also discuss relations with formal calculations based on statistical mechanics methods.

/pdf/message-passing-algorithms-for-compressed-sensing-i-19jvy4s6z6.pdf

Message passing algorithms for compressed sensing: I. motivation and construction

Let A be a d x n matrix and T = T(n-1) be the standard simplex in Rn. Suppose that d and n are both large and comparable: d approximately deltan, delta in (0, 1). We count the faces of the projected simplex AT when the projector A is chosen uniformly at random from the Grassmann manifold of d-dimensional orthoprojectors of Rn. We derive rhoN(delta) > 0 with the property that, for any rho or = f(k)(T)(1-epsilon). Vershik and Sporyshev previously showed existence of a threshold rhoVS(delta) > 0 at which phase transition occurs in k/d. We compute and display rhoVS and compare with rhoN. Corollaries are as follows. (1) The convex hull of n Gaussian samples in Rd, with n large and proportional to d, has the same k-skeleton as the (n-1) simplex, for k < rhoN (d/n)d(1 + oP(1)). (2) There is a "phase transition" in the ability of linear programming to find the sparsest nonnegative solution to systems of underdetermined linear equations. For most systems having a solution with fewer than rhoVS(d/n)d(1 + o(1)) nonzeros, linear programming will find that solution.

/pdf/neighborliness-of-randomly-projected-simplices-in-high-320g0f6pg9.pdf

Neighborliness of randomly-projected simplices in high dimensions

Recently, Candes and Donoho introduced the curvelet transform, a new multiscale representation suited for objects which are smooth away from discontinuities across curves. Their proposal was intended for functions f defined on the continuum plane R2. In this paper, we consider the problem of realizing this transform for digital data. We describe a strategy for computing a digital curvelet transform, we describe a software environment, Curvelet 256, implementing this strategy in the case of 256 X 256 images, and we describe some experiments we have conducted using it. Examples are available for viewing by web browser.

https://www.spiedigitallibrary.org/conference-proceedings-of-spie/4056/1/Digital-curvelet-transform-strategy-implementation-and-experiments/10.1117/12.381679.pdf

David L. Donoho

Papers

On minimum entropy deconvolution

Adapting to unknown sparsity by controlling the false discovery rate

Message passing algorithms for compressed sensing: I. motivation and construction

Neighborliness of randomly-projected simplices in high dimensions

Digital curvelet transform: strategy, implementation, and experiments