SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.

/pdf/regression-shrinkage-and-selection-via-the-lasso-6mx35txfr5.pdf

Regression Shrinkage and Selection via the Lasso

We present a new technique called “t-SNE” that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map. The technique is a variation of Stochastic Neighbor Embedding (Hinton and Roweis, 2002) that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map. t-SNE is better than existing techniques at creating a single map that reveals structure at many different scales. This is particularly important for high-dimensional data that lie on several different, but related, low-dimensional manifolds, such as images of objects from multiple classes seen from multiple viewpoints. For visualizing the structure of very large datasets, we show how t-SNE can use random walks on neighborhood graphs to allow the implicit structure of all of the data to influence the way in which a subset of the data is displayed. We illustrate the performance of t-SNE on a wide variety of datasets and compare it with many other non-parametric visualization techniques, including Sammon mapping, Isomap, and Locally Linear Embedding. The visualizations produced by t-SNE are significantly better than those produced by the other techniques on almost all of the datasets.

/pdf/visualizing-data-using-t-sne-2w1n304iq0.pdf

Visualizing Data using t-SNE

다중혈관 관상동맥 환자에서 y-문합을 이용하여 양쪽 내흉동맥만을 사용한 우회술의 조기 성적

Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

This paper presents a novel method for separating images into
texture and piecewise smooth parts. The proposed approach is based
on a combination of the Basis Pursuit Denoising (BPDN) algorithm
and the Total-Variation (TV) regularization scheme. The basic idea
promoted in this paper is the use of two appropriate dictionaries,
one for the representation of textures, and the other for the
natural scene parts. Each dictionary is designed for sparse
representation of a particular type of image-content (either
texture or piecewise smooth). The use of BPDN with the two
augmented dictionaries leads to the desired separation, along with
noise removal as a by-product. As the need to choose a proper
dictionary for natural scene is very hard, a TV regularization is
employed to better direct the separation process. Experimental
results validate the algorithm's performance.

Image decomposition: separation of texture from piecewise smooth content

Maximum entropy reconstruction has been used in several fields to produce visually striking reconstructions of positive objects (images, densities, spectra) from noisy, indirect measurements. In magnetic resonance spectroscopy, this technique is notable for its apparent noise suppression and its avoidance of the artifacts that affect discrete Fourier transform spectra of short (zero-extended) data records. In the general case where the length of the reconstructed spectrum exceeds that of the data record or where a convolution kernel is incorporated in the reconstruction, no known analytical solution to the reconstruction problem exists. Consequently, knowledge of the properties of maximum entropy reconstruction has been mainly anecdotal, based on a small selection of published reconstructions. However, in the limiting case where the lengths of the reconstructed spectrum and the data record are the same and a convolution kernel is not applied, the problem can be solved analytically. The solution has a simple structure that helps explain several commonly observed features of maximum entropy reconstructions--for example, the biases in the recovered intensities and the fact that noise near the baseline is more successfully suppressed than is noise superimposed on broad features in the spectrum. The solution also shows that the noise suppression offered by maximum entropy reconstruction could (in this special case) be equally well obtained by a "cosmetic" device: simply displaying the conventional Fourier transform reconstruction using a certain nonlinear plotting scale for the vertical (y) coordinate.

https://www.pnas.org/content/pnas/87/13/5066.full.pdf

Does the maximum entropy method improve sensitivity

Control of the False Discovery Rate (FDR) is an important development in multiple hypothesis testing, allowing the user to limit the fraction of rejected null hypotheses which correspond to false rejections (i.e. false discoveries). The FDR principle also can be used in multiparameter estimation problems to set thresholds for separating signal from noise when the signal is sparse. Success has been proven when the noise is Gaussian; see [3]. In this paper, we consider the application of FDR thresholding to a non-Gaussian setting, in hopes of learning whether the good asymptotic properties of FDR thresholding as an estimation tool hold more broadly than just at the standard Gaussian model. We consider a vector Xi, i = 1,...,n, whose coordinates are independent exponential with individual means µi. The vector µ is thought to be sparse, with most coordinates 1 and a small fraction significantly larger than 1. This models a situation where most coordinates are simply ‘noise’, but a small fraction of the coordinates contain ‘signal’. We develop an estimation theory working with log(µi) as the estimand, and use the percoordinate mean-squared error in recovering log(µi) to measure risk. We consider minimax estimation over parameter spaces defined by constraints on the per-coordinate ‘ p norm of log(µi): 1 n ( P n i=1 log p (µ i)) p . Members of such spaces are vectors (µ i) which are sparsely heterogeneous. We find that, for large n and small , FDR thresholding can be nearly minimax, increasingly so as decreases. The FDR control parameter 0 1 prevents near minimaxity. These conclusions mirror those found in the Gaussian case in [3]. The techniques developed here seem applicable to a wide range of other distributional assumptions, other loss measures, and non-i.i.d. dependency structures.

/pdf/asymptotic-minimaxity-of-false-discovery-rate-thresholding-1mm05vq90g.pdf

Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data

We present a discipline for verifiable computational scienti_c research. Our discipline revolves around three simple new concepts — verifiable computational result (VCR), VCR repository and Verifiable Result Identifier (VRI). These are web- and cloud-computing oriented concepts, which exploit today's web infrastructure to achieve standard, simple and automatic reproducibility in computational scientific research. The VCR discipline requires very slight modifications to the way researchers already conduct their computational research and authoring, and to the way publishers manage their content. In return, the discipline marks a significant step towards delivering on the long-anticipated promises of making scientific computation truly reproducible. A researcher practicing this discipline in everyday work produces computational scripts and word processor files that look very much like those they already produce today, but in which a few lines change very subtly and naturally. Those scripts produce a stream of verifiable results, which are the same tables, figures, charts and datasets the researcher traditionally would have produced, but which are watermarked for permanent identification by a VRI, and are automatically and permanently stored in a VCR repository. In a scientific community practicing Verifiable Computational Research, exchange of both ideas and data involves exchanging result identifiers—VRIs—rather than exchanging files. These identifiers are controlled, trusted and automatically generated strings that point to publicly available result as it was originally created by the computational process itself. When a verifiable result is included in a publication, its identifier can be used by any reader with a web browser to locate, browse and, where appropriate, re-execute the computation that produced the result. Journal readers can therefore scrutinize, dispute, understand and eventually trust these computational results, all to an extent impossible through textual explanations that constitute the core of scientific publications to date. In addition, the result identi_er can be used by subsequent computations to locate and retrieve both the published result (in graphical or numerical form) and the original datasets used by its generating computation. Colleagues can thus cite and import data into their own computations, just as traditional publications allow them to cite and import ideas. We describe an existing software implementation of the Verifiable Computational Research discipline, and argue that it solves many of the crucial problems commonly facing computer-based and computer-aided research in various scientific fields. Our system is secure, naturally adapted to large-scale and cloud computations and to modern massive data analysis, yet places effectively no additional workload on either the researcher or the publisher.

A Universal Identifier for Computational Results

We study the problem of estimating θ from data Y ~ N(θ,σ2) under squared-error loss. We define three new scalar minimax problems in which the risk is weighted by the size of θ. Simple thresholding gives asymptotically minimax estimates in all three problems. We indicate the relationships of the new problems to each other and to two other neo-classical problems: the problems of the bounded normal mean and of the risk-constrained normal mean. Via the wavelet transform, these results have implications for adaptive function estimation in two settings: estimating functions of unknown type and degree of smoothness in a global l2 norm; and estimating a function of unknown degree of local Holder smoothness at a fixed point. In the latter setting, the scalar minimax results imply: Lepskii's results that it is not possible fully to adapt the unknown degree of smoothness without incurring a performance cost; and that simple thresholding of the empirical wavelet transform gives an estimate of a function at a fixed point which is, to within constants, optimally adaptive to unknown degree of smoothness.

/pdf/neo-classical-minimax-problems-thresholding-and-adaptive-35gyduyfrn.pdf

David L. Donoho

Papers

Image decomposition: separation of texture from piecewise smooth content

Does the maximum entropy method improve sensitivity

Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data

A Universal Identifier for Computational Results

Neo-classical minimax problems, thresholding and adaptive function estimation