SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.

/pdf/regression-shrinkage-and-selection-via-the-lasso-6mx35txfr5.pdf

Regression Shrinkage and Selection via the Lasso

We present a new technique called “t-SNE” that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map. The technique is a variation of Stochastic Neighbor Embedding (Hinton and Roweis, 2002) that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map. t-SNE is better than existing techniques at creating a single map that reveals structure at many different scales. This is particularly important for high-dimensional data that lie on several different, but related, low-dimensional manifolds, such as images of objects from multiple classes seen from multiple viewpoints. For visualizing the structure of very large datasets, we show how t-SNE can use random walks on neighborhood graphs to allow the implicit structure of all of the data to influence the way in which a subset of the data is displayed. We illustrate the performance of t-SNE on a wide variety of datasets and compare it with many other non-parametric visualization techniques, including Sammon mapping, Isomap, and Locally Linear Embedding. The visualizations produced by t-SNE are significantly better than those produced by the other techniques on almost all of the datasets.

/pdf/visualizing-data-using-t-sne-2w1n304iq0.pdf

Visualizing Data using t-SNE

다중혈관 관상동맥 환자에서 y-문합을 이용하여 양쪽 내흉동맥만을 사용한 우회술의 조기 성적

Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

We consider recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE - square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level $\sigma$, the optimal nonlinearity for MSE loss simply shrinks each data singular value $y$ to $\sqrt{y^2-4n\sigma^2 }$ (or to 0 if $y 0.

Optimal Shrinkage of Singular Values

An unknown $m$ by $n$ matrix $X_{0}$ is to be estimated from noisy measurements $Y=X_{0}+Z$, where the noise matrix $Z$ has i.i.d. Gaussian entries. A popular matrix denoising scheme solves the nuclear norm penalization problem $\operatorname{min}_{X}\|Y-X\|_{F}^{2}/2+\lambda\|X\|_{*}$, where $\|X\|_{*}$ denotes the nuclear norm (sum of singular values). This is the analog, for matrices, of $\ell_{1}$ penalization in the vector case. It has been empirically observed that if $X_{0}$ has low rank, it may be recovered quite accurately from the noisy measurement $Y$. In a proportional growth framework where the rank $r_{n}$, number of rows $m_{n}$ and number of columns $n$ all tend to $\infty$ proportionally to each other ($r_{n}/m_{n}\rightarrow \rho$, $m_{n}/n\rightarrow \beta$), we evaluate the asymptotic minimax MSE $ \mathcal{M} (\rho,\beta)=\lim_{m_{n},n\rightarrow \infty}\inf_{\lambda}\sup_{\operatorname{rank}(X)\leq r_{n}}\operatorname{MSE}(X_{0},\hat{X}_{\lambda})$. Our formulas involve incomplete moments of the quarter- and semi-circle laws ($\beta=1$, square case) and the Marcenko–Pastur law ($\beta<1$, nonsquare case). For finite $m$ and $n$, we show that MSE increases as the nonzero singular values of $X_{0}$ grow larger. As a result, the finite-$n$ worst-case MSE, a quantity which can be evaluated numerically, is achieved when the signal $X_{0}$ is “infinitely strong.” The nuclear norm penalization problem is solved by applying soft thresholding to the singular values of $Y$. We also derive the minimax threshold, namely the value $\lambda^{*}(\rho)$, which is the optimal place to threshold the singular values. All these results are obtained for general (nonsquare, nonsymmetric) real matrices. Comparable results are obtained for square symmetric nonnegative-definite matrices.

/pdf/minimax-risk-of-matrix-denoising-by-singular-value-2gmwn2rpld.pdf

Minimax risk of matrix denoising by singular value thresholding

Image data are often composed of two or more geometrically distinct constituents; in galaxy catalogs, for instance, one sees a mixture of pointlike structures (galaxy superclusters) and curvelike structures (filaments). It would be ideal to process a single image and extract two geometrically `pure' images, each one containing features from only one of the two geometric constituents. This seems to be a seriously underdetermined problem, but recent empirical work achieved highly persuasive separations. We present a theoretical analysis showing that accurate geometric separation of point and curve singularities can be achieved by minimizing the $\ell_1$ norm of the representing coefficients in two geometrically complementary frames: wavelets and curvelets. Driving our analysis is a specific property of the ideal (but unachievable) representation where each content type is expanded in the frame best adapted to it. This ideal representation has the property that important coefficients are clustered geometrically in phase space, and that at fine scales, there is very little coherence between a cluster of elements in one frame expansion and individual elements in the complementary frame. We formally introduce notions of cluster coherence and clustered sparsity and use this machinery to show that the underdetermined systems of linear equations can be stably solved by $\ell_1$ minimization; microlocal phase space helps organize the calculations that cluster coherence requires.

Microlocal Analysis of the Geometric Separation Problem

The classical multivariate linear regression problem assumes p variables X 1 , X 2 , . . ., X p and a response vector y, each with n observations, and a linear relationship between the two: y = X β + z, where z ∼ N (0, σ2. We poin out that when p > n, there is a breakdown point for standard model selection schemes, such that model selection only works well below a certain critical complexity level depending on n/p. We apply this notion to some standard model selection algorithms (Forward Stepwise, LASSO, LARS) in the case where p ≫ n. We find that 1) the breakdown point is well-defined for random X-models and low noise, 2) increasing noise shifts the breakdown point to lower levels of sparsity, and reduces the model recovery ability of the algorithm in a systematic way, and 3) below breakdown, the size of coefficient errors follows the theoretical error distribution for the classical linear model.

/pdf/breakdown-point-of-model-selection-when-the-number-of-24ezv8nybi.pdf

Breakdown Point of Model Selection When the Number of Variables Exceeds the Number of Observations

M. Lustig, J. H. Lee, D. L. Donoho, J. M. Pauly Electrical Engineering, Stanford University, Stanford, CA, United States, Statistics, Stanford University, Stanford, CA, United States Introduction We propose a fast imaging method based on undersampled k-space spiral sampling and non-linear reconstruction. Our approach is inspired by theoretical results in sparse signal recovery [1,2] showing that compressible signals can be completely recovered from randomly undersampled frequency data. Since random sampling in frequency space is impractical for MRI hardware, we develop a practical strategy allowing 50% undersampling by adapting spiral MR imaging. We introduce randomness by perturbing our spiral trajectories. We reconstruct by minimizing the L1 norm of a transformed image subject to data fidelity constraints. Simulations and experimental results show good reconstruction particularly from heavily undersampled k-space data where conventional methods fail. This method can be used with other imaging/reconstruction methods such as SENSE[8]. Theory The goal of partial k-space reconstruction is to reconstruct an image from incomplete Fourier data -a highly under-determined problem. Medical images often have a sparse representation in some domain (such as finite differences, wavelets, etc.), where the number of coefficients needed to describe the image accurately is significantly smaller than the number of pixels in the image. We exploit sparsity by constraining our reconstruction to have a sparse representation and be consistent with the measured k-space data. Surprisingly, if the underlying true object has a sparse representation (see [1,2,3] for details) and the sampling in frequency is uniformly and randomly distributed, we can recover the signal accurately by solving the following constrained optimization problem, minimize ||Ψ(m)||1 (1) s.t ||Fm – y||2 < e Here, m is the image, Ψ transforms the image into a sparse representation, F is an undersampled Fourier matrix, y is the measured k-space data and e controls fidelity of the reconstruction to the measured data. e is usually set to the noise level. The objective enforces sparsity whereas the constraint enforces data consistency. Eq. 1 is a convex quadratic program (QP)[4] that can be solved efficiently by interior point methods. Note that the non-linearity of the L1 norm is crucial [4]. Our approach can also be used with SENSE reconstruction; simply substitute in place of the Fourier matrix an encoding matrix that includes both Fourier and coil sensitivity matrices. Methods We propose two different sparsifying transforms, the wavelet transform and finite differences -both widely used in image processing [7]. For finite differences, the objective becomes the total variation TV = ΣxΣy |∇m(x,y)| where ∇m(x,y) is the spatial gradient of the image, computed by finite differences. Random sampling, as advocated in [1,2,3] is not feasible in MR because of hardware limitations. However, spiral trajectories are a good candidate for approximating random sampling. They span kspace uniformly but on the other hand they are far from being as regular as a Cartesian grid. Furthermore spiral imaging is fast and time-efficient. To introduce more randomness we perturbed the individual spiral trajectories, slightly deviating from the deterministic spiral along each interleave; the interleave angles are also perturbed by a small random angle. To validate our approach we considered a 34 interleave perturbed spiral trajectory, designed for a 16 cm FOV 1 mm resolution. We undersampled by 50% by acquiring data only on a subset of 17 out of the 34 interleaves using a GRE sequence (TE=1.3ms, TR=8.24, RO=3ms, α=30°, ST= 4mm). The experiment was conducted on a 1.5T GE Signa scanner with gradients capable of 40mT/m and 150mT/m/ms maximum slew rate. The image was reconstructed by TV reconstruction implemented with finite derivatives, and with L1 wavelet (Daubechies 4) reconstruction. Results were compared to gridding and minimum-norm reconstructions. Our reconstructions used a primal-dual interior point solver [4] with min-max nuFFT [5,6]. Results and discussion Fig 1. illustrates the results of four reconstruction algorithms. As expected, the gridding and minimum norm reconstructions exhibit severe aliasing artifacts due to undersampling. On the other hand, the L1 reconstructions removed most aliasing artifacts while preserving resolution. TV penalization performs slightly better than the L1/wavelet penalty. This difference is attributable to the object’s being piece wise constant and so being sparser for the finite difference operator than for the wavelet transform. Fine structures that are severely corrupted by aliasing are well recovered by the L1 reconstructions. Note that the Fourier transform of all the reconstructed images in Fig.1 is the same (up to noise level) at the spiral sample points. The L1 method was able to recover the information because the correct image representation is sparse, and sparsity is being imposed. Conclusion In conclusion, L1 –penalized image reconstruction outperforms conventional linear reconstruction, recovering the image even with severe undersampling. The non-linearity of the L1 norm is the key; however our method is more computationally intensive than traditional linear methods. In the current, rather inefficient Matlab implementation we are able to reconstruct a 256x256 2D image in a matter of several minutes. Our simulations show that using perturbed spirals offers better reconstruction than just by uniformly undersampling k-space. This type of reconstruction can be used to speed up acquisition whenever there is sparsity to exploit. Applications such as angiography, time-resolved and contrast enhanced imaging are perfect candidates as such images can be have a very sparse representation. References

David L. Donoho

Papers

Optimal Shrinkage of Singular Values

Minimax risk of matrix denoising by singular value thresholding

Microlocal Analysis of the Geometric Separation Problem

Breakdown Point of Model Selection When the Number of Variables Exceeds the Number of Observations

Faster Imaging with Randomly Perturbed, Undersampled Spirals and |L|_1 Reconstruction