Showing papers on "Sparse approximation published in 2006"

$rm K$ -SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation

Abstract: In recent years there has been a growing interest in the study of sparse representation of signals. Using an overcomplete dictionary that contains prototype signal-atoms, signals are described by sparse linear combinations of these atoms. Applications that use sparse representation are many and include compression, regularization in inverse problems, feature extraction, and more. Recent activity in this field has concentrated mainly on the study of pursuit algorithms that decompose signals with respect to a given dictionary. Designing dictionaries to better fit the above model can be done by either selecting one from a prespecified set of linear transforms or adapting the dictionary to a set of training signals. Both of these techniques have been considered, but this topic is largely still open. In this paper we propose a novel algorithm for adapting dictionaries in order to achieve sparse signal representations. Given a set of training signals, we seek the dictionary that leads to the best representation for each member in this set, under strict sparsity constraints. We present a new method-the K-SVD algorithm-generalizing the K-means clustering process. K-SVD is an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data. The update of the dictionary columns is combined with an update of the sparse representations, thereby accelerating convergence. The K-SVD algorithm is flexible and can work with any pursuit method (e.g., basis pursuit, FOCUSS, or matching pursuit). We analyze this algorithm and demonstrate its results both on synthetic tests and in applications on real image data

Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries

Abstract: We address the image denoising problem, where zero-mean white and homogeneous Gaussian additive noise is to be removed from a given image. The approach taken is based on sparse and redundant representations over trained dictionaries. Using the K-SVD algorithm, we obtain a dictionary that describes the image content effectively. Two training options are considered: using the corrupted image itself, or training on a corpus of high-quality image database. Since the K-SVD is limited in handling small image patches, we extend its deployment to arbitrary image sizes by defining a global image prior that forces sparsity over patches in every location in the image. We show how such Bayesian treatment leads to a simple and effective denoising algorithm. This leads to a state-of-the-art denoising performance, equivalent and sometimes surpassing recently published leading alternative denoising methods

Sparse Principal Component Analysis

Abstract: Principal component analysis (PCA) is widely used in data processing and dimensionality reduction. However, PCA suffers from the fact that each principal component is a linear combination of all the original variables, thus it is often difficult to interpret the results. We introduce a new method called sparse principal component analysis (SPCA) using the lasso (elastic net) to produce modified principal components with sparse loadings. We first show that PCA can be formulated as a regression-type optimization problem; sparse loadings are then obtained by imposing the lasso (elastic net) constraint on the regression coefficients. Efficient algorithms are proposed to fit our SPCA models for both regular multivariate data and gene expression arrays. We also give a new formula to compute the total variance of modified principal components. As illustrations, SPCA is applied to real and simulated data with encouraging results.

On Model Selection Consistency of Lasso

Abstract: Sparsity or parsimony of statistical models is crucial for their proper interpretations, as in sciences and social sciences. Model selection is a commonly used method to find such models, but usually involves a computationally heavy combinatorial search. Lasso (Tibshirani, 1996) is now being used as a computationally feasible alternative to model selection. Therefore it is important to study Lasso for model selection purposes. In this paper, we prove that a single condition, which we call the Irrepresentable Condition, is almost necessary and sufficient for Lasso to select the true model both in the classical fixed p setting and in the large p setting as the sample size n gets large. Based on these results, sufficient conditions that are verifiable in practice are given to relate to previous works and help applications of Lasso for feature selection and sparse representation. This Irrepresentable Condition, which depends mainly on the covariance of the predictor variables, states that Lasso selects the true model consistently if and (almost) only if the predictors that are not in the true model are "irrepresentable" (in a sense to be clarified) by predictors that are in the true model. Furthermore, simulations are carried out to provide insights and understanding of this result.

Efficient sparse coding algorithms

Abstract: Sparse coding provides a class of algorithms for finding succinct representations of stimuli; given only unlabeled input data, it discovers basis functions that capture higher-level features in the data. However, finding sparse codes remains a very difficult computational problem. In this paper, we present efficient sparse coding algorithms that are based on iteratively solving two convex optimization problems: an L1-regularized least squares problem and an L2-constrained least squares problem. We propose novel algorithms to solve both of these optimization problems. Our algorithms result in a significant speedup for sparse coding, allowing us to learn larger sparse codes than possible with previously described algorithms. We apply these algorithms to natural images and demonstrate that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.

Just relax: convex programming methods for identifying sparse signals in noise

Abstract: This paper studies a difficult and fundamental problem that arises throughout electrical engineering, applied mathematics, and statistics. Suppose that one forms a short linear combination of elementary signals drawn from a large, fixed collection. Given an observation of the linear combination that has been contaminated with additive noise, the goal is to identify which elementary signals participated and to approximate their coefficients. Although many algorithms have been proposed, there is little theory which guarantees that these algorithms can accurately and efficiently solve the problem. This paper studies a method called convex relaxation, which attempts to recover the ideal sparse signal by solving a convex program. This approach is powerful because the optimization can be completed in polynomial time with standard scientific software. The paper provides general conditions which ensure that convex relaxation succeeds. As evidence of the broad impact of these results, the paper describes how convex relaxation can be used for several concrete signal recovery problems. It also describes applications to channel coding, linear regression, and numerical analysis

Direct Methods for Sparse Linear Systems

Abstract: Preface 1. Introduction 2. Basic algorithms 3. Solving triangular systems 4. Cholesky factorization 5. Orthogonal methods 6. LU factorization 7. Fill-reducing orderings 8. Solving sparse linear systems 9. CSparse 10. Sparse matrices in MATLAB Appendix: Basics of the C programming language Bibliography Index.

Efficient Learning of Sparse Representations with an Energy-Based Model

Abstract: We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces "stroke detectors" when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps.

For most large underdetermined systems of equations, the minimal 1-norm near-solution approximates the sparsest near-solution

Abstract: We consider inexact linear equations y ≈ Φx where y is a given vector in R n , Φ is a given n x m matrix, and we wish to find x 0,∈ as sparse as possible while obeying ∥y - Φx 0,∈ ∥ 2 ≤ ∈. In general, this requires combinatorial optimization and so is considered intractable. On the other hand, the l 1 -minimization problem min ∥x∥ 1 subject to ∥y - Φx∥ 2 ≤ e is convex and is considered tractable. We show that for most Φ, if the optimally sparse approximation x 0,∈ is sufficiently sparse, then the solution x 1,∈ of the l 1 -minimization problem is a good approximation to x 0,∈ . We suppose that the columns of Φ are normalized to the unit l 2 -norm, and we place uniform measure on such Φ. We study the underdetermined case where m ∼ τn and τ > 1, and prove the existence of p = p(r) > 0 and C = C(p, τ) so that for large n and for all Φ's except a negligible fraction, the following approximate sparse solution property of Φ holds: for every y having an approximation ∥y - Φx 0 ∥ 2 ≤ ∈ by a coefficient vector x 0 e R m with fewer than ρ · n nonzeros, ∥x 1,∈ - x 0 ∥ 2 ≤ C ≤ ∈. This has two implications. First, for most Φ, whenever the combinatorial optimization result x 0,∈ would be very sparse, x 1,∈ is a good approximation to x 0,∈ . Second, suppose we are given noisy data obeying y = Φx 0 + z where the unknown x 0 is known to be sparse and the noise ∥z∥ 2 ≤ ∈. For most Φ, noise-tolerant l 1 -minimization will stably recover x 0 from y in the presence of noise z. We also study the barely determined case m = n and reach parallel conclusions by slightly different arguments. Proof techniques include the use of almost-spherical sections in Banach space theory and concentration of measure for eigenvalues of random matrices.

Theoretical Results on Sparse Representations of Multiple-Measurement Vectors

Abstract: The sparse representation of a multiple-measurement vector (MMV) is a relatively new problem in sparse representation. Efficient methods have been proposed. Although many theoretical results that are available in a simple case-single-measurement vector (SMV)-the theoretical analysis regarding MMV is lacking. In this paper, some known results of SMV are generalized to MMV. Some of these new results take advantages of additional information in the formulation of MMV. We consider the uniqueness under both an lscr0-norm-like criterion and an lscr1-norm-like criterion. The consequent equivalence between the lscr0-norm approach and the lscr1-norm approach indicates a computationally efficient way of finding the sparsest representation in a redundant dictionary. For greedy algorithms, it is proven that under certain conditions, orthogonal matching pursuit (OMP) can find the sparsest representation of an MMV with computational efficiency, just like in SMV. Simulations show that the predictions made by the proved theorems tend to be very conservative; this is consistent with some recent advances in probabilistic analysis based on random matrix theory. The connections will be discussed

Sparse Representation for Signal Classification

Abstract: In this paper, application of sparse representation (factorization) of signals over an overcomplete basis (dictionary) for signal classification is discussed. Searching for the sparse representation of a signal over an overcomplete dictionary is achieved by optimizing an objective function that includes two terms: one that measures the signal reconstruction error and another that measures the sparsity. This objective function works well in applications where signals need to be reconstructed, like coding and denoising. On the other hand, discriminative methods, such as linear discriminative analysis (LDA), are better suited for classification tasks. However, discriminative methods are usually sensitive to corruption in signals due to lacking crucial properties for signal reconstruction. In this paper, we present a theoretical framework for signal classification with sparse representation. The approach combines the discrimination power of the discriminative methods with the reconstruction property and the sparsity of the sparse representation that enables one to deal with signal corruptions: noise, missing data and outliers. The proposed approach is therefore capable of robust classification with a sparse representation of signals. The theoretical results are demonstrated with signal classification tasks, showing that the proposed approach outperforms the standard discriminative methods and the standard sparse representation in the case of corrupted signals.

Image Denoising Via Learned Dictionaries and Sparse representation

Abstract: We address the image denoising problem, where zeromean white and homogeneous Gaussian additive noise should be removed from a given image. The approach taken is based on sparse and redundant representations over a trained dictionary. The proposed algorithm denoises the image, while simultaneously trainining a dictionary on its (corrupted) content using the K-SVD algorithm. As the dictionary training algorithm is limited in handling small image patches, we extend its deployment to arbitrary image sizes by defining a global image prior that forces sparsity over patches in every location in the image. We show how such Bayesian treatment leads to a simple and effective denoising algorithm, with state-of-the-art performance, equivalent and sometimes surpassing recently published leading alternative denoising methods.

k-t SPARSE: High frame rate dynamic MRI exploiting spatio-temporal sparsity

Abstract: M. Lustig, J. M. Santos, D. L. Donoho, J. M. Pauly Electrical Engineering, Stanford University, Stanford, CA, United States, Statistics, Stanford University, Stanford, CA, United States Introduction Recently rapid imaging methods that exploit the spatial sparsity of images using under-sampled randomly perturbed spirals and non-linear reconstruction have been proposed [1,2]. These methods were inspired by theoretical results in sparse signal recovery [1-5] showing that sparse or compressible signals can be recovered from randomly under-sampled frequency data. We propose a method for high frame-rate dynamic imaging based on similar ideas, now exploiting both spatial and temporal sparsity of dynamic MRI image sequences (dynamic scene). We randomly under-sample k-t space by random ordering of the phase encodes in time (Fig. 1). We reconstruct by minimizing the L1 norm of a transformed dynamic scene subject to data fidelity constraints. Unlike previously suggested linear methods [7, 8], our method does not require a known spatio-temporal structure nor a training set, only that the dynamic scene has a sparse representation. We demonstrate a 7-fold frame-rate acceleration both in simulated data and in vivo non-gated Cartesian balanced-SSFP cardiac MRI . Theory Dynamic MR images are highly redundant in space and time. By using linear transformations (such as wavelets, Fourier etc.), we can represent a dynamic scene using only a few sparse transform coefficients. Inadequate sampling of the spatial-frequency -temporal space (k-t space) results in aliasing in the spatial -temporal-frequency space (x-f space). The aliasing artifacts due to random under-sampling are incoherent as opposed to coherent artifacts in equispaced under sampling. More importantly the artifacts are incoherent in the sparse transform domain. By using the non-linear reconstruction scheme in [1-5] we can recover the sparse transform coefficients and as a consequence, recover the dynamic scene. We exploit sparsity by constraining our reconstruction to have a sparse representation and be consistent with the measured data by solving the constrained optimization problem: minimize ||Ψm||1 subject to: ||Fm – y||2 < e. Here m is the dynamic scene, Ψ transforms the scene into a sparse representation, F is randomized phase encode ordering Fourier matrix, y is the measured k-space data and e controls fidelity of the reconstruction to the measured data. e is usually set to the noise level. Methods For dynamic heart imaging, we propose using the wavelet transform in the spatial dimension and the Fourier transform in the temporal. Wavelets sparsify medical images [1] whereas the Fourier transform sparsifies smooth or periodic temporal behavior. Moreover, with random k-t sampling, aliasing is extremely incoherent in this particular transform domain. To validate our approach we considered a simulated dynamic scene with periodic heart-like motion. A random phase-encode ordered Cartesian acquisition (See Fig. 2) was simulated with a TR=4ms, 64 pixels, acquiring a total of 1024 phase encodes (4.096 sec). The data was reconstructed at a frame rate of 15FPS (a 4-fold acceleration factor) using the L1 reconstruction scheme implemented with non-linear conjugate gradients. The result was compared to a sliding window reconstruction (64 phase encodes in length). To further validate our method we considered a Cartesian balanced-SSFP dynamic heart scan (TR=4.4, TE=2.2, α=60°, res=2.5mm, slice=9mm). 1152 randomly ordered phase encodes (5sec) where collected and reconstructed using the L1 scheme at a 7-fold acceleration (25FPS). Result was compared to a sliding window (64 phase encodes) reconstruction. The experiment was performed on a 1.5T GE Signa scanner using a 5inch surface coil. Results and discussion Figs. 2 and 3 illustrate the simulated phantom and actual dynaic heart scan reconstructions. Note, that even at 4 to 7-fold acceleration, the proposed method is able to recover the motion, preserving the spatial frequencies and suppressing aliasing artifacts. This method can be easily extended to arbitrary trajectories and can also be easily integrated with other acceleration methods such as phase constrained partial k-space and SENSE [1]. In the current, Matlab implementation we are able to reconstruct a 64x64x64 scene in an hour. This can be improved by using newly proposed reconstruction techniques [5,6]. Previously proposed linear methods [7,8] exploit known or measured spatio-temporal structure. The advantage of the proposed method is that the signal need not have a known structure, only sparsity, which is a very realistic assumption in dynamic medical images [1,7,8]. Therefore, a training set is not required. References [1] Lustig et al. 13th ISMRM 2004:p605 [2] Lustig et al. ” Rapid MR Angiography ...” Accepted SCMR06’ [3] Candes et al. ”Robust Uncertainty principals". Manuscript. [4] Donoho D. “Compressesed Sensing”. Manuscript. [5] Candes et al. “Practical Signal Recovery from Random Projections” Manuscript. [6] M. Elad, "Why Simple Shrinkage is Still Relevant?" Manuscript. [7] Tsao et al.. Magn Reson Med. 2003 Nov;50(5):1031-42. [8] Madore et al. Magn Reson Med. 1999 Nov;42(5):813-28. Figure 2: Simulated dynamic data. (a) The transform domain of the cross section is truly sparse. (b) Ground truth crosssection. (c) L1 reconstruction from random phase encode ordering, 4-fold acceleration (d) Sliding window (64) reconstruction from random phase encode ordering.. Figure 1: (a) Sequential phase encode ordering. (b) Random Phase encode ordering. The k-t space is randomly sampled, which enables recovery of sparse spatio-temporal dynamic scenes using the L1 reconstruction.

Underdetermined blind source separation based on sparse representation

Abstract: This paper discusses underdetermined (i.e., with more sources than sensors) blind source separation (BSS) using a two-stage sparse representation approach. The first challenging task of this approach is to estimate precisely the unknown mixing matrix. In this paper, an algorithm for estimating the mixing matrix that can be viewed as an extension of the DUET and the TIFROM methods is first developed. Standard clustering algorithms (e.g., K-means method) also can be used for estimating the mixing matrix if the sources are sufficiently sparse. Compared with the DUET, the TIFROM methods, and standard clustering algorithms, with the authors' proposed method, a broader class of problems can be solved, because the required key condition on sparsity of the sources can be considerably relaxed. The second task of the two-stage approach is to estimate the source matrix using a standard linear programming algorithm. Another main contribution of the work described in this paper is the development of a recoverability analysis. After extending the results in , a necessary and sufficient condition for recoverability of a source vector is obtained. Based on this condition and various types of source sparsity, several probability inequalities and probability estimates for the recoverability issue are established. Finally, simulation results that illustrate the effectiveness of the theoretical results are presented.

Exactly Sparse Delayed-State Filters for View-Based SLAM

Abstract: This paper reports the novel insight that the simultaneous localization and mapping (SLAM) information matrix is exactly sparse in a delayed-state framework. Such a framework is used in view-based representations of the environment that rely upon scan-matching raw sensor data to obtain virtual observations of robot motion with respect to a place it has previously been. The exact sparseness of the delayed-state information matrix is in contrast to other recent feature-based SLAM information algorithms, such as sparse extended information filter or thin junction-tree filter, since these methods have to make approximations in order to force the feature-based SLAM information matrix to be sparse. The benefit of the exact sparsity of the delayed-state framework is that it allows one to take advantage of the information space parameterization without incurring any sparse approximation error. Therefore, it can produce equivalent results to the full-covariance solution. The approach is validated experimentally using monocular imagery for two datasets: a test-tank experiment with ground truth, and a remotely operated vehicle survey of the RMS Titanic

First-order methods for sparse covariance selection

Abstract: Given a sample covariance matrix, we solve a maximum likelihood problem penalized by the number of nonzero coefficients in the inverse covariance matrix. Our objective is to find a sparse representation of the sample data and to highlight conditional independence relationships between the sample variables. We first formulate a convex relaxation of this combinatorial problem, we then detail two efficient first-order algorithms with low memory requirements to solve large-scale, dense problem instances.

Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements

Abstract: This paper proves best known guarantees for exact reconstruction of a sparse signal f from few non-adaptive universal linear measurements. We consider Fourier measurements (random sample of frequencies of f) and random Gaussian measurements. The method for reconstruction that has recently gained momentum in the sparse approximation theory is to relax this highly non-convex problem to a convex problem, and then solve it as a linear program. What are best guarantees for the reconstruction problem to be equivalent to its convex relaxation is an open question. Recent work shows that the number of measurements k(r,n) needed to exactly reconstruct any r-sparse signal f of length n from its linear measurements with convex relaxation is usually O(r poly log (n)). However, known guarantees involve huge constants, in spite of very good performance of the algorithms in practice. In attempt to reconcile theory with practice, we prove the first guarantees for universal measurements (i.e. which work for all sparse functions) with reasonable constants. For Gaussian measurements, k(r,n) lsim 11.7 r [1.5 + log(n/r)], which is optimal up to constants. For Fourier measurements, we prove the best known bound k(r, n) = O(r log(n) middot log2(r) log(r log n)), which is optimal within the log log n and log3 r factors. Our arguments are based on the technique of geometric functional analysis and probability in Banach spaces.

Direct Methods for Sparse Linear Systems (Fundamentals of Algorithms 2)

Abstract: Iterative Methods for Sparse Linear Systems Direct methods for sparse linear systems IWR: Home Direct Methods for Sparse Linear Systems | Society for ... Direct Methods for Sparse Linear Systems: MATLAB sparse ... (PDF) Parallel Direct Methods For Sparse Linear Systems Direct Methods for Sparse Matrices Univerzita Karlova Iterative Methods for Sparse Linear Systems Second Edition Direct Methods for Sparse Linear Systems MATLAB ... Direct Methods for Sparse Linear Systems YouTube Direct Methods for Sparse Linear Systems by Timothy A ... A survey of direct methods for sparse linear systems Direct methods for sparse matrix solution Scholarpedia 01: direct methods for sparse linear systems (lecture 1 of 42) Direct Methods for Sparse Linear Systems (Fundamentals of ... Direct Methods For Sparse Linear CSPARSE A Concise Sparse Matrix Package in C [PDF] Download Direct Methods For Sparse Linear Systems ... Direct Methods for Sparse Linear Systems | Request PDF

Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part I: theory

Abstract: We study the robust estimation of missing regions in images and video using adaptive, sparse reconstructions. Our primary application is on missing regions of pixels containing textures, edges, and other image features that are not readily handled by prevalent estimation and recovery algorithms. We assume that we are given a linear transform that is expected to provide sparse decompositions over missing regions such that a portion of the transform coefficients over missing regions are zero or close to zero. We adaptively determine these small magnitude coefficients through thresholding, establish sparsity constraints, and estimate missing regions in images using information surrounding these regions. Unlike prevalent algorithms, our approach does not necessitate any complex preconditioning, segmentation, or edge detection steps, and it can be written as a sequence of denoising operations. We show that the region types we can effectively estimate in a mean-squared error sense are those for which the given transform provides a close approximation using sparse nonlinear approximants. We show the nature of the constructed estimators and how these estimators relate to the utilized transform and its sparsity over regions of interest. The developed estimation framework is general, and can readily be applied to other nonstationary signals with a suitable choice of linear transforms. Part I discusses fundamental issues, and Part II is devoted to adaptive algorithms with extensive simulation examples that demonstrate the power of the proposed techniques.

On the exponential convergence of matching pursuits in quasi-incoherent dictionaries

Abstract: The purpose of this correspondence is to extend results by Villemoes and Temlyakov about exponential convergence of Matching Pursuit (MP) with some structured dictionaries for "simple" functions in finite or infinite dimension. The results are based on an extension of Tropp's results about Orthogonal Matching Pursuit (OMP) in finite dimension, with the observation that it does not only work for OMP but also for MP. The main contribution is a detailed analysis of the approximation and stability properties of MP with quasi-incoherent dictionaries, and a bound on the number of steps sufficient to reach an error no larger than a penalization factor times the best m-term approximation error.

Nonnegative Sparse PCA

Abstract: We describe a nonnegative variant of the "Sparse PCA" problem. The goal is to create a low dimensional representation from a collection of points which on the one hand maximizes the variance of the projected points and on the other uses only parts of the original coordinates, and thereby creating a sparse representation. What distinguishes our problem from other Sparse PCA formulations is that the projection involves only nonnegative weights of the original coordinates — a desired quality in various fields, including economics, bioinformatics and computer vision. Adding nonnegativity contributes to sparseness, where it enforces a partitioning of the original coordinates among the new axes. We describe a simple yet efficient iterative coordinate-descent type of scheme which converges to a local optimum of our optimization criteria, giving good results on large real world datasets.

A Bayesian Approach for Blind Separation of Sparse Sources

Abstract: We present a Bayesian approach for blind separation of linear instantaneous mixtures of sources having a sparse representation in a given basis. The distributions of the coefficients of the sources in the basis are modeled by a Student t distribution, which can be expressed as a scale mixture of Gaussians, and a Gibbs sampler is derived to estimate the sources, the mixing matrix, the input noise variance and also the hyperparameters of the Student t distributions. The method allows for separation of underdetermined (more sources than sensors) noisy mixtures. Results are presented with audio signals using a modified discrete cosine transform basis and compared with a finite mixture of Gaussians prior approach. These results show the improved sound quality obtained with the Student t prior and the better robustness to mixing matrices close to singularity of the Markov chain Monte Carlo approach

A survey of Sparse Component Analysis for blind source separation: principles, perspectives, and new challenges

Abstract: In this survey, we highlight the appealing features and challenges of Sparse Component Analysis (SCA) for blind source separation (BSS). SCA is a simple yet powerful framework to separate several sources from few sensors, even when the independence assumption is dropped. So far, SCA has been most successfully applied when the sources can be represented sparsely in a given basis, but many other potential uses of SCA remain unexplored. Among other challenging perspectives, we discuss how SCA could be used to exploit both the spatial diversity corresponding to the mixing process and the morphological diversity between sources to unmix even underdetermined convolutive mixtures. This raises several challenges, including the design of both provably good and numerically efficient algorithms for large-scale sparse approximation with overcomplete signal dictionaries.

Learning weighted metrics to minimize nearest-neighbor classification error

Abstract: In order to optimize the accuracy of the nearest-neighbor classification rule, a weighted distance is proposed, along with algorithms to automatically learn the corresponding weights. These weights may be specific for each class and feature, for each individual prototype, or for both. The learning algorithms are derived by (approximately) minimizing the leaving-one-out classification error of the given training set. The proposed approach is assessed through a series of experiments with UCI/STATLOG corpora, as well as with a more specific task of text classification which entails very sparse data representation and huge dimensionality. In all these experiments, the proposed approach shows a uniformly good behavior, with results comparable to or better than state-of-the-art results published with the same data so far

Combinatorial Algorithms for Compressed Sensing

Abstract: In sparse approximation theory, the fundamental problem is to reconstruct a signal AisinRn from linear measurements (A,psii) with respect to a dictionary of psii's. Recently, there is focus on the novel direction of Compressed Sensing where the reconstruction can be done with very few-O(klogn)-linear measurements over a modified dictionary if the signal is compressible, that is, its information is concentrated in k coefficients with the original dictionary. In particular, the results prove that there exists a single O(klogn)timesn measurement matrix such that any such signal can be reconstructed from these measurements, with error at most O(1) times the worst case error for the class of such signals. Compressed sensing has generated tremendous excitement both because of the sophisticated underlying mathematics and because of its potential applications. In this paper, we address outstanding open problems in Compressed Sensing. Our main result is an explicit construction of a non-adaptive measurement matrix and the corresponding reconstruction algorithm so that with a number of measurements polynomial in k, logn, 1/epsiv, we can reconstruct compressible signals. This is the first known polynomial time explicit construction of any such measurement matrix. In addition, our result improves the error guarantee from O(1) to 1+epsiv and improves the reconstruction time from poly(n) to poly (klogn). Our second result is a randomized construction of O(kpolylog(n)) measurements that work for each signal with high probability and gives per-instance approximation guarantees rather than over the class of all signals. Previous work on compressed sensing does not provide such per-instance approximation guarantees; our result improves the best known number of measurements known from prior work in other areas including learning theory, streaming algorithms and complexity theory for this case. Our approach is combinatorial. In particular, we use two parallel sets of group tests, one to filter and the other to certify and estimate; the resulting algorithms are quite simple to implement.