Showing papers on "Sparse approximation published in 2000"

Sparse coding and decorrelation in primary visual cortex during natural vision.

Abstract: Theoretical studies suggest that primary visual cortex (area V1) uses a sparse code to efficiently represent natural scenes. This issue was investigated by recording from V1 neurons in awake behaving macaques during both free viewing of natural scenes and conditions simulating natural vision. Stimulation of the nonclassical receptive field increases the selectivity and sparseness of individual V1 neurons, increases the sparseness of the population response distribution, and strongly decorrelates the responses of neuron pairs. These effects are due to both excitatory and suppressive modulation of the classical receptive field by the nonclassical receptive field and do not depend critically on the spatiotemporal structure of the stimuli. During natural vision, the classical and nonclassical receptive fields function together to form a sparse representation of the visual world. This sparse code may be computationally efficient for both early vision and higher visual processing.

Boosting image retrieval

Abstract: We present an approach for image retrieval using a very large number of highly selective features and efficient online learning. Our approach is predicated on the assumption that each image is generated by a sparse set of visual "causes" and that images which are visually similar share causes. We propose a mechanism for computing a very large number of highly selective features which capture some aspects of this causal structure (in our implementation there are over 45,000 highly selective features). At query time a user selects a few example images, and a technique known as "boosting" is used to learn a classification function in this feature space. By construction, the boosting procedure learns a simple classifier which only relies on 20 of the features. As a result a very large database of images can be scanned rapidly, perhaps a million images per second. Finally we will describe a set of experiments performed using our retrieval system on a database of 3000 images.

Sparse approximation using least squares support vector machines

Abstract: In least squares support vector machines (LS-SVMs) for function estimation Vapnik's /spl epsiv/-insensitive loss function has been replaced by a cost function which corresponds to a form of ridge regression. In this way nonlinear function estimation is done by solving a linear set of equations instead of solving a quadratic programming problem. The LS-SVM formulation also involves less tuning parameters. However, a drawback is that sparseness is lost in the LS-SVM case. In this paper we investigate imposing sparseness by pruning support values from the sorted support value spectrum which results from the solution to the linear system.

On Algorithms For Permuting Large Entries to the Diagonal of a Sparse Matrix

Abstract: We consider bipartite matching algorithms for computing permutations of a sparse matrix so that the diagonal of the permuted matrix has entries of large absolute value. We discuss various strategies for this and consider their implementation as computer codes. We also consider scaling techniques to further increase the relative values of the diagonal entries. Numerical experiments show the effect of the reorderings and the scaling on the solution of sparse equations by a direct method and by preconditioned iterative techniques.

Blind Source Separation by Sparse Decomposition

Abstract: The blind source separation problem is to extract the underlying source signals from a set of their linear mixtures, where the mixing matrix is unknown. This situation is common, eg in acoustics, radio, and medical signal processing. We exploit the property of the sources to have a sparse representation in a corresponding signal dictionary. Such a dictionary may consist of wavelets, wavelet packets, etc., or be obtained by learning from a given family of signals. Starting from the maximum a posteriori framework, which is applicable to the case of more sources than mixtures, we derive a few other categories of objective functions, which provide faster and more robust computations, when there are an equal number of sources and mixtures. Our experiments with artificial signals and with musical sounds demonstrate significantly better separation than other known techniques.

Optimizing the Performance of Sparse Matrix-Vector Multiplication

Abstract: Sparse matrix operations dominate the performance of many scientific and engineering applications. In particular, iterative methods are commonly used in algorithms for linear systems, least squares problems, and eigenvalue problems, which involve a sparse matrix-vector product in the inner loop. The performance of sparse matrix algorithms is often disappointing on modern machines because the algorithms have poor temporal and spatial locality, and are therefore limited by the speed of main memory. Unfortunately, the performance gap between memory and processing is steadily increasing, as processor performance increases by roughly 60% every year, while memory latency drops by only 7%. Performance is also highly dependent on the nonzero structure of the sparse matrix, the organization of the data and its computation, and the exact parameters of the hardware memory system. This thesis presents a toolkit called Sparsity for the automatic optimization of sparse matrix-vector multiplication. We start with an extensive study of possible memory hierarchy optimizations, in particular, reorganization of the matrix and computation around blocks of the matrix. The research demonstrates that certain kinds of blocking can be effective for both registers and caches, although the nature of that tiling is quite different due to the differences in size between typical register sets and caches. Both types of blocking are shown to be highly effective in some matrices, but ineffective in others, and the choice of block size is also shown to be highly dependent on the matrix and the machine. Thus, to automatically determine when and how the optimizations should be applied, we employ a combination of search over a set of possible optimized versions, along with newly devised performance models to eliminate or constrain the search to make it practical. We also consider a common variation of basic sparse matrix-vector multiplication in which a sparse matrix is multiplied by a set of dense vectors. This operation arises, for example, when there are multiple right-hand sides in a linear solver, or when a higher level algorithm has been blocked. The introduction of multiple vectors offers enormous optimization opportunities, effectively changing a matrix-vector operation into a matrix-matrix operation. It is well known that for dense matrices, the latter algorithm has much higher data reuse than the former, and so can achieve much better performance; the same is true in the sparse case. The Sparsity system is designed as a web service, so scientists and engineers can easily obtain highly optimized sparse matrix routines without needing to understand the specifics of the optimization techniques or how they are selected. This thesis also reports on an extensive performance study of over 40 matrices on a variety of machines. The matrices are taken from various scientific and engineering problems, as well as from linear programming and data mining. The machines include the Alpha 21164, UltraSPARC I, MIPS R10000 and PowerPC 604e. These benchmark results are useful for understanding the performance differences across application domains, the effectiveness of the optimizations, and the costs associated with evaluating our performance models in applying the optimizations. The conclusion is that Sparsity is highly effective, producing routines that are up to 3.1 times faster for a single vector and 6.2 times faster for multiple vectors.

Sparse Representation for Gaussian Process Models

Abstract: We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the model. Experimental results on toy examples and large real-world data sets indicate the efficiency of the approach.

A subdivision-based algorithm for the sparse resultant

Abstract: Multivariate resultants generalize the Sylvester resultant of two polynomials and characterize the solvability of a polynomial system. They also reduce the computation of all common roots to a problem in linear algebra. We propose a determinantal formula for the sparse resultant of an arbitrary system of n + 1 polynomials in n variables. This resultant generalizes the classical one and has significantly lower degree for polynomials that are sparse in the sense that their mixed volume is lower than their Bezout number. Our algorithm uses a mixed polyhedral subdivision of the Minkowski sum of the Newton polytopes in order to construct a Newton matrix. Its determinant is a nonzero multiple of the sparse resultant and the latter equals the GCD of at most n + 1 such determinants. This construction implies a restricted version of an effective sparse Nullstellensatz. For an arbitrary specialization of the coefficients, there are two methods that use one extra variable and yield the sparse resultant. This is the first algorithm to handle the general case with complexity polynomial in the resultant degree and simply exponential in n. We conjecture its extension to producing an exact rational expression for the sparse resultant.

Variable order panel clustering

Abstract: We present a new version of the panel clustering method for a sparse representation of boundary integral equations. Instead of applying the algorithm separately for each matrix row (as in the classical version of the algorithm) we employ more general block partitionings. Furthermore, a variable order of approximation is used depending on the size of blocks.

Next-generation generic programming and its application to sparse matrix computations

Abstract: The contributions of this paper are the following.We introduce a new variety of generic programming in which algorithm implementors use a different API than data structure designers, the gap between the API's being bridged by restructuring compilers. One view of this approach is that it exploits restructuring compiler technology to perform a novel kind of template instantiation.We demonstrate the usefulness of this new generic programming technology by deploying it in a system that generates efficient sparse codes from high-level algorithms and specifications of sparse matrix formats.We argue that sparse matrix formats should be viewed as indexed-sequential access data structures (in the database sense), and show that appropriate abstractions of the index structure of common formats can be conveyed to a restructuring compiler through the type system of a modern language that supports inheritance and templates.

Reconstruction of scene models from sparse 3D structure

Abstract: In this paper we present a geometric theory for reconstruction of surface models from sparse 3D data captured from N camera views which are consistent with the data visibility. Sparse 3D measurements of real scenes are readily estimated from image sequences using structure-from-motion techniques. Currently there is no general method for reconstruction of 3D models of arbitrary scenes from sparse data. We introduce an algorithm for recursive integration of sparse 3D structure to obtain a consistent model. This algorithm is shown to converge to the real scene structure as the number of views increases and to have a computational cost which is linear in the number of views. Results are presented for real and synthetic image sequences which demonstrate correct reconstruction for scenes containing significant occlusions.

Towards a fast parallel sparse matrix-vector multiplication.

Abstract: The sparse matrix-vector product is an important computational kernel that runs ineffectively on many computers with super-scalar RISC processors. In this paper we analyse the performance of the sparse matrix-vector product with symmetric matrices originating from the FEM and describe techniques that lead to a fast implementation. It is shown how these optimisations can be incorporated into an efficient parallel implementation using messagepassing. We conduct numerical experiments on many different machines and show that our optimisations speed up the sparse matrix-vector multiplication substantially.

Fast methods for extraction and sparsification of substrate coupling

Abstract: The sudden increase in systems-on-a-chip designs has renewed interest in techniques for analyzing and eliminating substrate coupling problems. Previous work on the substrate coupling analysis has focused primarily on faster techniques for extracting coupling resistances, but has offered little help for reducing the resulting network whose number of resistors grows quadratically with the number of contacts. In this paper we show that an approach inspired by wavelets can be used in two ways. First, the wavelet method can be used to accurately sparsify the dense contact conductance matrix. In addition, we show that the method can be used to compute the sparse representation directly. Computational results are presented that show that for a problems with a few thousand contacts, the method can be almost ten times faster at constructing the matrix.

A Framework for Sparse Matrix Code Synthesis from High-level Specifications

Abstract: We present compiler technology for synthesizing sparse matrix code from (i) dense matrix code, and (ii) a description of the index structure of a sparse matrix. Our approach is to embed statement instances into a Cartesian product of statement iteration and data spaces, and to produce efficient sparse code by identifying common enumerations for multiple references to sparse matrices. The approach works for imperfectly-nested codes with dependences, and produces sparse code competitive with hand-written library code for the Basic Linear Algebra Subroutines (BLAS).

Frame based signal representation and compression

Abstract: The demand for efficient communication and data storage is continuously increasing and signal representation and compression are important factors in digital communication and storage systems.This work deals with Frame based signal representation and compression. The emphasis is on the design of frames suited for efficient representation, or for low bit rate compression, of classes of signals.Traditional signal decompositions such as transforms, wavelets, and filter banks, generate expansions using an analysis-synthesis setting. In this thesis we concentrate on the synthesis or reconstruction part of the signal expansion, having a system with no explicit analysis stage. We want to investigate the use of an overcomplete set of vectors, a frame or an overcomplete dictionary, for signal representations and allow sparse representations. Effective signal representations are desirable in many applications, where signal compression is one example. Others can be signal analysis for different purposes, reconstruction of signals from a limited observation set, feature extraction in pattern recognition and so forth.The lack of an explicit analysis stage originates some questions on finding the optimal representation. Finding an optimal sparse representation from an overcomplete set of vectors is NP-complete, and suboptimal vector selection methods are more practical. We have used some existing methods like different variations of the Matching Pursuit (MP) [52] algorithm, and we developed a robust regularized FOCUSS to be able to use FOCUSS (FOCal Underdetermined System Solver [29]) under lossy conditions.In this work we develop techniques for frame design, the Method of Optimal Directions (MOD), and propose methods by which such frames can successfully be used in frame based signal representation and in compression schemes. A Multi Frame Compression (MFC) scheme is presented and experiments with several signal classes show that the MFC scheme works well at low bit rates using MOD designed frames. Reconstruction experiments provides complimentary evidence of the good properties of the MOD algorithm.

Fast clustering with sparse data

Abstract: Efficient data modeling utilizing sparse representation of a data set. In one embodiment, a computer-implemented method such that a data set is first input. The data set has a plurality of records. Each record has at least one attribute, where each attribute has a default value. The method stores a sparse representation of each record, such that the value of each attribute of the record is stored only if the value of the attribute varies from the default value. A data model is then generated, utilizing the sparse representation, and the model is output. The generation of the data model in one embodiment is in accordance with the Expectation Maximization (EM) algorithm.

Recursive approach in sparse matrix LU factorization

Abstract: This paper describes a recursive method for the LU factorization of sparse matrices. The recursive formulation of common linear algebra codes has been proven very successful in dense matrix computations. An extension of the recursive technique for sparse matrices is presented. Performance results given here show that the recursive approach may perform comparable to leading software packages for sparse matrix factorization in terms of execution time, memory usage, and error estimates of the solution.

On the Enhancements of a Sparse Matrix Information Retrieval Approach.

Abstract: A novel approach to information retrieval is proposed and evaluated. By representing an inverted index as a sparse matrix, matrix-vector multiplication algorithms can be used to query the index. As many parallel sparse matrix multiplication algorithms exist, such an information retrieval approach lends itself to parallelism. This enables us to attack the problem of parallel information retrieval, which has resisted good scalability. We evaluate our proposed approach using several document collections from within the commonly used NIST TREC corpus. Our results indicate that our approach saves approximately 30% of the total storage requirements for the inverted index. Additionally, to improve accuracy, we develop a novel matrix based, relevance feedback technique as well as a proximity search algorithm.

Sparse algorithms for indefinite system of linear equations

Abstract: Sparse LDLT algorithms, based upon mixed forward-backward factorization strategies, are developed for direct solution of indefinite system of linear equations. A simple rotation matrix is also introduced and incorporated into 2×2 pivoting strategies. Several test problems have been conducted in order to evaluate the numerical performance of the proposed algorithms, and its associated FORTRAN codes.

Properties of basis functions generated by shift invariant sparse representations of natural images.

Abstract: The idea that a sparse representation is the computational principle of visual systems has been supported by Olshausen and Field [Nature (1996) 381: 607–609] and many other studies. On the other hand neurons in the inferotemporal cortex respond to moderately complex features called icon alphabets, and such neurons respond invariantly to the stimulus position. To incorporate this property into sparse representation, an algorithm is proposed that trains basis functions using sparse representations with shift invariance. Shift invariance means that basis functions are allowed to move on image data and that coefficients are equipped with shift invariance. The algorithm is applied to natural images. It is ascertained that moderately complex graphical features emerge that are not as simple as Gabor filters and not as complex as real objects. Shift invariance and moderately complex features correspond to the property of icon alphabets. The results show that there is another connection between visual information processing and sparse representations.

The exponent of discrepancy of sparse grids is at least 2:1933

Abstract: We study bounds on the exponents of sparse grids for L2‐discrepancy and average case d‐dimensional integration with respect to the Wiener sheet measure. Our main result is that the minimal exponent of sparse grids for these problems is bounded from below by 2.1933. This shows that sparse grids provide a rather poor exponent since, due to Wasilkowski and Woźniakowski [16], the minimal exponent of L2‐discrepancy of arbitrary point sets is at most 1.4778. The proof of the latter, however, is non‐constructive. The best known constructive upper bound is still obtained by a particular sparse grid and equal to 2.4526....

Initialized and guided EM-clustering of sparse binary data with application to text based documents

Abstract: We investigate an alternative way of combining classification and clustering techniques for sparse binary data in order to reduce the amount of training samples required. Initializing EM from the available labels also reduces the algorithms' known dependency on the initialization, which is more evident in the case of sparse data. In addition, the two-valued Poisson class-model is proposed in this paper as a sparse variant of the usual binomial assumption. Our method can be seen as a fusion between generalized logistic regression and parametric mixture modeling. Comparative simulation results on subsets of the 20 Newsgroups' binary coded text corpora and binary handwritten digits data demonstrate the potential usefulness of the suggested method.

A proof of the consistency of the finite difference technique on sparse grids

Abstract: In this paper, we give a proof of the consistency of the finite difference technique on regular sparse grids [7, 18]. We introduce an extrapolation-type discretization of differential operators on sparse grids based on the idea of the combination technique and we show the consistency of this discretization. The equivalence of the new method with that of [7, 18] is established.

Cryptographic Applications of Sparse Polynomials over Finite Rings

Abstract: This paper gives new examples that exploit the idea of using sparse polynomials with restricted coefficients over a finite ring for designing fast, reliable cryptosystems and identification schemes.

Color Opponency Constitutes a Sparse Representation for the Chromatic Structure of Natural Scenes

Abstract: The human visual system encodes the chromatic signals conveyed by the three types of retinal cone photoreceptors in an opponent fashion. This color opponency has been shown to constitute an efficient encoding by spectral decorrelation of the receptor signals. We analyze the spatial and chromatic structure of natural scenes by decomposing the spectral images into a set of linear basis functions such that they constitute a representation with minimal redundancy. Independent component analysis finds the basis functions that transforms the spatiochromatic data such that the outputs (activations) are statistically as independent as possible, i.e. least redundant. The resulting basis functions show strong opponency along an achromatic direction (luminance edges), along a blue-yellow direction, and along a red-blue direction. Furthermore, the resulting activations have very sparse distributions, suggesting that the use of color opponency in the human visual system achieves a highly efficient representation of colors. Our findings suggest that color opponency is a result of the properties of natural spectra and not solely a consequence of the overlapping cone spectral sensitivities.

Estimation of large-amplitude motion and disparity fields: application to intermediate view reconstruction

Abstract: This paper describes a method for establishing dense correspondence between two images in a video sequence (motion) or in a stereo pair (disparity) in case of large displacements In order to deal with large-amplitude motion or disparity fields, multi-resolution techniques such as blocks matching and optical flow have been used in the past Although quite successful, these techniques cannot easily cope with motion/disparity discontinuities as they do not explicitly exploit image structure Additionally, their computational complexity is high; block matching requires examination of numerous vector candidates while optical flow-based techniques are iterative In this paper, we propose a new approach that addresses both issues The approach combines feature matching with Delaunay triangulation, and thus reliable long-range correspondences result while the computational complexity is not high (sparse representation) In the proposed approach, feature points are found first using a simple intensity corner detector Then, correspondence pairs between two images are found by maximizing cross-correlation over a small window Finally, the Delaunay triangulation is applied to the resulting points, and a dense vector field is computed by planar interpolation over Delaunay triangles The resulting vector field is continuous everywhere, and thus does not reflect motion or depth discontinuities at object boundaries In order to improve the rendition of such discontinuities, we propose to further divide Delaunay triangles whenever the displacement vectors within a triangle do not allow good intensity match The approach has been extensively tested on stereoscopic images in the context of intermediate view reconstruction where the quality of estimated disparity fields is critical for final image rendering The first results are very encouraging as the reconstructed images are of high quality, especially at object boundaries, and the computational complexity is lower than that of multi- resolution block matching