Posted Content•
Convolutional Matching Pursuit and Dictionary Training
TL;DR: It is demonstrated that sparse coding by matching pursuit and dictionary learning via K-SVD can be used in the translation invariant setting.
Abstract: Here, {W,Z} are the dictionary and the coefficients, respectively, and zk is the kth column of Z. K, q, and λ are user selected parameters controlling the power of the model. More recently, many models with additional structure have been proposed. For example, in [9, 2], the dictionary elements are arranged in groups and the sparsity is on the group level. In [3, 5, 7], the dictionaries are constructed to be translation invariant. In the former work, the dictionary is constructed via a non-negative matrix factorization. In the latter two works, the construction is a convolutional analogue of 1.2 or an l variant, with 0 < p < 1. In this short note we work with greedy algorithms for solving the convolutional analogues of 1.1. Specifically, we demonstrate that sparse coding by matching pursuit and dictionary learning via K-SVD [1] can be used in the translation invariant setting.
Citations
More filters
TL;DR: New, efficient algorithms that substantially improve on the performance of other recent methods of sparse representation are presented, contributing to the development of this type of representation as a practical tool for a wider range of problems.
Abstract: When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but results in a representation that is multi-valued and not optimized with respect to the entire image. An alternative representation structure is provided by a convolutional sparse representation, in which a sparse representation of an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. The resulting representation is both single-valued and jointly optimized over the entire image. While this form of a sparse representation has been applied to a variety of problems in signal and image processing and computer vision, the computational expense of the corresponding optimization problems has restricted application to relatively small signals and images. This paper presents new, efficient algorithms that substantially improve on the performance of other recent methods, contributing to the development of this type of representation as a practical tool for a wider range of problems.
331 citations
Cites background from "Convolutional Matching Pursuit and ..."
...In most of these cases the primary focus of the work is not on sparse coding algorithm development, and only [18], [19], [36], [38], [42] discuss efficient convolutional extensions of these methods in any detail....
[...]
...tasks [11], [12], [16], [17], although some more recent work has specifically addressed the concept from a more traditional sparse representations perspective [18]–[21]....
[...]
...Solutions for the convolutional constrained forms of sparse coding have employed convolutional extensions of MP [18], [19], [31], [36], [38] or variants of OMP [32], [34], [42]–[44]....
[...]
...A number of different authors have considered the development of convolutional extensions of the K-SVD dictionary update [18], [19], [33], [36], [38], [43]....
[...]
...Most of these works [11], [12], [14], [16], [20], [21] have posed the sparse coding and dictionary learning problems in the form of CBPDN, the exceptions being probabilistic/Bayesian models [17], [19] and convolutional extensions [18] of MP and K-SVD [22]....
[...]
07 Jun 2015
TL;DR: The proposed method is the first efficient approach to allow for proper boundary conditions to be imposed and it also supports feature learning from incomplete data as well as general reconstruction problems.
Abstract: Convolutional sparse coding (CSC) has become an increasingly important tool in machine learning and computer vision. Image features can be learned and subsequently used for classification and reconstruction tasks. As opposed to patch-based methods, convolutional sparse coding operates on whole images, thereby seamlessly capturing the correlation between local neighborhoods. In this paper, we propose a new approach to solving CSC problems and show that our method converges significantly faster and also finds better solutions than the state of the art. In addition, the proposed method is the first efficient approach to allow for proper boundary conditions to be imposed and it also supports feature learning from incomplete data as well as general reconstruction problems.
253 citations
Cites methods from "Convolutional Matching Pursuit and ..."
...CSC was introduced in the context of modeling receptive fields in human vision [18], but it has recently been demonstrated to have important applications in a wide range of computer vision problems such as low/mid-level feature learning, lowlevel reconstruction [21, 7], as part of more complex hierarchical structures or networks in high-level computer vision challenges [13, 22, 23], and in physically-motivated computational imaging problems [12, 11]....
[...]
TL;DR: The theoretical aspects of the convolutional sparse model are addressed, providing the first meaningful answers to questions of uniqueness of solutions and success of pursuit algorithms, both greedy and convex relaxations, in ideal and noisy regimes.
Abstract: The celebrated sparse representation model has led to remarkable results in various signal processing tasks in the last decade. However, despite its initial purpose of serving as a global prior for entire signals, it has been commonly used for modeling low dimensional patches due to the computational constraints it entails when deployed with learned dictionaries. A way around this problem has been recently proposed, adopting a convolutional sparse representation model. This approach assumes that the global dictionary is a concatenation of banded circulant matrices. While several works have presented algorithmic solutions to the global pursuit problem under this new model, very few truly-effective guarantees are known for the success of such methods. In this paper, we address the theoretical aspects of the convolutional sparse model providing the first meaningful answers to questions of uniqueness of solutions and success of pursuit algorithms, both greedy and convex relaxations, in ideal and noisy regimes. To this end, we generalize mathematical quantities, such as the $\ell _0$ norm, mutual coherence, Spark and restricted isometry property to their counterparts in the convolutional setting, intrinsically capturing local measures of the global model. On the algorithmic side, we demonstrate how to solve the global pursuit problem by using simple local processing, thus offering a first of its kind bridge between global modeling of signals and their patch-based local treatment.
127 citations
TL;DR: This work represents a bridge between matrix factorization, sparse dictionary learning, and sparse autoencoders, and it is shown that the training of the filters is essential to allow for nontrivial signals in the model, and an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers.
Abstract: The recently proposed multilayer convolutional sparse coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of convolutional neural networks (CNNs). Under this framework, the forward pass in a CNN is equivalent to a pursuit algorithm aiming to estimate the nested sparse representation vectors from a given input signal. Despite having served as a pivotal connection between CNNs and sparse modeling, a deeper understanding of the ML-CSC is still lacking. In this paper, we propose a sound pursuit algorithm for the ML-CSC model by adopting a projection approach. We provide new and improved bounds on the stability of the solution of such pursuit and we analyze different practical alternatives to implement this in practice. We show that the training of the filters is essential to allow for nontrivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers. Last, but not least, we demonstrate the applicability of the ML-CSC model for several applications in an unsupervised setting, providing competitive results. Our work represents a bridge between matrix factorization, sparse dictionary learning, and sparse autoencoders, and we analyze these connections in detail.
127 citations
25 Mar 2015
TL;DR: In this article, an alternating direction method of multipliers (ADMM) framework was proposed to solve the convolutional sparse coding problem in the Fourier domain, and the theoretical computational cost was reduced from O(M 3 N) to O(MN log N, where N is the dimensionality of the data and M is the number of elements in the dictionary.
Abstract: Computationally efficient algorithms may be applied for fast dictionary learning solving the convolutional sparse coding problem in the Fourier domain. More specifically, efficient convolutional sparse coding may be derived within an alternating direction method of multipliers (ADMM) framework that utilizes fast Fourier transforms (FFT) to solve the main linear system in the frequency domain. Such algorithms may enable a significant reduction in computational cost over conventional approaches by implementing a linear solver for the most critical and computationally expensive component of the conventional iterative algorithm. The theoretical computational cost of the algorithm may be reduced from O(M 3 N) to O(MN log N), where N is the dimensionality of the data and M is the number of elements in the dictionary. This significant improvement in efficiency may greatly increase the range of problems that can practically be addressed via convolutional sparse representations.
125 citations
References
More filters
TL;DR: The authors introduce an algorithm, called matching pursuit, that decomposes any signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions, chosen in order to best match the signal structures.
Abstract: The authors introduce an algorithm, called matching pursuit, that decomposes any signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions. These waveforms are chosen in order to best match the signal structures. Matching pursuits are general procedures to compute adaptive signal representations. With a dictionary of Gabor functions a matching pursuit defines an adaptive time-frequency transform. They derive a signal energy distribution in the time-frequency plane, which does not include interference terms, unlike Wigner and Cohen class distributions. A matching pursuit isolates the signal structures that are coherent with respect to a given dictionary. An application to pattern extraction from noisy signals is described. They compare a matching pursuit decomposition with a signal expansion over an optimized wavepacket orthonormal basis, selected with the algorithm of Coifman and Wickerhauser see (IEEE Trans. Informat. Theory, vol. 38, Mar. 1992). >
9,380 citations
"Convolutional Matching Pursuit and ..." refers methods in this paper
...2 Matching Pursuit Matching pursuit [6] is a greedy algorithm for the solution of the sparse coding problem min z ||Wz − x||(2), ||z||0 ≤ q, where the d× k matrix W is the dictionary, the k× 1 z is the code, and x is an d× 1 data vector....
[...]
TL;DR: A novel algorithm for adapting dictionaries in order to achieve sparse signal representations, the K-SVD algorithm, an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data.
Abstract: In recent years there has been a growing interest in the study of sparse representation of signals. Using an overcomplete dictionary that contains prototype signal-atoms, signals are described by sparse linear combinations of these atoms. Applications that use sparse representation are many and include compression, regularization in inverse problems, feature extraction, and more. Recent activity in this field has concentrated mainly on the study of pursuit algorithms that decompose signals with respect to a given dictionary. Designing dictionaries to better fit the above model can be done by either selecting one from a prespecified set of linear transforms or adapting the dictionary to a set of training signals. Both of these techniques have been considered, but this topic is largely still open. In this paper we propose a novel algorithm for adapting dictionaries in order to achieve sparse signal representations. Given a set of training signals, we seek the dictionary that leads to the best representation for each member in this set, under strict sparsity constraints. We present a new method-the K-SVD algorithm-generalizing the K-means clustering process. K-SVD is an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data. The update of the dictionary columns is combined with an update of the sparse representations, thereby accelerating convergence. The K-SVD algorithm is flexible and can work with any pursuit method (e.g., basis pursuit, FOCUSS, or matching pursuit). We analyze this algorithm and demonstrate its results both on synthetic tests and in applications on real image data
8,905 citations
"Convolutional Matching Pursuit and ..." refers methods in this paper
...ers Given a set of x, we can learn the filters and the codes simultaneously. Several methods are available. A simple one is to alternate between updating the codes and updating the filters, as in K-SVD [1]: 1. Initialize k h f ×w f filters {w1,...,w k}. 2. Solve for z as above. 2 3. For each filter w j, •find all locations in all the data images where w j is activated •extract the h f ×w f patch E p from ...
[...]
... In this short note we work with greedy algorithms for solving the convolutional analogues of 1.1. Specifically, we demonstrate that sparse coding by matching pursuit and dictionary learning via K-SVD [1] can be used in the translation invariant setting. 2 Matching Pursuit Matching pursuit [6] is a greedy algorithm for the solution of the sparse coding problem min z ||Wz −x||2, ||z||0 ≤q, where the d×...
[...]
TL;DR: In this paper, instead of selecting factors by stepwise backward elimination, the authors focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection.
Abstract: Summary. We consider the problem of selecting grouped variables (factors) for accurate prediction in regression. Such a problem arises naturally in many practical situations with the multifactor analysis-of-variance problem as the most important and well-known example. Instead of selecting factors by stepwise backward elimination, we focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection. The lasso, the LARS algorithm and the non-negative garrotte are recently proposed regression methods that can be used to select individual variables. We study and propose efficient algorithms for the extensions of these methods for factor selection and show that these extensions give superior performance to the traditional stepwise backward elimination method in factor selection problems. We study the similarities and the differences between these methods. Simulations and real examples are used to illustrate the methods.
7,400 citations
"Convolutional Matching Pursuit and ..." refers background in this paper
...For example, in [9, 2], the dictionary elements are arranged in groups and the sparsity is on the group level....
[...]
TL;DR: These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.
3,840 citations
"Convolutional Matching Pursuit and ..." refers background in this paper
...1 Introduction One of the most succesful recent signal processing paradigms has been the sparse coding/dictionary design model [8, 4]....
[...]
TL;DR: The aim of this paper is to introduce a few key notions and applications connected to sparsity, targeting newcomers interested in either the mathematical aspects of this area or its applications.
Abstract: A full-rank matrix ${\bf A}\in \mathbb{R}^{n\times m}$ with $n
2,372 citations
"Convolutional Matching Pursuit and ..." refers background in this paper
...1 Introduction One of the most succesful recent signal processing paradigms has been the sparse coding/dictionary design model [8, 4]....
[...]