The following problem is considered: given a matrix $A$ in ${\bf R}^{m \times n}$, ($m$ rows and $n$ columns), a vector $b$ in ${\bf R}^m$, and ${\bf \epsilon} > 0$, compute a vector $x$ satisfying $\| Ax - b \|_2 \leq {\bf \epsilon}$ if such exists, such that $x$ has the fewest number of non-zero entries over all such vectors. It is shown that the problem is NP-hard, but that the well-known greedy heuristic is good in that it computes a solution with at most $\left\lceil 18 \mbox{ Opt} ({\bf \epsilon}/2) \|{\bf A}^+\|^2_2 \ln(\|b\|_2/{\bf \epsilon}) \right\rceil$ non-zero entries, where $\mbox{Opt}({\bf \epsilon}/2)$ is the optimum number of nonzero entries at error ${\bf \epsilon}/2$, ${\bf A}$ is the matrix obtained by normalizing each column of $A$ with respect to the $L_2$ norm, and ${\bf A}^+$ is its pseudo-inverse.

Sparse Approximate Solutions to Linear Systems

Sparse Bayesian learning (SBL) and specifically relevance vector machines have received much attention in the machine learning literature as a means of achieving parsimonious representations in the context of regression and classification. The methodology relies on a parameterized prior that encourages models with few nonzero weights. In this paper, we adapt SBL to the signal processing problem of basis selection from overcomplete dictionaries, proving several results about the SBL cost function that elucidate its general behavior and provide solid theoretical justification for this application. Specifically, we have shown that SBL retains a desirable property of the /spl lscr//sub 0/-norm diversity measure (i.e., the global minimum is achieved at the maximally sparse solution) while often possessing a more limited constellation of local minima. We have also demonstrated that the local minima that do exist are achieved at sparse solutions. Later, we provide a novel interpretation of SBL that gives us valuable insight into why it is successful in producing sparse representations. Finally, we include simulation studies comparing sparse Bayesian learning with basis pursuit and the more recent FOCal Underdetermined System Solver (FOCUSS) class of basis selection algorithms. These results indicate that our theoretical insights translate directly into improved performance.

/pdf/sparse-bayesian-learning-for-basis-selection-191qhmhsdw.pdf

Sparse Bayesian learning for basis selection

The accuracy of many schemes for interpolating scattered data with radial basis functions depends on a shape parameter c of the radial basis function. In this paper we study the effect of c on the quality of fit of the multiquadric, inverse multiquadric and Gaussian interpolants. We show, numerically, that the value of the optimal c (the value of c that minimizes the interpolation error) depends on the number and distribution of data points, on the data vector, and on the precision of the computation. We present an algorithm for selecting a good value for c that implicitly takes all the above considerations into account. The algorithm selects c by minimizing a cost function that imitates the error between the radial interpolant and the (unknown) function from which the data vector was sampled. The cost function is defined by taking some norm of the error vector E = (E 1, ... , EN)T where E k = Ek = fk - Sk xk) and S k is the interpolant to a reduced data set obtained by removing the point x k and the corresponding data value f k from the original data set. The cost function can be defined for any radial basis function and any dimension. We present the results of many numerical experiments involving interpolation of two dimensional data sets by the multiquadric, inverse multiquadric and Gaussian interpolants and we show that our algorithm consistently produces good values for the parameter c.

An algorithm for selecting a good value for the parameter c in radial basis function interpolation

A methodology is developed to derive algorithms for optimal basis selection by minimizing diversity measures proposed by Wickerhauser (1994) and Donoho (1994). These measures include the p-norm-like (l/sub (p/spl les/1)/) diversity measures and the Gaussian and Shannon entropies. The algorithm development methodology uses a factored representation for the gradient and involves successive relaxation of the Lagrangian necessary condition. This yields algorithms that are intimately related to the affine scaling transformation (AST) based methods commonly employed by the interior point approach to nonlinear optimization. The algorithms minimizing the (l/sub (p/spl les/1)/) diversity measures are equivalent to a previously developed class of algorithms called focal underdetermined system solver (FOCUSS). The general nature of the methodology provides a systematic approach for deriving this class of algorithms and a natural mechanism for extending them. It also facilitates a better understanding of the convergence behavior and a strengthening of the convergence results. The Gaussian entropy minimization algorithm is shown to be equivalent to a well-behaved p=0 norm-like optimization algorithm. Computer experiments demonstrate that the p-norm-like and the Gaussian entropy algorithms perform well, converging to sparse solutions. The Shannon entropy algorithm produces solutions that are concentrated but are shown to not converge to a fully sparse solution.

/pdf/an-affine-scaling-methodology-for-best-basis-selection-50hjmzu7gm.pdf

An affine scaling methodology for best basis selection

Many radial basis function (RBF) methods contain a free shape parameter that plays an important role for the accuracy of the method. In most papers the authors end up choosing this shape parameter by trial and error or some other ad hoc means. The method of cross validation has long been used in the statistics literature, and the special case of leave-one-out cross validation forms the basis of the algorithm for choosing an optimal value of the shape parameter proposed by Rippa in the setting of scattered data interpolation with RBFs. We discuss extensions of this approach that can be applied in the setting of iterated approximate moving least squares approximation of function value data and for RBF pseudo-spectral methods for the solution of partial differential equations. The former method can be viewed as an efficient alternative to ridge regression or smoothing spline approximation, while the latter forms an extension of the classical polynomial pseudo-spectral approach. Numerical experiments illustrating the use of our algorithms are included.

/pdf/on-choosing-optimal-shape-parameters-for-rbf-approximation-4673pg6gmn.pdf

On choosing “optimal” shape parameters for RBF approximation

Given N scattered data points, we examined the problem of finding N variable Multiquadric (MQ) shape-parameters, or R2 values. Because the problem of finding the optimal R2 values is a nonlinear one, we optimized these parameters numerically by minimizing the root-mean-square (RMS) errors. The resulting R2 values varied over many orders of magnitude. We have tested this approach on a number of univariate and bivariate (Franke's) problems, and found that the RMS error reduction was substantial.

Improved accuracy of multiquadric interpolation using variable shape parameters

The multiquadric and thin plate spline radial basis methods, together with the triangle-based minimum norm network algorithm and the modified quadratic Shepards' method, are applied to various sets of data that are sampled densely along tracks in the plane. The effectiveness of these methods on track data has been questioned in the past. We observe that both radial basis methods and the minimum norm network method performed well on smoothly varying track data sets, while the multiquadric method with a small value for the parameter R 2 was the only method that was effective on rapidly varying track data.

Interpolation of track data with radial basis methods

Recent developments in the theory and applications of radial basis functions are summarized. One intriguing property of such approximation methods is that rates of convergence improve with increasing dimensions. Also, such approximations, under favorable circumstances, can have exponential convergence. Numerical applications, especially on sparse scattered data sets of high dimensions, validate these theoretical claims.

Radial basis functions: A class of grid-free, scattered data approximations

Multiquadric interpolation is a technique for interpolating nonuniform samples of multivariate functions, in order to enable a variety of operations such as data visualization. We are interested in computing sparse but approximate interpolants, i.e., approximate interpolants with few coefficients. Such interpolants are useful since (1) the cost of evaluating the interpolant scales directly with the number of nonzero coefficients, and (2) the principle of Occam's Razor 2 suggests that the interpolant with fewer coefficients better approximates the underlying function. Since the number of coefficients in a multiquadric interpolant is, as is to be expected, equal to the number of data points in the given set, the problem can be abstracted thus: given a set S of samples of a function f : R k → R , and an error tolerance δ, find the smallest set of points T ⊆ S such that the multiquadric interpolant of T is within δ of f over S . Using some recent results on sparse solutions of linear systems, we show how T may be selected in a provably good fashion.

R.E. Carlson

Papers

Improved accuracy of multiquadric interpolation using variable shape parameters

Interpolation of track data with radial basis methods

Radial basis functions: A class of grid-free, scattered data approximations

Sparse approximate multiquadric interpolation