Showing papers in "Constructive Approximation in 2007"

On Early Stopping in Gradient Descent Learning

Abstract: In this paper we study a family of gradient descent algorithms to approximate the regression function from reproducing kernel Hilbert spaces (RKHSs), the family being characterized by a polynomial decreasing rate of step sizes (or learning rate) By solving a bias-variance trade-off we obtain an early stopping rule and some probabilistic upper bounds for the convergence of the algorithms We also discuss the implication of these results in the context of classification where some fast convergence rates can be achieved for plug-in classifiers Some connections are addressed with Boosting, Landweber iterations, and the online learning algorithms as stochastic approximations of the gradient descent method

Learning Theory Estimates via Integral Operators and Their Approximations

Abstract: The regression problem in learning theory is investigated with least square Tikhonov regularization schemes in reproducing kernel Hilbert spaces (RKHS). We follow our previous work and apply the sampling operator to the error analysis in both the RKHS norm and the L2 norm. The tool for estimating the sample error is a Bennet inequality for random variables with values in Hilbert spaces. By taking the Hilbert space to be the one consisting of Hilbert-Schmidt operators in the RKHS, we improve the error bounds in the L2 metric, motivated by an idea of Caponnetto and de Vito. The error bounds we derive in the RKHS norm, together with a Tsybakov function we discuss here, yield interesting applications to the error analysis of the (binary) classification problem, since the RKHS metric controls the one for the uniform convergence.

How to Compare Different Loss Functions and Their Risks

Abstract: Many learning problems are described by a risk functional which in turn is defined by a loss function, and a straightforward and widely known approach to learn such problems is to minimize a (modified) empirical version of this risk functional. However, in many cases this approach suffers from substantial problems such as computational requirements in classification or robustness concerns in regression. In order to resolve these issues many successful learning algorithms try to minimize a (modified) empirical risk of a surrogate loss function, instead. Of course, such a surrogate loss must be "reasonably related" to the original loss function since otherwise this approach cannot work well. For classification good surrogate loss functions have been recently identified, and the relationship between the excess classification risk and the excess risk of these surrogate loss functions has been exactly described. However, beyond the classification problem little is known on good surrogate loss functions up to now. In this work we establish a general theory that provides powerful tools for comparing excess risks of different loss functions. We then apply this theory to several learning problems including (cost-sensitive) classification, regression, density estimation, and density level detection.

Strong Asymptotics of Laguerre-Type Orthogonal Polynomials and Applications in Random Matrix Theory

Abstract: We consider polynomials orthogonal on [0,∞) with respect to Laguerre-type weights w(x) = xα e-Q(x), where α > -1 and where Q denotes a polynomial with positive leading coefficient. The main purpose of this paper is to determine Plancherel-Rotach-type asymptotics in the entire complex plane for the orthonormal polynomials with respect to w, as well as asymptotics of the corresponding recurrence coefficients and of the leading coefficients of the orthonormal polynomials. As an application we will use these asymptotics to prove universality results in random matrix theory. We will prove our results by using the characterization of orthogonal polynomials via a 2 × 2 matrix valued Riemann--Hilbert problem, due to Fokas, Its, and Kitaev, together with an application of the Deift-Zhou steepest descent method to analyze the Riemann-Hilbert problem asymptotically.

Matrix Valued Orthogonal Polynomials Arising from the Complex Projective Space

Abstract: The main purpose of this paper is to display new families of matrix valued orthogonal polynomials satisfying second-order differential equations, obtained from the representation theory of U(n). Given an arbitrary positive definite weight matrix W(t) one can consider the corresponding matrix valued orthogonal polynomials. These polynomials will be eigenfunctions of some symmetric second-order differential operator D only for very special choices of W(t). Starting from the theory of spherical functions associated to the pair (SU(n+1), U(n)) we obtain new families of such pairs {W,D}. These depend on enough integer parameters to obtain an immediate extension beyond these cases.

Numerical Integration over Spheres of Arbitrary Dimension

Abstract: In this paper, we study the worst-case error (of numerical integration) on the unit sphere $\\mathbb{S}^{d}$, $d\\geq 2$, for all functions in the unit ball of the Sobolev space $\\mathbb{H}^s(\\mathbb{S}^d)$, where $s>d/2$. More precisely, we consider infinite sequences $(Q_{m(n)})_{n\\in\\mathbb{N}}$ of $m(n)$-point numerical integration rules $Q_{m(n)}$, where (i) $Q_{m(n)}$ is exact for all spherical polynomials of degree $\\leq n$, and (ii) $Q_{m(n)}$ has positive weights or, alternatively to (ii), the sequence $(Q_{m(n)})_{n\\in\\mathbb{N}}$ satisfies a certain local regularity property. Then we show that the worst-case error (of numerical integration) $E(Q_{m(n)};\\mathbb{H}^s(\\matbb{S}^d))$ in $\\mathbb{H}^s(\\mathbb{S}^d)$ has the upper bound $c n^{-s}$, where the constant $c$ depends on $s$ and $d$ (and possibly the sequence $(Q_{m(n)})_{n\\in\\mathbb{N}}$). This extends the recent results for the sphere $\\mathbb{S}^2$ by K.Hesse and I.H.Sloan to spheres $\\mathbb{S}^d$ of arbitrary dimension $d\\geq2$ by using an alternative representation of the worst-case error. If the sequence $(Q_{m(n)})_{n\\in\\mathbb{N}}$ of numerical integration rules satisfies $m(n)=\\mathcal{O}(n^d)$ an order-optimal rate of convergence is achieved.

Universal Algorithms for Learning Theory. Part II: Piecewise Polynomial Functions

Abstract: This paper is concerned with estimating the regression function fρ in supervised learning by utilizing piecewise polynomial approximations on adaptively generated partitions. The main point of interest is algorithms that with high probability are optimal in terms of the least square error achieved for a given number m of observed data. In a previous paper [1], we have developed for each β > 0 an algorithm for piecewise constant approximation which is proven to provide such optimal order estimates with probability larger than 1- m-β. In this paper we consider the case of higher-degree polynomials. We show that for general probability measures ρ empirical least squares minimization will not provide optimal error estimates with high probability. We go further in identifying certain conditions on the probability measure ρ which will allow optimal estimates with high probability.

Monotonicity Properties of Determinants of Special Functions

Abstract: We prove the absolute monotonicity or complete monotonicity of some determinant functions whose entries involve $$\psi^{(m)}(x)=({d^m}/{dx^m}) [\Gamma'(x)/\Gamma(x)],$$ modified Bessel functions Iν, Kν, the confluent hypergeometric function Φ, and the Tricomi function Ψ Our results recover and generalize some known determinantal inequalities We also show that a certain determinant formed by the Fibonacci numbers are nonnegative while determinants involving Hermite polynomials of imaginary arguments are shown to be completely monotonic functions

Biorthogonal Laurent polynomials, Toplitz determinants, minimal Toda orbits and isomonodromic tau functions

Abstract: We consider the class of biorthogonal polynomials that are used to solve the inverse spectral problem associated to elementary co-adjoint orbits of the Borel group of upper triangular matrices; these orbits are the phase space of generalized integrable lattices of Toda type. Such polynomials naturally interpolate between the theory of orthogonal polynomials on the line and orthogonal polynomials on the unit circle and tie together the theory of Toda, relativistic Toda, Ablowitz-Ladik and Volterra lattices. We establish corresponding Christoffel-Darboux formulae. For all these classes of polynomials a 2 × 2 system of Differential-Difference-Deformation equations is analyzed in the most general setting of pseudo-measures with arbitrary rational logarithmic derivative. They provide particular classes of isomonodromic deformations of rational connections on the Riemann sphere. The corresponding isomonodromic tau function is explicitly related to the shifted Toplitz determinants of the moments of the pseudo-measure. In particular, the results imply that any (shifted) Toplitz (Hankel) determinant of a symbol (measure) with arbitrary rational logarithmic derivative is an isomonodromic tau function.

Coefficients of Orthogonal Polynomials on the Unit Circle and Higher-Order Szego Theorems

Abstract: Let \(\mu\) be a nontrivial probability measure on the unit circle \(\partial\mbox{\bf D},\ w\) the density of its absolutely continuous part, \(\alpha_n\) its Verblunsky coefficients, and \(\Phi_n\) its monic orthogonal polynomials. In this paper we compute the coefficients of \(\Phi_n\) in terms of the \(\alpha_n\). If the function \(\log w\) is in \(L^1(d\theta)\), we do the same for its Fourier coefficients. As an application we prove that if \(\alpha_n\in\ell^4\) and if \(Q(z) \equiv\sum_{m=0}^N q_m z^m\) is a polynomial, then with \(\bar Q(z) \equiv\sum_{m=0}^N \bar q_m z^m\) and S the left-shift operator on sequences we have $$| Q(e^{i\theta}) |^2\log w(\theta) \in L^1(d\theta) \quad \Leftrightarrow \quad \{\bar Q(S)\alpha\}_n\in\ell^2.$$ We also study relative ratio asymptotics of the reversed polynomials \(\Phi_{n+1}^*(\mu)/\Phi_n^*(\mu)-\Phi_{n+1}^*( u)/\Phi_n^*( u)\) and provide a necessary and sufficient condition in terms of the Verblunsky coefficients of the measures \(\mu\) and \( u\) for this difference to converge to zero uniformly on compact subsets of \(\mbox{\bf D}\).

On the Bernstein Constants of Polynomial Approximation

Abstract: Assume \(\alpha >0\) is not an integer. In papers published in 1913 and 1938, S.~N.~Bernstein established the limit $$\Lambda _{\infty ,\alpha }^{\ast }=\lim_{n\rightarrow \infty }n^{\alpha}E_{n}[ \vert x\vert ^{\alpha };L_{\infty }[ {-}1,1]] .$$ Here \(E_{n}[ \vert x\vert ^{\alpha };L_{\infty }[ -1,1] ] \) denotes the error in best uniform approximation of \(\left\vert x\right\vert ^{\alpha }$ on $\left[ {-}1,1\right] \) by polynomials of degree \(\leq n\). Bernstein proved that \(\Lambda _{\infty ,\alpha }^{\ast} \) is itself the error in best uniform approximation of \(\left\vert x\right\vert ^{\alpha }\) by entire functions of exponential type at most 1, on the whole real line. We prove that the best approximating entire function is unique, and satisfies an alternation property. We show that the scaled polynomials of best approximation converge to this unique entire function. We derive a representation for \(\Lambda _{\alpha ,\infty }^{\ast }\), as well as its \(L_{p}\) analogue for \(1\leq p<\infty \).

The Entropy in Learning Theory. Error Estimates

Abstract: We continue the investigation of some problems in learning theory in the setting formulated by F. Cucker and S. Smale. The goal is to find an estimator \(f_{\bf z}\) on the base of given data \(\mbox{\footnotesize\bf z}:=((x_1,y_1),\dots,(x_m,y_m))\) that approximates well the regression function \(f_\rho\) of an unknown Borel probability measure \(\rho\) defined on \(Z=X\times Y.\) We assume that \(f_\rho\) belongs to a function class \(\Theta.\) It is known from previous works that the behavior of the entropy numbers \(\epsilon_n(\Theta,{\cal C})\) of \(\Theta\) in the uniform norm \({\cal C}\) plays an important role in the above problem. The standard way of measuring the error between a target function \(f_\rho\) and an estimator \(f_{\bf z}\) is to use the \(L_2(\rho_X)\) norm (\(\rho_X\) is the marginal probability measure on X generated by \(\rho\)). This method has been used in previous papers. We continue to use this method in this paper. The use of the \(L_2(\rho_X)\) norm in measuring the error has motivated us to study the case when we make an assumption on the entropy numbers \(\epsilon_n(\Theta,L_2(\rho_X))\) of \(\Theta\) in the \(L_2(\rho_X)\) norm. This is the main new ingredient of thispaper. We construct good estimators in different settings: (1) we know both \(\Theta\) and \(\rho_X\); (2) we know \(\Theta\) but we do not know \(\rho_X;\) and (3) we only know that \(\Theta\) is from a known collection of classes but we do not know \(\rho_X.\) An estimator from the third setting is called a universal estimator.

Approximate Maximum a Posteriori with Gaussian Process Priors

Abstract: A maximum a posteriori method has been developed for Gaussian priors over infinite-dimensional function spaces. In particular, variational equations based on a generalisation of the representer theorem and an equivalent optimisation problem are presented. This amounts to a generalisation of the ordinary Bayesian maximum a posteriori approach which is nontrivial as infinite-dimensional domains do not admit any probability densities. Instead of the gradient of the density, the logarithmic gradient of the probability distribution is used. Galerkin methods are proposed for the approximate solution of the variational equations. In summary, a framework and some foundations are provided which are required for the application of numerical approximation to an important class of machine learning problems.

Approximation by Homogeneous Polynomials

Abstract: Let \(K\subset\mathbb{R}^d\) be the boundary of a convex domain symmetric to the origin. The conjecture that any continuous even function can be uniformly approximated by homogeneous polynomials of even degree on K is proven in the following cases: (a) if d = 2; (b) if K is twice continuously differentiable and has positive curvature in every point; or (c) if K is the boundary of a convex polytope.

General Results on the Convergence of Multipoint Hermite-Pade Approximants of Nikishin Systems

Abstract: We consider simultaneous approximation of Nikishin systems of functions by means of rational vector functions which are constructed interpolating along a prescribed table of points. We give general conditions for the uniform convergence of such approximants with a geometric rate under very weak assumptions.

Fourier–Padé Approximants for Angelesco Systems

Abstract: We study linear and nonlinear simultaneous Fourier-Pade approximation for Angelesco systems of functions and give the exact rate of convergence/divergence of the approximants in terms of the solution of associated vector equilibrium potential problems which differ for the linear and nonlinear cases.

Affine Density, Frame Bounds, and the Admissibility Condition for Wavelet Frames

Abstract: For a large class of irregular wavelet frames we derive a fundamental relationship between the affine density of the set of indices, the frame bounds, and the admissibility constant of the wavelet. Several implications of this theorem are studied. For instance, this result reveals one reason why wavelet systems do not display a Nyquist phenomenon analogous to Gabor systems, a question asked in Daubechies' Ten Lectures book. It also implies that the affine density of the set of indices associated with a tight wavelet frame has to be uniform. Finally, we show that affine density conditions can even be used to characterize the existence of wavelet frames, thus serving, in particular, as sufficient conditions.

Approximations of Set-Valued Functions by Metric Linear Operators

Abstract: In this paper we introduce new approximation operators for univariate set-valued functions with general compact images in Rn. We adapt linear approximation methods for real-valued functions by replacing linear combinations of numbers with new metric linear combinations of finite sequences of compact sets, thus obtaining "metric analogues" of these operators for set-valued functions. The new metric linear combination extends the binary metric average of Artstein to several sets and admits any real coefficients. Approximation estimates for the metric analogue operators are derived. As examples we study metric Bernstein operators, metric Schoenberg operators, and metric polynomial interpolants.

Cesaro Summability and Marchaud Inequality

Abstract: For Fourier expansions it will be shown that sharp Marchaud-type inequalities follow from the positivity of the Cesaro summability of some order. Some results of sharp Marchaud-type will be derived using this method when other known methods cannot be used.

Asymptotic Representation of Lp-Minimal Polynomials, 1 < p < ∞

Abstract: While the theory of asymptotics for L2-minimal polynomials is highly developed, so far not much is known about Lp-minimal polynomials, \(p\in (1,\infty) \backslash \{2\}.\) Indeed, Bernstein gave asymptotics for the minimum deviation, Fekete and Walsh gave nth root asymptotics and, recently, Lubinsky and Saff came up with asymptotics outside the support [-1,1]. But the main point of interest, the asymptotic representation on the support, still remains open. Here we settle it for weight functions of the form \(w(x)/\sqrt{1-x^2},\) where w is positive and \(w' \in {\rm Lip}\,\alpha\) on [-1,1] with \(\alpha \in (0,1)\ {\rm if}\ p\geq 2\) and \(\alpha > (2/p) - 1\ {\rm if}\ 1

Remainder Pade Approximants for the Exponential Function

Abstract: Following our earlier research, we propose a new method for obtaining the complete Pade table of the exponential function. It is based on an explicit construction of certain Pade approximants, not for the usual power series for exp at 0 but for a formal power series related in a simple way to the remainder term of the power series for exp. This surprising and nontrivial coincidence is proved more generally for type II simultaneous Pade approximants for a family \((\exp(a_jz))_{j=1,\ldots, r}\) with distinct complex a's and we recover Hermite's classical formulas. The proof uses certain discrete multiple orthogonal polynomials recently introduced by Arvesu, Coussement, and van Assche, which generalize the classical Charlier orthogonal polynomials.

Maximal Cluster Sets of L-Analytic Functions Along Arbitrary Curves

Abstract: Let Ώ be a domain in the N-dimensional real space, let L be an elliptic differential operator, and let (Tn) be a sequence whose members belong to a certain class of operators defined on the space of L-analytic functions on Ώ. This paper establishes the existence of a dense linear manifold of L-analytic functions all of whose nonzero members have maximal cluster sets under the action of every Tn along any curve ending at the boundary of Ώ such that its closure does not contain any component of the boundary. The above class contains all partial differentiation operators ∂α, hence the statement extends earlier results due to Boivin, Gauthier, and Paramonov, and due to the first, third, and fourth authors.

Remez-Type Inequalities in the Complex Plane

Abstract: We obtain a sharp upper bound on the modulus of a complex polynomial p(z) at any boundary point of a domain $G\subset\mbox{\bf C}$ bounded by the Jordan curve without cusps. The area of the subset of G, where $|p(z)|\le 1,$ is known.

Trivariate Cr Polynomial Macroelements

Abstract: Trivariate Cr macroelements defined in terms of polynomials of degree 8r + 1 on tetrahedra are analyzed. For r = 1,2, these spaces reduce to well-known macroelement spaces used in data fitting and in the finite-element method. We determine the dimension of these spaces, and describe stable local minimal determining sets and nodal minimal determining sets. We also show that the spaces approximate smooth functions to optimal order.

Thresholding in Learning Theory

Abstract: In this paper we investigate the problem of learning an unknown bounded function. We will emphasize special cases where it is possible to provide very simple (in terms of computation) estimates enjoying, in addition, the property of being universal, i.e. their construction does not depend on the a priori knowledge of regularity conditions on the unknown object and still they have almost optimal properties for a whole group of functions spaces. These estimates are constructed using a thresholding technique, which has proven in the last decade in statistics to have very good properties for recovering signals with inhomogeneous smoothness but has not been extensively developed in learning theory. We will basically consider two particular situations. In the first case, we consider the RKHS situation, where we produce a new algorithm and investigate its performances in \(L_2(\hat\rho_X)\). The exponential rates of convergences are proved to be almost optimal, and the regularity assumptions are expressed in simple terms. The second case considers a more specified situation where the Xi's are one-dimensional and the estimator is a wavelet thresholding estimate. The results are comparable in this setting to those obtained in the RKHS situation, as concerned the critical value and the exponential rates. The advantage here is that we are able to state the results in the \(L_2(\rho_X)\)-norm and the regularity conditions are expressed in terms of standard Holder spaces.

Density of the Set of Generators of Wavelet Systems

Abstract: Given a function ψ in \({\cal L}^2({\Bbb R}^d),\) the affine (wavelet) system generated by ψ, associated to an invertible matrix a and a lattice Γ, is the collection of functions \(\{|{\rm det}\, a|^{j/2} \psi(a^jx-\gamma): j \in {\Bbb Z}, \gamma \in {\Gamma}\}.\) In this paper we prove that the set of functions generating affine systems that are a Riesz basis of \({\cal L}^2({\Bbb R}^d).\)${\cal L}^2({\Bbb R}^d)$ is dense in We also prove that a stronger result is true for affine systems that are a frame of \({\cal L}^2({\Bbb R}^d).\) In this case we show that the generators associated to a fixed but arbitrary dilation are a dense set. Furthermore, we analyze the orthogonal case in which we prove that the set of generators of orthogonal (not necessarily complete) affine systems, that are compactly supported in frequency, are dense in the unit sphere of \({\cal L}^2({\Bbb R}^d)\) with the induced metric. As a byproduct we introduce the p-Grammian of a function and prove a convergence result of this Grammian as a function of the lattice. This result gives insight in the problem of oversampling of affine systems.

Multi-Box Splines

Abstract: We construct local generators, comprising r functions, for refinable spaces of bivariate Cn-1 spline functions of degree n on meshes comprising all lines through points of the integer lattice in the directions of n + r + 1 pairwise linearly independent vectors with integer components. The generators are characterised by their Fourier transforms. Their shifts are shown to form a Riesz basis if and only if at most r lines in the mesh intersect other than in the integer lattice, which can occur for n ≤ 2r - 1. The symmetry of these generators is studied and examples are given.

Some Representations of Translations of the Product of Two Functions for Hankel Transforms and Jacobi Transforms

Abstract: By introducing a series of operators Ty(m), m = 0,1,. . . , a representation of the generalized translation of the product of two functions associated to Hankel transforms is given. These operators have certain properties of finite differences. The analogous problem related to Jacobi transforms is also studied.

Perturbations of Roots under Linear Transformations of Polynomials

Abstract: Let \({\cal P}_n\) be the complex vector space of all polynomials of degree at most n. We give several characterizations of the linear operators \(T:{\cal P}_n\rightarrow{\cal P}_n\) for which there exists a constant C > 0 such that for all nonconstant \(f\in{\cal P}_n\) there exist a root u of f and a root v of Tf with \(|u-v|\leq C\). We prove that such perturbations leave the degree unchanged and, for a suitable pairing of the roots of f and Tf, the roots are never displaced by more than a uniform constant independent on f. We show that such "good" operators T are exactly the invertible elements of the commutative algebra generated by the differentiation operator. We provide upper bounds in terms of T for the relevant constants.

On the Kolmogorov Problem for the Upper Bounds of Four Consecutive Derivatives of a Multiply Monotone Function

Abstract: In this paper we shall give necessary and sufficient conditions for the system of positive numbers \(M_{k_0}, M_{k_1}, M_{k_2}, M_{k_3}, 0=k_0