There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Why is Nonparametric Regression Important? * How to Construct Nonparametric Regression Estimates * Lower Bounds * Partitioning Estimates * Kernel Estimates * k-NN Estimates * Splitting the Sample * Cross Validation * Uniform Laws of Large Numbers * Least Squares Estimates I: Consistency * Least Squares Estimates II: Rate of Convergence * Least Squares Estimates III: Complexity Regularization * Consistency of Data-Dependent Partitioning Estimates * Univariate Least Squares Spline Estimates * Multivariate Least Squares Spline Estimates * Neural Networks Estimates * Radial Basis Function Networks * Orthogonal Series Estimates * Advanced Techniques from Empirical Process Theory * Penalized Least Squares Estimates I: Consistency * Penalized Least Squares Estimates II: Rate of Convergence * Dimension Reduction Techniques * Strong Consistency of Local Averaging Estimates * Semi-Recursive Estimates * Recursive Estimates * Censored Observations * Dependent Observations

http://www.gbv.de/dms/hebis-darmstadt/toc/106925350.pdf

A Distribution-Free Theory of Nonparametric Regression

In this survey we discuss various approximation-theoretic problems that arise in the multilayer feedforward perceptron (MLP) model in neural networks. The MLP model is one of the more popular and practical of the many neural network models. Mathematically it is also one of the simpler models. Nonetheless the mathematics of this model is not well understood, and many of these problems are approximation-theoretic in character. Most of the research we will discuss is of very recent vintage. We will report on what has been done and on various unanswered questions. We will not be presenting practical (algorithmic) methods. We will, however, be exploring the capabilities and limitations of this model.

Approximation theory of the MLP model in neural networks

Preface to the second edition Preface to the first edition 1. Background 2. Orthogonal polynomials in two variables 3. General properties of orthogonal polynomials in several variables 4. Orthogonal polynomials on the unit sphere 5. Examples of orthogonal polynomials in several variables 6. Root systems and Coxeter groups 7. Spherical harmonics associated with reflection groups 8. Generalized classical orthogonal polynomials 9. Summability of orthogonal expansions 10. Orthogonal polynomials associated with symmetric groups 11. Orthogonal polynomials associated with octahedral groups and applications References Author index Symbol index Subject index.

/pdf/orthogonal-polynomials-of-several-variables-2cwkynqk1v.pdf

Orthogonal Polynomials of Several Variables

Zeros of orthogonal polynomials

We compute upper and lower bounds on the VC dimension and pseudodimension of feedforward neural networks composed of piecewise polynomial activation functions. We show that if the number of layers is fixed, then the VC dimension and pseudo-dimension grow as W log W, where W is the number of parameters in the network. This result stands in opposition to the case where the number of layers is unbounded, in which case the VC dimension and pseudo-dimension grow as W2 . We combine our results with recently established approximation error rates and determine error bounds for the problem of regression estimation by piecewise polynomial networks with unbounded weights.

/pdf/almost-linear-vc-dimension-bounds-for-piecewise-polynomial-4sel5kr3zh.pdf

Almost Linear VC Dimension Bounds for Piecewise Polynomial Networks

Lower bounds for approximation by MLP neural networks

On Best Approximation by Ridge Functions

Approximation by neural networks and learning theory

The pseudo-dimension of a real-valued function class is an extension of the VC dimension for set-indicator function classes. A class \( \cal H\) of finite pseudo-dimension possesses a useful statistical smoothness property. In [10] we introduced a nonlinear approximation width \( \rho_n({\cal F}, L_q) \) = \( inf_{{\cal H}^n} \mbox{dist}({\cal F}, {\cal H}^n, L_q) \) which measures the worst-case approximation error over all functions \( f\in {\cal F} \) by the best manifold of pseudo-dimension n . In this paper we obtain tight upper and lower bounds on ρn (Wr,dp, Lq) , both being a constant factor of n-r/d , for a Sobolev class Wr,dp , \( 1 \leq p, q \leq \infty \) . As this is also the estimate of the classical Alexandrov nonlinear n -width, our result proves that approximation of Wr,dp by the family of manifolds of pseudo-dimension n is as powerful as approximation by the family of all nonlinear manifolds with continuous selection operators.

Vitaly Maiorov

Papers

Almost Linear VC Dimension Bounds for Piecewise Polynomial Networks

Lower bounds for approximation by MLP neural networks

On Best Approximation by Ridge Functions

Approximation by neural networks and learning theory

On the Degree of Approximation by Manifolds of Finite Pseudo-Dimension