The problem of the approximation of nonlinear mapping, (especially continuous mappings) is considered. Regularization theory and a theoretical framework for approximation (based on regularization techniques) that leads to a class of three-layer networks called regularization networks are discussed. Regularization networks are mathematically related to the radial basis functions, mainly used for strict interpolation tasks. Learning as approximation and learning as hypersurface reconstruction are discussed. Two extensions of the regularization approach are presented, along with the approach's corrections to splines, regularization, Bayes formulation, and clustering. The theory of regularization networks is generalized to a formulation that includes task-dependent clustering and dimensionality reduction. Applications of regularization networks are discussed. >

Networks for approximation and learning

The affine rank minimization problem consists of finding a matrix of minimum rank that satisfies a given system of linear equality constraints. Such problems have appeared in the literature of a diverse set of fields including system identification and control, Euclidean embedding, and collaborative filtering. Although specific instances can often be solved with specialized algorithms, the general affine rank minimization problem is NP-hard because it contains vector cardinality minimization as a special case.

In this paper, we show that if a certain restricted isometry property holds for the linear transformation defining the constraints, the minimum-rank solution can be recovered by solving a convex optimization problem, namely, the minimization of the nuclear norm over the given affine space. We present several random ensembles of equations where the restricted isometry property holds with overwhelming probability, provided the codimension of the subspace is sufficiently large.

The techniques used in our analysis have strong parallels in the compressed sensing framework. We discuss how affine rank minimization generalizes this preexisting concept and outline a dictionary relating concepts from cardinality minimization to those of rank minimization. We also discuss several algorithmic approaches to minimizing the nuclear norm and illustrate our results with numerical examples.

https://faculty.washington.edu/mfazel/low-rank-v2.pdf

Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization

We give a simple technique for verifying the Restricted Isometry Property (as introduced by Candes and Tao) for random matrices that underlies Compressed Sensing. Our approach has two main ingredients: (i) concentration inequalities for random inner products that have recently provided algorithmically simple proofs of the Johnson–Lindenstrauss lemma; and (ii) covering numbers for finite-dimensional balls in Euclidean space. This leads to an elementary proof of the Restricted Isometry Property and brings out connections between Compressed Sensing and the Johnson–Lindenstrauss lemma. As a result, we obtain simple and direct proofs of Kashin’s theorems on widths of finite balls in Euclidean space (and their improvements due to Gluskin) and proofs of the existence of optimal Compressed Sensing measurement matrices. In the process, we also prove that these measurements have a certain universality with respect to the sparsity-inducing basis.

/pdf/a-simple-proof-of-the-restricted-isometry-property-for-34k5xh88z4.pdf

A Simple Proof of the Restricted Isometry Property for Random Matrices

We are interested in estimating the average effect of a binary treatment on a scalar outcome. If assignment to the treatment is exogenous or unconfounded, that is, independent of the potential outcomes given covariates, biases associated with simple treatmentcontrol average comparisons can be removed by adjusting for differences in the covariates. Rosenbaum and Rubin (1983) show that adjusting solely for differences between treated and control units in the propensity score removes all biases associated with differences in covariates. Although adjusting for differences in the propensity score removes all the bias, this can come at the expense of efficiency, as shown by Hahn (1998), Heckman, Ichimura, and Todd (1998), and Robins, Mark, and Newey (1992). We show that weighting by the inverse of a nonparametric estimate of the propensity score, rather than the true propensity score, leads to an efficient estimate of the average treatment effect. We provide intuition for this result by showing that this estimator can be interpreted as an empirical likelihood estimator that efficiently incorporates the information about the propensity score.

/pdf/efficient-estimation-of-average-treatment-effects-using-the-4nsrn8d919.pdf

Efficient estimation of average treatment effects using the estimated propensity score

Possibility of Approximation: 1 Basic notions 2 Linear operators 3 Approximation theorems 4 The theorem of Stone 5 Notes Polynomials of Best Approximation: 1 Existence of polynomials of best approximation 2 Characterization of polynomials of best approximation 3 Applications of convexity 4 Chebyshev systems 5 Uniqueness of polynomials of best approximation 6 Chebyshev's theorem 7 Chebyshev polynomials 8 Approximation of some complex functions 9 Notes Properties of Polynomials and Moduli of Continuity: 1 Interpolation 2 Inequalities of Bernstein 3 The inequality of Markov 4 Growth of polynomials in the complex plane 5 Moduli of continuity 6 Moduli of smoothness 7 Classes of functions 8 Notes The Degree of Approximation by Trigonometric Polynomials: 1 Generalities 2 The theorem of Jackson 3 The degree of approximation of differentiable functions 4 Inverse theorems 5 Differentiable functions 6 Notes The Degree of Approximation by Algebraic Polynomials: 1 Preliminaries 2 The approximation theorems 3 Inequalities for the derivatives of polynomials 4 Inverse theorems 5 Approximation of analytic functions 6 Notes Approximation by Rational Functions Functions of Several Variables: 1 Degree of rational approximation 2 Inverse theorems 3 Periodic functions of several variables 4 Approximation by algebraic polynomials 5 Notes Approximation by Linear Polynomial Operators: 1 Sums of de la Vallee-Poussin Positive operators 2 The principle of uniform boundedness 3 Operators that preserve trigonometric polynomials 4 Trigonometric saturation classes 5 The saturation class of the Bernstein polynomials 6 Notes Approximation of Classes of Functions: 1 Introduction 2 Approximation in the space 3 The degree of approximation of the classes 4 Distance matrices 5 Approximation of the classes 6 Arbitrary moduli of continuity Approximation by operators 7 Analytic functions 8 Notes Widths: 1 Definitions and basic properties 2 Sets of continuous and differentiable functions 3 Widths of balls 4 Applications of theorem 2 5 Differential operators 6 Widths of the sets 7 Notes Entropy: 1 Entropy and capacity 2 Sets of continuous and differentiable functions 3 Entropy of classes of analytic functions 4 More general sets of analytic functions 5 Relations between entropy and widths 6 Notes Representation of Functions of Several Variables by Functions of One Variable: 1 The Theorem of Kolmogorov 2 The fundamental lemma 3 The completion of the proof 4 Functions not representable by superpositions 5 Notes Bibliography Index

Approximation of Functions

/pdf/a-contribution-to-the-theory-of-divergent-sequences-442cfsauqm.pdf

A contribution to the theory of divergent sequences

Some New Functional Spaces

1. Problems of Polynomial Approximation.- 1. Examples of Polynomials of Best Approximation.- 2. Distribution of Alternation Points of Polynomials of Best Approximation.- 3. Distribution of Zeros of Polynomials of Best Approximation.- 4. Error of Approximation.- 5. Approximation on (-?, ?) by Linear Combinations of Functions (x - c)-1.- 6. Weighted Approximation by Polynomials on (-?, ?).- 7. Spaces of Approximation Theory.- 8. Problems and Notes.- 2. Approximation Problems with Constraints.- 1. Introduction.- 2. Growth Restrictions for the Coefficients.- 3. Monotone Approximation.- 4. Polynomials with Integral Coefficients.- 5. Determination of the Characteristic Sets.- 6. Markov-Type Inequalities.- 7. The Inequality of Remez.- 8. One-sided Approximation by Polynomials.- 9. Problems.- 10. Notes.- 3. Incomplete Polynomials.- 1. Incomplete Polynomials.- 2. Incomplete Chebyshev Polynomials.- 3. Incomplete Trigonometric Polynomials.- 4. Sequences of Polynomials with Many Real Zeros.- 5. Problems.- 6. Notes.- 4. Weighted Polynomials.- 1. Essential Sets of Weighted Polynomials.- 2. Weighted Chebyshev Polynomials.- 3. The Equilibrium Measure.- 4. Determination of Minimal Essential Sets.- 5. Weierstrass Theorems and Oscillations.- 6. Weierstrass Theorem for Freud Weights.- 7. Problems.- 8. Notes.- 5. Wavelets and Orthogonal Expansions.- 1. Multiresolutions and Wavelets.- 2. Scaling Functions with a Monotone Majorant.- 3. Periodization.- 4. Polynomial Schauder Bases.- 5. Orthonormal Polynomial Bases.- 6. Problems and Notes.- 6. Splines.- 1. General Facts.- 2. Splines of Best Approximation.- 3. Periodic Splines.- 4. Convergence of Some Spline Operators.- 5. Notes.- 7. Rational Approximation.- 1. Introduction.- 2. Best Rational Approximation.- 3. Rational Approximation of |x|.- 4. Approximation of e-xon [-1,1].- 5. Rational Approximation of e-x on [0, ?).- 6. Approximation of Classes of Functions.- 7. Theorems of Popov.- 8. Properties of the Operator of Best Rational Approximation in C and Lp.- 9. Approximation by Rational Functions with Arbitrary Powers.- 10. Problems.- 11. Notes.- 8. StahPs Theorem.- 1. Introduction and Main Result.- 2. A Dirichlet Problem on [1/2, l/pn].- 3. The Second Approach to the Dirichlet Problem.- 4. Proof of Theorem 1.1.- 5. Notes.- 9. Pade Approximation.- 1. The Pade Table.- 2. Convergence of the Rows of the Pade Table.- 3. The Nuttall-Pommerenke Theorem.- 4. Problems.- 5. Notes.- 10. Hardy Space Methods in Rational Approximation.- 1. Bernstein-Type Inequalities for Rational Functions.- 2. Uniform Rational Approximation in Hardy Spaces.- 3. Approximation by Simple Functions.- 4. The Jackson-Rusak Operator Rational Approximation of Sums of Simple Functions.- 5. Rational Approximation on T and on [-1,1].- 6. Relations Between Spline and Rational Approximation in the Spaces 0 ?.- 7. Problems.- 8. Notes.- 11. Muntz Polynomials.- 1. Definitions and Simple Properties.- 2. Muntz-Jackson Theorems.- 3. An Inverse Muntz-Jackson Theorem.- 4. The Index of Approximation.- 5. Markov-Type Inequality for Muntz Polynomials.- 6. Problems.- 7. Notes.- 12. Nonlinear Approximation.- 1. Definitions and Simple Properties.- 2. Varisolvent Families.- 3. Exponential Sums.- 4. Lower Bounds for Errors of Nonlinear Approximation.- 5. Continuous Selections from Metric Projections.- 6. Approximation in Banach Spaces: Suns and Chebyshev Sets.- 7. Problems.- 8. Notes.- 13. Widths I.- 1. Definitions and Basic Properties.- 2. Relations Between Different Widths.- 3. Widths of Cubes and Octahedra.- 4. Widths in Hilbert Spaces.- 5. Applications of Borsuk's Theorem.- 6. Variational Problems and Spectral Functions.- 7. Results of Buslaev and Tikhomirov.- 8. Classes of Differentiate Functions on an Interval.- 9. Classes of Analytic Functions.- 10. Problems.- 11. Notes.- 14. Widths II: Weak Asymptotics for Widths of Lipschitz Balls, Random Approximants.- 1. Introduction.- 2. Discretization.- 3. Weak Equivalences for Widths. Elementary Methods.- 4. Distribution of Scalar Products of Unit Vectors.- 5. Kashin's Theorems.- 6. Gaussian Measures.- 7. Linear Widths of Finite Dimensional Balls.- 8. Linear Widths of the Lipschitz Classes.- 9. Problems.- 10. Notes.- 15. Entropy.- 1. Entropy and Capacity.- 2. Elementary Estimates.- 3. Linear Approximation and Entropy.- 4. Relations Between Entropy and Widths.- 5. Entropy of Classes of Analytic Functions.- 6. The Birman-Solomyak Theorem.- 7. Entropy Numbers of Operators.- 8. Notes.- 16. Convergence of Sequences of Operators.- 1. Introduction.- 2. Simple Necessary and Sufficient Conditions.- 3. Geometric Properties of Dominating Sets.- 4. Strict Dominating Systems Minimal Systems Examples.- 5. Shadows of Sets of Continuous Functions.- 6. Shadows in Banach Function Spaces.- 7. Positive Contractions.- 8. Contractions.- 9. Notes.- 17. Representation of Functions by Superpositions.- 1. The Theorems of Kolmogorov.- 2. Proof of the Theorems.- 3. Functions Not Representable by Superpositions.- 4. Linear Superpositions.- 5. Notes.- Appendix 1. Theorems of Borsuk and of Brunn-Minkowski.- 1. Borsuk's Theorem.- 2. The Brunn-Minkowski Inequality.- Appendix 2. Estimates of Some Elliptic Integrals.- Appendix 3. Hardy Spaces and Blaschke Products.- 1. Hardy Spaces.- 2. Conjugate Functions and Cauchy Integrals.- 3. Atomic Decompositions in Hardy Spaces.- 4. Blaschke Products.- Appendix 4. Potential Theory and Logarithmic Capacity.- 1. Logarithmic Potentials.- 2. Equilibrium Distribution and Logarithmic Capacity.- 3. The Dirichlet Problem and Green's Function.- 4. Balayage Methods.- Author Index.

Constructive approximation : advanced problems

/pdf/metric-entropy-and-approximation-4l2g46irch.pdf

George G. Lorentz

Papers

Approximation of Functions

A contribution to the theory of divergent sequences

Some New Functional Spaces

Constructive approximation : advanced problems

Metric entropy and approximation