In this survey we discuss various approximation-theoretic problems that arise in the multilayer feedforward perceptron (MLP) model in neural networks. The MLP model is one of the more popular and practical of the many neural network models. Mathematically it is also one of the simpler models. Nonetheless the mathematics of this model is not well understood, and many of these problems are approximation-theoretic in character. Most of the research we will discuss is of very recent vintage. We will report on what has been done and on various unanswered questions. We will not be presenting practical (algorithmic) methods. We will, however, be exploring the capabilities and limitations of this model.

Approximation theory of the MLP model in neural networks

In the chemical sciences, many laboratory experiments, environmental and industrial processes, as well as modeling exercises, are characterized by large numbers of input variables. A general objective in such cases is an exploration of the high-dimensional input variable space as thoroughly as possible for its impact on observable system behavior, often with either optimization in mind or simply for achieving a better understanding of the phenomena involved. An important concern when undertaking these explorations is the number of experiments or modeling excursions necessary to effectively learn the system input → output behavior, which is typically a nonlinear relationship. Although simple logic suggests that the number of runs could grow exponentially with the number of input variables, broadscale evidence indicates that the required effort often scales far more comfortably. This paper considers an emerging family of high dimensional model representation concepts and techniques capable of dealing with s...

High Dimensional Model Representations

We formulate the general theory of random walks in continuum, essential for treating a collision rate which depends smoothly upon direction of motion We also consider a smooth probability distribution of correlations between the directions of motion before and after collisions, as well as orientational Brownian motion between collisions These features lead to an effective Smoluchowski equation Such random walks involving an infinite number of distinct directions of motion cannot be treated on a lattice, which permits only a finite number of directions of motion, nor by Langevin methods, which make no reference to individual collisions The effective Smoluchowski equation enables a description of the biased random walk of the bacterium Escherichia coli during chemotaxis, its search for food The chemotactic responses of cells which perform temporal comparisons of the concentration of a chemical attractant are predicted to be strongly positive, whereas those of cells which measure averages of the ambient attractant concentration are predicted to be negative The former prediction explains the observed behavior of wild-type (naturally occurring) cells; however, the latter behavior has yet to be observed, even in cells defective in adaption

Theory of continuum random walks and application to chemotaxis

We formulate the general theory of random walks in continuum, essential for treating a collision rate which depends smoothly upon direction of motion. We also consider a smooth probability distribution of correlations between the directions of motion before and after collisions, as well as orientational Brownian motion between collisions. These features lead to an effective Smoluchowski equation. Such random walks involving an infinite number of distinct directions of motion cannot be treated on a lattice, which permits only a finite number of directions of motion, nor by Langevin methods, which make no reference to individual collisions. The effective Smoluchowski equation enables a description of the biased random walk of the bacterium Escherichia coli during chemotaxis, its search for food

The theory of scale relativity is a new approach to the problem of the origin of fundamental scales and of scaling laws in physics, that consists of generalizing Einstein's principle of relativity (up to now applied to motion laws) to scale transformations. Namely, we redefine space-time resolutions as characterizing the state of scale of the reference system and require that the equations of physics keep their form under resolution transformations (i.e. be scale covariant). We recall in the present review paper how the development of the theory is intrinsically linked to the concept of fractal space-time, and how it allows one to recover quantum mechanics as mechanics on such a non-differentiable space-time, in which the Schrodinger equation is demonstrated as a geodesies equation. We recall that the standard quantum behavior is obtained, however, as a manifestation of a “Galilean” version of the theory, while the application of the principle of relativity to linear scale laws leads to the construction of a theory of special scale relativity, in which there appears impassable, minimal and maximal scales, invariant under dilations. The theory is then applied to its preferential domains of applications, namely very small and very large length- and time-scales, i.e. high energy physics, cosmology and chaotic systems.

Scale relativity and fractal space-time: applications to quantum physics, cosmology and chaotic systems.

We consider the motion of a classical particle in (1+1)-dimensional space-time. Four probability distributions govern the trajectory of the particle; these give the probability of moving to the left or right in space while moving backwards or forwards in time. If these probabilities are randomly distributed and if the probability of moving backwards in time is related to the probability of moving forwards in time in a prescribed manner, then the master equations for these probabilities give rise to the Dirac equation without recourse to direct analytic continuation. In contrast, when a particle always moves forward in time, an analytic continuation is required to recover the Dirac equation.

Time reversal in stochastic processes and the Dirac equation.

We present a procedure of practical use for representing functions of several variables as superpositions of functions of only one variable. We show how the procedure works when applied, for example, to the location of global minima. Our numerical examples are restricted here, for simplicity, to functions of two variables. The straightforward extension to functions of more variables will be discussed elsewhere

G. N. Ord

Papers

Time reversal in stochastic processes and the Dirac equation.

Approximate representation of functions of several variables in terms of functions of one variable.