Nonlinear black-box modeling in system identification: a unified overview

TLDR: What are the common features in the different approaches, the choices that have to be made and what considerations are relevant for a successful system-identification application of these techniques are described, from a user's perspective.

About: This article is published in Automatica.The article was published on 1995-12-01 and is currently open access. It has received 2031 citations till now. The article focuses on the topics: Basis function & Nonlinear system.

Figures

Figure Illustrating the backpropagation The gure shows two graphs The graph on the left encodes a formula in the way expressions are encoded into abstract graphs in syntactic analyzers in computer science for instance the graph above encodes g P i i gi i where denotes multiplication i e the general one layer network formula Nodes of this graph can be interpreted both as operators e g gi and as the evaluation of the expression encoded by the subtree located below this node The triangle same pattern indicates that each component l of could be the result of evaluating again the same function encoded by the same graph This would immediately encode a two and then multi layer network On the right hand side we have expanded once the same pattern showing the case of two layers We have shown in thick the paths linking the root g to one particular parameter say k The semantic of the thickening is the following each node on a thick path which is an operator e g gi is replaced by its partial derivative with respect to the node just below it on the thick path here we regard such node as the evaluation of the expression encoded by the subtree located below it For instance in the gure gi is replaced by gi and gk by gk k By the chain rule for di erentiation it turns out that the graph on the right hand side encodes the partial derivative g k This graphical representation also explains why intermediate calculations can be shared for di erent partial derivatives and this is indeed the nice feature of backpropagating the gradient This presentation is due to G Cybenko

Frequently asked questions

Q1. What are the contributions in this paper?

There has been considerable recent interest in this area with structures based on neural networks radial basis networks wavelet networks hinging hyperplanes as well as wavelet transform based methods and models based on fuzzy sets and fuzzy rules This paper describes all these approaches in a common framework from a user s perspective These mappings are discussed separately This is handled by making the number of used parameters consid erably less than the number of o ered parameters by regularization shrinking pruning or regressor selection A more mathematically comprehensive treatment is given in a companion paper Judit sky et al The expansion from the scalar argument to the regressor space is achieved by a radial or a ridge type approach Basic techniques for estimating the parameters in the structures are criterion minimiza tion as well as two step procedures where rst the relevant basis functions are determined using data and then a linear least squares step to determine the coordinates of the func tion approximation A particular problem is to deal with the large number of potentially necessary parameters 

Q2. what is the way to build a linear model for the system?

The parameterized functions f and g can be chosen to be linear or nonlinear by a neural net A further motivation for this model is that it becomes easier to develop controllers from than from the models discussed earlierIn McAvoy it is suggested rst to build a linear model for the system 

Q3. What is the common method of terminating the iterations when the model t is?

This method of terminating the iterations when the model t evaluated for the validation data starts to increase will be called stopped searchRegularization implemented as stopped search is called implicit regularization in contrast to the explicit regularization which is obtained by minimizing the modi ed criterionLocal MinimaA fundamental problem with minimization tasks like is that VN may have several or many local non global minima where local search algorithms may get caught 

Q4. how do ridge constructions deal with the curse of dimensionality?

Ridge constructions like the ones used in sigmoidal neural networks and the hinging hyperplanes networks deal with the curse of dimensionality by extrapolation 

Q5. What is the equivalent of shrinking in connection with neural nets?

The equivalent of shrinking in connection with neural nets is called pruning and it has attracted much interest lately See e g Reed for an overview and further references therein 

Q6. What are the advantages of using a fuzzy model as an initial guess?

It would also be possible to use their prior model as initial guess but allow other rules to be introduced via learning corresponding experiments are under progressAnother advantage of describing the model via fuzzy rules is the possibility to decompile the model after learning again in the form of fuzzy rules for return to the user doctor or patient Returning a mathematical model would be of little use for the average user having no training in mathematicsSummary and recommendations 

Q7. how many variables are used to determine the glyc)mic level of a diabet?

Bo t which is a ash injection to assimilate a recent mealNevertheless despite doctor s experience it is very di cult to manually obtain a more or less constant glyc)mic level in part because a good control should take into account up to six input variables which is far beyond human control capability 

Q8. how can i estimate wavelet coe cients?

For d the wavelet basis function expansion would be an excellent choice since the wavelet coe cients can be estimated very e ciently 

Q9. what is the dimension dependent computational cost of a function constructed by the other two methods?

For a function constructed by the other two methods the dimension dependent computational cost stays only in the evaluation of the norm of k or the inner product 

Q10. What is the way to approach the problem of minimizing with respect to m?

It is well known that orthonormal wavelets form orthonormal basis of L Rd Mallat Daubechies Several authors have shown that one hidden layer sigmoid network can approximate any continuous functions with an arbitrary accuracy provided the number of basis functions used in the net is su ciently large and some error bounds are known see e g Cybenko Barron Juditsky et al Similar results can be obtained for other one hidden layer networks by using similar techniquesParameters O ered and Parameters UsedThere is a natural way to approach the problem of minimizing with respect to m