The integration of optimization methodologies with computational analyses/simulations has a profound impact on the product design. Such integration, however, faces multiple challenges. The most eminent challenges arise from high-dimensionality of problems, computationally-expensive analysis/simulation, and unknown function properties (i.e., black-box functions). The merger of these three challenges severely aggravates the difficulty and becomes a major hurdle for design optimization. This paper provides a survey on related modeling and optimization strategies that may help to solve High-dimensional, Expensive (computationally), Black-box (HEB) problems. The survey screens out 207 references including multiple historical reviews on relevant subjects from more than 1,000 papers in a variety of disciplines. This survey has been performed in three areas: strategies tackling high-dimensionality of problems, model approximation techniques, and direct optimization strategies for computationally-expensive black-box functions and promising ideas behind non-gradient optimization algorithms. Major contributions in each area are discussed and presented in an organized manner. The survey exposes that direct modeling and optimization strategies to address HEB problems are scarce and sporadic, partially due to the difficulty of the problem itself. Moreover, it is revealed that current modeling research tends to focus on sampling and modeling techniques themselves and neglect studying and taking the advantages of characteristics of the underlying expensive functions. Based on the survey results, two promising approaches are identified to solve HEB problems. Directions for future research are also discussed.

/pdf/survey-of-modeling-and-optimization-strategies-to-solve-high-1tg25zsec9.pdf

Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions

In recent years, multi-layer feedforward neural networks have been popularly used for pattern classification, function approximation and regression problems. Methods using standard back-propagation learning algorithm perform gradient descent only in the weight space of a network with fixed topology. Recently, various researchers have investigated different approaches that alter the network topology as learning proceeds. In this thesis, I concentrate on constructive algorithms for structure learning in feedforward neural networks for regression problems. The basic idea of constructive algorithms is to start with a small network, then add hidden units and weights incrementally until a satisfactory solution is found. There are hurdles that constructive algorithms have to overcome, including: (1) How to train the new hidden unit? (2) Whether the constructive algorithms can produce a neural network function that is as close to an arbitrary target function as desired? (3) How to control the complexity of the new hidden unit?
To address these issues, I develop a number of objective functions for training new hidden units. The theoretical convergence properties of a number of constructive algorithms, in which hidden units are added one by one in a greedy manner, are also examined. Moreover, I study how the integration of Bayesian regularization and constructive algorithms can lead to improved network performance. The approach is promising in that the regularization parameters can be automatically controlled in a disciplined manner without requiring manual setting.* ftn*Originally published in DAI Vol. 58, No. 11. Reprinted here with corrected author name.

Constructive algorithms for structure learning in feedforward neural networks

In this survey paper, we review the constructive algorithms for structure learning in feedforward neural networks for regression problems. The basic idea is to start with a small network, then add hidden units and weights incrementally until a satisfactory solution is found. By formulating the whole problem as a state-space search, we first describe the general issues in constructive algorithms, with special emphasis on the search strategy. A taxonomy, based on the differences in the state transition mapping, the training algorithm, and the network architecture, is then presented.

Constructive algorithms for structure learning in feedforward neural networks for regression problems

In this paper, we study a number of objective functions for training new hidden units in constructive algorithms for multilayer feedforward networks. The aim is to derive a class of objective functions the computation of which and the corresponding weight updates can be done in O(N) time, where N is the number of training patterns. Moreover, even though input weight freezing is applied during the process for computational efficiency, the convergence property of the constructive algorithms using these objective functions is still preserved. We also propose a few computational tricks that can be used to improve the optimization of the objective functions under practical situations. Their relative performance in a set of two-dimensional regression problems is also discussed.

/pdf/objective-functions-for-training-new-hidden-units-in-1c4r0nk1mo.pdf

Objective functions for training new hidden units in constructive neural networks

Function approximation has been found in many applications. The radial basis function (RBF) network is one approach which has shown a great promise in this sort of problems because of its faster learning capacity. A traditional RBF network takes Gaussian functions as its basis functions and adopts the least-squares criterion as the objective function, However, it still suffers from two major problems. First, it is difficult to use Gaussian functions to approximate constant values. If a function has nearly constant values in some intervals, the RBF network will be found inefficient in approximating these values. Second, when the training patterns incur a large error, the network will interpolate these training patterns incorrectly. In order to cope with these problems, an RBF network is proposed in this paper which is based on sequences of sigmoidal functions and a robust objective function. The former replaces the Gaussian functions as the basis function of the network so that constant-valued functions can be approximated accurately by an RBF network, while the latter is used to restrain the influence of large errors. Compared with traditional RBF networks, the proposed network demonstrates the following advantages: (1) better capability of approximation to underlying functions; (2) faster learning speed; (3) better size of network; (4) high robustness to outliers.

https://www.umbc.edu/rssipl/pdf/SMC_12_99.pdf

Robust radial basis function neural networks

This paper concerns neural network approaches to function approximation and optimization using linear superposition of Gaussians (or what are popularly known as radial basis function (RBF) networks). The problem of function approximation is one of estimating an underlying function f, given samples of the form ((y/sub i/, x/sub i/); i=1,2,...,n; with y/sub i/=f(x/sub i/)). When the dimension of the input is high and the number of samples small, estimation of the function becomes difficult due to the sparsity of samples in local regions. The authors find that this problem of high dimensionality can be overcome to some extent by using linear transformations of the input in the Gaussian kernels. Such transformations induce intrinsic dimension reduction, and can be exploited for identifying key factors of the input and for the phase space reconstruction of dynamical systems, without explicitly computing the dimension and delay. They present a generalization that uses multiple linear projections onto scalars and successive RBF networks (MLPRBF) that estimate the function based on these scaler values. They derive some key properties of RBF networks that provide suitable grounds for implementing efficient search strategies for nonconvex optimization within the same framework. >

Chuan-Lin Wu

Papers

Approximation, dimension reduction, and nonconvex optimization using linear superpositions of Gaussians