https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Forecasting with artificial neural networks: the state of the art

A survey is presented of current methods for decision tree classifier (DTC) designs and the various existing issues. After considering potential advantages of DTCs over single-state classifiers, the subjects of tree structure design, feature selection at each internal node, and decision and search strategies are discussed. The relation between decision trees and neutral networks (NN) is also discussed. >

/pdf/a-survey-of-decision-tree-classifier-methodology-3kolfz0lgh.pdf

A survey of decision tree classifier methodology

/pdf/neural-networks-and-deep-learning-3ccs3ain8p.pdf

Neural Networks and Deep Learning

Artificial Neural Networks (ANNs) are being used increasingly to predict and forecast water resources variables. In this paper, the steps that should be followed in the development of such models are outlined. These include the choice of performance criteria, the division and pre-processing of the available data, the determination of appropriate model inputs and network architecture, optimisation of the connection weights (training) and model validation. The options available to modellers at each of these steps are discussed and the issues that should be considered are highlighted. A review of 43 papers dealing with the use of neural network models for the prediction and forecasting of water resources variables is undertaken in terms of the modelling process adopted. In all but two of the papers reviewed, feedforward networks are used. The vast majority of these networks are trained using the backpropagation algorithm. Issues in relation to the optimal division of the available data, data pre-processing and the choice of appropriate model inputs are seldom considered. In addition, the process of choosing appropriate stopping criteria and optimising network geometry and internal network parameters is generally described poorly or carried out inadequately. All of the above factors can result in non-optimal model performance and an inability to draw meaningful comparisons between different models. Future research efforts should be directed towards the development of guidelines which assist with the development of ANN models and the choice of when ANNs should be used in preference to alternative approaches, the assessment of methods for extracting the knowledge that is contained in the connection weights of trained ANNs and the incorporation of uncertainty into ANN models.

Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications

We propose a robust learning algorithm and apply it to recurrent neural networks. This algorithm is based on filtering outliers from the data and then estimating parameters from the filtered data. The filtering removes outliers from both the target function and the inputs of the neural network. The filtering is soft in that some outliers are neither completely rejected nor accepted. To show the need for robust recurrent networks, we compare the predictive ability of least squares estimated recurrent networks on synthetic data and on the Puget Power Electric Demand time series. These investigations result in a class of recurrent neural networks, NARMA(p,q), which show advantages over feedforward neural networks for time series with a moving average component. Conventional least squares methods of fitting NARMA(p,q) neural network models are shown to suffer a lack of robustness towards outliers. This sensitivity to outliers is demonstrated on both the synthetic and real data sets. Filtering the Puget Power Electric Demand time series is shown to automatically remove the outliers due to holidays. Neural networks trained on filtered data are then shown to give better predictions than neural networks trained on unfiltered time series. >

/pdf/recurrent-neural-networks-and-robust-time-series-prediction-10igzaep85.pdf

Recurrent neural networks and robust time series prediction

The important differences between multilayer perceptrons and classification trees are considered. A number of empirical tests on three real-world problems in power-system load forecasting, power-system security prediction, and speaker-independent vowel recognition are presented. The load-forecasting problem, which is partially a regression problem, uses past trends to predict the critical needs of future power generation. The power-security problem uses the classifier as an interpolator of previously known states of the system. The vowel-recognition problem is representative of the difficulties in automatic speech recognition caused by variability across speakers and phonetic context. In all cases even with various sizes of training sets, the multilayer perceptron performed as well as or better than the trained classification trees. It is therefore concluded that there is not enough theoretical basis to demonstrate clear-cut superiority of one technique over the other. >

A performance comparison of trained multilayer perceptrons and trained classification trees

Uses a parametric statistical framework to understand the effect of input representation on performance for nonlinear prediction of time series. In particular, considerations of input representation lead directly to choices between feedforward and recurrent neural networks. It is shown that feedforward networks are nonlinear autoregressive models and that recurrent networks can model a larger class of processes, including nonlinear autoregressive moving average models. For some processes, feedback allows recurrent networks to achieve better predictions than can be made with a feedforward network with a finite number of inputs. The results are confirmed on a problem in power system regional load forecasting. >

Recurrent neural networks and time series prediction

There exist large classes of time series, such as those with nonlinear moving average components, that are not well modeled by feedforward networks or linear models, but can be modeled by recurrent networks. We show that recurrent neural networks are a type of nonlinear autoregressive-moving average (NARMA) model. Practical ability will be shown in the results of a competition sponsored by the Puget Sound Power and Light Company, where the recurrent networks gave the best performance on electric load forecasting.

/pdf/recurrent-networks-and-narma-modeling-3kb2g29ncr.pdf

Recurrent Networks and NARMA Modeling

Multilayer perceptrons and trained classification trees are two very different techniques which have recently become popular. Giving enough data and time, both methods are capable of performing arbitrary nonlinear classification. The two techniques have not previously been compared on real-world problems. The authors first consider the important differences between multilayer perceptrons and classification trees and conclude that there is not enough theoretical basis for the clear-cut superiority of one technique over the other. They then present results of a number of empirical tests on quite different problems in power system load forecasting and speaker-independent vowel identification. They compare the performance for classification and prediction in terms of accuracy outside the training set. In all cases, even with various sizes of training sets, the multilayer perceptron performed as well as or better than the trained classification trees. The authors are confident that the univariate version of the trained classification trees do not perform as well as the multilayer perceptron. More studies are needed, however, on the comparative performance of the linear combination version of the classification trees. >

J. Connor

Papers

Recurrent neural networks and robust time series prediction

A performance comparison of trained multilayer perceptrons and trained classification trees

Recurrent neural networks and time series prediction

Recurrent Networks and NARMA Modeling

A performance comparison of trained multilayer perceptrons and trained classification trees