https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.

/pdf/learning-to-learn-by-gradient-descent-by-gradient-descent-21i3y06n6m.pdf

Learning to learn by gradient descent by gradient descent

This paper presents an approach and solution to the IEEE 2008 Prognostics and Health Management conference challenge problem. The solution utilizes an advanced recurrent neural network architecture to estimate the remaining useful life of the system. The recurrent neural network is trained with back-propagation through time gradient calculations, an Extended Kalman Filter training method, and evolutionary algorithms to generate an accurate and compact algorithm. This solution placed second overall in the competition with a very small margin between the first and second place finishers.

Recurrent neural networks for remaining useful life estimation

Three types of functionally different artificial neural network (ANN) models are calibrated using a relatively short length of groundwater level records and related hydrometeorological data to simulate water table fluctuations in the Gondo aquifer, Burkina Faso. Input delay neural network (IDNN) with static memory structure and globally recurrent neural network (RNN) with inherent dynamical memory are proposed for monthly water table fluctuations modeling. The simulation performance of the IDNN and the RNN models is compared with results obtained from two variants of radial basis function (RBF) networks, namely, a generalized RBF model (GRBF) and a probabilistic neural network (PNN). Overall, simulation results suggest that the RNN is the most efficient of the ANN models tested for a calibration period as short as 7 years. The results of the IDNN and the PNN are almost equivalent despite their basically different learning procedures. The GRBF performs very poorly as compared to the other models. Furthermore, the study shows that RNN may offer a robust framework for improving water supply planning in semiarid areas where aquifer information is not available. This study has significant implications for groundwater management in areas with inadequate groundwater monitoring network.

/pdf/artificial-neural-network-modeling-of-water-table-depth-27z6akjcwf.pdf

Artificial neural network modeling of water table depth fluctuations

From the Publisher:
The use of neural networks is permeating every area of signal processing. They can provide powerful means for solving many problems, especially in nonlinear, real-time, adaptive, and blind signal processing. The Handbook of Neural Network Signal Processing brings together applications that were previously scattered among various publications to provide an up-to-date, detailed treatment of the subject from an engineering point of view.The authors cover basic principles, modeling, algorithms, architectures, implementation procedures, and well-designed simulation examples of audio, video, speech, communication, geophysical, sonar, radar, medical, and many other signals. The subject of neural networks and their application to signal processing is constantly improving. You need a handy reference that will inform you of current applications in this new area. The Handbook of Neural Network Signal Processing provides this much needed service for all engineers and scientists in the field.

Handbook of Neural Network Signal Processing

We present a coherent neural net based framework for solving various signal processing problems. It relies on the assertion that time-lagged recurrent networks possess the necessary representational capabilities to act as universal approximators of nonlinear dynamical systems. This applies to system identification, time-series prediction, nonlinear filtering, adaptive filtering, and temporal pattern classification. We address the development of models of nonlinear dynamical systems, in the form of time-lagged recurrent neural nets, which can be used without further training. We employ a weight update procedure based on the extended Kalman filter (EKF). Against the tendency for a net to forget earlier learning as it processes new examples, we develop a technique called multistream training. We demonstrate our framework by applying it to 4 problems. First, we show that a single time-lagged recurrent net can be trained to produce excellent one-time-step predictions for two different time series and also to be robust to severe errors in the input sequence. Second, we model stably a complex system containing significant process noise. The remaining two problems are drawn from real-world automotive applications. One involves input-output modeling of the dynamic behavior of a catalyst-sensor system which is exposed to an operating engine's exhaust stream, the other the real-time and continuous detection of engine misfire.

G.V. Puskorius

Papers

A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification