https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

On the difficulty of training Recurrent Neural Networks

Survey: Reservoir computing approaches to recurrent neural network training

This tutorial/survey paper: (1) provides a concise point of departure for researchers and practitioners alike wishing to assess the current state of the art in the control and monitoring of civil engineering structures; and (2) provides a link between structural control and other fields of control theory, pointing out both differences and similarities, and points out where future research and application efforts are likely to prove fruitful. The paper consists of the following sections: section 1 is an introduction; section 2 deals with passive energy dissipation; section 3 deals with active control; section 4 deals with hybrid and semiactive control systems; section 5 discusses sensors for structural control; section 6 deals with smart material systems; section 7 deals with health monitoring and damage detection; and section 8 deals with research needs. An extensive list of references is provided in the references section.

Structural control: past, present, and future

Recently, research has picked up a fervent pace in the area of fault diagnosis of electrical machines. The manufacturers and users of these drives are now keen to include diagnostic features in the software to improve salability and reliability. Apart from locating specific harmonic components in the line current (popularly known as motor current signature analysis), other signals, such as speed, torque, noise, vibration etc., are also explored for their frequency contents. Sometimes, altogether different techniques, such as thermal measurements, chemical analysis, etc., are also employed to find out the nature and the degree of the fault. In addition, human involvement in the actual fault detection decision making is slowly being replaced by automated tools, such as expert systems, neural networks, fuzzy-logic-based systems; to name a few. It is indeed evident that this area is vast in scope. Hence, keeping in mind the need for future research, a review paper describing different types of faults and the signatures they generate and their diagnostics' schemes will not be entirely out of place. In particular, such a review helps to avoid repetition of past work and gives a bird's eye view to a new researcher in this area.

Condition Monitoring and Fault Diagnosis of Electrical Motors&#8212;A Review

How to efficiently train recurrent networks remains a challenging and active research topic. Most of the proposed training approaches are based on computational ways to efficiently obtain the gradient of the error function, and can be generally grouped into five major groups. In this study we present a derivation that unifies these approaches. We demonstrate that the approaches are only five different ways of solving a particular matrix equation. The second goal of this paper is develop a new algorithm based on the insights gained from the novel formulation. The new algorithm, which is based on approximating the error gradient, has lower computational complexity in computing the weight update than the competing techniques for most typical problems. In addition, it reaches the error minimum in a much smaller number of iterations. A desirable characteristic of recurrent network training algorithms is to be able to update the weights in an online fashion. We have also developed an online version of the proposed algorithm, that is based on updating the error gradient approximation in a recursive manner.

New results on recurrent network training: unifying the algorithms and accelerating convergence

In this paper, a method is proposed which enables the simulation of the air-gap eccentricity in induction machines. The method is based on the coupled magnetic circuit approach. The model is derived by means of winding functions, and no symmetry in windings layout is assumed. The parameters of the model are calculated directly from the geometry and winding layout of the machine. The effect of eccentricity is included in calculation of machine inductances. It is shown that by proper modeling of the induction motor it is possible to determine the effect of eccentricity on the machine startup, whether the machine is running off a sinusoidal supply or a converter. Theoretical foundations of the technique, as well as the detailed differential equations describing the machine performance under rotor eccentricity are presented.

A method for dynamic simulation of air-gap eccentricity in induction machines

Early detection and diagnosis of incipient faults is desirable for online condition assessment, product quality assurance and improved operational efficiency of induction motors running off power supply mains. In this paper, a model-based fault diagnosis system is developed for induction motors, using recurrent dynamic neural networks for transient response prediction and multi-resolution signal processing for nonstationary signal feature extraction. In addition to nameplate information required for the initial setup, the proposed diagnosis system uses measured motor terminal currents and voltages, and motor speed. The effectiveness of the diagnosis system is demonstrated through staged motor faults of electrical and mechanical origin. The developed system is scalable to different power ratings and it has been successfully demonstrated with data from 2.2-, 373-, and 597-kW induction motors. Incremental tuning is used to adapt the diagnosis system during commissioning on a new motor, significantly reducing the system development time.

Induction motor fault diagnosis based on neuropredictors and wavelet signal processing

A nonlinear dynamic model is developed for a process system, namely a heat exchanger, using the recurrent multilayer perceptron network as the underlying model structure. The perceptron is a dynamic neural network, which appears effective in the input-output modeling of complex process systems. Dynamic gradient descent learning is used to train the recurrent multilayer perceptron, resulting in an order of magnitude improvement in convergence speed over a static learning algorithm used to train the same network. In developing the empirical process model the effects of actuator, process, and sensor noise on the training and testing sets are investigated. Learning and prediction both appear very effective, despite the presence of training and testing set noise, respectively. The recurrent multilayer perceptron appears to learn the deterministic part of a stochastic training set, and it predicts approximately a moving average response of various testing sets. Extensive model validation studies with signals that are encountered in the operation of the process system modeled, that is steps and ramps, indicate that the empirical model can substantially generalize operational transients, including accurate prediction of instabilities not in the training set. However, the accuracy of the model beyond these operational transients has not been investigated. Furthermore, online learning is necessary during some transients and for tracking slowly varying process dynamics. Neural networks based empirical models in some cases appear to provide a serious alternative to first principles models. >

Alexander G. Parlos

Papers

New results on recurrent network training: unifying the algorithms and accelerating convergence

A method for dynamic simulation of air-gap eccentricity in induction machines

Induction motor fault diagnosis based on neuropredictors and wavelet signal processing

Application of the recurrent multilayer perceptron in modeling complex process dynamics

Multi-step-ahead prediction using dynamic recurrent neural networks