The temporal distance between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While Hidden Markov Models tend to ignore this information, recurrent neural networks (RNNs) can in principle learn to make use of it. We focus on Long Short-Term Memory (LSTM) because it has been shown to outperform other RNNs on tasks involving long time lags. We find that LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars. Without external resets or teacher forcing, our LSTM variant also learns to generate stable streams of precisely timed spikes and other highly nonlinear periodic patterns. This makes LSTM a promising approach for tasks that require the accurate measurement or generation of time intervals.

/pdf/learning-precise-timing-with-lstm-recurrent-networks-55hmjdbaed.pdf

Learning precise timing with lstm recurrent networks

Control of chaos: Methods and applications in engineering

The problems and methods of control of chaos, which in the last decade was the subject of intensive studies, were reviewed. The three historically earliest and most actively developing directions of research such as the open-loop control based on periodic system excitation, the method of Poincare map linearization (OGY method), and the method of time-delayed feedback (Pyragas method) were discussed in detail. The basic results obtained within the framework of the traditional linear, nonlinear, and adaptive control, as well as the neural network systems and fuzzy systems were presented. The open problems concerned mostly with support of the methods were formulated. The second part of the review will be devoted to the most interesting applications.

/pdf/control-of-chaos-methods-and-applications-i-methods-3zi2ru8whq.pdf

Control of Chaos: Methods and Applications. I. Methods

https://bura.brunel.ac.uk/bitstream/2438/4939/3/Fulltext.pdf

A delay-dependent LMI approach to dynamics analysis of discrete-time recurrent neural networks with time-varying delays

In this paper, a fractional-order recurrent neural network is proposed and several topics related to the dynamics of such a network are investigated, such as the stability, Hopf bifurcations, and undamped oscillations. The stability domain of the trivial steady state is completely characterized with respect to network parameters and orders of the commensurate-order neural network. Based on the stability analysis, the critical values of the fractional order are identified, where Hopf bifurcations occur and a family of oscillations bifurcate from the trivial steady state. Then, the parametric range of undamped oscillations is also estimated and the frequency and amplitude of oscillations are determined analytically and numerically for such commensurate-order networks. Meanwhile, it is shown that the incommensurate-order neural network can also exhibit a Hopf bifurcation as the network parameter passes through a critical value which can be determined exactly. The frequency and amplitude of bifurcated oscillations are determined.

Undamped Oscillations Generated by Hopf Bifurcations in Fractional-Order Recurrent Neural Networks With Caputo Derivative

We study a particular class of n-node recurrent neural networks (RNNs). In the 3-node case we use monotone dynamical systems theory to show, for a well-defined set of parameters, that, generically, every orbit of the RNN is asymptotic to a periodic orbit. We then investigate whether RNNs of this class can adapt their internal parameters so as to "learn" and then replicate autonomously (in feedback) certain external periodic signals. Our learning algorithm is similar to the identification algorithms in adaptive control theory. The main feature of the algorithm is that global exponential convergence of parameters is guaranteed. We also obtain partial convergence results in the n-node case.

/pdf/existence-and-learning-of-oscillations-in-recurrent-neural-2y7u599xfp.pdf

Existence and learning of oscillations in recurrent neural networks

We study a particular class of n-node recurrent neural networks (RNNs). In the 3-node case we use monotone dynamical systems theory to show, for a well-defined set of parameters, that, generically, every orbit of the RNN is asymptotic to a periodic orbit. We then investigate whether RNNs of this class can adapt their internal parameters so as to ?learn? and then replicate autonomously (in feedback) certain external periodic signals. Our learning algorithm is similar to the identification algorithms in adaptive control theory. The main feature of the algorithm is that global exponential convergence of parameters is guaranteed. We also obtain partial convergence results in the n-node case

M. G. Weiss

Papers

Existence and learning of oscillations in recurrent neural networks

Existence and learning of oscillations in recurrent neural networks