Journal ArticleDOI
Learning long-term dependencies with gradient descent is difficult
TLDR
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.Abstract:
Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. >read more
Citations
More filters
Posted Content
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
Nils Reimers,Iryna Gurevych +1 more
TL;DR: This paper evaluates the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or theNumber of recurrent units, are of minor importance.
Book ChapterDOI
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
John F. Kolen,Stefan C. Kremer +1 more
TL;DR: This chapter contains sections titled: Introduction Exponential Error Decay Dilemma: Avoiding Aradient Decay Prevents Long-Term Latching Remedies and Conclusion.
Posted Content
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
Colin Raffel,Daniel P. W. Ellis +1 more
TL;DR: A simplified model of attention is proposed which is applicable to feed-forward neural networks and can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.
Journal ArticleDOI
Deep learning-based feature engineering for stock price movement prediction
Wen Long,Zhichen Lu,Lingxiao Cui +2 more
TL;DR: Experimental results show that the proposed novel end-to-end multi-filters neural network outperforms traditional machine learning models, statistical models, and single-structure networks in terms of the accuracy, profitability, and stability.
Journal ArticleDOI
Drawing and Recognizing Chinese Characters with Recurrent Neural Network
TL;DR: Wang et al. as mentioned in this paper proposed a framework by using the recurrent neural network (RNN) as both a discriminative model for recognizing Chinese characters and a generator model for drawing (generating) Chinese characters.
References
More filters
Journal ArticleDOI
Optimization by Simulated Annealing
TL;DR: There is a deep and useful connection between statistical mechanics and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters), and a detailed analogy with annealing in solids provides a framework for optimization of very large and complex systems.
Book ChapterDOI
Learning internal representations by error propagation
TL;DR: This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion.
Book
Learning internal representations by error propagation
TL;DR: In this paper, the problem of the generalized delta rule is discussed and the Generalized Delta Rule is applied to the simulation results of simulation results in terms of the generalized delta rule.
Journal ArticleDOI
A learning algorithm for continually running fully recurrent neural networks
Ronald J. Williams,David Zipser +1 more
TL;DR: The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
Journal ArticleDOI
Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithm—Corrigenda for this article is available here
TL;DR: A new global optimization algorithm for functions of continuous variables is presented, derived from the “Simulated Annealing” algorithm recently introduced in combinatorial optimization, which is quite costly in terms of function evaluations, but its cost can be predicted in advance, depending only slightly on the starting point.