Journal ArticleDOI
Learning long-term dependencies with gradient descent is difficult
TLDR
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.Abstract:
Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. >read more
Citations
More filters
Journal ArticleDOI
Aspect-Based Opinion Mining on Student’s Feedback for Faculty Teaching Performance Evaluation
Irum Sindhu,Sher Muhammad Daudpota,Kamal Badar,Maheen Bakhtyar,Junaid Baber,Mohammad Nurunnabi +5 more
TL;DR: This study proposes a supervised aspect based opinion mining system based on two-layered LSTM model for performing aspect based sentiment analysis on students’ feedback for evaluating faculty teaching performance.
Proceedings ArticleDOI
Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks
TL;DR: This work proposes to apply the Layer-wise Relevance Propagation algorithm to explain clinical decisions proposed by deep modern neural networks, and shows that the features, which are identified by the algorithm to be relevant, largely agree with clinical knowledge and guidelines.
Proceedings ArticleDOI
The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models
TL;DR: Experimental results have shown that without using any recurrent feedbacks, FOFE based FNNLMs can significantly outperform not only the standard fixed-input FNN-LMs but also the popular recurrent neural network (RNN) LMs.
Journal ArticleDOI
Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series
TL;DR: In this article, an unsupervised approach based on reservoir computing is proposed to learn vectorial representations of multivariate time series (MTS) by encoding each MTS within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics.
Journal ArticleDOI
Pathways and challenges of the application of artificial intelligence to geohazards modelling
TL;DR: In this article, the authors analyzed the various domains of geohazards which have benefited from classical machine learning approaches and highlighted the future course of direction in this field and emphasized that the future studies should focus on consecutive events along with integration of physical models.
References
More filters
Journal ArticleDOI
Optimization by Simulated Annealing
TL;DR: There is a deep and useful connection between statistical mechanics and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters), and a detailed analogy with annealing in solids provides a framework for optimization of very large and complex systems.
Book ChapterDOI
Learning internal representations by error propagation
TL;DR: This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion.
Book
Learning internal representations by error propagation
TL;DR: In this paper, the problem of the generalized delta rule is discussed and the Generalized Delta Rule is applied to the simulation results of simulation results in terms of the generalized delta rule.
Journal ArticleDOI
A learning algorithm for continually running fully recurrent neural networks
Ronald J. Williams,David Zipser +1 more
TL;DR: The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
Journal ArticleDOI
Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithm—Corrigenda for this article is available here
TL;DR: A new global optimization algorithm for functions of continuous variables is presented, derived from the “Simulated Annealing” algorithm recently introduced in combinatorial optimization, which is quite costly in terms of function evaluations, but its cost can be predicted in advance, depending only slightly on the starting point.