Dropout Improves Recurrent Neural Networks for Handwriting Recognition
Vu Pham,Theodore Bluche,Christopher Kermorvant,Jérôme Louradour +3 more
- pp 285-290
TLDR
In this article, the authors show that RNNs with Long Short-Term Memory (LSTM) cells can be improved using dropout, a recently proposed regularization method for deep architectures.Abstract:
Recurrent neural networks (RNNs) with Long Short-Term memory cells currently hold the best known results in unconstrained handwriting recognition. We show that their performance can be greatly improved using dropout - a recently proposed regularization method for deep architectures. While previous works showed that dropout gave superior performance in the context of convolutional networks, it had never been applied to RNNs. In our approach, dropout is carefully used in the network so that it does not affect the recurrent connections, hence the power of RNNs in modeling sequences is preserved. Extensive experiments on a broad range of handwritten databases confirm the effectiveness of dropout on deep architectures even when the network mainly consists of recurrent and shared connections.read more
Citations
More filters
Journal ArticleDOI
Deep learning in neural networks
TL;DR: This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.
Posted Content
Recurrent Neural Network Regularization
TL;DR: This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
Proceedings Article
A theoretically grounded application of dropout in recurrent neural networks
Yarin Gal,Zoubin Ghahramani +1 more
TL;DR: The authors apply this variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks, and to the best of their knowledge improve on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity).
Journal ArticleDOI
Named Entity Recognition with Bidirectional LSTM-CNNs
Jason P.C. Chiu,Eric Nichols +1 more
TL;DR: In this article, a hybrid bidirectional LSTM and CNN architecture was proposed to automatically detect word and character-level features, eliminating the need for feature engineering and lexicons to achieve high performance.
Posted Content
Learning to Diagnose with LSTM Recurrent Neural Networks
TL;DR: In this paper, a simple LSTM network was used to recognize patterns in multivariate time series of clinical measurements for classification of diagnoses, training a model to classify 128 diagnoses given 13 frequently but irregularly sampled clinical measurements.
References
More filters
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Journal ArticleDOI
Long short-term memory
TL;DR: A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Journal ArticleDOI
Learning long-term dependencies with gradient descent is difficult
TL;DR: This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.
Posted Content
Improving neural networks by preventing co-adaptation of feature detectors
TL;DR: The authors randomly omits half of the feature detectors on each training case to prevent complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors.
Proceedings ArticleDOI
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
TL;DR: This paper presents a novel method for training RNNs to label unsegmented sequences directly, thereby solving both problems of sequence learning and post-processing.