Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
Citations
[...]
38,208 citations
14,635 citations
Cites background or result from "Connectionist temporal classificati..."
...Currently successful techniques: LSTM RNNs and GPU-MPCNNs Most competition-winning or benchmark record-setting Deep Learners actually use one of two supervised techniques: (a) recurrent LSTM (1997) trained by CTC (2006) (Sections 5.13, 5.17, 5.21, 5.22), or (b) feedforward GPU-MPCNNs (2011, Sections 5.19, 5.21, 5.22) based on CNNs (1979, Section 5.4) with MP (1992, Section 5.11) trained through BP (1989–2007, Sections 5.8, 5.16)....
[...]
...2009: first official competitionswon by RNNs, andwithMPCNNs Stacks of LSTM RNNs trained by CTC (Sections 5.13, 5.16) became the first RNNs to win official international pattern recognition contests (with secret test sets known only to the organizers)....
[...]
...CTC-LSTM also helped to score first at NIST’s OpenHaRT2013 evaluation (Bluche et al., 2014)....
[...]
...Unlike traditional methods for automatic sequential program synthesis (e.g., Balzer, 1985; Deville & Lau, 1994; Soloway, Abbreviations in alphabetical order AE: Autoencoder AI: Artificial Intelligence ANN: Artificial Neural Network BFGS: Broyden–Fletcher–Goldfarb–Shanno BNN: Biological Neural Network BM: Boltzmann Machine BP: Backpropagation BRNN: Bi-directional Recurrent Neural Network CAP: Credit Assignment Path CEC: Constant Error Carousel CFL: Context Free Language CMA-ES: Covariance Matrix Estimation ES CNN: Convolutional Neural Network CoSyNE: Co-Synaptic Neuro-Evolution CSL: Context Sensitive Language CTC: Connectionist Temporal Classification DBN: Deep Belief Network DCT: Discrete Cosine Transform DL: Deep Learning DP: Dynamic Programming DS: Direct Policy Search EA: Evolutionary Algorithm EM: Expectation Maximization ES: Evolution Strategy FMS: Flat Minimum Search FNN: Feedforward Neural Network FSA: Finite State Automaton GMDH: Group Method of Data Handling GOFAI: Good Old-Fashioned AI GP: Genetic Programming GPU: Graphics Processing Unit GPU-MPCNN: GPU-Based MPCNN HMM: Hidden Markov Model HRL: Hierarchical Reinforcement Learning HTM: Hierarchical Temporal Memory HMAX: Hierarchical Model ‘‘and X’’ LSTM: Long Short-Term Memory (RNN) MDL: Minimum Description Length MDP: Markov Decision Process MNIST: Mixed National Institute of Standards and Technol- ogy Database MP: Max-Pooling MPCNN: Max-Pooling CNN NE: NeuroEvolution NEAT: NE of Augmenting Topologies NES: Natural Evolution Strategies NFQ: Neural Fitted Q-Learning NN: Neural Network OCR: Optical Character Recognition PCC: Potential Causal Connection PDCC: Potential Direct Causal Connection PM: Predictability Minimization POMDP: Partially Observable MDP RAAM: Recursive Auto-Associative Memory RBM: Restricted Boltzmann Machine ReLU: Rectified Linear Unit RL: Reinforcement Learning RNN: Recurrent Neural Network R-prop: Resilient Backpropagation SL: Supervised Learning SLIM NN: Self-Delimiting Neural Network SOTA: Self-Organizing Tree Algorithm SVM: Support Vector Machine TDNN: Time-Delay Neural Network TIMIT: TI/SRI/MIT Acoustic-Phonetic Continuous Speech Corpus UL: Unsupervised Learning WTA: Winner-Take-All 1986; Waldinger & Lee, 1969), RNNs can learn programs that mix sequential and parallel information processing in a natural and efficient way, exploiting the massive parallelism viewed as crucial for sustaining the rapid decline of computation cost observed over the past 75 years....
[...]
...CTC-LSTM performs simultaneous segmentation (alignment) and recognition (Section 5.22)....
[...]
12,299 citations
11,936 citations
Cites methods from "Connectionist temporal classificati..."
...The Connectionist Sequence Classification is another popular technique for mapping sequences to sequences with neural networks, but it assumes a monotonic alignment between the inputs and the outputs [11]....
[...]
7,316 citations
Cites background or methods from "Connectionist temporal classificati..."
...In the past CTC networks have been decoded using either a form of bestfirst decoding known as prefix search, or by simply taking the most active output at every timestep [8]....
[...]
...Instead of combining RNNs with HMMs, it is possible to train RNNs ‘end-to-end’ for speech recognition [8, 9, 10]....
[...]
...possible alignments and determine the normalised probability Pr(z|x) of the target sequence given the input sequence [8]....
[...]
...The first method, known as Connectionist Temporal Classification (CTC) [8, 9], uses a softmax layer to define a separate output distribution Pr(k|t) at every step t along the input sequence....
[...]
References
72,897 citations
"Connectionist temporal classificati..." refers background in this paper
...BLSTM combines the ability of Long Short-Term Memory (LSTM; Hochreiter & Schmidhuber, 1997) to bridge long time lags with the access of bidirectional RNNs (BRNNs; Schuster & Paliwal, 1997) to past and future context....
[...]
...Schmidhuber, 2005). BLSTM combines the ability of Long Short-Term Memory (LSTM; Hochreiter & Schmidhuber, 1997 ) to bridge long time lags with the access of bidirectional RNNs (BRNNs; Schuster & Paliwal, 1997) to past and future context....
[...]
21,819 citations
19,056 citations
"Connectionist temporal classificati..." refers background in this paper
...Note that this is the same principle underlying the standard neural network objective functions (Bishop, 1995)....
[...]
13,190 citations
"Connectionist temporal classificati..." refers background in this paper
...Currently, graphical models such as hidden Markov Models (HMMs; Rabiner, 1989), conditional random fields (CRFs; Lafferty et al., 2001) and their variants, are the predominant framework for sequence la- Appearing in Proceedings of the 23 rd International Conference on Machine Learning, Pittsburgh,…...
[...]