A fast learning algorithm for deep belief nets
Citations
16,717 citations
14,799 citations
Cites background or methods from "A fast learning algorithm for deep ..."
...Restricted Boltzmann machines (RBMs) have been used as generative models of many different types of data including labeled or unlabeled images (Hinton et al., 2006), sequences of mel-cepstral coefficients that represent speech (Mohamed & Hinton, 2010), bags of words that represent documents (Salakhutdinov & Hinton, 2009), and user ratings of movies (Salakhutdinov et al....
[...]
...Restricted Boltzmann machines (RBMs) have been used as generative models of many different types of data including labeled or unlabeled images (Hinton et al., 2006), sequences of mel-cepstral coefficients that represent speech (Mohamed & Hinton, 2010), bags of words that represent documents…...
[...]
...Pretraining is done for 300 epochs (in both layers), using mini-batches of 100 training examples with a learning rate of 10 applied to the average per-case CD update, along with momentum (Hinton et al., 2006)....
[...]
...Their most important use is as learning modules that are composed to form deep belief nets (Hinton et al., 2006)....
[...]
...Pretraining is done for 300 epochs (in both layers), using mini-batches of 100 training examples with a learning rate of 10−3 applied to the average per-case CD update, along with momentum (Hinton et al., 2006)....
[...]
14,635 citations
12,783 citations
Cites background or methods from "A fast learning algorithm for deep ..."
...As such, this is a form of supervised pre-training, which contrasts with the unsupervised pre-training methods popularized by (Hinton et al., 2006) and others (Bengio et al....
[...]
...As such, this is a form of supervised pre-training, which contrasts with the unsupervised pre-training methods popularized by (Hinton et al., 2006) and others (Bengio et al., 2007; Vincent et al., 2008)....
[...]
11,201 citations
Cites background or methods from "A fast learning algorithm for deep ..."
...Object Recognition The beginnings of deep learning in 2006 have focused on the MNIST digit image classification problem (Hinton et al., 2006; Bengio et al., 2007), breaking the supremacy of SVMs (1.4% error) on this dataset3....
[...]
...The first proposal was to stack pre-trained RBMs into a Deep Belief Network (Hinton et al., 2006) or DBN, where the top layer is interpreted as an RBM and the lower layers as a directed sigmoid belief network....
[...]
...One option is the wake-sleep algorithm (Hinton et al., 2006) but more work should be done to assess the efficiency of this procedure in terms of improving the generative model....
[...]
...Contrastive Divergence (Hinton, 1999; Hinton et al., 2006), Stochastic Maximum Likelihood (Younes, 1999; Tieleman, 2008) and fast-weights persistent contrastive divergence or FPCD (Tieleman and Hinton, 2009) are all ways to avoid or reduce the need for burn-in....
[...]
...In 2006, a breakthrough in feature learning and deep learning was initiated by Geoff Hinton and quickly followed up in the same year (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), and soon after by Lee et al. (2008) and many more later....
[...]
References
42,067 citations
15,671 citations
6,693 citations
5,150 citations
2,783 citations
"A fast learning algorithm for deep ..." refers methods in this paper
...Using local elastic deformations in a convolutional neural network, Simard, Steinkraus, and Platt (2003) achieve 0.4%, which is slightly better than the 0.63% achieved by the best hand-coded recognition algorithm (Belongie, Malik, & Puzicha, 2002)....
[...]
...Using local elastic deformations in a convolutional neural network, Simard, Steinkraus, and Platt (2003) achieve 0.4%, which is slightly better than the 0.63% achieved by the best hand-coded recognition algorithm (Belongie, Malik, & Puzicha, 2002)....
[...]