Open AccessPosted Content
Neural Architecture Search with Reinforcement Learning
Barret Zoph,Quoc V. Le +1 more
Reads0
Chats0
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.Abstract:
Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.read more
Citations
More filters
Book ChapterDOI
Real Image Super Resolution Via Heterogeneous Model Ensemble using GP-NAS
TL;DR: In this paper, a Gaussian process based neural architecture search (GP-NAS) scheme is utilized to find candidate network architectures using a large search space by varying the number of dense residual blocks, the block size and the number features.
Proceedings Article
Auto-scaling Vision Transformers without Training
TL;DR: As-ViT is proposed, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner and proposes a progressive tokenization strategy to train ViTs faster and cheaper.
Journal Article
A Surgery of the Neural Architecture Evaluators
Xuefei Ning,Wenshuo Li,Zixuan Zhou,Tianchen Zhao,Shuang Liang,Yin Zheng,Huazhong Yang,Yu Wang +7 more
TL;DR: This paper conducts an extensive assessment of both the one-shot and predictor-based evaluator on the NAS-Bench-201 benchmark search space, and breaks up how and why different factors influence the evaluation correlation and other NAS-oriented criteria.
Posted Content
Applications and Techniques for Fast Machine Learning in Science
Allison McCarn Deiana,Nhan Tran,Joshua Agar,Michaela Blott,Giuseppe Di Guglielmo,Javier Duarte,Philip Harris,Scott Hauck,Mia Liu,Mark Neubauer,Jennifer Ngadiuba,Seda Ogrenci-Memik,Maurizio Pierini,Thea Klaeboe Aarrestad,Steffen Bahr,Jurgen Becker,Anne-Sophie Berthold,Richard J. Bonventre,Tomas E. Muller Bravo,Markus Diefenthaler,Zhen Dong,Nick Fritzsche,Amir Gholami,Ekaterina Govorkova,Kyle J Hazelwood,Christian Herwig,Babar Khan,Sehoon Kim,Thomas Klijnsma,Yaling Liu,Kin Ho Lo,Tri Minh Nguyen,Gianantonio Pezzullo,Seyedramin Rasoulinezhad,Ryan A. Rivera,Kate Scholberg,Justin Selig,Sougata Sen,Dmitri Strukov,William Tang,Savannah Thais,Kai Lukas Unger,Ricardo Vilalta,Belinavon Krosigk,Thomas K. Warburton,Maria Acosta Flechas,Anthony Aportela,Thomas Calvet,Leonardo Cristella,Daniel Diaz,Caterina Doglioni,Maria Domenica Galati,Elham E Khoda,Farah Fahim,Davide Giri,Benjamin Hawks,Duc Hoang,Burt Holzman,Shih-Chieh Hsu,Sergo Jindariani,Iris Johnson,Raghav Kansal,Ryan Kastner,Erik Katsavounidis,Jeffrey Krupa,Pan Li,Sandeep Madireddy,Ethan Marx,Patrick McCormack,Andres Meza,Jovan Mitrevski,Mohammed Attia Mohammed,Farouk Mokhtar,Eric Moreno,Srishti Nagu,Rohin Narayan,Noah Palladino,Zhiqiang Que,Sang Eon Park,Subramanian Ramamoorthy,Dylan Rankin,Simon Rothman,Ashish Sharma,Sioni Summers,Pietro Vischia,Jean-Roch Vlimant,Olivia Weng +86 more
TL;DR: In this article, the authors discuss applications and techniques for fast machine learning (ML) in science, the concept of integrating power ML methods into the real-time experimental data processing loop to accelerate scientific discovery.
Journal ArticleDOI
Discriminating glaucomatous and compressive optic neuropathy on spectral-domain optical coherence tomography with deep learning classifier
Jinho Lee,Jin Soo Kim,Haeng Jin Lee,Seong Joon Kim,Young Kook Kim,Ki-Ho Park,Jin Wook Jeoung +6 more
TL;DR: The deep learning classifier can outperform the conventional diagnostic parameters for discrimination of GON and CON on SD-OCT and achieve a sensitivity and specificity higher than expected.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Journal ArticleDOI
Gradient-based learning applied to document recognition
Yann LeCun,Léon Bottou,Léon Bottou,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio,Patrick Haffner +6 more
TL;DR: In this article, a graph transformer network (GTN) is proposed for handwritten character recognition, which can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters.
Proceedings ArticleDOI
Histograms of oriented gradients for human detection
Navneet Dalal,Bill Triggs +1 more
TL;DR: It is shown experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection, and the influence of each stage of the computation on performance is studied.