R
Roman Novak
Researcher at Google
Publications - 24
Citations - 3332
Roman Novak is an academic researcher from Google. The author has contributed to research in topics: Artificial neural network & Gaussian process. The author has an hindex of 16, co-authored 20 publications receiving 2106 citations.
Papers
More filters
Proceedings Article
Deep Neural Networks as Gaussian Processes
Jaehoon Lee,Yasaman Bahri,Roman Novak,Samuel S. Schoenholz,Jeffrey Pennington,Jascha Sohl-Dickstein +5 more
TL;DR: The exact equivalence between infinitely wide deep networks and GPs is derived and it is found that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite- width networks.
Journal ArticleDOI
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Jaehoon Lee,Lechao Xiao,Samuel S. Schoenholz,Yasaman Bahri,Roman Novak,Jascha Sohl-Dickstein,Jeffrey Pennington +6 more
TL;DR: In this article, the authors show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters.
Journal Article
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava,Abhinav Rastogi,Abhishek Rao,Abu Awal Md Shoeb,Abubakar Abid,Adam Fisch,Adam R. Brown,Adam Santoro,Aditya Gupta,Adrià Garriga-Alonso,Agnieszka Kluska,Aitor Lewkowycz,Akshat Agarwal,Alethea. Power,Alex Ray,Alex Warstadt,Alexander W. Kocurek,Ali Safaya,Ali Tazarv,Alice Xiang,Alicia Parrish,Allen Nie,Aman Hussain,Amanda Askell,Amanda Dsouza,Ameet Annasaheb Rahane,Anantharaman S. Iyer,Anders Andreassen,Andrea Santilli,Andreas Stuhlmuller,Andrew M. Dai,Andrew D. La,Andrew K. Lampinen,Andy Zou,Angela Jiang,Angelica Chen,Anh Vuong,Animesh Gupta,Anna Gottardi,Antonio Norelli,Anushree Venkatesh,Arash Gholamidavoodi,Arfa Tabassum,Arul Menezes,Arun Kirubarajan,Asher Mullokandov,Ashish Sabharwal,Austin Herrick,Avia Efrat,Aykut Erdem,Ayla Karakacs,Bridget R. Roberts,Bao Sheng Loe,Barret Zoph,Bartlomiej Bojanowski,Batuhan Ozyurt,Behnam Hedayatnia,Behnam Neyshabur,Benjamin Inden,Benno Stein,Berk Ekmekci,Bill Yuchen Lin,Blake Howald,Cameron Diao,Cameron Dour,Catherine Stinson,Cedrick Argueta,C'esar Ferri Ram'irez,Chandan Singh,Charles Rathkopf,Chenlin Meng,Chitta Baral,Chiyu Wu,Chris Callison-Burch,Chris Waites,Christian Voigt,Christopher D. Manning,C. W. Potts,Cindy Tatiana Ramirez,Clara E. Rivera,Clemencia Siro,Colin Raffel,Courtney Ashcraft,Cristina Gârbacea,Damien Sileo,Daniel H Garrette,Dan Hendrycks,D. Kilman,Dan Roth,Daniel Freeman,Daniel Khashabi,Daniel Levy,Daniel Gonz'alez,Danny Hernandez,Danqi Chen,Daphne Ippolito,Dar Gilboa,David Dohan,D. Drakard,David A. Jurgens,Debajyoti Datta,Deep Ganguli,Denis Emelin,Denis Kleyko,Deniz Yuret,Derek Chen,Derek Tam,Dieuwke Hupkes,Diganta Misra,Dilyar Buzan,Dimitri Coelho Mollo,Diyi Yang,Dongho Lee,E P Shutova,Ekin D. Cubuk,Elad Segal,Eleanor Hagerman,Elizabeth A. Barnes,Elizabeth P. Donoway,Ellie Pavlick,Emanuele Rodolà,Emma F C Lam,Eric Chu,Eric Tang,Erkut Erdem,Ernie Chang,Ethan A. Chi,Ethan Dyer,Ethan Jerzak,Ethan Kim,Eunice Engefu Manyasi,Evgenii Zheltonozhskii,Fan Xia,F. Siar,Fernando Mart'inez-Plumed,Francesca Happ'e,Francois Chollet,Frieda Rong,Gaurav Mishra,Genta Indra Winata,Gerard de Melo,G. Kruszewski,Giambattista Parascandolo,Giorgio Mariani,Gloria Wang,Gonzalo Jaimovitch-L'opez,Gregor Betz,Guy Gur-Ari,Hana Galijasevic,Han Sol Kim,Hannah Rashkin,Hanna Hajishirzi,Harsh Mehta,H. Bogar,Henry Shevlin,Hinrich Schuetze,Hiromu Yakura,Hongming Zhang,Hubert Wong,Ian A. S. Ng,Isaac Noble,Julien Jumelet,Jack Geissinger,John Kernion,Jacob Hilton,Jae-Hoon Lee,Jaime F. Fisac,J. Brooker Simon,James Koppel,James Zheng,James Zou,Jan Koco'n,Jana Thompson,Jared Kaplan,Jarema Radom,Jascha Sohl-Dickstein,Jason Phang,Jason Loh Seong Wei,Jason Yosinski,Jekaterina Novikova,Jelle Bosscher,Jennifer Violet Marsh,Jeremy Kim,Jeroen Taal,Jesse Engel,Jesujoba O. Alabi,Jiacheng Xu,Jiaming Song,Jillian Tang,Jane W Waweru,John Burden,John A. Miller,John U. Balis,Jonathan Berant,Jorg Frohberg,Jos Rozen,José Hernández-Orallo,Joseph Boudeman,Josephine R. Jones,Joshua B. Tenenbaum,Joshua Rule,Joyce Hui Ping Chua,Kamil Kanclerz,Karen Livescu,Karl Krauth,Karthik Gopalakrishnan,Katerina Ignatyeva,Katja Markert,Kaustubh Dhole,K Gimpel,Kevin O Omondi,K. Mathewson,Kristen Chiafullo,Ksenia Shkaruta,Kumar Shridhar,Kyle McDonell,Kyle Richardson,Laria Reynolds,Leo Gao,Li Zhang,Liam Dugan,Lianhui Qin,Lidia Contreras-Ochando,Louis-Philippe Morency,Luca Moschella,Luca Lam,Lucy Noble,Ludwig Schmidt,Luheng He,Luis Oliveros Col'on,Luke Metz,Lutfi Kerem cSenel,Maarten Bosma,Maarten Sap,Maartje ter Hoeve,Madotto Andrea,Maheen Saleem Farooqi,Manaal Faruqui,Mantas Mazeika,Marco Baturan,Marco Marelli,Marco Maru,M Quintana,Marie Tolkiehn,Mario Giulianelli,M. Lewis,M. Potthast,Matthew Leavitt,Matthias Hagen,M. Schubert,Medina Baitemirova,M Arnaud,Melvin Andrew McElrath,Michael A Yee,Michael Cohen,Mi-jin Gu,Michael I. Ivanitskiy,Michael Starritt,Michael Strube,Michal Swkedrowski,Michele Bevilacqua,Michihiro Yasunaga,Mihir Kale,Michael D. Cain,Mimee Xu,Mirac M. Suzgun,Monica Tiwari,Mohit Bansal,Moin Aminnaseri,Mor Geva,Mozhdeh Gheini,T. MukundVarma,Nanyun Peng,Nathan Chi,Nayeon Lee,Neta Gur-Ari Krakover,Nicholas Cameron,Nicholas S. Roberts,Nicholas Doiron,Nikita Nangia,Niklas Deckers,Niklas Muennighoff,Nitish Shirish Keskar,Niveditha Iyer,Noah Constant,Noah Fiedel,Nuan Wen,Oliver Zhang,Omar Agha,Omar Elbaghdadi,Omer Levy,Owain Evans,P. A. M. Casares,P. S. Doshi,Pascale Fung,Paul Pu Liang,Paul Adrian Vicol,Pegah Alipoormolabashi,Peiyuan Liao,Percy Liang,Peter W. Chang,Peter Eckersley,Phu Mon Htut,Pi-Bei Hwang,Piotr Miłkowski,P. T. Patil,Pouya Pezeshkpour,Priti Oli,Qiaozhu Mei,Qing Lyu,Qinlang Chen,Rabin Banjade,Rachel E. Rudolph,Raefer Gabriel,Rahel Habacker,Ramon Delgado,Raphael Milliere,Rhythm Garg,Richard Barnes,Rif A. Saurous,Riku Arakawa,Robbe Raymaekers,Robert Frank,Rohan Sikand,Roman Novak,Roman Sitelew,Ronan LeBras,Rosanne Liu,Rowan Jacobs,Rui Zhang,Ruslan Salakhutdinov,Ryan Chi,Ryan Lee,Ryan Stovall,Ryan Teehan,Rylan Yang,Sahib Singh,Saif M. Mohammad,Sajant Anand,Sam Dillavou,Sam Shleifer,Sam Wiseman,Samuel Gruetter,Sam W. Bowman,Samuel S. Schoenholz,Sanghyun Han,Sanjeev Kwatra,Sarah Adler Rous,Sarik Ghazarian,Sayan Ghosh,Sean Casey,Sebastian Bischoff,Sebastian Gehrmann,Sebastian Schuster,Sepideh Sadeghi,Shadi Sameh Hamdan,Sharon Zhou,S. K. Srivastava,Sherry Shi,Shikhar Kumar Singh,Shima Asaadi,Shixiang Gu,Shubh Pachchigar,Shubham Toshniwal,Shyam Upadhyay,Shyamolima Debnath,Siamak Shakeri,Simon Thormeyer,Simone Melzi,Siva Koti Reddy,Sneha Priscilla Makini,Soo-Hwan Lee,Spencer Bradley Torene,Sriharsha Hatwar,Stanislas Dehaene,Stefan Divic,Stefano Ermon,Stella Biderman,Stephanie C. Lin,S. Prasad,Steven Piantadosi,Stuart M. Shieber,Summer Misherghi,Svetlana Kiritchenko,Swaroop Mishra,Tal Linzen,T. Schuster,Tao Li,Tariq Ali,Tatsuo Hashimoto,Teng Wu,Theo Desbordes,Theodore Rothschild,Thomas Ngoc Phan,Tianle Wang,Tiberius Tabulu Nkinyili,Timo Schick,T. N. Kornev,Timothy Telleen-Lawton,T. Tunduny,Tobias Gerstenberg,T.P. Chang,Trishala Neeraj,Tushar Khot,T. Shultz,Vedant Misra,Vera Demberg,Victoria Nyamai,Vikas Raunak,Vinay Ramasesh,Vinay Uday Prabhu,Vishakh Padmakumar,V. Srikumar,William Fedus,William Saunders,William Zhang,W. Vossen,Xiangyuan Ren,Xiaoyu Tong,Xinyi Wu,Xudong Shen,Yadollah Yaghoobzadeh,Yair Lakretz,Yang Song,Yasaman Bahri,Ye Ji Choi,Yichi Yang,Yiding Hao,Yifu Chen,Yonatan Belinkov,Yu Hou,Yuntao Bai,Zachary Seid,Zhao Xin-ran,Zhuoye Zhao,Zi Fu Wang,Zijie J. Wang,Ziyi Wu,Sahib Singh,Uri Shaham +439 more
TL;DR: Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.
Proceedings Article
Sensitivity and Generalization in Neural Networks: an Empirical Study
TL;DR: In this article, the authors investigate the tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations, and demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points.
Proceedings Article
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Roman Novak,Lechao Xiao,Jaehoon Lee,Yasaman Bahri,Greg Yang,Jiri Hron,Daniel A. Abolafia,Jeffrey Pennington,Jascha Sohl-Dickstein +8 more
TL;DR: This work derives an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and introduces a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.