H
HyoukJoong Lee
Researcher at Google
Publications - 30
Citations - 3328
HyoukJoong Lee is an academic researcher from Google. The author has contributed to research in topics: Compiler & Domain-specific language. The author has an hindex of 23, co-authored 30 publications receiving 2539 citations. Previous affiliations of HyoukJoong Lee include Stanford University.
Papers
More filters
Posted Content
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang,Youlong Cheng,Ankur Bapna,Orhan Firat,Mia Xu Chen,Dehao Chen,HyoukJoong Lee,Jiquan Ngiam,Quoc V. Le,Yonghui Wu,Zhifeng Chen +10 more
TL;DR: GPipe is introduced, a pipeline parallelism library that allows scaling any network that can be expressed as a sequence of layers by pipelining different sub-sequences of layers on separate accelerators, resulting in almost linear speedup when a model is partitioned across multiple accelerators.
Proceedings Article
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang,Youlong Cheng,Ankur Bapna,Orhan Firat,Dehao Chen,Mia Xu Chen,HyoukJoong Lee,Jiquan Ngiam,Quoc V. Le,Yonghui Wu,Zhifeng Chen +10 more
TL;DR: TensorPipe as mentioned in this paper is a pipeline parallelism library that allows scaling any network that can be expressed as a sequence of layers by pipelining different sub-sequences of layers on separate accelerators.
Posted Content
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Dmitry Lepikhin,HyoukJoong Lee,Yuanzhong Xu,Dehao Chen,Orhan Firat,Yanping Huang,Maxim Krikun,Noam Shazeer,Zhifeng Chen +8 more
TL;DR: GShard enabled us to scale up multilingual neural machine translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600 billion parameters using automatic sharding and it is demonstrated that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art.
Posted Content
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Jonathan Shen,Patrick Nguyen,Yonghui Wu,Zhifeng Chen,Mia Xu Chen,Ye Jia,Anjuli Kannan,Tara N. Sainath,Yuan Cao,Chung-Cheng Chiu,Yanzhang He,Jan Chorowski,Smit Hinsu,Stella Marie Laurenzo,James Qin,Orhan Firat,Wolfgang Macherey,Suyog Gupta,Ankur Bapna,Shuyuan Zhang,Ruoming Pang,Ron Weiss,Rohit Prabhavalkar,Qiao Liang,Benoit Jacob,Bowen Liang,HyoukJoong Lee,Ciprian Chelba,Sébastien Jean,Bo Li,Melvin Johnson,Rohan Anil,Rajat Tibrewal,Xiaobing Liu,Akiko Eriguchi,Navdeep Jaitly,Naveen Ari,Colin Cherry,Parisa Haghani,Otavio Good,Youlong Cheng,Raziel Alvarez,Isaac Caswell,Wei-Ning Hsu,Zongheng Yang,Kuan-Chieh Wang,Ekaterina Gonina,Katrin Tomanek,Ben Vanik,Zelin Wu,Llion Jones,Mike Schuster,Yanping Huang,Dehao Chen,Kazuki Irie,George Foster,John Richardson,Klaus Macherey,Antoine Bruguier,Heiga Zen,Colin Raffel,Shankar Kumar,Kanishka Rao,David Rybach,Matthew Murray,Vijayaditya Peddinti,Maxim Krikun,Michiel Bacchiani,Thomas B. Jablin,Robert Suderman,Ian Williams,Benjamin N. Lee,Deepti Bhatia,Justin Carlson,Semih Yavuz,Yu Zhang,Ian McGraw,Max Galkin,Qi Ge,Golan Pundak,Chad Whipkey,Todd Wang,Uri Alon,Dmitry Lepikhin,Ye Tian,Sara Sabour,William Chan,Shubham Toshniwal,Baohua Liao,Michael Nirschl,Pat Rondon +90 more
TL;DR: This document outlines the underlying design of Lingvo and serves as an introduction to the various pieces of the framework, while also offering examples of advanced features that showcase the capabilities of the Framework.
Proceedings Article
OptiML: An Implicitly Parallel Domain-Specific Language for Machine Learning
Arvind K. Sujeeth,HyoukJoong Lee,Kevin J. Brown,Tiark Rompf,Hassan Chafi,Michael Wu,Anand R. Atreya,Martin Odersky,Kunle Olukotun +8 more
TL;DR: OptiML is an implicitly parallel, expressive and high performance alternative to MATLAB and C++ and shows that OptiML outperforms explicitly parallelized MATLAB code in nearly all cases.