Nikoli Dryden

TL;DR: This work port two existing quantization approaches, one-bit and threshold, and develop their own adaptive quantization algorithm, which is comparable or superior for large layers without sacrificing accuracy and achieves near-linear speedup in data-parallel training.

TL;DR: This paper introduces a new approach to building distributed-memory graph analytics systems that exploits heterogeneity in processor types (CPU and GPU), partitioning policies, and programming models, and Gluon, a communication-optimizing substrate that enables these programs to run on heterogeneous clusters and optimizes communication in a novel way.

TL;DR: A mixed model that uses only a subset of the original weather trajectories combined with a post-processing step using deep neural networks enables the model to account for non-linear relationships that are not captured by current numerical models or post- processing methods.

TL;DR: This work finds that data movement is the key bottleneck when training, and presents a recipe for globally optimizing data movement in transformers to achieve a 1.30x performance improvement over state-of-the-art frameworks when training BERT.

TL;DR: This work introduces three algorithms that partition channel or filter data to exploit parallelism beyond the sample dimension, and partition the parameters of convolutional layers, replacing global all reduces with segmented allreduces among disjoint processor sets.