Simran Arora

TL;DR: The authors provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e. g.g. model architectures, training procedures, data, systems, security, evaluation, theory) to their applications.

TL;DR: This paper develops an understanding of the effective prompt formats and proposes to use weak supervision, a procedure for combining the noisy predictions, to produce the final predictions of the GPT-Neo-6B model.

TL;DR: This work defines core reasoning patterns for disambiguation, creates a learning procedure to encourage the self-supervised model to learn the patterns, and shows how to use weak supervision to enhance the signals in the training data.

TL;DR: This article study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pre-trained embedding, and an even simpler baseline (random word embedding) focusing on the impact of the training set size and the linguistic properties of the task.

TL;DR: Surprisingly, both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks.