Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

In Distributed Algorithms, Nancy Lynch provides a blueprint for designing, implementing, and analyzing distributed algorithms. She directs her book at a wide audience, including students, programmers, system designers, and researchers.



Distributed Algorithms contains the most significant algorithms and impossibility results in the area, all in a simple automata-theoretic setting. The algorithms are proved correct, and their complexity is analyzed according to precisely defined complexity measures. The problems covered include resource allocation, communication, consensus among distributed processes, data consistency, deadlock detection, leader election, global snapshots, and many others.



The material is organized according to the system model-first by the timing model and then by the interprocess communication mechanism. The material on system models is isolated in separate chapters for easy reference.



The presentation is completely rigorous, yet is intuitive enough for immediate comprehension. This book familiarizes readers with important problems, algorithms, and impossibility results in the area: readers can then recognize the problems when they arise in practice, apply the algorithms to solve them, and use the impossibility results to determine whether problems are unsolvable. The book also provides readers with the basic mathematical tools for designing new algorithms and proving new impossibility results. In addition, it teaches readers how to reason carefully about distributed algorithms-to model them formally, devise precise specifications for their required behavior, prove their correctness, and evaluate their performance with realistic measures.


Table of Contents

1 Introduction 
2 Modelling I; Synchronous Network Model 
3 Leader Election in a Synchronous Ring 
4 Algorithms in General Synchronous Networks 
5 Distributed Consensus with Link Failures 
6 Distributed Consensus with Process Failures 
7 More Consensus Problems 
8 Modelling II: Asynchronous System Model 
9 Modelling III: Asynchronous Shared Memory Model 
10 Mutual Exclusion 
11 Resource Allocation 
12 Consensus 
13 Atomic Objects 
14 Modelling IV: Asynchronous Network Model 
15 Basic Asynchronous Network Algorithms 
16 Synchronizers 
17 Shared Memory versus Networks 
18 Logical Time 
19 Global Snapshots and Stable Properties 
20 Network Resource Allocation 
21 Asynchronous Networks with Process Failures 
22 Data Link Protocols 
23 Partially Synchronous System Models 
24 Mutual Exclusion with Partial Synchrony 
25 Consensus with Partial Synchrony

Distributed algorithms

An Integrated Agent Architecture for Software Defined Radio. Rapid-prototype cognitive radio, CR1, was developed to apply these.The modern software defined radio has been called the heart of a cognitive radio. Cognitive radio: an integrated agent architecture for software defined radio. Http:bwrc.eecs.berkeley.eduResearchMCMACR White paper final1.pdf. The cognitive radio, built on a software-defined radio, assumes. Radio: An Integrated Agent Architecture for Software Defined Radio, Ph.D. The need for software-defined radios is underlined and the most important notions used for such. Mitola III, Cognitive radio: an integrated agent architecture for software defined radio, Ph.D. This results in the set-theoretic ontology of radio knowledge defined in the. Cognitive Radio An Integrated Agent Architecture for Software.This article first briefly reviews the basic concepts about cognitive radio CR. Cognitive Radio-An Integrated Agent Architecture for Software Defined Radio. Cognitive Radio RHMZ 2007. Software-defined radio SDR idea 1. Cognitive radio: An integrated agent architecture for software.Cognitive Radio SOFTWARE DEFINED RADIO, AND ADAPTIVE WIRELESS SYSTEMS2 Cognitive Networks. 3 Joseph Mitola III, Cognitive Radio: An Integrated Agent Architecture for Software Defined Radio Stockholm.

/pdf/cognitive-radio-an-integrated-agent-architecture-for-1v855bviue.pdf

Cognitive Radio An Integrated Agent Architecture for Software Defined Radio

Motivation: Counting the number of occurrences of every k-mer (substring of length k) in a long string is a central subproblem in many applications, including genome assembly, error correction of sequencing reads, fast multiple sequence alignment and repeat detection. Recently, the deep sequence coverage generated by next-generation sequencing technologies has caused the amount of sequence to be processed during a genome project to grow rapidly, and has rendered current k-mer counting tools too slow and memory intensive. At the same time, large multicore computers have become commonplace in research facilities allowing for a new parallel computational paradigm.

Results: We propose a new k-mer counting algorithm and associated implementation, called Jellyfish, which is fast and memory efficient. It is based on a multithreaded, lock-free hash table optimized for counting k-mers up to 31 bases in length. Due to their flexibility, suffix arrays have been the data structure of choice for solving many string problems. For the task of k-mer counting, important in many biological applications, Jellyfish offers a much faster and more memory-efficient solution.

Availability: The Jellyfish software is written in C++ and is GPL licensed. It is available for download at http://www.cbcb.umd.edu/software/jellyfish.

Contact: [email protected]

Supplementary information:Supplementary data are available at Bioinformatics online.

/pdf/a-fast-lock-free-approach-for-efficient-parallel-counting-of-3tjx1nqa3f.pdf

A fast, lock-free approach for efficient parallel counting of occurrences of k-mers

The transactional memory programming paradigm is gaining momentum as the approach of choice for replacing locks in concurrent programming. This paper introduces the transactional locking II (TL2) algorithm, a software transactional memory (STM) algorithm based on a combination of commit-time locking and a novel global version-clock based validation technique. TL2 improves on state-of-the-art STMs in the following ways: (1) unlike all other STMs it fits seamlessly with any systems memory life-cycle, including those using malloc/free (2) unlike all other lock-based STMs it efficiently avoids periods of unsafe execution, that is, using its novel version-clock validation, user code is guaranteed to operate only on consistent memory states, and (3) in a sequence of high performance benchmarks, while providing these new properties, it delivered overall performance comparable to (and in many cases better than) that of all former STM algorithms, both lock-based and non-blocking. Perhaps more importantly, on various benchmarks, TL2 delivers performance that is competitive with the best hand-crafted fine-grained concurrent structures. Specifically, it is ten-fold faster than a single lock. We believe these characteristics make TL2 a viable candidate for deployment of transactional memory today, long before hardware transactional support is available.

Transactional locking II

The transactional memory programming paradigm is gaining momentum as the approach of choice for replacing locks in concurrent programming. This paper introduces the transactional locking II (TL2) algorithm, a software transactional memory (STM) algorithm based on a combination of commit-time locking and a novel global version-clock based validation technique. TL2 improves on state-of-the-art STMs in the following ways: (1) unlike all other STMs it fits seamlessly with any system's memory life-cycle, including those using malloc/free (2) unlike all other lock-based STMs it efficiently avoids periods of unsafe execution, that is, using its novel version-clock validation, user code is guaranteed to operate only on consistent memory states, and (3) in a sequence of high performance benchmarks, while providing these new properties, it delivered overall performance comparable to (and in many cases better than) that of all former STM algorithms, both lock-based and non-blocking. Perhaps more importantly, on various benchmarks, TL2 delivers performance that is competitive with the best hand-crafted fine-grained concurrent structures. Specifically, it is ten-fold faster than a single lock. We believe these characteristics make TL2 a viable candidate for deployment of transactional memory today, long before hardware transactional support is available.

/pdf/transactional-locking-ii-1n416uini4.pdf

We give necessary and sufficient combinatorial conditions characterizing the tasks that can be solved by asynchronous processes, of which all but one can fail, that communicate by reading and writing a shared memory. We introduce a new formalism for tasks, based on notions from classical algebraic and combinatorial topology, in which a task''s possible input and output values are each associated with high-dimensional geometric structures called simplicial complexes. We characterize computability in terms of the topological properties of these complexes. This characterization has a surprising geometric interpretation: a task is solvable if and only if the complex representing the task''s allowable inputs can be mapped to the complex representing the task''s allowable outputs by a function satisfying certain simple regularity properties. Our formalism thus replaces the ``operational'''' notion of a wait-free decision task, expressed in terms of interleaved computations unfolding in time, by a static ``combinatorial'''' description expressed in terms of relations among topological spaces, allowing us to exploit powerful theorems from the classic literature on algebraic and combinatorial topology. This approach yields the first impossibility results for several long-standing open problems in distributed computing, such as the ``renaming'''' problem of Attiya et al., and the ``$k$-set agreement'''' problem of Chaudhuri.

/pdf/the-topological-structure-of-asynchronous-computability-1ey1twqclg.pdf

The topological structure of asynchronous computability

Deep neural networks trained on large supervised datasets have led to impressive results in image classification and other tasks. However, well-annotated datasets can be time-consuming and expensive to collect, lending increased interest to larger but noisy datasets that are more easily obtained. In this paper, we show that deep neural networks are capable of generalizing from training data for which true labels are massively outnumbered by incorrect labels. We demonstrate remarkably high test performance after training on corrupted data from MNIST, CIFAR, and ImageNet. For example, on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples. Such behavior holds across multiple patterns of label noise, even when erroneous labels are biased towards confusing classes. We show that training in this regime requires a significant but manageable increase in dataset size that is related to the factor by which correct labels have been diluted. Finally, we provide an analysis of our results that shows how increasing noise decreases the effective batch size.

/pdf/deep-learning-is-robust-to-massive-label-noise-4om9xxhu48.pdf

Deep Learning is Robust to Massive Label Noise

This paper introduces a general formulation of atomic snapshot memory, a shared memory partitioned into words written (updated) by individual processes, or instantaneously read (scanned) in its entirety. This paper presents three wait-free implementations of atomic snapshot memory. The first implementation in this paper uses unbounded (integer) fields in these registers, and is particularly easy to understand. The second implementation uses bounded registers. Its correctness proof follows the ideas of the unbounded implementation. Both constructions implement a single-writer snapshot memory, in which each word may be updated by only one process, from single-writer, n-reader registers. The third algorithm implements a multi-writer snapshot memory from atomic n-writer, n-reader registers, again echoing key ideas from the earlier constructions. All operations require Θ(n2) reads and writes to the component shared registers in the worst case. —Authors' Abstract

Nir Shavit

Papers

Transactional locking II

Transactional locking II

The topological structure of asynchronous computability

Deep Learning is Robust to Massive Label Noise

Atomic snapshots of shared memory