Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning , which drastically reduces the number of task-speciﬁc training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model (PaLM). We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly eﬃcient training across multiple TPU Pods. We demonstrate continued beneﬁts of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the ﬁnetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A signiﬁcant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

PaLM: Scaling Language Modeling with Pathways

Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit code's abundance of patterns. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.

A Survey of Machine Learning for Big Code and Naturalness

Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit the abundance of patterns of code. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.

https://dl.acm.org/doi/pdf/10.1145/3212695

When two single Rydberg atoms—those with electrons in highly excited states—interact, one can be used to control the quantum state of the other. Two independent experiments demonstrate such ‘Rydberg blockade’, an effect that might make long-range quantum gates between neutral atoms possible.

Collective excitation of two individual atoms in the Rydberg blockade regime

A machine equivalent to Turing machine, that is more intuitive in its working is defined. Three derivation rules are added to Elementary Arithmetic of Godel and his incompleteness theorems are proved without using any metalanguage. Two axioms are added to Zermelo-Fraenkel theory to derive the Continuum Hypothesis and to split the unit interval into infinitesimals.

Foundations of Computer Science

The problem of automatically fixing programming errors is a very active research topic in software engineering. This is a challenging problem as fixing even a single error may require analysis of the entire program. In practice, a number of errors arise due to programmer's inexperience with the programming language or lack of attention to detail. We call these common programming errors. These are analogous to grammatical errors in natural languages. Compilers detect such errors, but their error messages are usually inaccurate. In this work, we present an end-to-end solution, called DeepFix, that can fix multiple such errors in a program without relying on any external tool to locate or fix them. At the heart of DeepFix is a multi-layered sequence-to-sequence neural network with attention which is trained to predict erroneous program locations along with the required correct statements. On a set of 6971 erroneous C programs written by students for 93 programming tasks, DeepFix could fix 1881 (27%) programs completely and 1338 (19%) programs partially.

https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/download/14603/13921

DeepFix: Fixing Common C Language Errors by Deep Learning

We experimentally study the response of star-shaped clusters of initially unentangled N=4, 10, and 37 nuclear spin-1/2 moments to an inexact π-pulse sequence and show that an Ising coupling between the center and the satellite spins results in robust period-2 magnetization oscillations. The period is stable against bath effects, but the amplitude decays with a timescale that depends on the inexactness of the pulse. Simulations reveal a semiclassical picture in which the rigidity of the period is due to a randomizing effect of the Larmor precession under the magnetization of surrounding spins. The timescales with stable periodicity increase with net initial magnetization, even in the presence of perturbations, indicating a robust temporal ordered phase for large systems with finite magnetization per spin.

Temporal Order in Periodically Driven Spins in Star-Shaped Clusters.

Machine learning models are increasingly being deployed in practice. Machine Learning as a Service (MLaaS) providers expose such models to queries by third-party developers through application programming interfaces (APIs). Prior work has developed model extraction attacks, in which an attacker extracts an approximation of an MLaaS model by making black-box queries to it. We design ActiveThief – a model extraction framework for deep neural networks that makes use of active learning techniques and unannotated public datasets to perform model extraction. It does not expect strong domain knowledge or access to annotated data on the part of the attacker. We demonstrate that (1) it is possible to use ActiveThief to extract deep classifiers trained on a variety of datasets from image and text domains, while querying the model with as few as 10-30% of samples from public datasets, (2) the resulting model exhibits a higher transferability success rate of adversarial examples than prior work, and (3) the attack evades detection by the state-of-the-art model extraction detection method, PRADA.

/pdf/activethief-model-extraction-using-active-learning-and-5ffu0ynlsb.pdf

ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data

Machine learning models trained on confidential datasets are increasingly being deployed for profit. Machine Learning as a Service (MLaaS) has made such models easily accessible to end-users. Prior work has developed model extraction attacks, in which an adversary extracts an approximation of MLaaS models by making black-box queries to it. However, none of these works is able to satisfy all the three essential criteria for practical model extraction: (1) the ability to work on deep learning models, (2) the non-requirement of domain knowledge and (3) the ability to work with a limited query budget. We design a model extraction framework that makes use of active learning and large public datasets to satisfy them. We demonstrate that it is possible to use this framework to steal deep classifiers trained on a variety of datasets from image and text domains. By querying a model via black-box access for its top prediction, our framework improves performance on an average over a uniform noise baseline by 4.70x for image tasks and 2.11x for text tasks respectively, while using only 30% (30,000 samples) of the public dataset at its disposal.

A framework for the extraction of deep neural networks by leveraging public data

Quantum entanglement is a form of correlation between quantum particles that cannot be increased via local operations and classical communication. It has therefore been proposed that an increment of quantum entanglement between probes that are interacting solely via a mediator implies non-classicality of the mediator. Indeed, under certain assumptions regarding the initial state, entanglement gain between the probes indicates quantum coherence in the mediator. Going beyond such assumptions, there exist other initial states which produce entanglement between the probes via only local interactions with a classical mediator. In this process the initial entanglement between any probe and the rest of the system "flows through" the classical mediator and gets localised between the probes. Here we theoretically characterise maximal entanglement gain via classical mediator and experimentally demonstrate, using liquid-state NMR spectroscopy, the optimal growth of quantum correlations between two nuclear spin qubits interacting through a mediator qubit in a classical state. We additionally monitor, i.e., dephase, the mediator in order to emphasise its classical character. Our results indicate the necessity of verifying features of the initial state if entanglement gain between the probes is used as a figure of merit for witnessing non-classical mediator. Such methods were proposed to have exemplary applications in quantum optomechanics, quantum biology and quantum gravity.

/pdf/experimental-localisation-of-quantum-entanglement-through-25ybhn0vsw.pdf

Soham Pal

Papers

DeepFix: Fixing Common C Language Errors by Deep Learning

Temporal Order in Periodically Driven Spins in Star-Shaped Clusters.

ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data

A framework for the extraction of deep neural networks by leveraging public data

Experimental localisation of quantum entanglement through monitored classical mediator