Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning , which drastically reduces the number of task-speciﬁc training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model (PaLM). We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly eﬃcient training across multiple TPU Pods. We demonstrate continued beneﬁts of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the ﬁnetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A signiﬁcant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

PaLM: Scaling Language Modeling with Pathways

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

A Survey of Large Language Models

We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and disconvergence. In this paper, we introduce the training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B -- the largest Chinese language model -- across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization, without quantization aware training and with almost no performance loss, making it the first among 100B-scale models. More importantly, the property allows its effective inference on 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the most ever affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at https://github.com/THUDM/GLM-130B .

GLM-130B: An Open Bilingual Pre-trained Model

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\alpha$, with up to 200 billion parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$\alpha$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$\alpha$ in performing various tasks under few-shot or zero-shot settings.

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation.

The utility model discloses a flowerpot easy for plant repotting and capable of displaying water content of soil, the flowerpot is composed of a pot bottom and a pot body, the pot bottom and the pot body are connected to form a flowerpot body, the pot body is provided with a transparent visual window which is through up and down, and the inner wall of the visual window is sequentially covered witha color changing layer, a bonding layer and a water absorbing layer; an isolation layer is adsorbed on the inner wall of the flowerpot body, a hollow area corresponding to the visual window is arranged on the isolation layer, and the outer edge of the isolation layer is higher than the pot body; according to the utility model, the isolation layer is arranged, so that plants can be conveniently moved to pots; according to the flowerpot, soil scattering and plant root system damage are avoided, different colors can be shown through the arranged visual window, the color changing layer, the bonding layer and the water absorbing layer, the color changing layer, the bonding layer and the water absorbing layer are arranged in the visual window, the humidity of soil can be judged in time, and thewatering operation time and the watering amount are easier to control when plants are planted.

Flowerpot facilitating plant pot replacement and capable of displaying soil water content condition

The invention relates to a turf roll and a production method thereof, in particular to an environment-friendly and efficient turf roll and a preparation method thereof. The turf roll comprises a surface soil layer, a water retaining layer, a cultivation soil layer, a sandstone water seepage layer and a fixing layer which are adjacent to one another in sequence; the fixing layer comprises resin fibers and sodium alginate; the cultivation soil layer comprises perlite, vermiculite, agricultural waste, modified calcium carbonate and an adhesive; grass seeds are fixed in the cultivation soil layer.When the turf roll is in use, the turf roll can be integrated with the land at the lower portion, so that turf grows more stably, does not easily fall down and more easily survives, and meanwhile theeffect of sand stabilization is achieved; the ingredients of the turf roll are environment-friendly natural ingredients and cannot cause influence on the environment or the like, the grass seeds aredirectly mixed with a growth matrix, the conveying and laying processes are more convenient and quick, and the demands of various places are met.

Environment-friendly and efficient turf roll and preparation method thereof

The invention discloses a flowerpot decomposable through water absorption and a preparation method thereof. The flowerpot is prepared from, by weight, 20-45 parts of unhusked rice, 15-25 parts of humic acid, 10-20 parts of slag, 5-10 parts of modified starch, 3-5 parts of sodium alginate, 20-30 parts of clay minerals, 0.5-1 part of cellulase, 2-5 parts of a pectin substance, 0.1-0.5 part of pectinase and 5-10 parts of resin fiber. The flowerpot adopts the unhusked rice, the humic acid, the slag and other organic matter, an easily dissolved and softened material is formed through two times of firing, and an isolating membrane on the inner wall of the flowerpot can also prevent the flowerpot from being dissolved during planting watering. The prepared flowerpot is soaked in water and then issoftened and decomposed, the flowerpot and plants are both put in a larger container, the initial flowerpot is dissolved and forms an organic fertilizer easily absorbed by plants, meanwhile the step of taking out the plants to achieve pot change is omitted, and plant damage in the pot changing process is avoided.

Wang Jin

Papers

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation.

Flowerpot facilitating plant pot replacement and capable of displaying soil water content condition

Environment-friendly and efficient turf roll and preparation method thereof

Flowerpot decomposable through water absorption and preparation method thereof