Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

Pre-Trained Models: Past, Present and Future

This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems, namely, 1. Task-oriented dialog Modeling with unstructured knowledge access, 2. Multi-domain task-oriented dialog, 3. Interactive evaluation of dialog, and 4. Situated interactive multi-modal dialog. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Training machine learning models in a meaningful order, from the easy samples to the hard ones, using curriculum learning can provide performance improvements over the standard training approach based on random data shuffling, without any additional computational costs. Curriculum learning strategies have been successfully employed in all areas of machine learning, in a wide range of tasks. However, the necessity of finding a way to rank the samples from easy to hard, as well as the right pacing function for introducing more difficult data can limit the usage of the curriculum approaches. In this survey, we show how these limits have been tackled in the literature, and we present different curriculum learning instantiations for various tasks in machine learning. We construct a multi-perspective taxonomy of curriculum learning approaches by hand, considering various classification criteria. We further build a hierarchical tree of curriculum learning methods using an agglomerative clustering algorithm, linking the discovered clusters with our taxonomy. At the end, we provide some interesting directions for future work.

Curriculum Learning: A Survey

Nowadays, open-domain dialogue models can generate acceptable responses according to the historical context based on the large-scale pre-trained language models. However, they generally concatenate the dialogue history directly as the model input to predict the response, which we named as the flat pattern and ignores the dynamic information flow across dialogue utterances. In this work, we propose the DialoFlow model, in which we introduce a dynamic flow mechanism to model the context flow, and design three training objectives to capture the information dynamics across dialogue utterances by addressing the semantic influence brought about by each utterance in large-scale pre-training. Experiments on the multi-reference Reddit Dataset and DailyDialog Dataset demonstrate that our DialoFlow significantly outperforms the DialoGPT on the dialogue generation task. Besides, we propose the Flow score, an effective automatic metric for evaluating interactive human-bot conversation quality based on the pre-trained DialoFlow, which presents high chatbot-level correlation ($r=0.9$) with human ratings among 11 chatbots. Code and pre-trained models will be public. \footnote{\url{this https URL}}

Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances

Training machine learning models in a meaningful order, from the easy samples to the hard ones, using curriculum learning can provide performance improvements over the standard training approach based on random data shuffling, without any additional computational costs. Curriculum learning strategies have been successfully employed in all areas of machine learning, in a wide range of tasks. However, the necessity of finding a way to rank the samples from easy to hard, as well as the right pacing function for introducing more difficult data can limit the usage of the curriculum approaches. In this survey, we show how these limits have been tackled in the literature, and we present different curriculum learning instantiations for various tasks in machine learning. We construct a multi-perspective taxonomy of curriculum learning approaches by hand, considering various classification criteria. We further build a hierarchical tree of curriculum learning methods using an agglomerative clustering algorithm, linking the discovered clusters with our taxonomy. At the end, we provide some interesting directions for future work. 

To build a high-quality open-domain chatbot, we introduce the effective training process of PLATO-2 via curriculum learning. There are two stages involved in the learning process. In the first stage, a coarse-grained generation model is trained to learn response generation under the simplified framework of one-to-one mapping. In the second stage, a fine-grained generative model augmented with latent variables and an evaluation model are further trained to generate diverse responses and to select the best response, respectively. PLATO-2 was trained on both Chinese and English data, whose effectiveness and superiority are verified through comprehensive evaluations, achieving new state-of-the-art results.

PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning

/pdf/plato-2-towards-building-an-open-domain-chatbot-via-24z6jeegqv.pdf

The invention discloses a scene knowledge graph generation method, a man-machine conversation method and related equipment, and relates to the technical field of knowledge graphs. The specific implementation scheme of the method comprises the steps: acquiring a plurality of knowledge text content fragments of a to-be-constructed scene knowledge graph, and determining scene names of the knowledge text content fragments for the knowledge text content fragments; obtaining scene types corresponding to the scene names; determining scene element information in the knowledge text content fragments according to the scene types; and according to the scene name, the scene type and the scene element information of each knowledge text content fragment, generating the scene knowledge graph. Therefore,the scene knowledge graph based on the scene is generated in combination with the scene names, the scene types and the scene element information of the knowledge text content fragments, the relationship between knowledge is understood in combination with the scene knowledge graph conveniently, and subsequent machines can perform subsequent related processing in combination with the scene knowledgegraph conveniently.

Scene knowledge graph generation method, man-machine conversation method and related equipment

To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conversations. To train such large models, we adopt the architecture of unified transformer with high computation and parameter efficiency. In addition, we carry out multi-party aware pre-training to better distinguish the characteristic information in social media conversations. With such designs, PLATO-XL successfully achieves superior performances as compared to other approaches in both Chinese and English chitchat. We further explore the capacity of PLATO-XL on other conversational tasks, such as knowledge grounded dialogue and task-oriented conversation. The experimental results indicate that PLATO-XL obtains state-of-the-art results across multiple conversational tasks, verifying its potential as a foundation model of conversational AI.

PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation

The invention discloses a method and device for establishing a label labeling model, electronic equipment and a readable storage medium, and relates to the technical field of natural language processing. The implementation scheme adopted when the label labeling model is established comprises the steps: acquiring text data, and determining words to be labeled in all the text data; constructing a first training sample, corresponding to a word replacement task, of each piece of text data and a second training sample, corresponding to a label labeling task, according to the to-be-labeled words; and respectively using the first training sample and the second training sample to train a neural network model until the loss functions of the word replacement task and the label labeling task meet preset conditions to obtain a label labeling model. According to the method and the device, the word labeling accuracy of the label labeling model can be improved, and the label labeling model can adaptto various labeling scenes.

Xu Xinchao

Papers

PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning

PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning

Scene knowledge graph generation method, man-machine conversation method and related equipment

PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation

Method and device for establishing label labeling model, electronic equipment and readable storage medium