Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions.

/pdf/learning-to-rank-for-information-retrieval-2q6t3viyug.pdf

Learning to Rank for Information Retrieval

The paper is concerned with learning to rank, which is to construct a model or a function for ranking objects. Learning to rank is useful for document retrieval, collaborative filtering, and many other applications. Several methods for learning to rank have been proposed, which take object pairs as 'instances' in learning. We refer to them as the pairwise approach in this paper. Although the pairwise approach offers advantages, it ignores the fact that ranking is a prediction task on list of objects. The paper postulates that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning. The paper proposes a new probabilistic method for the approach. Specifically it introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning. Neural Network and Gradient Descent are then employed as model and algorithm in the learning method. Experimental results on information retrieval show that the proposed listwise approach performs better than the pairwise approach.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2007-40.pdf

Learning to rank: from pairwise approach to listwise approach

The mitochondrial calcium uniporter imports cytoplasmic Ca2+ into the mitochondrial matrix to regulate cell bioenergetics, Ca2+ signaling, and apoptosis. The uniporter contains the pore-forming MCU subunit, an EMRE protein that binds to MCU, and the regulatory MICU1/MICU2 subunits. Structural and biochemical studies have suggested that MICU1 gates MCU by blocking and unblocking the Ca2+ pore. However, mitoplast patch-clamp experiments argue that MICU1 does not block Ca2+ transport but instead potentiates MCU. To address this direct clash of proposed MICU1 function, we applied purified MICU1 to Ca2+-conducting MCU-EMRE subcomplexes in outside-out patches excised from Xenopus oocytes. MICU1 strongly inhibits Ca2+ currents, and the inhibition is abolished by mutating an MCU-interacting K126 residue in MICU1. Further experiments show that MICU1 block was not observed in mitoplasts because MICU1 dissociates from the uniporter complex. These results firmly establish that MICU1 shuts the uniporter in resting cellular conditions.

https://biorxiv.org/content/10.1101/2021.11.15.468501v1.full.pdf

MICU1 occludes MCU in the mitochondrial calcium uniporter complex

In this paper, we report our methods and experiments for the TREC Conversational Assistance Track (CAsT) 2021. In this work, We aim to improve the conversational information seeking system by reforming the modules in the existing multi-stage conversational search pipeline. In the first-stage document retrieval, we proposed the Paraphrase Query Expansion (PQE), which is adapted to the less-training-data scenario like CAsT. As for the second-stage re-ranking, we adopt the T5 point-wise ranking model with multi-view learning framework (monoT5M) to avoid the underlying overfitting problems. To further elucidate the effectiveness of our proposed methods, we also report the ablation studies of our proposed modules.

An Exploration Study of Multi-stage Conversational Passage Retrieval: Paraphrase Query Expansion and Multi-view Point-wise Ranking

The accuracy of recommendation trends to be worse towards the long tail of the popularity distribution of items, but items in the long tail are generally considered to be valuable as they occupy a majority part of entire data. In this paper, we develop an instance-level cost-sensitive Factorization Machine (FM) to tackle the problem. The new algorithm allows the FM model to automatically leverage the trade-off between item popularity and recommendation quality. Specifically, by adding a cost criterion to the loss function, the FM model is now able to discriminate the relative importance of popularity from massive data. In addition, we convert several well-known functions into the popularity weighting functions, thereby demonstrating that the proposed method can fit the model parameters to various kinds of measurements. In the experiments, we assess the performance on a real-world music dataset which is collected from an online music streaming service, KKBOX. The dataset contains 1,800,000 listening records that cover 5,000 users and 30,000 songs. The results show that, the proposed method not only keeps the performance as primitive model but also avoids retrieving too much popular music in the top recommendations.

Ming-Feng Tsai

Papers

MICU1 occludes MCU in the mitochondrial calcium uniporter complex

An Exploration Study of Multi-stage Conversational Passage Retrieval: Paraphrase Query Expansion and Multi-view Point-wise Ranking

Leverage Item Popularity and Recommendation Quality via Cost-Sensitive Factorization Machines

Quantitative assays to measure the transport activity of the mitochondrial calcium uniporter in cell lines or Xenopus oocytes

Molecular mechanisms of the mitochondrial calcium uniporter part 1: structural and functional mechanisms of MICU-dependent Ca2+activation