Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions.

/pdf/learning-to-rank-for-information-retrieval-2q6t3viyug.pdf

Learning to Rank for Information Retrieval

The paper is concerned with learning to rank, which is to construct a model or a function for ranking objects. Learning to rank is useful for document retrieval, collaborative filtering, and many other applications. Several methods for learning to rank have been proposed, which take object pairs as 'instances' in learning. We refer to them as the pairwise approach in this paper. Although the pairwise approach offers advantages, it ignores the fact that ranking is a prediction task on list of objects. The paper postulates that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning. The paper proposes a new probabilistic method for the approach. Specifically it introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning. Neural Network and Gradient Descent are then employed as model and algorithm in the learning method. Experimental results on information retrieval show that the proposed listwise approach performs better than the pairwise approach.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2007-40.pdf

Learning to rank: from pairwise approach to listwise approach

Data sparsity is a well-known challenge in recommender systems. One way to alleviate this problem is to leverage knowledge from relevant domains. In this paper, we focus on an important real-world scenario in which some users overlap two different domains but items of the two domains are distinct. Although several studies leverage side information (e.g., user reviews) for cross-domain recommendation, side information is not always available or easy to obtain in practice. To this end, we propose cross-domain preference ranking (CPR) with a simple yet effective user transformation that leverages only user interactions with items in the source and target domains to transform the user representation. Given the proposed user transformation, CPR not only successfully enhances recommendation performance for users having interactions with target-domain items but also yields superior performance for cold-start users in comparison with state-of-the-art cross-domain recommendation approaches. Extensive experiments conducted on three pairs of cross-domain recommendation datasets demonstrate the effectiveness of the proposed method in comparison with existing cross-domain recommendation approaches. Our codes are available at https://github.com/cnclabs/codes.crossdomain.rec . 

CPR: Cross-Domain Preference Ranking with User Transformation

Deep Learning-Based Clinical Wound Image Analysis Using a Mask R-CNN Architecture

Explicit feedback---user input regarding their interest in an item---is the most helpful information for recommendation as it comes directly from the user and shows their direct interest in the item. Most approaches either treat the recommendation given such feedback as a typical regression problem or regard such data as implicit and then directly adopt approaches for implicit feedback; both methods, however,tend to yield unsatisfactory performance in top-k recommendation. In this paper, we propose interaction-level preference ranking(IPR), a novel pairwise ranking embedding learning approach to better utilize explicit feedback for recommendation. Experiments conducted on three real-world datasets show that IPR yields the best results compared to six strong baselines. 

Abstract Background Reuse of cardiac implantable electronic devices (CIEDs) can reduce the cost of using these expensive devices. However, whether resterilized CIEDs will increase the risk of reinfection in patients with previous device infection remains unknown. The aim of the present study is to compare the reinfection rates in patients who had initial CIED infection and underwent reimplantation of resterilized CIEDs or new devices. Methods Data from patients with initial CIED infection who received debridement of the infected pocket and underwent reimplantation of new or resterilized CIEDs at MacKay Memorial Hospital, Taipei, Taiwan, between January 2014 and June 2019 were retrospectively analyzed. Patient characteristics, relapse rates of infection, and potential contributing factors to the infection risk were examined. Results Twenty-seven patients with initial CIED infection and reimplanted new CIEDs (n = 11) or resterilized CIEDs (n = 16) were included. During the 2-year follow-up, there were 1 (9.1%) and 2 (12.5%) infection relapses in the new and resterilized CIED groups, respectively. No relapse occurred for either group if the lead was completely removed or cut short. The median duration between debridement and device reimplantation in patients with infection relapse vs patients without relapse was 97 vs 4.5 days for all included patients, and 97 vs 2 days and 50.5 vs 5.5 days for the new and resterilized CIED groups, respectively. Conclusions Subpectoral reimplanting of resterilized CIEDs in patients with previous device infection is safe and efficacious. With delicate debridement and complete extraction of the leads, the CIED pocket infection relapse risk can be greatly decreased.

Safety and Efficacy of Submuscular Implantation With Resterilized Cardiac Implantable Electronic Device in Patients With Device Infection: A Retrospective Observational Study in Taiwan

In this demonstration, we present RecDelta, an interactive tool for the cross-model evaluation of top-k recommendation. RecDelta is a web-based information system where people visually compare the performance of various recommendation algorithms and their recommended items. In the proposed system, we visualize the distribution of the δ scores between algorithms--a distance metric measuring the intersection between recommendation lists. Such visualization allows for rapid identification of users for whom the items recommended by different algorithms diverge or vice versa; then, one can further select the desired user to present the relationship between recommended items and his/her historical behavior. RecDelta benefits both academics and practitioners by enhancing model explainability as they develop recommendation algorithms with their newly gained insights. Note that while the system is now online at https://cfda.csie.org/recdelta, we also provide a video recording at https://tinyurl.com/RecDelta to introduce the concept and the usage of our system. 

Ming-Feng Tsai

Papers

CPR: Cross-Domain Preference Ranking with User Transformation

Deep Learning-Based Clinical Wound Image Analysis Using a Mask R-CNN Architecture

Ipr

Safety and Efficacy of Submuscular Implantation With Resterilized Cardiac Implantable Electronic Device in Patients With Device Infection: A Retrospective Observational Study in Taiwan

RecDelta