Data Mining Practical Machine Learning Tools and Techniques

We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MSCOCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/ mjhucla/Google_Refexp_toolbox.

/pdf/generation-and-comprehension-of-unambiguous-object-1fflts31e4.pdf

Generation and Comprehension of Unambiguous Object Descriptions

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past two decades, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artifical intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of nlp, with an emphasis on different evaluation methods and the relationships between them.

/pdf/survey-of-the-state-of-the-art-in-natural-language-26vdvzdbx0.pdf

Survey of the state of the art in natural language generation: core tasks, applications and evaluation

Dimensions of Reinforcement Learning

The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

/pdf/why-we-need-new-evaluation-metrics-for-nlg-4u9qet9ukp.pdf

Why We Need New Evaluation Metrics for NLG

Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility. In this paper, we present (i) our dataset of 165 NLG papers with human evaluations, (ii) the annotation scheme we developed to label the papers for different aspects of evaluations, (iii) quantitative analyses of the annotations, and (iv) a set of recommendations for improving standards in evaluation reporting. We use the annotations as a basis for examining information included in evaluation reports, and levels of consistency in approaches, experimental design and terminology, focusing in particular on the 200+ different terms that have been used for evaluated aspects of quality. We conclude that due to a pervasive lack of clarity in reports and extreme diversity in approaches, human evaluation in NLG presents as extremely confused in 2020, and that the field is in urgent need of standard methods and terminology.

Twenty Years of Confusion in Human Evaluation : NLG Needs Evaluation Sheets and Standardised Definitions

In this paper we present a snapshot of endto-end NLG system evaluations as presented in conference and journal papers1 over the last ten years in order to better understand the nature and type of evaluations that have been undertaken. We find that researchers tend to favour specific evaluation methods, and that their evaluation approaches are also correlated with the publication venue. We further discuss what factors may influence the types of evaluation used for a given NLG system.

/pdf/a-snapshot-of-nlg-evaluation-practices-2005-2014-3d19wru754.pdf

A Snapshot of NLG Evaluation Practices 2005 - 2014

Decision-making is often dependent on
uncertain data, e.g. data associated with
confidence scores or probabilities. We
present a comparison of different informa-
tion presentations for uncertain data and,
for the first time, measure their effects
on human decision-making. We show
that the use of Natural Language Genera-
tion (NLG) improves decision-making un-
der uncertainty, compared to state-of-the-
art graphical-based representation meth-
ods. In a task-based study with 442 adults,
we found that presentations using NLG
lead to 24% better decision-making on av-
erage than the graphical presentations, and
to 44% better decision-making when NLG
is combined with graphics. We also show
that women achieve significantly better re-
sults when presented with NLG output
(an 87% increase on average compared to
graphical presentations).

Natural Language Generation enhances human decision-making with uncertain information.

We present a novel approach for automatic report generation from time-series data, in the context of student feedback generation. Our proposed methodology treats content selection as a multi-label (ML) classification problem, which takes as input time-series data and outputs a set of templates, while capturing the dependencies between selected templates. We show that this method generates output closer to the feedback that lecturers actually generated, achieving 3.5% higher accuracy and 15% higher F-score than multiple simple classifiers that keep a history of selected templates. Furthermore, we compare a ML classifier with a Reinforcement Learning (RL) approach in simulation and using ratings from real student users. We show that the different methods have different benefits, with ML being more accurate for predicting what was seen in the training data, whereas RL is more exploratory and slightly preferred by the students.

/pdf/comparing-multi-label-classification-with-reinforcement-1bzw4vrc7a.pdf

Comparing Multi-label Classification with Reinforcement Learning for Summarisation of Time-series Data

Decision-making is often dependent on uncertain data, e.g. data associated with confidence scores or probabilities. This article presents a comparison of different information presentations for uncertain data and, for the first time, measures their effects on human decision-making, in the domain of weather forecast generation. We use a game-based setup to evaluate the different systems. We show that the use of Natural Language Generation (NLG) enhances decision-making under uncertainty, compared to state-of-the-art graphicalbased representation methods. In a task-based study with 442 adults, we found that presentations using NLG led to 24% better decision-making on average than the graphical presentations, and to 44% better decision-making when NLG is combined with graphics. We also show that women achieve significantly better results when presented with NLG output (an 87% increase on average compared to graphical presentations). Finally, we present a further analysis of demographic data and its impact on decision-making, and we discuss implications for future NLG systems.

/pdf/data-to-text-generation-improves-decision-making-under-4zieha3onc.pdf

Dimitra Gkatzia

Papers

Twenty Years of Confusion in Human Evaluation : NLG Needs Evaluation Sheets and Standardised Definitions

A Snapshot of NLG Evaluation Practices 2005 - 2014

Natural Language Generation enhances human decision-making with uncertain information.

Comparing Multi-label Classification with Reinforcement Learning for Summarisation of Time-series Data

Data-to-Text Generation Improves Decision-Making Under Uncertainty