Anya Belz

TL;DR: Due to a pervasive lack of clarity in reports and extreme diversity in approaches, human evaluation in NLG presents as extremely confused in 2020, and that the field is in urgent need of standard methods and terminology.

TL;DR: This work proposes a classification system for evaluations based on disentangling what is being evaluated, and how it is evaluated in specific evaluation modes and experimental designs and shows that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.

TL;DR: The authors provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, with a focus on the NLP field, focusing on how to define, measure, and address the Reproducibility crisis in science.

TL;DR: An extensive human evaluation study of consultation notes finds that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore.

TL;DR: The Human Evaluation Datasheet as mentioned in this paper is a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP) and is intended to facilitate the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation, and reproducibility tests.