We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

/pdf/hypothesis-only-baselines-in-natural-language-inference-2p1z2f2yq6.pdf

Hypothesis Only Baselines in Natural Language Inference

Automatic short answer grading (ASAG) is the task of assessing short natural language responses to objective questions using computational methods. The active research in this field has increased enormously of late with over 80 papers fitting a definition of ASAG. However, the past efforts have generally been ad-hoc and non-comparable until recently, hence the need for a unified view of the whole field. The goal of this paper is to address this aim with a comprehensive review of ASAG research and systems according to history and components. Our historical analysis identifies 35 ASAG systems within 5 temporal themes that mark advancement in methodology or evaluation. In contrast, our component analysis reviews 6 common dimensions from preprocessing to effectiveness. A key conclusion is that an era of evaluation is the newest trend in ASAG research, which is paving the way for the consolidation of the field.

/pdf/the-eras-and-trends-of-automatic-short-answer-grading-59suzs1qw8.pdf

The Eras and Trends of Automatic Short Answer Grading

The lack of large-scale question matching corpora greatly limits the development of matching methods in question answering (QA) system, especially for non-English languages To ameliorate this situation, in this paper, we introduce a large-scale Chinese question matching corpus (named LCQMC), which is released to the public1 LCQMC is more general than paraphrase corpus as it focuses on intent matching rather than paraphrase How to collect a large number of question pairs in variant linguistic forms, which may present the same intent, is the key point for such corpus construction In this paper, we first use a search engine to collect large-scale question pairs related to high-frequency words from various domains, then filter irrelevant pairs by the Wasserstein distance, and finally recruit three annotators to manually check the left pairs After this process, a question matching corpus that contains 260,068 question pairs is constructed In order to verify the LCQMC corpus, we split it into three parts, ie, a training set containing 238,766 question pairs, a development set with 8,802 question pairs, and a test set with 12,500 question pairs, and test several well-known sentence matching methods on it The experimental results not only demonstrate the good quality of LCQMC but also provide solid baseline performance for further researches on this corpus

LCQMC:A Large-scale Chinese Question Matching Corpus

We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on ten distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

Neural approaches to automated essay scoring have recently shown state-of-the-art performance The automated essay scoring task typically involves a broad notion of writing quality that encompasses content, grammar, organization, and conventions This differs from the short answer content scoring task, which focuses on content accuracy The inputs to neural essay scoring models – ngrams and embeddings – are arguably well-suited to evaluate content in short answer scoring tasks We investigate how several basic neural approaches similar to those used for automated essay scoring perform on short answer scoring We show that neural architectures can outperform a strong non-neural baseline, but performance and optimal parameter settings vary across the more diverse types of prompts typical of short answer scoring

/pdf/investigating-neural-architectures-for-short-answer-scoring-3aygwoq5dv.pdf

Investigating neural architectures for short answer scoring

We present the results of the Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge, aiming to bring together researchers in educational NLP technology and textual entailment. The task of giving feedback on student answers requires semantic inference and therefore is related to recognizing textual entailment. Thus, we offered to the community a 5-way student response labeling task, as well as 3-way and 2way RTE-style tasks on educational data. In addition, a partial entailment task was piloted. We present and compare results from 9 participating teams, and discuss future directions.

/pdf/semeval-2013-task-7-the-joint-student-response-analysis-and-57v1002ob8.pdf

SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

In the 2009 Recognizing Textual Entailment challenge a Search Pilot task has been introduced, aimed at finding all the sentences in a corpus which entail a set of given hypotheses. The preparation of the data set for this task has provided an opportunity to better understand some phenomena concerning textual entailment recognition in a natural setting. This paper focuses on some problematic issues related to resolving coreferences to entities, space, time and events at the corpus level, as emerged during the annotation of the data set for the textual entailment Search Pilot.

/pdf/considering-discourse-references-in-textual-entailment-nwcslnhdtf.pdf

Hoa Trang Dang

Papers

SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

Considering Discourse References in Textual Entailment Annotation | NIST