The Righteous Mind: Why Good People Are Divided by Politics and Religion Jonathan Haidt Pantheon Books, 2012One has likely heard that, for the sake of decorum, religion and politics should never be topics of conversation with strangers. Even amongst friends or even when it is known that others hold opposing political or religious views, why is it that discussion of religion and politics leads to visceral-level acrimony and that one's views are right and the other's views are wrong? Professor Jonathan Haidt of the University of Virginia examines the psychological basis of our "righteous minds" without resorting to any of the pejorative labeling that is usually found in a book on politics and religion and eschews a purely comparative approach. Haidt proposes the intriguing hypothesis that our visceral reaction to competing ideologies is a subconscious, rather than leaned, reaction that evolved over human evolution to innate senses of suffering, fairness, cheating and disease, and that moral foundations facilitated intra-group cooperation which in turn conferred survival advantages over other groups. These psychological mechanisms are genetic in origin and not necessarily amenable to rational and voluntary control - this is in part the reason debating one's ideological opposite more often leads to frustration rather than understanding. Haidt also suggests that morality is based on six "psychological systems" or foundations (Moral Foundations Theory), similar to the hypothesized adaptive mental modules which evolved to solve specific problems of survival in the human ancestral environment.While decorum pleads for more civility, it would be better, as Haidt suggests, dragging the issue of partisan politics out into the open in order to understand it and work around our righteous minds. Haidt suggests a few methods by which the level of rhetoric in American politics can be reduced, such that the political parties can at least be cordial as they have been in the past and work together to solve truly pressing social problems.There are a number of fascinating points raised in the current book, but most intriguing is the one that morality, ideology and religion are products of group selection, as adaptations that increased individual cooperation and suppressed selfishness, thereby increasing individual loyalty to the group. That morality, political ideology and religion buttress group survival is probably highly intuitive. However, given the contemporary focus on the individual as the source of adaptations, to the exclusion of all else, to suggest that adaptations such as religion and political ideology arose to enhance survival of groups is heresy or, as Haidt recounts, "foolishness". While previous rejection of group selection itself was due in part to conceptual issues, one could also point out the prevailing individualist social sentiment, "selfish gene" mentality and unrelenting hostility against those who supported the view that group selection did indeed apply to humans and not just to insects. Haidt gives a lengthy and convincing defense of group selection, his main point being that humans can pursue self- interest at the same time they promote self-interest within a group setting - humans are "90 percent chimp, 10 percent bees". One can readily observe in the news and entertainment mediate that religion is a frequent target of derision, even within the scientific community - Haidt points to the strident contempt that the "New Atheists" hold for religion. They claim that religion is purely a by-product of an adaptive psychological trait and as a mere by-product religion serves no useful purpose. However, the religious "sense" has somehow managed to persist in the human psyche. One explanation by the New Atheists of how religion propagated itself is that it is a "parasite" or "virus" which latches onto a susceptible host and induces the host to "infect" others. As a "virus" or "parasite" that is merely interested in its own survival, religion causes people to perform behaviors that do not increase their own reproductive fitness and may even be detrimental to survival, but religion spreads nonetheless. …

The Righteous Mind: Why Good People Are Divided by Politics and Religion

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at this https URL.

Language Models as Knowledge Bases

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.

/pdf/language-models-as-knowledge-bases-2bs7zawas9.pdf

Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc. This has led to format-specialized models, and even to an implicit division in the QA community. We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format. As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UNIFIEDQA, that performs well across 19 QA datasets spanning 4 diverse formats. UNIFIEDQA performs on par with 8 different models that were trained on individual datasets themselves. Even when faced with 12 unseen datasets of observed formats, UNIFIEDQA performs surprisingly well, showing strong generalization from its outof-format training data. Finally, simply finetuning this pre trained QA model into specialized models results in a new state of the art on 10 factoid and commonsense question answering datasets, establishing UNIFIEDQA as a strong starting point for building QA systems.

UNIFIEDQA: Crossing Format Boundaries with a Single QA System

State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the `reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER\, a benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of ``rationales'' (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/

ERASER: A Benchmark to Evaluate Rationalized NLP Models

We present ATOMIC, an atlas of everyday commonsense reasoning, organized through 877k textual descriptions of inferential knowledge. Compared to existing resources that center around taxonomic knowledge, ATOMIC focuses on inferential knowledge organized as typed if-then relations with variables (e.g., “if X pays Y a compliment, then Y will likely return the compliment”). We propose nine if-then relation types to distinguish causes vs. effects, agents vs. themes, voluntary vs. involuntary events, and actions vs. mental states. By generatively training on the rich inferential knowledge described in ATOMIC, we show that neural models can acquire simple commonsense capabilities and reason about previously unseen events. Experimental results demonstrate that multitask models that incorporate the hierarchical structure of if-then relation types lead to more accurate inference compared to models trained in isolation, as measured by both automatic and human evaluation.

/pdf/atomic-an-atlas-of-machine-commonsense-for-if-then-reasoning-1wzj0zsj7r.pdf

ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning

When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments on four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics. First, our data maps show the presence of "ambiguous" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are "easy to learn" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds "hard to learn"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.

/pdf/dataset-cartography-mapping-and-diagnosing-datasets-with-4ya0mj0o5l.pdf

Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

Non-textual components such as charts, diagrams and tables provide key information in many scientific documents, but the lack of large labeled datasets has impeded the development of data-driven methods for scientific figure extraction. In this paper, we induce high-quality training labels for the task of figure extraction in a large number of scientific documents, with no human intervention. To accomplish this we leverage the auxiliary data provided in two large web collections of scientific documents (arXiv and PubMed) to locate figures and their associated captions in the rasterized PDF. We share the resulting dataset of over 5.5 million induced labels---4,000 times larger than the previous largest figure extraction dataset---with an average precision of 96.8%, to enable the development of modern data-driven methods for this task. We use this dataset to train a deep neural network for end-to-end figure detection, yielding a model that can be more easily extended to new domains compared to previous work. The model was successfully deployed in Semantic Scholar,\footnote\urlhttps://www.semanticscholar.org/ a large-scale academic search engine, and used to extract figures in 13 million scientific documents.\footnoteA demo of our system is available at \urlhttp://labs.semanticscholar.org/deepfigures/,and our dataset of induced labels can be downloaded at \urlhttps://s3-us-west-2.amazonaws.com/ai2-s2-research-public/deepfigures/jcdl-deepfigures-labels.tar.gz. Code to run our system locally can be found at \urlhttps://github.com/allenai/deepfigures-open.

Nicholas Lourie

Papers

ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Extracting Scientific Figures with Distantly Supervised Neural Networks