Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

/pdf/teaching-machines-to-read-and-comprehend-381zg0jrq1.pdf

Teaching machines to read and comprehend

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

Gaussian Error Linear Units (GELUs)

Child psychiatrists, pediatricians, and other child clinicians need to have a solid understanding of child language development. There are at least four important reasons that make this necessary. First, slowing, arrest, and deviation of language development are highly associated with, and complicate the course of, child psychopathology. Second, language competence plays a crucial role in emotional and mood regulation, evaluation, and therapy. Third, language deficits are the most frequent underpinning of the learning disorders, ubiquitous in our clinical populations. Fourth, clinicians should not confuse the rich linguistic and dialectal diversity of our clinical populations with abnormalities in child language development. The challenge for the clinician becomes, then, how to get immersed in the captivating field of child language acquisition without getting overwhelmed by its conceptual and empirical complexity. In the past 50 years and since the seminal works of Roger Brown, Jerome Bruner, and Catherine Snow, child language researchers (often known as developmental psycholinguists) have produced a remarkable body of knowledge. Linguists such as Chomsky and philosophers such as Grice have strongly influenced the science of child language. One of the major tenets of Chomskian linguistics (known as generative grammar) is that children’s capacity to acquire language is “hardwired” with “universal grammar”—an innate language acquisition device (LAD), a language “instinct”—at its core. This view is in part supported by the assertion that the linguistic input that children receive is relatively dismal and of poor quality relative to the high quantity and quality of output that they manage to produce after age 2 and that only an advanced, innate capacity to decode and organize linguistic input can enable them to “get from here (prelinguistic infant) to there (linguistic child).” In “Constructing a Language,” Tomasello presents a contrasting theory of how the child acquires language: It is not a universal grammar that allows for language development. Rather, human cognition universals of communicative needs and vocal-auditory processing result in some language universals, such as nouns and verbs as expressions of reference and predication (p. 19). The author proposes that two sets of cognitive skills resulting from biological/phylogenetic adaptations are fundamental to the ontogenetic origins of language. These sets of inherited cognitive skills are intentionreading on the one hand and pattern-finding, on the other. Intention-reading skills encompass the prelinguistic infant’s capacities to share attention to outside events with other persons, establishing joint attentional frames, to understand other people’s communicative intentions, and to imitate the adult’s communicative intentions (an intersubjective form of imitation that requires symbolic understanding and perspective-taking). Pattern-finding skills include the ability of infants as young as 7 months old to analyze concepts and percepts (most relevant here, auditory or speech percepts) and create concrete or abstract categories that contain analogous items. Tomasello, a most prominent developmental scientist with research foci on child language acquisition and on social cognition and social learning in children and primates, succinctly and clearly introduces the major points of his theory and his views on the origins of language in the initial chapters. In subsequent chapters, he delves into the details by covering most language acquisition domains, namely, word (lexical) learning, syntax, and morphology and conversation, narrative, and extended discourse. Although one of the remaining domains (pragmatics) is at the core of his theory and permeates the text throughout, the relative paucity of passages explicitly devoted to discussing acquisition and proBOOK REVIEWS

Constructing a language: A usage-based theory of language acquisition

The Cambridge Grammar of the English Language

People tweet more than 100 Million times daily, yielding a noisy, informal, but sometimes informative corpus of 140-character messages that mirrors the zeitgeist in an unprecedented manner. The performance of standard NLP tools is severely degraded on tweets. This paper addresses this issue by re-building the NLP pipeline beginning with part-of-speech tagging, through chunking, to named-entity recognition. Our novel T-ner system doubles F1 score compared with the Stanford NER system. T-ner leverages the redundancy inherent in tweets to achieve this performance, using LabeledLDA to exploit Freebase dictionaries as a source of distant supervision. LabeledLDA outperforms co-training, increasing F1 by 25% over ten common entity types.

Our NLP tools are available at: http://github.com/aritter/twitter_nlp

/pdf/named-entity-recognition-in-tweets-an-experimental-study-4a5ky783nf.pdf

Named Entity Recognition in Tweets: An Experimental Study

The subject of our research is textual evidence for coherence relations between clauses (aka discourse relations) and what relations they provide evidence for. We call explicit indicators of coherence relations discourse connectives.1 Coherence relations have been the object of study in semantics and psycholinguistics, including (Knott, 1996; Knott & Mellish, 1996; Knott & Sanders, 1998; Spooren, 1997), and a focus of discourse parsing applications in Natural Language Processing (Lin et al, 2014; Xue et al, 2015; Xue et al, 2016). In all this work, the standard assumption has been, simply put, that explicit discourse connectives signal discourse relations, and that only where they are absent or ambiguous is inference used to establish coherence. This assumption bears questioning, however, and work we have carried out to date on concurrent discourse relations and on how listeners interpret discourse connectives shows this assumption to be false (Rohde et al, 2015; Rohde et al, 2016; Rohde et al, 2017). We believe that correcting this assumption can both improve computational modelling (say, for machine translation and relation extraction) and support deeper cross-lingual studies and more effective design and interpretation of psycholinguistic experimentation. As such, we believe that our work should be of interest to both the existing ACL and new SCiL communities. Our work was undertaken with the computational goal of improving inter-clausal relation mining. Work on the Penn Discourse TreeBank (PDTB) (Prasad et al, 2014) had already shown that annotators sometimes chose to annotate more than one coherence relation as holding between the same discourse spans, and further annotation (Webber et al, 2016) has shown even more frequent annotation of more than one discourse relation between the same pair of spans. But PDTB annotation has also hinted at another possible source of multiple discourse relations holding concurrently. Specifically, annotators sense-labelled some of the sentences containing the discourse adverbial “instead” as if the sentences started with the conjunction “but” (as in Ex. 1), annotating “instead” with a sense normally associated with “but” (CONTRAST) rather than its own more common label (CHOSEN ALTERNATIVE). Since “instead” can co-occur with conjunctions other than “but” (as in Ex. 2–4)

/pdf/explicit-discourse-connectives-implicit-discourse-relations-2ox8ybub37.pdf

Nathan Schneider

Papers

Explicit Discourse Connectives / Implicit Discourse Relations