scispace - formally typeset
Search or ask a question
Journal ArticleDOI

A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001--2010

01 Sep 2013-Lingua (North- Holland)-Vol. 134, Iss: 134, pp 219-248
TL;DR: The authors compared the performance of informal and formal judgment collection methods and reported a convergence rate of 95% with a margin of error of 5.3-5.8% between the two methods, and discussed the implications of this convergence rate for future research into syntactic methodology.
About: This article is published in Lingua.The article was published on 2013-09-01 and is currently open access. It has received 159 citations till now. The article focuses on the topics: Sample (statistics) & Margin of error.

Summary (1 min read)

A comparison of informal and formal acceptability judgments using a random sample from

  • There is at least one potential confound in such an analysis that could only be overcome with a different experimental design.
  • Because each of these tasks are viable candidates for use in any given formal acceptability judgment experiment, and because each provides slightly different information that may be of interest to syntacticians, the authors decided to test the sample of 150 phenomena three distinct times: once each using ME, LS, and FC.
  • Anticipating the discussion in section 4.3, the authors wish to stress that the convergence rates they have observed carry no information concerning which method is superior (if that question even has a general answer).

Acknowledgments

  • The authors would like to thank audiences at the following universities for helpful comments on earlier stages of this project: Harvard University, Johns Hopkins University, Michigan State University, Pomona College, Princeton University, University of Connecticut, University of Michigan, and the attendees of TEAL 7 at Hiroshima University.
  • The authors would also like to thank two anonymous reviewers for helpful comments on an earlier draft.

Did you find this useful? Give us your feedback

Figures (8)
Citations
More filters
Journal ArticleDOI
TL;DR: This article used a corpus of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature to test the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence.
Abstract: This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.

903 citations

Journal ArticleDOI
TL;DR: It is proposed that structural priming provides a new basis for understanding the nature of language and provides evidence about the consistency of representations across languages and about language development.
Abstract: Within the cognitive sciences, most researchers assume that it is the job of linguists to investigate how language is represented, and that they do so largely by building theories based on explicit judgments about patterns of acceptability – whereas it is the task of psychologists to determine how language is processed, and that in doing so, they do not typically question the linguists' representational assumptions. We challenge this division of labor by arguing that structural priming provides an implicit method of investigating linguistic representations that should end the current reliance on acceptability judgments. Moreover, structural priming has now reached sufficient methodological maturity to provide substantial evidence about such representations. We argue that evidence from speakers' tendency to repeat their own and others' structural choices supports a linguistic architecture involving a single shallow level of syntax connected to a semantic level containing information about quantification, thematic relations, and information structure, as well as to a phonological level. Many of the linguistic distinctions often used to support complex (or multilevel) syntactic structure are instead captured by semantics; however, the syntactic level includes some specification of “missing” elements that are not realized at the phonological level. We also show that structural priming provides evidence about the consistency of representations across languages and about language development. In sum, we propose that structural priming provides a new basis for understanding the nature of language.

126 citations

Journal ArticleDOI
TL;DR: It is argued that, on the computational side, it is important to move from toy problems to the full complexity of the learning situation, and take as input as faithful reconstructions of the sensory signals available to infants as possible.

122 citations

Journal ArticleDOI
TL;DR: It is argued that most evidence is consistent with the one-system view and how to relate grammatical theories and language processing models to each other is addressed.
Abstract: We address two important questions about the relationship between theoretical linguistics and psycholinguistics. First, do grammatical theories and language processing models describe separate cognitive systems, or are they accounts of different aspects of the same system? We argue that most evidence is consistent with the one-system view. Second, how should we relate grammatical theories and language processing models to each other?

107 citations

Posted Content
TL;DR: This paper introduces the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature, and trains several recurrent neural network models on acceptability classification, and finds that the authors' models outperform unsupervised models by Lau et al. (2016) on CoLA.
Abstract: This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.'s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.

97 citations

References
More filters
Journal Article
TL;DR: Copyright (©) 1999–2012 R Foundation for Statistical Computing; permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and permission notice are preserved on all copies.
Abstract: Copyright (©) 1999–2012 R Foundation for Statistical Computing. Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the R Core Team.

272,030 citations


"A comparison of informal and formal..." refers background in this paper

  • ...…for many types of language experiments, treating them as random will result in less statistical power, thereby creating a greater risk of false negative results (Wike and Church, 1976; Cohen, 1976; Keppel, 1976; Smith, 1976; Wickens and Keppel, 1983; Raaijmakers et al., 1999; Raaijmakers, 2003)....

    [...]

Journal ArticleDOI
TL;DR: Methodological preliminaries of generative grammars as theories of linguistic competence; theory of performance; organization of a generative grammar; justification of grammar; descriptive and explanatory theories; evaluation procedures; linguistic theory and language learning.

12,586 citations


"A comparison of informal and formal..." refers background in this paper

  • ...Acceptability judgments provide the primary empirical foundation of many syntactic theories (Chomsky, 1965; Schütze, 1996)....

    [...]

Book
01 May 1965
TL;DR: Generative grammars as theories of linguistic competence as discussed by the authors have been used as a theory of performance for language learning. But they have not yet been applied to the problem of language modeling.
Abstract: : Contents: Methodological preliminaries: Generative grammars as theories of linguistic competence; theory of performance; organization of a generative grammar; justification of grammars; formal and substantive grammars; descriptive and explanatory theories; evaluation procedures; linguistic theory and language learning; generative capacity and its linguistic relevance Categories and relations in syntactic theory: Scope of the base; aspects of deep structure; illustrative fragment of the base component; types of base rules Deep structures and grammatical transformations Residual problems: Boundaries of syntax and semantics; structure of the lexicon

12,225 citations

Journal ArticleDOI
TL;DR: In this article, the authors provide an introduction to mixed-effects models for the analysis of repeated measurement data with subjects and items as crossed random effects, and a worked-out example of how to use recent software for mixed effects modeling is provided.

6,853 citations


"A comparison of informal and formal..." refers methods in this paper

  • ...…because the items were lexically matched across conditions), we nonetheless constructed linear mixed effects models treating both participants and items as crossed random effects for the ME and LS experiments, and simulated p-values using the languageR package (Baayen, 2007; Baayen et al., 2008)....

    [...]

Journal ArticleDOI
Jacob Cohen1
TL;DR: The authors reviewed the problems with null hypothesis significance testing, including near universal misinterpretation of p as the probability that H is false, the misinterpretation that its complement is the probability of successful replication, and the mistaken assumption that if one rejects H₀ one thereby affirms the theory that led to the test.
Abstract: After 4 decades of severe criticism, the ritual of null hypothesis significance testing (mechanical dichotomous decisions around a sacred .05 criterion) still persists. This article reviews the problems with this practice, including near universal misinterpretation of p as the probability that H₀ is false, the misinterpretation that its complement is the probability of successful replication, and the mistaken assumption that if one rejects H₀ one thereby affirms the theory that led to the test. Exploratory data analysis and the use of graphic methods, a steady improvement in and a movement toward standardization in measurement, an emphasis on estimating effect sizes using confidence intervals, and the informed use of available statistical methods are suggested. For generalization, psychologists must finally rely, as has been done in all the older sciences, on replication. (PsycINFO Database Record (c) 2012 APA, all rights reserved)

3,838 citations


"A comparison of informal and formal..." refers background in this paper

  • ...…are true of the world, and ask how likely a given hypothesis would be under that assumption (e.g., Gallistel, 2009; Kruschke, 2011; and for accessible reviews of the controversies surrounding NHT, see Shaver, 1993; Cohen, 1994; Nickerson, 2000; Balluerka et al., 2005; Hubbard and Lindsay, 2008)....

    [...]