A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001--2010

TLDR: The authors compared the performance of informal and formal judgment collection methods and reported a convergence rate of 95% with a margin of error of 5.3-5.8% between the two methods, and discussed the implications of this convergence rate for future research into syntactic methodology.

About: This article is published in Lingua.The article was published on 2013-09-01 and is currently open access. It has received 159 citations till now. The article focuses on the topics: Sample (statistics) & Margin of error.

Figures

Table 2: Descriptive analysis of the directionality of the responses. For ME and LS, these counts are based on the difference between means for each phenomenon. For FC, these counts are based on the difference between the number of choices in each direction.

Table 3: Categorized results of statistical tests for ME. Significant p-values are defined at p<.05 in each direction; marginal p-values are defined at p≤.1 in each direction. Significant Bayes factors are defined at BF>3 in each direction; marginal Bayes factors are defined at BF>1 in each direction.

Table 1: Estimated counts of the number of US-English data points in Linguistic Inquiry from 2001 through 2010. The margin of error for these estimates is maximally 6.9% (see section 2.3 for details).

Table 9: Summary of divergent results in the data from Figure 1, based on thresholds that minimize divergences

Table 6: Convergence rates (in percentage) between each analysis and the traditional results reported in Linguistic Inquiry 2001-2010. In cells with slashes (/) the percentage on the left assumes that marginal results are non-significant; the percentage on the right assumes that marginal results are significant. All rates are estimates based on random sampling, resulting in a margin of error of 5.3-5.6%.

Table 7: The eight phenomena that led to divergent results between informal and formal methods based on the forced-choice task and a mixed logit analysis. We report the mixed logit analysis for FC, and the linear mixed effects models and two-tailed t-tests for ME and LS, where 0 indicates a null result (p > .1), – indicates a sign-reversal (p < .05), + indicates a significant result in the same direction as the informal result (p < .05), and parentheses indicate a marginal effect (.05 ≤ p ≤ .1) in the direction indicated by the symbol inside the parentheses. A lowercase g in the items column indicates that we created the control condition based on the discussion in the text.