In psychology, attempts to replicate published findings are less successful than expected. For properly powered studies replication rate should be around 80%, whereas in practice less than 40% of the studies selected from different areas of psychology can be replicated. Researchers in cognitive psychology are hindered in estimating the power of their studies, because the designs they use present a sample of stimulus materials to a sample of participants, a situation not covered by most power formulas. To remedy the situation, we review the literature related to the topic and introduce recent software packages, which we apply to the data of two masked priming studies with high power. We checked how we could estimate the power of each study and how much they could be reduced to remain powerful enough. On the basis of this analysis, we recommend that a properly powered reaction time experiment with repeated measures has at least 1,600 word observations per condition (e.g., 40 participants, 40 stimuli). This is considerably more than current practice. We also show that researchers must include the number of observations in meta-analyses because the effect sizes currently reported depend on the number of stimuli presented to the participants. Our analyses can easily be applied to new datasets gathered.

/pdf/power-analysis-and-effect-size-in-mixed-effects-models-a-1ox1118bc1.pdf

Power Analysis and Effect Size in Mixed Effects Models: A Tutorial

The purpose of this study was to determine the relation between reading and working memory (WM) in the context of 3 major theories: the domain-specificity theory (debate) of WM, the intrinsic cognitive load theory, and the dual process theory. A meta-analysis of 197 studies with 2026 effect sizes found a significant moderate correlation between reading and WM, r = .29, 95% CI [.27, .31]. Moderation analyses indicated that after controlling for publication type, bilingual status, domains of WM, and grade level, the relation between WM and reading was not affected by types of reading. The effects of WM domains were associated with grade level: before 4th grade, different domains of WM were related to reading to a similar degree, whereas verbal WM showed the strongest relations with reading at or beyond 4th grade. Further, the effect of WM on reading comprehension was partialed out when decoding and vocabulary were controlled for. Taken together, the findings are generally compatible with aspects of the domain-specificity theory of WM and the dual process theory, but, importantly, add a developmental component that is not currently reflected in models of the relation between reading and WM. The findings suggest that the domain-general central executive of WM is implicated in early reading acquisition, and verbal WM is more strongly implicated in later reading performance as readers gain more experience with reading. The implications of these findings for reading instruction and WM training are also discussed. (PsycINFO Database Record

A meta-analysis on the relation between reading and working memory.

Within the cognitive sciences, most researchers assume that it is the job of linguists to investigate how language is represented, and that they do so largely by building theories based on explicit judgments about patterns of acceptability – whereas it is the task of psychologists to determine how language is processed, and that in doing so, they do not typically question the linguists' representational assumptions. We challenge this division of labor by arguing that structural priming provides an implicit method of investigating linguistic representations that should end the current reliance on acceptability judgments. Moreover, structural priming has now reached sufficient methodological maturity to provide substantial evidence about such representations. We argue that evidence from speakers' tendency to repeat their own and others' structural choices supports a linguistic architecture involving a single shallow level of syntax connected to a semantic level containing information about quantification, thematic relations, and information structure, as well as to a phonological level. Many of the linguistic distinctions often used to support complex (or multilevel) syntactic structure are instead captured by semantics; however, the syntactic level includes some specification of “missing” elements that are not realized at the phonological level. We also show that structural priming provides evidence about the consistency of representations across languages and about language development. In sum, we propose that structural priming provides a new basis for understanding the nature of language.

/pdf/an-experimental-approach-to-linguistic-representation-4yw02lvg7p.pdf

An experimental approach to linguistic representation

With the arrival of the R packages nlme and lme4, linear mixed models (LMMs) have come to be widely used in experimentally-driven areas like psychology, linguistics, and cognitive science. This tutorial provides a practical introduction to fitting LMMs in a Bayesian framework using the probabilistic programming language Stan. We choose Stan (rather than WinBUGS or JAGS) because it provides an elegant and scalable framework for fitting models in most of the standard applications of LMMs. We ease the reader into fitting increasingly complex LMMs, first using a two-condition repeated measures self-paced reading study, followed by a more complex $2\times 2$ repeated measures factorial design that can be generalized to much more complex designs.

Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists

No evidential value in samples of transcranial direct current stimulation (tDCS) studies of cognition and working memory in healthy populations.

A meta-analysis of syntactic priming in language production

Verbal working memory predicts co-speech gesture: evidence from individual differences.

Language comprehension requires successfully navigating linguistic variability. One hypothesis for how listeners manage variability is that they rapidly update their expectations of likely linguistic events in new contexts. This process, called adaptation, allows listeners to better predict the upcoming linguistic input. In previous work, Fine, Jaeger, Farmer, and Qian (PLoS ONE, 8, e77661, 2013) found evidence for syntactic adaptation. Subjects repeatedly encountered sentences in which a verb was temporarily ambiguous between main verb (MV) and reduced relative clause (RC) interpretations. They found that subjects who had higher levels of exposure to the unexpected RC interpretation of the sentences had an easier time reading the RC sentences but a more difficult time reading the MV sentences. They concluded that syntactic adaptation occurs rapidly in unexpected structures and also results in difficulty with processing the previously expected alternative structures. This article presents two experiments. Experiment 1 was designed as a follow-up to Fine et al.'s study and failed to find evidence of adaptation. A power analysis of Fine et al.'s raw data revealed that a similar study would need double the items and four times the subjects to reach 95% power. In Experiment 2 we designed a close replication of Fine et al.'s experiment using these sample size guidelines. No evidence of rapid syntactic adaptation was found in this experiment. The failure to find evidence of adaptation in both experiments calls into question the robustness of the effect.

/pdf/a-failure-to-replicate-rapid-syntactic-adaptation-in-2ovlhsnkci.pdf

A failure to replicate rapid syntactic adaptation in comprehension.

Individual differences in syntactic processing: Is there evidence for reader-text interactions?

Experience-based theories of language processing suggest that listeners use the properties of their previous linguistic input to constrain comprehension in real time (e.g. MacDonald & Christiansen, 2002; Smith & Levy, 2013; Stanovich & West, 1989; Mishra, Pandey, Singh, & Huettig, 2012). This project investigates the prediction that individual differences in experience will predict differences in sentence comprehension. Participants completed a visual world eye-tracking task following Altmann and Kamide (1999) which manipulates whether the verb licenses the anticipation of a specific referent in the scene (e.g. The boy will eat/move the cake). Within this paradigm, we ask (1) are there reliable individual differences in language-mediated eye movements during this task? If so, (2) do individual differences in language experience correlate with these differences, and (3) can this relationship be explained by other, more general cognitive abilities? Study 1 finds evidence that language experience predicts an overall facilitation in fixating the target, and Study 2 replicates this effect and finds that it remains when controlling for working memory, inhibitory control, phonological ability, and perceptual speed.

/pdf/language-experience-predicts-eye-movements-during-online-19d1f54v.pdf

Ariel N. James

Papers

A meta-analysis of syntactic priming in language production

Verbal working memory predicts co-speech gesture: evidence from individual differences.

A failure to replicate rapid syntactic adaptation in comprehension.

Individual differences in syntactic processing: Is there evidence for reader-text interactions?

Language Experience Predicts Eye Movements During Online Auditory Comprehension