Fundamental statistics in psychology and education.

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0140525X00011183

Peer-review practices of psychological journals: The fate of published articles, submitted again

An academic scientist’s professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive—getting it right—competitive with the more tangible and concrete incentive—getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

/pdf/scientific-utopia-ii-restructuring-incentives-and-practices-4ll5s2mnqk.pdf

Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability

An academic scientist’s professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive – getting it right – competitive with the more tangible and concrete incentive – getting it published. We develop strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and self-serving biases.

Scientific Utopia: II - Restructuring Incentives and Practices to Promote Truth Over Publishability

The inferences made in integrative research reviews are as central to the validity of behavioral science knowledge as those made in primary research. Therefore, research reviewers must pay the same attention to rigorous methodology that is required of primary researchers. This article conceptualizes the research review as a scientific inquiry involving five stages that parallel those of primary research. The functions, sources of variance, and potential threats to validity associated with each stage are described.

http://faculty.metrostate.edu/barrerma/Summer2015/3%20REVIEWCooper%20%20-291-302.pdf

Scientific Guidelines for Conducting Integrative Research Reviews

Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments.

This study compares, within the context of parole decision-making, the predictive utility of five statistical methods commonly employed to develop correctional risk-screening devices: (a) two general linear additive models, (b) two configural models, and (c) a model based on a multivariate contingency approach. Devices felt to be operationally useful (at least potentially) were developed and cross-validated using large samples of federal releases. Results suggest that no apparent empirical advantage accrues given use of different models. Even when large amounts of error are randomly added to the predictor item-pool, substantive conclusions do not change. Further, most devices developed are highly intercorrelated. Implications for the practical development of screening instruments are discussed, and further research is suggested.

Stephen D. Gottfredson

Papers

Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments.

Screening for Risk: A Comparison of Methods.

Criminal Justice and (Reviewer) Behavior: How to Get Papers Published