DOI•
Reproducibility Project: Psychology
About: The article was published on 2012-04-01 and is currently open access. It has received 14 citations till now. The article focuses on the topics: Reproducibility Project & Replication (statistics).
Citations
More filters
TL;DR: The authors show that the average sample size in social-personality research is 104 and that the power to detect the typical effect size in the field is approximately 50%, and show that there is considerable variation among journals in sample sizes and power of the studies they publish.
Abstract: The authors evaluate the quality of research reported in major journals in social-personality psychology by ranking those journals with respect to their N-pact Factors (NF)—the statistical power of the empirical studies they publish to detect typical effect sizes. Power is a particularly important attribute for evaluating research quality because, relative to studies that have low power, studies that have high power are more likely to (a) to provide accurate estimates of effects, (b) to produce literatures with low false positive rates, and (c) to lead to replicable findings. The authors show that the average sample size in social-personality research is 104 and that the power to detect the typical effect size in the field is approximately 50%. Moreover, they show that there is considerable variation among journals in sample sizes and power of the studies they publish, with some journals consistently publishing higher power studies than others. The authors hope that these rankings will be of use to authors who are choosing where to submit their best work, provide hiring and promotion committees with a superior way of quantifying journal quality, and encourage competition among journals to improve their NF rankings.
292 citations
Cites background from "Reproducibility Project: Psychology..."
...(Fortunately, recent systematic replication efforts in social psychology have used sample sizes that are larger than those used in the original studies [15,16]....
[...]
Proceedings Article•
25 Apr 2018TL;DR: The reproducibility scores decrease with increased documentation requirements, but one of the metrics show statistically significant increase over time while the others show no change.
Abstract: Background: Research results in artificial intelligence (AI) are criticized for not being reproducible. Objective: To quantify the state of reproducibility of empirical AI research using six reproducibility metrics measuring three different degrees of reproducibility. Hypotheses: 1) AI research is not documented well enough to reproduce the reported results. 2) Documentation practices have improved over time. Method: The literature is reviewed and a set of variables that should be documented to enable reproducibility are grouped into three factors: Experiment, Data and Method. The metrics describe how well the factors have been documented for a paper. A total of 400 research papers from the conference series IJCAI and AAAI have been surveyed using the metrics. Findings: None of the papers document all of the variables. The metrics show that between 20% and 30% of the variables for each factor are documented. One of the metrics show statistically significant increase over time while the others show no change. Interpretation: The reproducibility scores decrease with in- creased documentation requirements. Improvement over time is found. Conclusion: Both hypotheses are supported.
155 citations
Cites result from "Reproducibility Project: Psychology..."
...This is even the case for results published in the most prestigious journals; even the original researchers cannot reproduce their own results (Aarts et al. 2016; Begley and Ellis 2012; Begley and Ioannidis 2014; Prinz et al. 2011)....
[...]
Journal Article•
46 citations
Journal Article•
TL;DR: It is concluded that systematic evaluation of educational programs not only allow for the appraisal of instructional effectiveness but also allows for progressive refinement of educational initiatives.
Abstract: Of the many interventions that might be used to improve the responsible conduct of research, educational interventions are among the most frequently employed. However, educational interventions come in many forms and have proven of varying effectiveness. Recognition of this point has led to calls for the systematic evaluation of responsible conduct of research educational programs. In the present effort, the basic principles underlying evaluation of educational programs are discussed. Subsequently, the application of these principles in the evaluation of responsible conduct of research educational programs is described. It is concluded that systematic evaluation of educational programs not only allow for the appraisal of instructional effectiveness but also allows for progressive refinement of educational initiatives. Ethics in the sciences and engineering is of concern not only because of its impact on progress in the research enterprise but also because the work of scientists and engineers impacts the lives of many people. Recognition of this point has led to a number of initiatives intended to improve the ethical conduct of investigators (National Academy of Engineering, 2009; National Institute of Medicine, 2002; National Academy of Sciences, 1992). Although a number of interventions have been proposed as a basis for improving ethical conduct, for example development of ethical guidelines, open data access, and better mentoring, perhaps the most widely applied approach has been ethics education (Council of Graduate Schools, 2012)—an intervention often referred to as training in the responsible conduct of research (RCR). When one examines the available literature on RCR training, it is apparent that a wide variety of approaches have been employed. Some RCR courses are based on a self-paced, online, instructional framework (e.g. Braunschweiger and Goodman, 2007). Other RCR courses involve face-to-face instruction over longer periods of time using realistic exercises and cases (e.g. Kligyte, Marcy, Waples, Sevier, Godfrey, Mumford, and Hougen, 2008). Some RCR courses 1 As the committee launched this study, members realized that questions related to the effectiveness of Responsible Conduct of Research education programs and how they might be improved were an essential part of the study task. A significant amount of work has been done to explore these topics. This work has yielded important insights, but additional research is needed to strengthen the evidence base relevant to several key policy questions. The committee asked one of the leading researchers in this field, Michael D. Mumford, to prepare a review characterizing the current state of knowledge and describing future priorities and pathways for assessing and improving RCR education programs. The resulting review constitutes important source material for Chapter 10 of the report. The committee also believes that the review adds value to this report a as a standalone document, and is including it as an appendix.
42 citations
Cites methods from "Reproducibility Project: Psychology..."
...The replication effort was undertaken as an open, global collaborative and involved contacting the original authors for materials and asking them to review the replication study protocol, public registration of the protocol, and public archiving of the replication materials and data (Aarts et al., 2015)....
[...]
TL;DR: The study advances G-complexity for defining and comparing decidable and undecidable knowledge and suggests that AI and related computational expressions of knowledge could benefit from the awareness of what distinguishes the dynamics of life from any other expressions of change.
Abstract: The current assumptions of knowledge acquisition brought about the crisis in the reproducibility of experiments. A complementary perspective should account for the specific causality characteristic of life by integrating past, present, and future. A “second Cartesian revolution,” informed by and in awareness of anticipatory processes, should result in scientific methods that transcend the theology of determinism and reductionism. In our days, science, itself an expression of anticipatory activity, makes possible alternative understandings of reality and its dynamics. For this purpose, the study advances G-complexity for defining and comparing decidable and undecidable knowledge. AI and related computational expressions of knowledge could benefit from the awareness of what distinguishes the dynamics of life from any other expressions of change.
17 citations