Protocols of systematic reviews and meta-analyses allow for planning and documentation of review methods, act as a guard against arbitrary decision making during review conduct, enable readers to assess for the presence of selective reporting against completed reviews, and, when made publicly available, reduce duplication of efforts and potentially prompt collaboration. Evidence documenting the existence of selective reporting and excessive duplication of reviews on the same or similar topics is accumulating and many calls have been made in support of the documentation and public availability of review protocols. Several efforts have emerged in recent years to rectify these problems, including development of an international register for prospective reviews (PROSPERO) and launch of the first open access journal dedicated to the exclusive publication of systematic review products, including protocols (BioMed Central's Systematic Reviews). Furthering these efforts and building on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, an international group of experts has created a guideline to improve the transparency, accuracy, completeness, and frequency of documented systematic review and meta-analysis protocols--PRISMA-P (for protocols) 2015. The PRISMA-P checklist contains 17 items considered to be essential and minimum components of a systematic review or meta-analysis protocol.This PRISMA-P 2015 Explanation and Elaboration paper provides readers with a full understanding of and evidence about the necessity of each item as well as a model example from an existing published protocol. This paper should be read together with the PRISMA-P 2015 statement. Systematic review authors and assessors are strongly encouraged to make use of PRISMA-P when drafting and appraising review protocols.

/pdf/preferred-reporting-items-for-systematic-review-and-meta-mgb1uy0yle.pdf

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation.

Quantitative synthesis of data from single-case designs (SCDs) is becoming increasingly common in psychology and education journals. Because researchers do not ordinarily report numerical data in addition to graphical displays, reliance on plot digitizing tools is often a necessary component of this research. Intercoder reliability of data extraction is a commonly overlooked, but potentially important, step of this process. The purpose of this study was to examine the intercoder reliability and validity of WebPlotDigitizer (Rohatgi, 2015), a web-based plot digitizing tool for extracting data from a variety of plots, including XY coordinates of interrupted time-series data. Two coders extracted 3,596 data points from 168 data series in 36 graphs across 18 studies. Results indicated high levels of intercoder reliability and validity. Implications of and recommendations based on these results are discussed in relation to researchers involved in quantitative synthesis of data from SCDs.

Intercoder Reliability and Validity of WebPlotDigitizer in Extracting Graphed Data

This article reports the results of a study that located, digitized, and coded all 809 single-case designs appearing in 113 studies in the year 2008 in 21 journals in a variety of fields in psychology and education. Coded variables included the specific kind of design, number of cases per study, number of outcomes, data points and phases per case, and autocorrelations for each case. Although studies of the effects of interventions are a minority in these journals, within that category, single-case designs are used more frequently than randomized or nonrandomized experiments. The modal study uses a multiple-baseline design with 20 data points for each of three or four cases, where the aim of the intervention is to increase the frequency of a desired behavior; but these characteristics vary widely over studies. The average autocorrelation is near to but significantly different from zero; but autocorrelations are significantly heterogeneous. The results have implications for the contributions of single-case designs to evidence-based practice and suggest a number of future research directions.

https://link.springer.com/content/pdf/10.3758%2Fs13428-011-0111-y.pdf

Characteristics of single-case designs used to assess intervention effects in 2008

The articles in the previous, special issue of Evidence-Based Communication Assessment and Intervention provided an excellent review of the meta-analysis of single-case designs. This article weaves commentary about those articles into a larger narrative about two major lines of attack on this problem: the use of parametric approaches like regression and multilevel modeling, and the development of parametric and nonparametric effect-size estimators. On each of these two topics, we describe an agenda of research topics that need to be addressed; and we also introduce a new effect-size estimator that may prove to be comparable to the usual standardized mean difference statistics (d) widely used in between-groups analysis. The article ends with observations about ways in which developments in the meta-analysis of single-case designs may have far wider implications than previously appreciated. Source of funding: U.S. Department of Education, Institute of Education Science, Grant # H324U050001-06

The state of the science in the meta-analysis of single-case experimental designs

A Systematic Evaluation of Token Economies as a Classroom Management Tool for Students with Challenging Behavior.

Certain research tasks require extracting data points from graphs and charts Using 91 graphs that presented results from single-case designs, we investigated whether pairs of coders extract similar data from the same graphs (reliability), and whether the extracted data match numerical descriptions of the graph that the original author may have presented in tables or text (validity) Coders extracted data using the UnGraph computer program Extraction proved highly reliable over several different kinds of analyses Coders nearly always extracted identical numbers of data points, and the values they assigned to those data points were nearly identical Extraction also proved highly valid, with the means of extracted data correlating nearly perfectly with means reported in tables or text and with very few discrepancies in any single case These results suggest that researchers can use extracted data with a high degree of confidence that they are nearly identical to the original data

/pdf/using-ungraph-to-extract-data-from-image-files-verification-5atgmllpcc.pdf

Using UnGraph to extract data from image files: verification of reliability and validity.

Theories of how people value and search for information share the assumption that beliefs give rise to the perceived value of information. However, few studies have directly addressed the pre-search processes that influence information-foraging behavior. This experiment examined the influence of pre-search belief updating on the perceived value of information sources. A sample of college students completed a hypothesis-testing, medical-diagnosis task. The experiment used medical tests with equal objective informative value before unveiling a presenting symptom intended to alter the strength of belief in different disease hypotheses. The observed patterns of test selection suggest that changes in beliefs about disease hypotheses result in systematic and predictable changes in test preference—a notion we refer to as the principle of hypothesis-guided search. We also present a simulation of how pre-search processes (e.g., hypothesis generation and working memory capacity) and task variables (e.g., time pressure) influence subsequent information search.

David A. Illingworth

Papers

Using UnGraph to extract data from image files: verification of reliability and validity.

Strength of Belief Guides Information Foraging