OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.

/pdf/evaluating-non-randomised-intervention-studies-2k25cz7x6r.pdf

Evaluating non-randomised intervention studies.

Titles & abstract s Titles & abstracts Include N=54 Include N=121 Include N=89 Exclude N = 0 Exclude N = 1,024 Exclude N = 21

Guidance on the conduct of narrative synthesis in systematic Reviews. A Product from the ESRC Methods Programme. Version 1

Guidance on the conduct of narrative synthesis in sytematic reviews

Writing the proposal for a qualitative research methodology study is a double challenge because of the emergent nature of qualitative research design and because a methodology study entails describing a process to produce a process. How the authors addressed this challenge is shown in the annotated text of the grant proposal—“Analytic Techniques for Qualitative Metasynthesis”—funded by the National Institute of Nursing Research. Appealing qualitative research proposals adhere to principles that engage writers and readers in an informative and mutually respectful interaction.

/pdf/writing-the-proposal-for-a-qualitative-research-methodology-3uqa2l1wh4.pdf

Writing the Proposal for a Qualitative Research Methodology Project

BACKGROUND

Little is known about the effects of chemotherapy on cognitive function. The purposes of this metaanalysis were to estimate the effect sizes for the effect of chemotherapy on each domain of cognitive function and to differentiate effect sizes by each method of comparison of effects (i.e., normative data, control group, or baseline data).



METHODS

Sixteen studies that evaluated cognitive function in chemotherapy patients were included in the study. DSTAT metaanalysis software was used to calculate an effect size and confidence intervals for each neuropsychologic test. Tests were assigned to a specific cognitive domain, and an average effect size was determined for each domain.



RESULTS

Only one domain of cognitive function (i.e., visual memory) had significant chemotherapy-induced impairment across all comparison types. However, when the neuropsychologic test scores of chemotherapy patients were compared with normative data, significant effect sizes were found for four domains of cognitive function (i.e., executive function, information processing speed, verbal memory, visual memory). In addition, significant, albeit small, effect sizes were found for language and verbal memory when chemotherapy patients' test scores were compared with test scores of healthy matched controls. All significant averaged effect sizes were in the negative direction, indicating that mean scores on neuropsychologic tests for patients who had received chemotherapy were on average lower than comparison scores.



CONCLUSION

Data from this metaanalysis supported the hypothesis that chemotherapy can have a negative impact on cognitive function. However, most deficits in this study ranged from small to moderate and were nonsignificant. Cancer 2005. © 2005 American Cancer Society.

https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.1002/cncr.21469

A metaanalysis of studies of the effects of cancer chemotherapy on various domains of cognitive function

Proposed nurse educator competencies: Development and validation of a model

Meta-analysis has gained international recognition as a statistical approach to research synthesis. What is missing in research reviews, however, is the systematic compilation of results from non-experimentally designed studies. An approach is described which extends the rigours of integrative research review and meta-analysis to descriptive studies. A model for conceptualizing a systematically derived research synthesis is presented with selected accompanying generic elements. The model serves as a prototype for application to a variety of topical areas of interest. Generic elements contained in the prototype may be tailored to specific topics with relative ease. Resources and time required for topic development from conceptualization through to dissemination can be considerably reduced by adaptation of the generic elements.

A prototype for integrative review and meta-analysis of nursing research

The systematic accumulation of widely scattered research results is essential to developing a scientific basis for clinical practice. The purposes of this study were to describe 10 years of patient-related oncology nursing research in the United States, assess effectiveness of interventions, and test a prototype for research synthesis projects. Using a nursing framework and strategies of integrative research review and meta-analysis, 428 studies were selected for the study. The studies were found to be predominantly descriptive (90%), with convenience samples (79%), and a median sample size of 48. Subjects were predominantly in treatment or rehabilitation (52%), with demographic and stage of disease characteristics largely unreported. Of the 428 studies, 42 met the criteria for meta-analytic treatment. Interventions were found to be significantly effective, with membership in the intervention group improving success rate by 22%. The prototype was judged highly for adherence to standards and usefulness for adaptation to other research syntheses. Because of wide diversity in topics studied, missing information on subjects, and lack of replication, implications for research-based practice are premature at this time. Recommendations for future research include new studies, replications, and new research syntheses with an international scope for nurses studying and providing cancer care.

An integrative review and meta-analysis of oncology nursing research: 1981-1990.

Graduate education for nursing students who have English as a second language.

This article takes the flow-chart approach used in a previous article to dcscribe the process of conducting an integrative research review/meta-analysis (IRR/MA) and applies it on a practical level. Using an actual topic on head injured adults, the authors present the steps that the investigators used from the beginning grant application idea through Cooper's five-stage process. The glossary and coding forms used during this study are presented so that the reader can gaina clearer idea of the detial that goes into any IRR/MA project.

Elizabeth Stullenbarger

Papers

Proposed nurse educator competencies: Development and validation of a model

A prototype for integrative review and meta-analysis of nursing research

An integrative review and meta-analysis of oncology nursing research: 1981-1990.

Graduate education for nursing students who have English as a second language.

Integrative Review and Meta-Analysis: An Application