OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.

/pdf/evaluating-non-randomised-intervention-studies-2k25cz7x6r.pdf

Evaluating non-randomised intervention studies.

A meta-analysis was conducted on controlled clinical trials investigating adaptations of motivational interviewing (AMIs), a promising approach to treating problem behaviors. AMIs were equivalent to other active treatments and yielded moderate effects (from .25 to .57) compared with no treatment and/or placebo for problems involving alcohol, drugs, and diet and exercise. Results did not support the efficacy of AMIs for smoking or HIV-risk behaviors. AMIs showed clinical impact, with 51% improvement rates, a 56% reduction in client drinking, and moderate effect sizes on social impact measures (d=0.47). Potential moderators (comparative dose, AMI format, and problem area) were identified using both homogeneity analyses and exploratory multiple regression. Results are compared with other review results and suggestions for future research are offered.

/pdf/the-efficacy-of-motivational-interviewing-a-meta-analysis-of-2hycrwhc30.pdf

The efficacy of motivational interviewing: a meta-analysis of controlled clinical trials.

Research is reviewed on the association between alcohol outcome expectancies and consumption which has led many to argue that manipulating expectancies might be a route to manipulating consumption for problem prevention and treatment. Studies indirectly and directly evaluating this latter position are reviewed. Expectancies predicting treatment outcome: two studies have shown that the more positive expectancies held at treatment, the poorer is treatment outcome, but five other studies have failed to find this. Three related studies have shown that the more negative expectancies held at treatment, the better the treatment outcome. This evaluation provides evidence inconsistent with the main position for positive expectancy and limited support for negative. Expectancy manipulations and ad libitum consumption: three studies in the laboratory have shown that increasing positive expectancies through word priming increases subsequent consumption and two studies have shown that increasing negative expectancies decreases it. A single study in the field showed a similar relationship. This evaluation provides evidence consistent with the main position but is limited by measuring consumption changes over only 1-2 hours. Prevention programmes with expectancy components: seven projects are reviewed in which positive expectancies were targeted, but only two report an expectancy change analysis and in both cases the expectancy change did not relate to subsequent consumption. This evaluation provides evidence inconsistent with the main position. Expectancy challenge: two related studies are reviewed in which positive expectancy challenges reduce subsequent consumption but changes in expectancy were not evaluated as predictors of consumption change. Two studies are reviewed which found a reduction in positive expectancy following expectancy challenge but no reduction in consumption. One study is reviewed in which when negative expectancy was increased in treatment there was a better treatment outcome at 3 months follow-up than when it was not. This evaluation provides evidence inconsistent with the main position for positive expectancy and limited consistent evidence for negative. It is concluded that the research has still to be done that securely links expectancy manipulations with subsequent changes in consumption, and fulfils the early promise from association studies.

A review of expectancy theory and alcohol consumption.

This article evaluates the efficacy, effectiveness, and clinical significance of empirically supported couple and family interventions for treating marital distress and individual adult disorders, including anxiety disorders, depression, sexual dysfunctions, alcoholism and problem drinking, and schizophrenia. In addition to consideration of different theoretical approaches to treating these disorders, different ways of including a partner or family in treatment are highlighted: (a) partner-family-assisted interventions, (b) disorder-specific partner-family interventions, and (c) more general couple-family therapy. Findings across diagnostic groups and issues involved in applying efficacy criteria to these populations are discussed.

/pdf/empirically-supported-couple-and-family-interventions-for-2x1ny48v4l.pdf

Empirically Supported Couple and Family Interventions for Marital Distress and Adult Mental Health Problems

Aims : (1) To examine whether or not motivational interviewing (MI) is more efficacious than no intervention in reducing alcohol consumption; (2) to examine whether or not MI is as efficacious as other interventions. Method : A literature search followed by a meta-analytic review of randomized control trials of MI interventions. Aggregated between-group effect sizes and confidence intervals were calculated for each study. Results : Literature search revealed 22 relevant studies, of which nine compared brief MI with no treatment, and met methodological criteria for inclusion. In these, the aggregate effect size was 0.18 (95% C.I. 0.07, 0.29), but was greater 0.60 (95% C.I. 0.36, 0.83) when, in a post-hoc analysis, the follow-up period was three months or less. Its efficacy also increased when dependent drinkers were excluded. There were nine studies meeting methodological criteria for inclusion which compared brief MI with another treatment (one of a diverse set of interventions), yielding an aggregate effect size of 0.43(95% C.I. 0.17, 0.70). The literature review pointed to several factors which may influence MI's long-term efficacy effectiveness of MI. Conclusions : Brief MI is effective. Future studies should focus on possible predictors of efficacy such as gender, age, employment status, marital status, mental health, initial expectations, readiness to change, and whether the population is drawn from treatment-seeking or non-treatment-seeking populations. Also, the components of MI should be compared to determine which are most responsible for maintaining long-term changes.

( Received 11 July 2005; first review notified 24 July 2005; in revised form 13 February 2006; accepted 16 February 2006)

/pdf/the-efficacy-of-motivational-interviewing-as-a-brief-1ni744udgw.pdf

TH Bien

Papers

What works? A methodological analysis of the alcohol treatment outcome literature.