OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.

/pdf/evaluating-non-randomised-intervention-studies-2k25cz7x6r.pdf

Evaluating non-randomised intervention studies.

What is Qualitative Interviewing? is an accessible and comprehensive ‘what is’ and ‘how to’ methods book. It is distinctive in emphasizing the importance of good practice in understanding and undertaking qualitative interviews within the framework of a clear philosophical position. Rosalind Edwards and Janet Holland provide clear and succinct explanations of relevant philosophies and theories of how to know about the social world, and a thorough discussion of how to go about researching it using interviews. A series of short chapters explain a range of interview types and practices. Drawing on their own and colleagues’ experiences Edwards and Holland provide real research examples as informative illustrations of qualitative interviewing in practice and the use of creative interview tools. They discuss the use of new technologies as well as tackling enduring issues around asking and listening, and power dynamics in research. Written in a clear and accessible style the book concludes with a useful annotated bibliography of key texts and journals in the field. What is Qualitative Interviewing? provides a vital resource for both new and experienced researchers across the social science disciplines.

What is Qualitative Interviewing

Despite increasing incidence of HIV/AIDS, there has been no systematic review of correlates of condom use among heterosexual samples. To rectify this, the present study used meta-analysis to quantify the relationship between psychosocial variables and self-reported condom use. Six hundred sixty correlations distributed across 44 variables were derived from 121 empirical studies. Variables were organized in terms of the labeling, commitment, and enactment stages of the AIDS Risk Reduction Model (Catania, Kegeles, & Coates, 1990). Findings showed that demographic, personality, and labeling stage variables had small average correlations with condom use. Commitment and enactment stage variables fared better, with attitudes toward condoms, behavioral intentions, and communication about condoms being the most important predictors. Overall, findings support a social psychological model of condom use highlighting the importance of behavior-specific cognitions, social interaction, and preparatory behaviors rather than knowledge and beliefs about the threat of infection.

Psychosocial correlates of heterosexual condom use: a meta-analysis.

This critical review assesses whether evaluation studies can answer three key questions about behaviour change interventions: ‘Do they work? How well do they work? How do they work?’ Reviews of intervention evaluations are examined, particularly those addressing decreasing unprotected sexual intercourse and smoking. Selection of outcome measures and calculation of effect sizes are discussed. The article also considers the extent to which evaluation reports specify (i) discrete intervention techniques and (ii) psychological mechanisms that account for observed behavioural change. It is concluded that intervention descriptions are often not specific about the techniques employed and that there is no clear correspondence between theoretical inspiration and adoption of particular change techniques. The review calls for experimental testing of specific theory-based techniques, separately and in combination.

/pdf/interventions-to-change-health-behaviours-evidence-based-or-4wpdpcpuv9.pdf

Interventions to change health behaviours: evidence-based or evidence-inspired?

We need less research, better research, and research done for the right reasons

What should we think about a doctor who uses the wrong treatment, either wilfully or through ignorance, or who uses the right treatment wrongly (such as by giving the wrong dose of a drug)? Most people would agree that such behaviour was unprofessional, arguably unethical, and certainly unacceptable.

What, then, should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature, in both general and specialist journals, have shown that all of the above phenomena are common.1 2 3 4 5 6 7 This is surely a scandal.

When I tell friends outside medicine that many papers published in medical journals are misleading because of methodological weaknesses they are rightly shocked. Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods …

/pdf/the-scandal-of-poor-medical-research-aar1pxmrh5.pdf

The scandal of poor medical research.

The `Inventing Adulthoods' study seeks to document transitions to adulthood reported by over 100 young people living in five contrasting communities in the UK over a five-year period. A principal a...

Critical moments: Choice, chance and opportunity in young people's narratives of transition

Objectives: To locate reports of sexual health education interventions for young people, assess the methodological quality of evaluations, identify the subgroup with a methodologically sound design, and assess the evidence with respect to the effectiveness of different approaches to promoting young people9s sexual health. Design: Survey of reports in English by means of electronic databases and hand searches for relevant studies conducted in the developed world since 1982. Papers were reviewed for eight methodological qualities. The evidence on effectiveness generated by studies meeting four core criteria was assessed. Judgments on effectiveness by reviewers and authors were compared. Papers: 270 papers reporting sexual health interventions. Main outcome measure: The methodological quality of evaluations. Results: 73 reports of evaluations of sexual health interventions examining the effectiveness of these interventions in changing knowledge, attitudes, or behavioural outcomes were identified, of which 65 were separate outcome evaluations. Of these studies, 45 (69%) lacked random control groups, 44 (68%) failed to present preintervention and 38 (59%) postintervention data, and 26 (40%) omitted to discuss the relevance of loss of data caused by drop outs. Only 12 (18%) of the 65 outcome evaluations were judged to be methodologically sound. Academic reviewers were more likely than authors to judge studies as unclear because of design faults. Only two of the sound evaluations recorded interventions which were effective in showing an impact on young people9s sexual behaviour. Conclusions: The design of evaluations in sexual health intervention needs to be improved so that reliable evidence of the effectiveness of different approaches to promoting young people9s sexual health may be generated. Key messages Key messages There should be greater awareness among health education researchers, practitioners, and policy makers of the need to base sexual health education strategies for young people on sound evidence of effectiveness Journals should refuse to accept methodo- logically flawed papers Funders should refuse to support studies with methodologically flawed designs By using the best evidence available a randomised controlled trial should be under- taken of a sexual health education intervention for young people with 5–10 years of follow up

/pdf/sexual-health-education-interventions-for-young-people-a-1s7o4w9ked.pdf

Sexual health education interventions for young people: a methodological review

'Inventing Adulthoods' offers a groundbreaking new perspective on young people's experiences of growing up at the turn of the 21st century, arguing that a biographical approach is vital to understanding the holistic and dynamic character of their lives. Based on a study of a diverse group of young people over a 10 year period, the book explores high profile policy issues - education, employment, drugs, violence and well-being. It also considers the significance of things that mean the most to young people themselves: mobility, home, belonging, intimacy and social life.

Inventing Adulthoods: A Biographical Approach to Youth Transitions

Traditionally adulthood and citizenship have been synonymous. Yet adulthood is changing. In this paper we explore how young people's evolving understandings of adulthood may contribute towards an understanding of citizenship within the broader context of increasingly extended and fragmented transitions. The paper draws on a unique qualitative longitudinal data set in which 100 young people, from contrasting social backgrounds in the United Kingdom, have been followed over a five-year period using repeat biographical interviews. We present first the themes that emerged from a cross-cut analysis of the first of three rounds of interviews distinguishing between relational and individualised understandings of adulthood. We then present a model we developed to capture the ways that young people sought out opportunities for competence and recognition in different fields of their lives. Finally a case study that follows a young woman through her three interviews illustrates how these themes can appear in an indi...

Inventing adulthoods: a biographical approach to understanding youth citizenship

The `Inventing Adulthoods' study seeks to document transitions to adulthood reported by over 100 young people living in five contrasting communities in the UK over a five-year period. A principal aim of the study is to identify `critical moments' in young people's biographies and to explore how these moments are implicated in processes of social inclusion and exclusion. This article reports on an analysis of the first of three rounds of one-to-one interviews. We begin by mapping young people's critical moments, exploring the relationship between the social and geographical location in which they live and the kinds of events that they report as having particular biographical significance. We suggest that the character of these `critical moments' is socially structured, as are young people's responses to them. The argument is illustrated by case studies that show the interaction of choice, chance and opportunity in three young people's lives.

Sheena McGrellis

Papers

Critical moments: Choice, chance and opportunity in young people's narratives of transition

Sexual health education interventions for young people: a methodological review

Inventing Adulthoods: A Biographical Approach to Youth Transitions

Inventing adulthoods: a biographical approach to understanding youth citizenship

Critical moments: choice, chance and opportunity in young people's narratives of transition to adulthood