Complex interventions are more than the sum of their parts, and interventions need to be better theorised to reflect this 

Many people think that standardisation and randomised controlled trials go hand in hand. Having an intervention look the same as possible in different places is thought to be paramount. But this may be why some community interventions have had weak effects. We propose a radical departure from the way large scale interventions are typically conceptualised. This could liberate interventions to be responsive to local context and potentially more effective while still allowing meaningful evaluation in controlled designs. The key lies in looking past the simple elements of a system to embrace complex system functions and processes.

The suitability of cluster randomised trials for evaluating interventions directed at whole communities or organisations remains vexed.1 It need not be.2 Some health promotion advocates (including the WHO European working group on health promotion evaluation) believe randomised controlled trials are inappropriate because of the perceived requirement for interventions in different sites to be standardised or look the same.1 3 4 They have abandoned randomised trials because they think context level adaptation, which is essential for interventions to work, is precluded by trial designs. An example of context level adaptation might be adjusting educational materials to suit various local learning styles and literacy levels.

Lead thinkers in complex interventions, such as the UK's Medical Research Council, also think that trials of complex interventions must “consistently provide as close to the same intervention as possible” by “standardising the content and delivery of the intervention.”5 By contrast, however, they do not see this as a reason to reject randomised controlled trials.

These divergent views have led to problems on two fronts. Firstly, the field of health promotion is being turned away from randomised …

/pdf/complex-interventions-how-out-of-control-can-a-randomised-4h53nwygb9.pdf

Complex interventions: how "out of control" can a randomised controlled trial be?

Public health interventions tend to be complex, programmatic, and context dependent. The evidence for their effectiveness must be sufficiently comprehensive to encompass that complexity. This paper asks whether and to what extent evaluative research on public health interventions can be adequately appraised by applying well established criteria for judging the quality of evidence in clinical practice. It is adduced that these criteria are useful in evaluating some aspects of evidence. However, there are other important aspects of evidence on public health interventions that are not covered by the established criteria. The evaluation of evidence must distinguish between the fidelity of the evaluation process in detecting the success or failure of an intervention, and the success or failure of the intervention itself. Moreover, if an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently faulty (that is, failure of intervention concept or theory), or just badly delivered (failure of implementation). Furthermore, proper interpretation of the evidence depends upon the availability of descriptive information on the intervention and its context, so that the transferability of the evidence can be determined. Study design alone is an inadequate marker of evidence quality in public health intervention evaluation.

https://jech.bmj.com/content/jech/56/2/119.full.pdf

Criteria for evaluating evidence on public health interventions

This article addresses various aspects of health promotion evaluation that are under debate. The paper opens by noting the recent growth in research directed toward health promotion and its evaluation. Next the definition of "success" in health promotion is discussed in terms of 1) valued outcomes and valued processes 2) health outcomes 3) health outcomes and health promotion outcomes and 4) measurement of outcomes. The next section describes how best to evaluate "success" by presenting a hierarchy of evaluation stages (problem definition solution generation testing innovations intervention demonstration intervention dissemination and program management). Analysis of ways to assess cause and effect in evaluating outcome considers experimental designs to attribute cause and effect alternatives to experimental designs strategic issues in evaluating community/population interventions community interventions and social movements and building evidence using multiple methods and multiple sources of data. Examination of the evaluation of the process looks at program reach program acceptability and program integrity. The article concludes that the four key challenges to the evaluation of health promotion are 1) using research evidence more systematically while planning activities 2) improving the definition and measurement of outcome 3) adopting an appropriate evaluation intensity and 4) adopting an appropriate evaluation design.

Evaluating Health Promotion—Progress, Problems and solutions

SUMMARY AND CONCLUSION Faced with the demand for evidence-based practice on one side and the supplyof idiosyncratic sources of evidence onthe other, health promotion practitioners,planners, and policy makers have had toadapt the evidence to their regional, lo-cal, or organizational circumstances onthe fly. With professional judgment andcommunity input, most local programshave made good use of evidence when itseemed relevant, but more often ignoredit because it felt foreign. The foreign feelof the evidence stems from its unrepre-sentative sources, either in the artifici-ality of the circumstances of the researchor in the socioeconomic character of thesubjects or the setting. The CDC-derived“best practices” for tobacco control, forexample, came in the first edition fromthe states of California, Massachusetts,and Oregon. Other states have had diffi-culty identifying with those three states,their populations, and the circumstancesunder which they were able to mounttheir programs in tobacco control.Such misfitting of evidence and prac-tice is pervasive and probably inherent inhealth promotion. Even if the number ofsettings in which the research could bereplicated were multiplied by the numberof states, provinces, or countries in whichit might be applied, the changing politi-cal, economic, and other time-dependentcircumstances between the research andthe application in practice might makethe research suspect.These observations need not lead us toa nihilistic or postpositivist position ofdismissing all evidence that is not localand immediate. I have suggested at least6 ways to cope with these problems ofpractice adapting to evidence or adaptingevidence to practice. Each has prece-dents in practice and some lines of devel-opment in social, behavioral and healthpromotion research. These suggestionsare entirely consistent with the philoso-phies and tenets underpinning healthpromotion practice, and each warrants aresearch agenda of its own.

Deirdre Degeling

Papers

Evaluating health promotion : a health worker's guide

Evaluating Health Promotion

Evaluación en promoción de la salud: guía para trabajadores de la salud