Getting what you pay for: the challenge of measuring success in conservation

TL;DR: Howe and Milner-Gulland as mentioned in this paper investigated how much agreement there was in rankings of project success as evaluated using different indices (one based on reported outputs, and two based on subjective scoring of information about outcomes) and also, which explanatory variables best predicted success as defined by different indices.

Abstract: Imagine a construction project whose objective is to build a bridge. The quality of the resulting bridge might be debated and disputed, but that it has been delivered, and that the project is responsible for its delivery, is usually relatively simple to verify. It is much more difficult to evaluate success where a project aims to deliver something whose existence is costly or technically challenging to monitor, and something whose status may be affected (positively or negatively) by a range of influences that have nothing to do with the project. Conservation interventions may seek to influence the conservation status of a range of species, improve the management of an area of habitat, or change the attitudes and behaviour of a population of people. All of these things are challenging to measure in themselves. Importantly, they can also be affected by a myriad of factors external to a project (global commodity prices, national politics or law enforcement, and shifting social norms) making it challenging to tease apart the influence of the project on the sought after outcome. Conservation projects have widely been criticized in the past for poor evaluation (Saterson et al., 2004; Brooks et al., 2006). In their paper, Howe & Milner-Gulland (2012) look at the question of what indices are appropriate for evaluating success in conservation. Howe & Milner-Gulland (2012) follow others by distinguishing outputs (the amount of something delivered by a project, e.g. number of workshops held, papers published or posters distributed) and outcomes (the long-term consequences of the project, e.g. change in population size of a target species). Outcomes are what the project ultimately aims to deliver, but they can be very costly to measure. A recent study of the costs of monitoring presence or absence of a variety of species of conservation concern in the dry forests of Madagascar illustrates the challenge of monitoring outcomes directly. Sommerville, Milner-Gulland & Jones (2011) found that monitoring that could robustly detect change over time would be unrealistically costly for the vast majority of species as would cost more than the budget for the entire intervention. The UK government launched its Darwin Initiative at the Rio summit in 1992. Since then, it has invested £88 million in biodiversity conservation projects in 154 countries (DEFRA, 2012). This fantastic programme provided Howe & Milner-Gulland (2012) with an unrivalled opportunity to investigate how much agreement there was in rankings of project success as evaluated using different indices (one based on reported outputs, and two based on subjective scoring of information about outcomes) and also, which explanatory variables best predicted success as defined by the different indices. Their finding that ranking of projects using the outputs-based indicator was well correlated with the ranking from the subjective outcomes measure is interesting and worthy of further exploration. However, as the authors themselves note, there is no quantitative, independent data on outcomes available against which to measure the success of the various indices. Because outcomes are so difficult to measure directly, and may also not be achieved over the small timescale of a funded project, indices based on outputs will always be needed. Underlying this approach is an assumption that there is a mechanism that links delivery of the outputs with delivery of outcomes. This is often not explicit. If assumptions as to linkages between outputs and outcomes were more explicitly spelt out, both in project proposals and reports, alongside the evidence upon which the assumption is based, output measures would become more valuable for assessing project success. Howe & Milner-Gulland (2012) also investigated the internal consistency (how different assessors would score an individual project using the same index) of two of the possible indices. While they found a high level of agreement between different assessors scoring the same projects with the same index, the existence of an outlier was revealing. The majority of their assessors came from a very similar academic background, while the one from a different discipline (pest-management rather than conservation), scored projects quite differently. It is likely that world view plays an important role in what an individual considers as success in bs_bs_banner