Showing papers in "Evaluation in 2008"
TL;DR: The authors proposes ways to use program theory for evaluating aspects of programs that are complicated or complex. But they do not discuss the differences between what is complicated (multiple components) and what is complex (emergent).
Abstract: This article proposes ways to use programme theory for evaluating aspects of programmes that are complicated or complex. It argues that there are useful distinctions to be drawn between aspects that are complicated and those that are complex, and provides examples of programme theory evaluations that have usefully represented and address both of these. While complexity has been defined in varied ways in previous discussions of evaluation theory and practice, this article draws on Glouberman and Zimmerman's conceptualization of the differences between what is complicated (multiple components) and what is complex (emergent). Complicated programme theory may be used to represent interventions with multiple components, multiple agencies, multiple simultaneous causal strands and/or multiple alternative causal strands. Complex programme theory may be used to represent recursive causality (with reinforcing loops), disproportionate relationships (where at critical levels, a small change can make a big difference ...
755 citations
TL;DR: The authors make the point that significant complementarities exist between evaluation and performance measurement and therefore the boundaries between these practices may need to be redefined and evaluators will need to enter into a constructive dialogue with performance management practitioners.
Abstract: This article focuses on what some sceptics see as disillusionment with conventional evaluation practice, in that many governments experience only limited use of evaluation findings.This has contributed to a significant increase in results-based performance measurement. Yet not everyone in the evaluation community welcomes this development.The authors make the point that significant complementarities exist between evaluation and performance measurement and therefore the boundaries between these practices may need to be redefined. In other words, evaluators will need to enter into a constructive dialogue with performance management practitioners. By investigating their methodological similarities and differences, the authors argue that evaluation studies and performance measurement are highly complementary forms of knowledge production. Finally, they argue that evaluation tools may in fact strengthen a number of the identified shortcomings of performance measurement systems when applied in performance manag...
175 citations
TL;DR: Four criteria that help characterize systems of evaluation are outlined, one of which is the existence of a distinctive epistemological perspective; another is that, in order to be labelled a system, evaluation activities are carried out byevaluators within organizational structures and institutions and not only by `lonely' or sole-trader evaluators.
Abstract: Increasingly, in the world of evaluation, `systems of evaluation' have been developed.This article outlines four criteria that help characterize such systems. One criterion is the existence of a di...
100 citations
TL;DR: It is proposed that the theory of negotiated order offers a basis to fill that gap in evaluation practice and to provide promise for theory-informed policy.
Abstract: This article argues that complexity theory has the potential to bring important insights in reframing of the role and practice of evaluation but that the utility of complexity theory needs to be developed to support its application in evaluation research.This article focuses on the implications of a reformed relationship between theory and the empirical setting for what we can learn in evaluating policy. It suggests that while complexity theory provides a new way of looking at causal relationships and how we should approach them, we need to develop complexity-consistent approaches to understanding the interplay of agency and structure at the local level, the level at which explanation is deemed possible in complexity theory. It is proposed that the theory of negotiated order offers a basis to fill that gap in evaluation practice and to provide promise for theory-informed policy.
69 citations
TL;DR: In this paper, transaction cost economics (TCE) is used to evaluate the performance of a hybrid organizational form, where a partnership's governance structure is misaligned with its transactions and higher transaction costs will decrease the partnership's performance.
Abstract: Partnerships are often considered an alternative way to deliver programs provided by governments and organizations (potentially) more cost effectively. However, this assumption needs to be verified. Evaluators and auditors now face a challenge: how to assess the performance of this hybrid organizational form.This article suggests one powerful way of evaluating partnerships: transaction cost economics (TCE). A key hypothesis of TCE is that partners choose a governance structure that minimizes transaction costs (TCs). If a partnership's governance structure is misaligned with its transactions, higher TCs will decrease the partnership's performance. Hence, measuring the partnership's TCs is essential. After defining what constitutes a partnership, the article introduces the TC framework. It then identifies relevant factors in the literature affecting partnership performance. It concludes with key steps in applying the framework and shows how it fits into partnership performance evaluation.
61 citations
TL;DR: In this paper, the authors argue that there has been a move away from a project-based approach to a broader vision of development centred on the Millennium Development Goals, which has led to a change of the evaluation approach: from an assessment of outputs to an evaluation of results.
Abstract: Over the past 20 years, the development industry has changed rapidly and now presents evaluators with a series of new challenges.This article considers this process of change. It argues that there has been a move away from a `project-based' approach to a broader vision of development centred on the Millennium Development Goals, which has led to a change of the evaluation approach: from an assessment of outputs to an evaluation of results. Furthermore, the increasing complexity of ways in which the development process is conceptualized demands increasingly sophisticated and interpretive approaches to evaluation.Whilst there will remain space for positivist approaches in certain sectors of the industry, it is increasingly likely that other approaches will become more dominant.
57 citations
TL;DR: In this paper, the authors discuss the importance of social relations in evaluation, and argue that social relations are critical for evaluating the performance of the evaluation process among stakeholders in the evaluation setting.
Abstract: This article discusses the importance of social relations in evaluation. It focuses first on social relations among stakeholders in the evaluation setting. It is argued that social relations are cr...
48 citations
TL;DR: In this paper, the authors investigate if realist evaluation can be extended to large-scale public sector reforms, in casu: the reform of the electricity market in Denmark, in which five areas of regulation under the reform were selected and hypotheses on the context, mechanism and outcome (CMO) for each intervention were formulated.
Abstract: The realist evaluation (RE) approach is a promising contribution to the literature on theory-driven evaluation.Thus far, RE has mainly been applied to the evaluation of human service programmes, but the aim of this article is to investigate if it can be extended to large-scale public sector reforms, in casu: the reform of the electricity market in Denmark. Five areas of regulation under the reform were selected and hypotheses on the context, mechanism and outcome (CMO) for each intervention were formulated. The evaluation of three of these interventions is summarized: opening markets to foreign trade, price regulation of monopolies and consumer protection. The article concludes that the CMO framework is applicable not just to interventions at the programme level, but also to large-scale public sector reform characterized by high technical complexity and strong policy coalitions. However, it only serves as a general framework, which has to be substantiated.This is done, first, by generating theories — i.e....
39 citations
TL;DR: In this article, the authors examine the relationship of the National Audit Office (NAO) with a range of third parties that shape the performance audit work the NAO undertakes.
Abstract: This article examines the relationships of the National Audit Office (NAO) — the state audit institution (SAI) of the United Kingdom — with a range of third parties that shape the performance audit work the NAO undertakes. In particular, it considers from a practitioner perspective how the NAO has sought to balance its independence with the desire to be responsive to the expectations of others. It concludes that, while independence remains crucial to the credibility of the value for money auditor, examining the connections made by the NAO in its value for money work also helps to explain the hybrid discipline that performance auditing has become in recent years.
39 citations
TL;DR: The US federalist system has been at the origin of evaluation in many ways: providing a laboratory for experimentation of innovative policies, and requiring evaluation of the growing federal intervention from the War on Poverty onwards as discussed by the authors.
Abstract: The US federalist system has been at the origin of evaluation in many ways: providing a laboratory for experimentation of innovative policies, and requiring evaluation of the growing federal intervention from the War on Poverty onwards. Evaluation approaches have been developed that took into consideration the multiple forms of collaboration activated by the federalist system.This experience offers a benchmark for considering evaluation of EU policies and programmes, as it has been elicited by the complex system of EU governance, encompassing both federalist trends — towards concentration of powers at the centre and towards autonomy at the lower levels. First, the article analyses the effects of the centralistic administrative culture of the EU on the way evaluations are conducted at the different levels. Second, it focuses on how the European value added is assessed in the evaluation of two main mechanisms of EU governance that imply a partnership between different jurisdictions: the Structural Funds pro...
27 citations
TL;DR: In this paper, an evaluation conducted in a region of northern Italy, based on the realistic evaluation approach, is described, which is related to the application at regional level of national level of this paper.
Abstract: This article describes an evaluation conducted in a region of northern Italy, based on the realistic evaluation approach.This case study is related to the application at regional level of national ...
TL;DR: In this article, the main actors and patterns of evaluation development patterns differ from those presented so far in this work, and a contribution portraying the institutionalization and practice of evaluation in Switzerland is presented.
Abstract: This contribution portrays the institutionalization and practice of evaluation in Switzerland.The main actors and patterns of evaluation development patterns differ from those presented so far in r...
TL;DR: In this article, the authors focus on negotiation and discuss its relevance for evaluators in participatory evaluation, and make recommendations on how to utilize elements of negotiation in community programme evaluation, including attention to emotional situations, face-giving, rapport and creativity, timing, perceptions and improvisation.
Abstract: This article focuses on negotiation and discusses its relevance for evaluators. Given the impetus for participatory evaluation, evaluators would benefit from improving skills that enable them to make collaborative decisions and work alongside stakeholders, in particular in community evaluations. Negotiation skills are explored through post hoc reflection of a Sure Start programme evaluation in a UK setting. Literature on stakeholder involvement and negotiation is discussed together with the UK evaluation. Recommendations are made on how to utilize elements of negotiation in community programme evaluation. Key skills are highlighted, including attention to: working with emotional situations, face-giving, rapport and creativity, timing, perceptions and improvisation.
TL;DR: In this article, the authors argue that processes of change in local government present specific challenges for evaluation and propose a framework for evaluation based on evaluation research on intermunicipal cooperation.
Abstract: This article, grounded in evaluation research on intermunicipal cooperation, contends that processes of change in local government present specific challenges for evaluation. In the course of data ...
TL;DR: This article reports on the structured literature survey methodology, which was used to identify how and at what stage users of medical devices are engaged in the medical device technology cycle, including methods and tools for evaluating device outcomes.
Abstract: This article reports on the structured literature survey methodology, which was used to identify how and at what stage users of medical devices are engaged in the medical device technology cycle, including methods and tools for evaluating device outcomes.This was not a conventional systematic review of the literature.The stated purpose of the survey is broader than synthesizing best evidence to inform an area of practice and policy, as undertaken in a conventional systematic review.The survey was systematic in the sense that an explicit search strategy was used with inclusion and exclusion criteria to minimize sampling bias. An established qualitative methodology, framework analysis, was used to organize and synthesize major findings from a broad range of healthcare literature.The search strategy and thematic analysis are presented, to contribute to the literature on review strategies, together with the major findings concerning users and medical devices
TL;DR: It is found that in non-evidence-based interventions, program fidelity and adaptation may not be competitors but serve to complement each other, which leads to expanded roles of evaluation in program design and implementation.
Abstract: Based on our experience in helping design, implement, and evaluate a non-evidence-based program, we explore a non-conventional fidelity and adaptation relationship and its implications for program evaluation.We find that in non-evidence-based interventions, program fidelity and adaptation may not be competitors but serve to complement each other.This non-conventional relationship leads to expanded roles of evaluation in program design and implementation.
TL;DR: Complex, area-based initiatives are a widely adopted policy response to problems of health inequalities and social exclusion in industrialized countries such as the UK and the US as mentioned in this paper, and have been widely adopted in the UK.
Abstract: Complex, area-based initiatives are a widely adopted policy response to problems of health inequalities and social exclusion in industrialized countries such as the UK and the US.This article explo...
TL;DR: In this article, the authors argue that in settings permeated by accountability concerns, evaluators need to understand these accountability relationships that so significantly shape their work, especially if they hope to improve accountability systems or create the space for evaluation to serve other important purposes.
Abstract: While evaluators have considered accountability pressures and examined the consequences of performance measurement for organizations, they have paid less attention to the accountability relationship in which their work is used. Drawing on theoretical literature and empirical data, this article shows how these accountability relationships drive the use of evaluative frameworks, like performance measurement, and shape the consequences for the organizations involved.The article argues that in settings permeated by accountability concerns, evaluators need to understand these accountability relationships that so significantly shape their work, especially if they hope to improve accountability systems or create the space for evaluation to serve other important purposes.This article provides evaluators with a set of questions and points of analysis that they can use to examine accountability relationships and expand what is possible for evaluation in settings where accountability is the dominant concern.
TL;DR: In this paper, the authors explore how the challenges and contradictions of such monitoring present themselves in Latin American advocacy activities supported by a Danish NGO and conclude that the improvement of learning-oriented monitoring first and foremost need to be addressed.
Abstract: Advocacy has become an important area of development support. Simultaneously, the interest in learning-oriented monitoring of advocacy programmes has increased. Starting from the premise that learning has sociopolitical dimensions, this article explores how the challenges and contradictions of such monitoring present themselves in Latin American advocacy activities supported by a Danish NGO. Our case-study demonstrates that two largely separate monitoring systems coexist. Side by side with a conventional formal and indicator-based monitoring system, project staff and stakeholders have developed a more informal and dialogical mode of monitoring advocacy. Although the latter has potential advantages from a learning perspective, the article demonstrates that its actual contribution to organizational learning is suboptimal and points to several sociopolitical obstacles and influences that lead to limited learning.The article concludes that the improvement of learning-oriented monitoring first and foremost req...
TL;DR: This study indicates that the concept of HTA has gained acceptance and that several groups want their concerns and priorities to be included in the agencies' agendas, and proposes strategies that may help improve relationships and exchanges between evaluators and their stakeholders.
Abstract: This article uses a sociopolitical perspective to analyse the results of a case study of six Canadian Health Technology Assessment (HTA) agencies and their stakeholders: health care administrators, provider associations, patient associations and the biomedical industry. A total of 40 HTA agency representatives and 46 stakeholders were interviewed.A self-administered survey was also filled out by a larger number of respondents belonging to the four groups (n = 405). Our study indicates that the concept of HTA has gained acceptance and that several groups want their concerns and priorities to be included in the agencies' agendas. Nevertheless, stakeholders question the political autonomy of HTA agencies and their ability to bring about concrete change.This article proposes strategies that may help improve relationships and exchanges between evaluators and their stakeholders. Evaluators must not be afraid to ask for more accountability in the way health technology policies are drafted.
TL;DR: In this article, the authors present experiences from 10 evaluation workshops conducted in western Sweden, where welfare work professionals meet in order to conduct evaluations together with researchers/professional evaluators.
Abstract: Ever increasing demands are being made on welfare organizations to display efficiency. Evaluation workshops constitute a form of learning for the purposes of building up competence to conduct evaluations within welfare organizations, with the support of research and development units. In workshops of this kind, welfare work professionals meet in order to conduct evaluations together with researchers/professional evaluators.This article presents experiences from 10 such evaluation workshops conducted in western Sweden.The workshops were perceived very positively by the participants. While the evaluations are being conducted, the participants also develop a more general competence in this field.The evaluations conducted at the workshops are primarily internal, but with external support, with all the limitations this involves in relation to the possibilities for critical scrutiny. Evaluation workshops have a beneficial effect on the learning of evaluation methods by directly combining learning and conducting...
TL;DR: A Learner-Focused Evaluation Strategy for Developing Medical Education through a Deliberative Dialogue with Stakeholders is presented in this paper, focusing on developing medical education through a dialogue with stakeholders.
Abstract: A Learner-Focused Evaluation Strategy : Developing Medical Education through a Deliberative Dialogue with Stakeholders
TL;DR: In this article, the authors introduce an interpretation method that acknowledges the eclecticism of prevention practice and is also a feasible instrument to enhance the quality of meta-studies, which is illustrated by an evaluation of practices in two Dutch crime prevention programs.
Abstract: Impact studies of prevention programmes, in particular meta-analyses, usually interpret outcome and impact statistics as tests of an underlying theory of prevention. However, these programmes usually combine various interventions linked to different theoretical perspectives. Consequently, the effects of the programme can easily be misinterpreted.This article introduces an interpretation method that acknowledges the eclecticism of prevention practice and is also a feasible instrument to enhance the quality of meta-studies. First, the interventions of the programme are identified and analysed separately. Second, an assessment is made of the arguments that link interventions and the espoused theories.Third, each intervention is represented by a set of scores indicating the types of links and core theoretical assumptions.These scores are aggregated to programme scores and included as independent variables in meta-analyses.The method is illustrated by an evaluation of practices in two Dutch crime prevention in...