For over thirty years now, many evaluators have recommended making explicit the underlying assumptions about how programs are expected to work-the program theory-and then using this theory to guide the eval- uation. In this chapter, we provide an overview of program theory evalu- ation (PTE), based on our search to find what was value-added to evaluations that used this approach. We found fewer clear-cut, full-blown examples in practice than expected, but we found many interesting vari- ations of PTE in practice and much to recommend it. And elements of PTE, whether the evaluators use the terminology or not, are being used in a wide range of areas of concern to evaluators. Based on this review, in this chapter we discuss the practice, promise, and problems of PTE.

Program theory evaluation: Practice, promise, and problems

International aid agencies face major problems when attempting to evaluate their achievements. Activities, intended beneficiaries, partner institutions and social contexts are diverse because of their global scale. How can agencies’ ‘theories of change’ be adequately represented in summary forms that respect the complexity and diversity? This article, the first of two, looks at a range of types of change processes and how they can be represented. It starts with linear processes, traditionally represented via the Logical Framework, and ends with network processes. The solutions proposed are informed by three related cross-disciplinary theoretical perspectives: evolutionary theory; complex adaptive systems; and social network analysis. Examples draw on the author’s consultancy experience with international development aid programmes in Vietnam, Bangladesh, Cameroon and Burkina Faso.

Scale, Complexity and the Representation of Theories of Change

/pdf/targeting-outcomes-of-programs-a-hierarchy-for-targeting-4uth85m6uy.pdf

Targeting Outcomes of Programs: A Hierarchy for Targeting Outcomes and Evaluating Their Achievement

The views expressed and the conclusions reached in this publication are those of the author and not necessarily those of persons consulted. RIRDC shall not be responsible in any way whatsoever to any person who relies in whole or in part on the contents of this report. ii Foreword Greater emphasis on accountability in the delivery of extension programs has put new focus on techniques for evaluating such programs. This report discusses the discipline of program evaluation and then reviews five main forms of evaluation and illustrates them with case studies drawn from agricultural extension in Australia. It is hoped that improved access to sound information on evaluation methods will be reflected in better planning of agricultural extension projects, more effective monitoring and evaluation of programs, and hence more efficient use of industry funds and greater benefits for clients of all research and development programs in rural Australia. This report belongs to RIRDC's range of over 250 research publications and forms part of our Human Capital, Communications and Information Systems R&D program, which aims to enhance human capital and facilitate innovation in rural industries and communities. iii Acknowledgments Firstly we would like to thank all the people who contributed to the Review by sending us evaluation reports and answering our inquiries. All contributors are listed in Appendix 4. Special acknowledgment must be given to the practitioners who conducted the evaluations that make up the case studies in this report. Without their work we would have no concrete examples of evaluation to illustrate the various approaches to evaluation that have been taken in agricultural extension today. Secondly we would like to thank and acknowledge the support of Dr. Barrie Bardsley for his timely advice and input into the Review. His balanced critique of early drafts greatly influenced the final product. We would also like to thank John Owen, whose book Program Evaluation: Forms and Approaches greatly helped us to understand the various approaches to evaluation from the discipline of program evaluation per se. Also a big thank you to Jerome Winston from RMIT for his advice and encouragement to a group of non-professional evaluators exploring the vast field of evaluation.

Review of evaluation in agricultural extension

Lifetime Ewe Management is an extension program designed to assist sheep producers to improve their understanding of ewe nutrition and to develop the skills and confidence to improve their management. The course is based on a small-group extension model and was developed by the Lifetimewool project as a way to incorporate the research findings, economic modelling and producer guidelines developed by the project. Lifetime Ewe Management commenced in Victoria in the spring of 2006 and by the end of 2010, 221 producers had completed the 2-year program. The changes in knowledge, attitudes, skills, aspirations and management practices of 182 of these participants were examined. Participants of the Lifetime Ewe Management program increased their whole-farm stocking rates by 14%, increased lamb marking percentages by 11–13% depending on enterprise type, and decreased ewe mortality rates by 43%. These improvements resulted from a significant change in the perceived importance of managing ewes to condition-score targets to improve profitability and increases in the ability of participants to condition score ewes, assess pasture quantity and quality and feed budget. These changes were consistent regardless of how innovative the participants were at the beginning of the program. The appeal and success of the program was attributed largely to the small-group model where producers worked with their own flock under the guidance of a skilled facilitator and with access to effective decision-making tools. The Lifetime Ewe Management program design provides a blueprint for future extension programs striving to achieve widespread practice change.

/pdf/participation-in-lifetime-ewe-management-results-in-changes-54nks9esep.pdf

Paul J. Viel

Papers

Up the Hierarchy