Showing papers in "American Journal of Evaluation in 2011"

A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009

Abstract: Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen’s 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended.

Known Knowns, Known Unknowns, Unknown Unknowns: The Predicament of Evidence-Based Policy

Abstract: The authors present a case study examining the potential for policies to be “evidence-based.” To what extent is it possible to say that a decision to implement a complex social intervention is warranted on the basis of available empirical data? The case chosen is whether there is sufficient evidence to justify banning smoking in cars carrying children. The numerous assumptions underpinning such legislation are elicited, the weight and validity of evidence for each is appraised, and a mixed picture emerges. Certain propositions seem well supported; others are not yet proven and possibly unknowable. The authors argue that this is the standard predicament of evidence-based policy. Evidence does not come in finite chunks offering certainty and security to policy decisions. Rather, evidence-based policy is an accumulative process in which the data pursue but never quite capture unfolding policy problems. The whole point is the steady conversion of “unknowns” to “knowns.”

Measures of Program Effectiveness Based on Retrospective Pretest Data: Are All Created Equal?

Abstract: This study analyzed data from four evaluation designs incorporating the retrospective pretest (ie, thentest), analyzing the effects of self-report pretesting and post-program survey format on a set of self-report measures Validity of self-report data was assessed by comparing the criterion validity of then ratings to established benchmarks and by including a control condition This study found that designs incorporating separate posttest and thentest surveys yielded the most comparable levels of criterion validity for then ratings and the least biased measures of program effectiveness Conversely, designs that incorporated a single post-program survey, with adjacent post and then items, yielded the least comparable levels of criterion validity for then ratings and the most biased measures of program effectiveness This study also found a slight interaction between the effects of self-report pretesting and post-program survey format Implications for program evaluation are discussed

Heterogeneous Roles and Heterogeneous Practices: Understanding the Adoption and Uses of Nonprofit Performance Evaluations

Abstract: Evaluating the performance of nonprofit organizations has been of growing importance for the last several decades. Although there is much good that can come out of self-improvement, there is substantial heterogeneity within the sector that calls into question the usefulness of ‘‘across the board’’ evaluation tools. In this article, the authors assess nonprofit evaluation practices, at both the organizational and the programmatic levels. Through a multitheoretical framework, the authors explore the extent to which the adoption and uses of evaluation reflect strategic alignment with heterogeneous nonprofit roles or the institutionalization of the organizational environment. The authors find evidence that institutional perceptions are consistent predictors of whether nonprofit organizations adopt particular practices. Diverse roles are not consistently associated with the adoption of particular practices, but the uses of evaluation are diverse and specific to nonprofit organizations’ unique strategies and roles.

Measuring Evaluation Capacity—Results and Implications of a Danish Study:

Abstract: Evaluation capacity building (ECB) has become an area in the evaluation field that is attracting both scholarly attention and the practical work of evaluators. Despite some recent contributions, differing conceptions still exist concerning what constitutes ECB, let alone the nature of the capacity being built. Responding to an earlier call on the need for conceptual clarity regarding evaluation capacity (EC), this article discusses a model and a measurement tool used to map EC in Danish public sector organizations. Using empirical evidence as its basis, this article discusses the strengths and limitations of the conceptual model in relation to its face, content, and construct validity. Finally, the article discusses the implications arising from empirical findings and recent theoretical developments and suggests potential steps for improving conceptual clarity in the future.

A Catalyst-for-Change Approach to Evaluation Capacity Building

Abstract: Evaluation capacity building (ECB) has become a popular approach for helping community-based organizations (CBOs) to meet their funders’ demands for accountability. This case study reports the ECB process with one staff member using a catalyst-for-change approach. The authors analyzed the role of the catalyst in diffusing evaluation knowledge and skills and in promoting evaluation mainstreaming and use within the CBO; and the outcomes achieved as a result of the catalyst’s role in the process of building evaluation capacity. Implications based on the study findings are discussed.

The Politics and Consequences of Including Stakeholders in International Development Evaluation

Abstract: Participatory evaluation approaches have a relatively long history of advocacy and application in the international development evaluation community. Despite widespread use and apparent resonance with practitioners and donors alike, very little empirical research exists on why and how participatory evaluation approaches are used in international development settings. In this article, we present results derived from a mixed method investigation of a sample of practicing international development evaluators regarding their perceptions of how and why stakeholders are included in international development evaluations. Findings suggest that participatory evaluation approaches are interpreted and practiced in widely differing ways. Implications for international development evaluation practice and future research are discussed.

Politics and Evaluation: More Than Methodology

Abstract: In the 1960s, evaluation thrived in the sunshine of political support for our work. We also experienced the dark side of the force of politics: efforts to introduce partisan biases favorable to special interests. In response to politics as contamination, at least two families of stances developed: populist and public interest. The populist stances embraced the politics of extensive stakeholder involvement, democratic deliberation, collaboration, social justice, and empowering the disenfranchised. The public-interest-stances continued to explore strategies for carrying out evaluations of high-stakes, controversial, large-scale national policy issues, being wary of political bias in any form and from any source. How have these stances developed and played out over the past 25 years? Whether the originating theory is populist or public interest, we may have more in common in practice than we may think. These communalities could form a necessary political toolkit to join our methodological knowledge.

Grounding Evaluations in Culture.

Abstract: The emergence of and the attention given to culture in the evaluation field over the last decade has created a heightened awareness of and need for evaluators to understand the complexity and multidimensionality of evaluations within multicultural, multiracial, and cross-cultural contexts. In this article, the authors discuss how cultural considerations can be incorporated within other evaluation approaches (e.g., school-based evaluation). The authors begin with a selective review on the rapidly growing literature on culture and evaluation, with particular attention to culturally responsive evaluation. After sketching the foundations and implementation of this approach, designated as culturally relevant democratic inquiry (CDI), the authors present an excerpt from a CDI pilot the authors have conducted. The article concludes with an analysis of the potential benefits and current challenges to implementing CDI and integrating culture into evaluations in general.

Evaluating Methods for Estimating Program Effects.

Abstract: I define a treatment effect in terms of a comparison of outcomes and provide a typology of all possible comparisons that can be used to estimate treatment effects, including comparisons that are relatively unknown in both the literature and practice. I then assess the relative merit, worth, and value of all possible comparisons based on the criteria of bias, precision, generalizability, ease of implementation, and cost. Which comparison is best depends on how these criteria are weighted and on the constraints of the specific research setting. I hope readers come to recognize a wider range of comparisons than they had previously, appreciate the value of considering all possible comparisons, and see how my typology of comparisons provides the basis for making fair appraisals of the relative strengths and weaknesses of different types of comparisons in the presence of the contingencies that are most likely to arise in practice.

The Good, the Bad, and the Evaluator: 25 Years of AJE Ethics

Abstract: This essay explores the contributions of Evaluation Practice and the American Journal of Evaluation to our understanding of evaluation ethics. The review is organized according to the five Guiding Principles for Evaluators (Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for General and Public Welfare) and identifies key themes within each domain. The value of case-based ethical analysis is highlighted, as is the need for increased empirical research on ethical dimensions of evaluation, especially as these dimensions are perceived by stakeholders with whom evaluators interact. Over the past 25 years, evaluators have learned a great deal about the ethical challenges they are likely to encounter during any given stage of an evaluation, even as they continue to engage in debate over the roles through which evaluators can best serve the public interest.

Escaping the Dusty Shelf: Legislative Evaluation Offices’ Efforts to Promote Utilization

Abstract: Although scholars have proposed many steps to increase evaluation use, there has been little comparative empirical study of whether researchers follow these recommendations and whether specific steps are associated with greater utilization. This study of state legislative evaluators finds that those that regularly meet with stakeholders and provide readily actionable products were considered by senior legislative staff to have more impact, as were larger offices. Legislative evaluators working in auditing organizations were viewed less favorably than those working in other units; this appears to be related to their adherence to Government Auditing Standards that prescribe organizational independence. Evaluators following these standards had less stakeholder engagement than did those working in other legislative units that adhered to research standards that stress meeting stakeholder needs. Environmental factors such as changes in party control may also play a role in how the work of evaluators is valued a...

Policy Workaround Stories are Valuable Evaluative Indicators: But Should They be Told?

Abstract: In evaluating local policy implementation, workaround stories—accounts of deliberate efforts to evade, subvert, or even break rules or procedures in order to get the job done—are valuable evaluative indicators. By “backward mapping” (Elmore, 1979) from frequently mentioned workarounds, we can identify flawed elements of policy- or grant-making systems. However, workaround stories captured in confidential interviews pose an ethical dilemma for evaluators. If reported intact, these stories potentially expose their tellers to repercussions from compliance-oriented superiors in the bureaucratic chain of command. But unless these stories—in all their vivid details—are told, it is less likely that those superiors will be motivated to fix the often intricate implementation problems which create the need for workarounds in the first place. We explore options for resolving this dilemma in light of the American Evaluation Association guiding principles and the trade-off between the competing public goods of innovat...

Developmental Impact Evaluation for Facilitating Learning in Innovation Networks

Abstract: Innovations and new project ideas often emerge in networks of researchers, users of research, and societal actors. This article analyzes and discusses how research scientists learn to conduct research that has an impact on their clients, scientific community, and society. We describe a new developmental impact evaluation method based on the theory of expansive learning. The pilot process was conducted for bioenergy researchers at MTT Agrifood Research Finland. Developmental impact evaluation offers networks a process for evaluating completed projects, learning from them, and constructing a new research concept via pilot projects. The method opens up the concept of ‘research impact’ as a qualitative learning challenge, rather than an accountable target to be judged. It may be adopted as a continuous way of managing and renewing cross-disciplinary research programs, research groups, or networks.

Revising Swiss policies: the influence of efficiency analyses

Abstract: Evidence-based policy-making and other recent reforms in public steering emphasize the role systematic evidence can play in improving decision making and public policies. Increasing deficits heighten the pressure on public authorities to legitimate public spending and to find savings. Existing studies show that the influence of research-based information on decision making is shaped by several factors, but they typically do not distinguish between different types of information. Our contribution aims to compare the influence of efficiency analysis to information about performance effectiveness. We do so by looking at 10 cases in which public policies are being revised at the federal level in Switzerland, and do so by tracing the entire policy reform process. This qualitative analysis sheds light on which actors use efficiency information, how and under which conditions, and highlights the contribution of efficiency analysis for evidence-based policy-making.

Challenges and Dilemmas in Implementing Random Assignment in Educational Research

Abstract: In the contentious debate over the use of randomized clinical trials (RCTs) in education research, little attention has been given to how methodological issues arise and/or are resolved in the impl...

Evaluating the Science of Discovery in Complex Health Systems

Abstract: Complex health problems such as chronic disease or pandemics require knowledge that transcends disciplinary boundaries to generate solutions. Such transdisciplinary discovery requires researchers to work and collaborate across boundaries, combining elements of basic and applied science. At the same time, calls for more interdisciplinary health science acknowledge that there are few metrics to evaluate the products associated with these new ways of working. The Research on Academic Research (RoAR) initiative was established to evaluate the process of discovery and impact of collaboration that emerged through the Life Sciences Institute (LSI) at the University of British Columbia, a state-of-the-art facility designed to support researchers—self-organized around specific health problems rather than disciplines. A logic model depicting the factors influencing such collaboration is presented along with a multimethod evaluation plan to assist understanding of the discovery process in this new environment and de...

Planning for Evaluation’s Future Undergraduate Students' Interest in Program Evaluation

Abstract: Undergraduate students are a potential pool of future evaluators, but little is known about their level of interest in pursuing a career in program evaluation (PE). This two-stage study collected s...

Using Large-Scale Databases in Evaluation: Advances, Opportunities, and Challenges.

Abstract: Major advances in the number, capabilities, and quality of state, national, and transnational databases have opened up new opportunities for evaluators. Both large-scale data sets collected for administrative purposes and those collected by other researchers can provide data for a variety of evaluation-related activities. These include (a) identifying or highlighting issues that invite greater client attention; (b) establishing the plausibility of policy theories of action; and (c) program evaluation. The authors illustrate through examples from the fields of education, social services, and public health both the opportunities provided by higher quality, interoperable data systems and the challenges encountered when using databases to identify issues of concern, test the plausibility of a proposed theory of action, or evaluate an existing program. The authors then explore implications for roles evaluators may need to play to support these uses of databases and for funding and infrastructure to support gre...

Looking Ahead: The Future of Evaluation

Abstract: 2011 is the 25th anniversary of the American Evaluation Association. Such occasions are often recognized both by looking back to assess and celebrate accomplishments of the past and by looking ahead to anticipate the problems and potential of the future. It is surprisingly difficult to think in fresh ways about the future. Our first inclination is to assume that what is happening now will somehow continue, perhaps in a slightly altered form, into the future. This expectation reflects the comforting assumption that the past predicts the future, an assumption sorely tested unfortunately when one tries to predict stock prices, winning sports teams, or political events. While it is important to consider how current activities may be leading us into future conditions, such speculations are not only unreliable, they do little to stimulate radically alternative visions of the future.

Evaluator Characteristics and Methodological Choice

Abstract: This study addresses the central question “How do evaluators' background characteristics relate to their evaluation design choices?” Evaluators were provided with a fictitious description of a scho...

Using Citation Analysis Methods to Assess the Influence of Science, Technology, Engineering, and Mathematics Education Evaluations

Abstract: This study explores the use of citation analysis methods to assess the influence of program evaluations conducted within the area of science, technology, engineering, and mathematics (STEM) education. Citation analysis is widely used within scientific research communities to measure the relative influence of scientific research enterprises and/or individual scientists, however it has not been used frequently to study the influence of evaluations. This study examines the levels, content, and networks of citations arising from products produced by four large-scale, multi-site STEM evaluations. This article explores the usefulness of citation data in understanding the impact of evaluations and finds that citation analysis methods provide information to help understand, to a limited extent, the influence of large-scale, multi-site program evaluations on the fields of STEM education and evaluation.

Understanding Evaluation Training in Schools and Programs of Public Health

Abstract: This study provides an understanding of how the coursework required for attaining a Masters of Public Health (MPH) degree in epidemiology or health education from accredited schools or programs of public health prepares students to evaluate programs or interventions. Study data were generated using a content analysis of required coursework documented on websites of academic institutions and an online survey of professors who instruct courses that focus on educating students about program or intervention evaluation. Information was abstracted from websites for over 1,000 required courses associated with completing an MPH degree in epidemiology or health education at 51 academic institutions. The survey response rate for courses with a primary focus on evaluation was 60.5% and for courses with a partial focus on evaluation was 39.8%. One academic institution required that students pursuing an MPH degree in epidemiology successfully complete a specific course that has a primary focus on program or interventi...

Testing Conceptual Frameworks of Nonexperimental Program Evaluation Designs Using Structural Equation Modeling

Abstract: A theory-driven approach to evaluation (TDE) emphasizes the development and empirical testing of conceptual models to understand the processes and mechanisms through which programs achieve their in...

Evaluating Capacity Building for Policy Research Organizations.

Abstract: The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. The evaluation of the impact of these capacity building and mentoring programs has been inadequate. This paper sets out some general points for the development of effective program evaluation and suggests concrete ways to measure two more important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena.

Promoting Diversity in the Field of Evaluation: Reflections on the First Year of the Robert Wood Johnson Foundation Evaluation Fellowship Program

Abstract: In this paper we describe an evaluation training program sponsored by the Robert Wood Johnson Foundation and led by Duquesne University and OMG Center for Collaborative Learning designed to meet th...

“Honor the Context”: Opening Lines for a Critical Multicultural Evaluative Practice

Abstract: Racial minority evaluators increase the democratic capacity and potential of institutions to the extent that they draw on their experiences and those of other marginalized communities. Their reflection on these experiences helps to disclose institutional power dynamics that are often veiled in neutrality. Of course, this depends on sound consideration of the context in and around institutions. This essay has two general aims. The first is to outline the components of this argument in relation to pertinent scholarship. The second is to demonstrate the ways that the author might bring this understanding to bear on the evaluation of new school systems such as charter schools. Along the way, the author hopes to contribute to the discussion on culturally competent evaluation.

More Than 25 Years of the American Journal of Evaluation The Recollections of Past Editors in Their Own Words

Abstract: This article explores the important role that past editors of the American Journal of Evaluation and its predecessors, Evaluation Practice and Evaluation News, played in the development of the American Evaluation Association (AEA). In interviews presented here, the editors recount the history of the association, the journal, and the field as they experienced it. Using a variety of sources, the author provides additional context for their remarks, describing the pioneering efforts of early evaluation associations, the contentious mergers that resulted in a single national association, the emergence of dedicated evaluation publications, and the challenges posed by an increasingly diverse membership. The author concludes by discussing the ways in which the editors, throughout their 35 years of service, strengthened not only professional associations but the larger field of evaluation.

Aspects of Successful Evaluation Practice at an Established Private Evaluation Firm.

Abstract: This article is third in a series of exemplary cases under the two current section editors. The first two cases (Brandon, Smith, Trenholm, & Devaney, 2010; Smith, Brandon, Lawton, & Krohn-Ching, 2010) profiled evaluations that had been designated as exemplary by professional associations. The present case profiles an evaluation organization: Social Program Evaluators and Consultants, Inc., which goes by the name of SPEC Associates. This Exemplars case reflects our intent, stated in our editorial statement, ‘‘to broaden the nature of evaluation practice to be examined in this section to include not only discrete studies but also such forms of evaluation practice as long-term consulting arrangements’’ (Brandon & Smith, 2010, p. 252). Our focus is on the lessons learned by SPEC’s seasoned Chief Executive Officer (CEO), Dr Melanie Hwalek, about the key aspects of evaluation work that account for successful evaluation practice at a private firm. We use three evaluations identified by Dr. Hwalek as illustrative of SPEC’s evaluation practices to highlight these aspects of SPEC’s work. These evaluations are intended to help enlighten readers about aspects of SPEC and its evaluation practice, as well as the contextual characteristics that typically contribute to or hinder the design and conduct of SPEC’s evaluations. As is our usual practice, we introduce the evaluation organization and three of its evaluations and then present a discussion in the form of an interview that, in reality, is the result of the compilation of information from SPEC’s website, an exchange of documents between the editors and Dr. Hwalek, conversations among the three authors at two AEA conferences, Dr. Hwalek’s presentation about exemplary evaluation in a symposium organized by the editors at the 2010 AEA conference, the results of the editors’ formal oral telephone interview of Dr. Hwalek, and multiple reviews of the article by all three authors. We conclude with reflections by the three authors.