scispace - formally typeset
Search or ask a question
Journal ArticleDOI

The Evaluation of Broad-Aim Programs: Experimental Design, Its Difficulties, and an Alternative.

TL;DR: The broad-aim program as mentioned in this paper is an approach to evaluate social action programs that aims to achieve nonspecific forms of change-for-the-better and which also involve unstandardized, large-scale interventions and are evaluated in only a few sites.
Abstract: There is an approach to the evaluation of social action programs which seems so sensible that it has been accepted without question. The underlying assumption is that action programs are designed to achieve specific ends and that their success can be established by demonstrating cause-effect relationships between the pro­ grams and their aims. In consequence, the preferred research design is an experi­ mental one in which aspects of the situation to be changed are measured before and after implementation of the action program. To support the argument that the program is responsible for the observed changes, the anticipated effects may be measured simultaneously in a control situation that does not receive the program (Campbell & Stanley, 1966). This plausible approach misleads when the action programs have broad aims and take unstandardized forms. The term broad-aim program is intended to describe programs that hope to achieve nonspecific forms of change-for-the-better and which also, because of their ambition and magnitude, involve unstandardized, large-scale interventions and are evaluated in only a few sites. These characteristics have been shared by a
Citations
More filters
Journal ArticleDOI
TL;DR: For instance, this paper examined the utility of multisite qualitative policy research in the context of public self-criticism with an eye to improving their methods, and pointed out the benefits of cross-site comparison without sacrificing within-site understanding.
Abstract: The classical qualitative educational research design is the case study. Studies of school life (Cusick, 1973; Wolcott, 1973), of the larger social forces affecting schooling (Ogbu, 1974), and of efforts to promote planned educational change (Smith & Keith, 1971) have used qualitative data in describing a single social setting. Typically, such studies emphasize in-depth description but provide a weak basis for generalization to other settings. The last decade, however, has seen the emergence of a new form of qualitative research, one intended to strengthen its ability to generalize while preserving indepth description. These multisite qualitative studies address the same research question in a number of settings using similar data collection and analysis procedures in each setting. They consciously seek to permit cross-site comparison without necessarily sacrificing within-site understanding. Although having some roots in academic social sciences (e.g., see Clark, 1970; Whiting, 1963; Whiting & Whiting, 1975), multisite qualitative research arose primarily in response to pressures from the federal government in the 1970s for studies that could overcome some of the weaknesses of large quantitative evaluations without being limited by the particularism of the single-site case study. Like many hybrids, it is today quite robust. However, these multisite qualitative studies were typically expensive endeavors and were done for specific policy purposes, which the current federal administration seems neither to value nor to feel it can afford. There are two important reasons for reflecting on the historical development and potential utility of multisite qualitative policy research at this time. Although it is unwelcomed by most social scientists, the current hiatus in commissioning policy research at the federal level provides researchers and policymakers with an opportunity to consider these issues in some detail. Further, the field of policy research has matured to the point where such considerations can be very fruitful. In recent years qualitative researchers have moved beyond the need to defend the legitimacy of their craft in the policy arena (Rist, 1977; Smith, 1978; Stake, 1978). Moreover, quantitative researchers are beginning to acknowledge a role for qualitative research in policy and evaluation studies (Cronbach, 1982; Hoaglin, Light, McPeek, Mosteller, & Stoto, 1982) and to consider the proper balance of qualitative and quantitative techniques (Cook & Reichardt, 1979; Smith & Louis, 1982). In addition, practitioners of multisite qualitative policy research now exhibit sufficient confidence in their craftsmanship to begin a process of public self-criticism with an eye to improving their methods (Firestone & Herriott, 1982; Miles, 1979; Smith & Louis, 1982; Yin, 1981). Efforts to examine multisite

916 citations


Cites background from "The Evaluation of Broad-Aim Program..."

  • ...…independent variables, particularly the degree to which programs implemented with federal funds were sufficiently faithful to the intentions of their des igners , or enacted in a sufficiently uniform m a n n e r across most sites, to permit a meaningful test of their effects (Weiss & Rein, 1970)....

    [...]

Journal ArticleDOI
TL;DR: In this paper, the authors report on program impact assessment methodology as it is developing in the United States today, and assume that social project evaluation methodology is one of the fields of science that has enough universality to make scientific sharing mutually beneficial.

737 citations


Cites background from "The Evaluation of Broad-Aim Program..."

  • ...Inevitably, this split has spilled over into evaluation research, taking the form of a controversy over the legitimacy of the quantitative-experimental paradigm for program evaluation (e.g., Weiss & Rein, 1969, 1970; Guttentag, 1971, 1973; Campbell, 1970, 1973)....

    [...]

  • ...So far, the qualitative-knowing alternatives suggested (e.g., Weiss & Rein, 1969, 1970; Guttentag, 1971, 1973) have not been persuasive to me....

    [...]

Journal ArticleDOI
TL;DR: Yin and Heald as discussed by the authors analyzed the content of case studies by using a closed-ended questionnaire to assess the quality of each case study in a reliable and replicable manner.
Abstract: Robert K. Yin and Karen A. Heald A common feature of most policy literatures is that the bulk of the empirical evidence is embodied in case studies. This presents a problem for subsequent analysis. Although each case study may provide rich insights into a specific situation, it is difficultto generalize aboutthe studies as a whole. The following article describes one means of dealing with the problem: analyzing the content of case studies by using a closed-ended questionnaire. The resulting case survey method allows an analyst to aggregate the case study experiences and to assess the quality of each case study in a reliable and replicable manner.1

310 citations

Journal ArticleDOI
TL;DR: In this article, the authors aim to encourage the use of quasi-experimentation by identifying five key benefits: (a) strengthening causal inference when random assignment and controlled manipulation are not possible or ethical; (b) building better theories of time and temporal progression; (c) minimizing ethical dilemmas of harm, inequity, paternalism, and deception; (d) facilitating collaboration with practitioners; and (e) using context to explain conflicting findings.
Abstract: Although quasi-experiments can facilitate causal inferences by combining good internal validity with high external validity, organizational scholars underutilize them. In this article, the authors aim to encourage the novel use of quasi-experimentation by identifying five of its key benefits: (a) strengthening causal inference when random assignment and controlled manipulation are not possible or ethical; (b) building better theories of time and temporal progression; (c) minimizing ethical dilemmas of harm, inequity, paternalism, and deception; (d) facilitating collaboration with practitioners; and (e) using context to explain conflicting findings. We offer advice and illustrative examples to guide future research, and provide recommendations for gaining access to organizations to open doors for collaborating on quasi-experiments.

213 citations

Journal ArticleDOI
TL;DR: In this article, the authors propose a theory-driven approach for dealing with validity, where a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research.

196 citations

References
More filters
Book
01 Jan 1967
TL;DR: In this paper, the discovery of grounded theory is discussed and grounded theory can be found in the form of a grounded theory discovery problem, where the root cause of the problem is identified.
Abstract: The discovery of grounded theory , The discovery of grounded theory , کتابخانه مرکزی دانشگاه علوم پزشکی تهران

22,245 citations

Journal ArticleDOI
TL;DR: In this article, the authors discuss ways of placing objects in a way that they can be placed according to the philosophy of the Philosophic Schools and the Dialectic of CONSTITUTIONS.
Abstract: Introduction Part One: Ways of Placement I. CONTAINER AND THING CONTAINED II. ANTINOMIES OF DEFINITION III. SCOPE AND REDUCTION Part Two: The Philosophic Schools I. SCENE II. AGENT IN GENERAL III. ACT IV. AGENCY AND PURPOSE Part Three: On Dialectic I. THE DIALECTIC OF CONSTITUTIONS II. DIALECTIC IN GENERAL Appendix Index

3,197 citations

Book
01 Jan 1945
TL;DR: In this article, the authors discuss ways of placing objects in a way that they can be placed according to the philosophy of the Philosophic Schools and the Dialectic of CONSTITUTIONS.
Abstract: Introduction Part One: Ways of Placement I. CONTAINER AND THING CONTAINED II. ANTINOMIES OF DEFINITION III. SCOPE AND REDUCTION Part Two: The Philosophic Schools I. SCENE II. AGENT IN GENERAL III. ACT IV. AGENCY AND PURPOSE Part Three: On Dialectic I. THE DIALECTIC OF CONSTITUTIONS II. DIALECTIC IN GENERAL Appendix Index

2,797 citations

Journal ArticleDOI
TL;DR: The following article and its companions in the next issue epitomize general systems behavior theory as presented in the author's Living Systems, to be published in a few months.
Abstract: The first issue of this journal began with an editorial which said in part: “Our present thinking—which may alter with time—is that a general theory will deal with structural and behavioral properties of systems. The diversity of systems is great. The molecule, the cell, the organ, the individual, the group, the society are all examples of systems. Besides differing in the level of organization, systems differ in many other crucial respects. They may be living, nonliving, or mixed; material or conceptual; and so forth.” A decade later, the thinking has not altered greatly. Every year the structure and process of many sorts of systems have been analyzed in these pages. The following article and its companions in the next issue epitomize general systems behavior theory as presented in the author's Living Systems, to be published in a few months.

383 citations