Showing papers in "Educational Evaluation and Policy Analysis in 1986"

Qualitative data analysis: a sourcebook of new methods

Abstract: Part One: Introduction Part Two: Focusing and Bounding the Collection of Data Part Three: Analysis During Data Collection Part Four: Within-Site Analysis Part Five: Cross-Site Analysis Part Six: Matrix Displays: Some General Suggestions Part Seven: Drawing and Verifying Conclusions Part Eight: Concluding Remarks

How Textbooks Affect Achievement in Developing Countries: Evidence From Thailand

Abstract: For the past decade, researchers have documented the effects of textbooks on achievement in developing countries, but no research has explored the mechanisms that account for this contribution. This paper analyzes longitudinal data from a national sample of eighth-grade mathematics classrooms in Thailand and explores the effects of textbooks and other factors on student achievement gain. The results indicate that textbooks may affect achievement by substituting for additional postsecondary mathematics education of teachers and by delivering a more comprehensive curriculum.

The Valid Use of Student Performance Measures for Teacher Evaluation

Abstract: Student achievement test scores appear promising as indicators of teacher performance, but their use carries significant risks. Inappro­ priate tests improperly used may encourage undesirable shifts in curricular focus or poor teaching practices, and may unfairly favor teachers of more able classes. It is often said that standardized achievement test batteries are unsuitable for teacher evaluation, but few systematic alternatives have been suggested. The purposes of this paper are to analyze some problems in using student test scores to evaluate teachers and to propose an achievement-b ased model for teacher evaluation that is effective, affordable, fair, legally defensi­ ble, and politically acceptable. The system is designed only for detecting and documenting poor teacher performance; rewarding excellence in teaching is viewed as a separate problem, and is not addressed in this paper. In addition to pretesting and statistical adjustments for student aptitude differences, the proposed system relies upon attendance data and portfolios of student work to distin­ guish alternative explanations for poor test scores. While no single set of procedures can eliminate all errors, the proposed system, if carefully implemented, could expose teaching to constructive scru­ tiny, organize objective information about teaching adequacy, and help to guide its improvement.

The Context of State Level Policy Formation.

Abstract: To identify the relative power and influence of various state education policy groups, data were collected by elite interviewing as part of a larger study by the authors. This article displays the ...

The Effects of Alternative School Programs on High School Completion and Labor Market Outcomes

Abstract: This research probes two questions regarding participation in alternative high school programs: Does participation reduce the likelihood of students dropping out? Does participation lead to enhanced experiences in the labor market after students leave school? Using the California subsample of the 1980 and 1982 High School and Beyond surveys (involving nearly 3,000 sophomores and 3,000 seniors), vocational education and participation in other alternatives are scrutinized. Our findings regarding the dropout-preventing effects of these programs are mixed: The assessment varies across different procedures used to control for prior propensity to dropout. Our findings for labor market effects are more definite. Participants in vocational and other alternative programs have generally higher employment rates and, for some, higher wages. Suggested extensions of this work are offered.

Levels of Evaluation

Abstract: The formative-summative paradigm is useful in that it draws attention to the importance of process and outcome measures in evaluation. The paradigm is not helpful in practical evaluation planning, however, because it unnecessarily separates qualitative and quantitative methods, and because it does not differentiate among various types of formative evaluation. The article presents a strategy for organizing the evaluation effort into three levels. Each level combines aspects of formative and summative evaluations and includes a variety of methods for addressing the evaluation questions. Resources and skills needed by evaluators at each level are recommended, and a plan for implementing the strategy is suggested.

The Effect of Mode Discourse on Student Writing Performance: Implications for Policy

Abstract: To determ ine the reliαbility of evalua tion decisions about student writing compeenncy across modes of discourse, a large sample of students was randomly assigned a writing task in one of four modes of discourse. Analysis of the data revealed significat differences between the mean score of narrative writing and scores for descrip­ tive, expository, and persuasive writing. The decision consissency analysis also found that on the average, 30% of the students were misclassified across the modes of discourse. It follows from these results that modes of discourse do not appear to be interchangeable for making decisions about the overall writing competency of either groups of, or individual, students.

Naturalistic Evaluation: Potential Conflicts Between Evaluation Standards and Criteria for Conducting Naturalistic Inquiry

Abstract: Although the use of naturalistic methods in educational evaluation is growing, the practical implications of this practice are unclear. Through a description and comparison of standards for evaluation and criteria for judging naturalistic inquiries, a few potential con­ flicts are identified. Analysis of the problems evaluators could en­ counter while using naturalistic methods suggests that compromises in the use of evaluation standards and criteria for naturalistic pro­ cedures are usually necessary. However, it is argued that when the evaluation needs warrant the use of naturalistic techniques, cautious flexibility in using the few standards and criteria in potential conflict seems well justified. at are some practical implications of using naturalistic techniques to con­ duct evaluations? By juxtaposing criteria for judging naturalistic inquiries with evaluation standards, this analysis iden­ tifies ways the two perspectives support and conflict with one another. The purposes and procedures of natu­ ralistic inquiry and evaluation tradition­ ally have been divergent. Identified by a variety of labels (qualitative research, eth­ nography, participant observation, field work, phenomenology), naturalistic in­ quiry evolved through anthropology, so­ ciology, and other disciplines as a way of "understanding" people and the meaning behind their activities. The focus is on

Accountability in California Public Schools.

Abstract: The California Department of Education has adopted a school accountability program with three main components: a statewide summary report, individual school performance reports, and local school se...

Evaluating Preschool Programs: A Meta-Analytic Approach

Abstract: There are many individual evaluation studies regarding the effec­ tiveness of preschool intervention programs. In this study, a meta­ analysis of such evaluation studies is performed to arrive at an overall effect size of preschool intervention programs. The results of this analysis indicate that despite the diversities in intervention sites, ages of subjects at intervention, lengths of intervention, and curricula models, there is a positive homogeneous effect on the variables IQ, math and reading achievement, and percent of students not meeting educational requirements. The methods used in this analysis have implications for policymakers wishing to arrive at decisions about the overall merits of educational programs. In the literature on preschool interven­ tion programs with culturally disadvan­ taged children there has been much dispute as to the magnitude of the effec­ tiveness of these programs. Among the criticisms is that the initial gains wear off as the children proceed through school and that the wide diversity among these programs limits the possibility of arriving at overall conclusions about the effective­ ness and merits of preschool intervention. Despite these limitations in evaluating preschool intervention programs, this pa­ per shows that combining preschool eval­ uation studies to arrive at a single effect size can be important for policymakers when more general questions are ad­ dressed. For example, it would be useful for policymakers, administrators, and pol­ iticians to know whether there is one underlying positive effect of preschool in

Social Judgment Theory and Cognitive Feedback: A General Model for Analyzing Educational Policies and Decisions

Abstract: Applications of social judgment theory (SJT) are discussed with respect to policy formation and decisionmaking in educational con­ texts. SJT serves to externalize, quantitatively and graphically, policymakers1 covert judgment processes through the provision of cognitive process feedback. This can give policymakers a clearer understanding of the processes directing their judgments and thus helps to illuminate sources of conflict over policy parameters and outcomes. SJT procedures function equally well for policy analysis at the "micro" level (e.g., within a classroom) and at the "macro" level (e.g., within schools or regions). Interactive computer technology is described, which can execute social judgment analyses, provide cognitive feedback to the policymaker on selected policy parameters, compare the policies of several policymakers, and incorporate de­ sired policy revisions.

Long-Term Trends in Youth Conduct and the Revival of Traditional Value Patterns.

Abstract: A considerable body of data disclosed a long-term deterioration in the conduct of typical adolescents. In the last 5 to 8 years, some of these trends have changed slightly for the better. After pre...

On the Trail of the Omnibeast: Evaluating Omnibus Education Reforms in the 1980s

Abstract: In this study we investigate curricular changes in California comprehensive high schools from 1982–83 to 1984–85. During this period a number of educational reforms occurred, all aimed at bolstering the academic demands of secondary schools. Senate Bill 813 mandated more extensive statewide graduation requirements for high schools, while California state universities and the University of California altered their entrance requirements. In addition, a number of national reports found America’s high schools lacking in academic rigor. Although it is difficult to ascertain the precise causes of change, we find changes in the course offerings of a sample of 20 California comprehensive high schools. In almost all cases, our data mirror statewide curricular changes documented in California Basic Education Data System data. The general pattern in both sets of data involves increased offerings in academic areas, particularly in math, science, and advanced placement, and decreased offerings in industrial arts, home...

Academic Work and Educational Excellence. Raising Student Productivity.

Abstract: Read casually or passively, this collec­ tion of essays could easily be mistaken for just another contingent in the perennial humdrum parade of complaints, nos­ trums, and aspirations concerning Amer­ ican public education. A wide-awake and critical attitude, however, can make read­ ing this book—this book in particular— disquieting as well as intensely thoughtprovoking. Although I am generally dis­ inclined to mark up a book, by the time I had finished reading this one, many of its pages were replete with my underlinings and marginal comments. Some of the es­ says abound with quotable statements, many because they seem pithily sound, and quite a few others because they seem such patent educationist Pollyanna. One example is the notion, repeated several times in various chapters, that any pupil can learn anything if only given enough time. This seems to be an unqual­ ified and overextended interpretation of the systematic relation between time-ontask and amount learned, as demon­ strated in a rather limited variety of psy­ chological laboratory learning experi­ ments that have minimized such factors as developmental readiness, individual differences, and the importance of insight or understanding in the acquisition of in­ tellectual skills and conceptual knowl­ edge—the sine qua non of academic achievement. In view of the education establishment's proclivity for solutions that too often turn out to be unfruitful fads fashioned from overinterpretation of perhaps scientifically valid but narrowly limited psychological discoveries, critical caution, if not outright skepticism, is in order when psychological principles are generalized to classroom applications. This book, sponsored by the National Society for the Study of Education (NSSE), consists of 10 papers selected from among some 40 commissioned by the National Commission on Excellence in Education as background information in preparing its recent widely publicized report, A Na­ tion at Risk: The Imperative for Educαtional Reform. The editors have written the introductory and final chapters spe­ cially for this volume. A good way to approach the book is to read these two chapters first, as they present an excellent overview of the problems, concepts, and recommendations that are the gist of A Nation at Risk (henceforth referred to as At Risk). The chapters by the editors, in­ cidentally, pretty much contain the es­ sence of the whole book, but without all the detail and literature citations of the other 10 more specialized papers, which deal with the social context of the Com­ mission's concerns (Adelson & Zimilies) and with the educational goals of elemen­ tary (Good & Ward), secondary (Cusick), and college education (Neumann). Four chapters explicate theories of academic work (Doyle), motivation to learn (Stipek),

Gifted and Talented Education: A National Test Case in Peoria:

Abstract: This article presents a study of a program in Peoria, Illinois for the gifted and talented that serves as a national test case for gifted education and minority enrollment. An analysis of the program, and the referral, identification, and selection mechanisms, was conducted. This case study concludes that low minority enrollment need not suggest that the local school district engages in discriminatory practices or that low enrollment is explained by genetic differences between races. Instead, the study points to the impact of the community’s socioeconomic characteristics on gifted enrollment. Fundamentally, the study addresses the issues of equal opportunity, ability, and achievement in American education.

The Contributions of the Social Sciences to Educational Policy and Practice: 1965-1985.

Abstract: the first half of the present century the study of public school finance was closely wedded to the study of public school administration. The early finance specialists, Cubberly, Strayer, Mort, and others, were as highly regarded as writers and teachers in general education administration fields as they were in school finance. In that era public school finance was an eclectic field, drawing on the concepts, principles, and methods from education, public finance, law, and accounting. After the Second World War we saw the emergence of more narrowly disciplinedbased school finance specialists. Major contributions to the field were made by scholars in fields like economics, political science, and law who had an interest in educational matters. Research was

An Evaluation Model of Training Outcomes for Higher Education.

Abstract: A performance evaluation model is presented to address the need for measuring the effectiveness of educational training programs in institutions of higher learning. Described as value-added, this model is pioneering and yet practical; it is based on an award-winning experiment at Northeast Missouri State University. It emphasizes outcomes by assessing the quality of change that occurs in individual and program performance. The model may be adapted to other educational and noneducational settings. 1 he idea of value-added evaluation is not new, but as used by Northeast Mis­ souri State University (Northeast) it is unique. The Northeast method of assess­ ment addresses needs identified by many educators as well as by private and public

The Determinants of Public Revenues for Higher and Lower Education: A Thirty-Year Perspective.

Abstract: This longitudinal study examines the determinants of higher education revenues per capita, lower education revenues per capita, and the ratio of the two over a 30-year period. Demographic variables (percentage of population in the appropriate age group, education of the adult population, etc.) are unimportant, as is personal income per capita. The most important variables are a historical one and two political variables. The historical variable, density of population in 1860, measures the propensity of states to develop private higher education before passage of the Morrill Act. The fact that the two political variables (percentage of taxes raised at the state level and taxes as a percentage of income) are important implies that the dependent variables are capable of being manipulated by the political process.

Living to Tell the Tale: Researching Politically Controversial Topics and Communicating the Findings

Abstract: Doing politically controversial research on education issues at the statewide level presents a host of human problems in addition to technical, scientific ones. Drawing on research experience in a highly charged political context, this paper suggests practical strategies for gaining access, developing legitimacy, and communicating information in order to influence policymakers. We emphasize the importance of forming a steering committee to represent major political interest groups.

Adaptive Organizational Behavior of School Organizations: An Exploratory Study

Abstract: Although school administrators continually have to introduce internal organizational changes that allow them to cope with changes in their environments, little research on the adaptive organizational behavior of school organizations is available. This study collected exploratory data on a group of organizational adjustment variables (procedural, personnel, process, structural, and strategic) among a group of schools and school districts. The results provide a preliminary basis for suggesting that the adaptive organizational behavior of schools and school districts may be influenced by perceived environmental uncertainty and, to a lesser extent, by organization size. School administrators appeared to choose courses of actions or adjustments that solve their problems with minimum cost to the organization. Adaptation as a factor essential to organizational performance is discussed.

Teaching in Public and Private Schools: The Significance of Race:

Abstract: This paper explores variations in the employment of minority teach­ ers in elementary and secondary public, Catholic, and private (nonCatholicl schools. Our data are drawn from a survey of public and nonpublic schools in the six counties surrounding San Francisco Bay. The survey was conducted during 1981-82 by the Institute for Re­ search on Educational Finance and Governance, Stanford University. We analyze several determinants of minority employment in public and private schools to explain variation among the three sectors. Subsequently, we present a more detailed analysis of the different employment experiences of black and Hispanic teachers within two of these sectors, public and Catholic schools. Based on these analyses, we conclude with a discussion of implications for public policy.

Assessing National Data on Education.

Abstract: Efforts to improve education may be misdirected without adequate data to monitor developments and elucidate key issues. This assessment evaluates some 35 data bases, federal and privately sponsored, that report national education statistics in terms of their potential for informing policy deliberations. It argues that for all the national data collected, information is largely lacking on several major concerns, often inadequate statistically, and sometimes contradictory. From this evaluation, various improvements are proposed to upgrade the collection of policy-oriented data.

Three Evaluation-Related Behaviors of the School Principal

Abstract: Largely because of the drive to improve school productivity, evaluation has touched almost every aspect of education. This paper focuses on evaluation-related behavior of school principals. First, the role of principals as clients of professional evaluation is examined. In the second section, research that views school principals as evaluators in their own right is reported. In the last section, concepts drawn from the evaluation literature are used to explore new ways of studying the rationality that principals employ to make decisions.

The Convening Process: A New Technique for Applying Knowledge to Practice

Abstract: U.S. Department of Education staff created and evaluated a Conven- ing Process to help educational administrators and policymakers make sound decisions based on practical experience, research, and evaluation in short periods of time. This article describes the back- ground and rationale for the Convening Process and its relation to other efforts to increase the use of research and evaluation informa- tion. Convening Process experiences with the District of Columbia Public Schools and the State of Mississippi Department of Education are summarized and the impact of each experience is discussed. To guide future uses of the Convening Process, situational and proce- dural differences are analyzed. This analysis provides the basis for specific advice on replicating the Convening Process by explaining the operation of the key elements: (a) the identification of practical problems that can be resolved with currently available knowledge; (b) the use of consultant colleagues with appropriate expertise; and (c) the use of consultant groups to acquire, analyze, and exchange information with local stakeholders to arrive at recommendations based on consensus.

Early Research on the Federal Education Block Grant: Themes and Unanswered Questions.

Abstract: This paper reviews early research at the national level on the implementation and effects of the Department of Education’s block grant, Chapter 2 of the Education Consolidation and Improvement Act of 1981. The review is intended to synthesize what is known about the block grant in its first two years of operation (the 1982–83 and 1983–84 school years) and to suggest directions for future research on this policy mechanism. Findings from the early studies are summarized under six topics: funds allocation, educational service delivery, program administration, program assessment, decisionmaking, and intergovernmental relations. Although it is preoccupied with short-term impacts, this body of research brings important issues into focus and points the way toward further investigation.

Legal Education as a Model for the Education of Evaluators

Abstract: While the skills of quantitative analysis are commonly a part of an evaluator's training, the skills of logical analysis and argumentation are little emphasized. For example, an evaluator needs to argue from evidence to conclusions or recommendatio ns. Such skills typically are taught to lawyers during their training. There are two parts to this paper. The first part is a discussion of legal education, which uses the Socratic method of pedagogy and the study of case histories. The second part examines what aspects of legal education might be used as a model for the education of evaluators. There is a discussion of what information an evaluation case history might contain.

Blackwell’s Commentaries, Engineering’s Handbooks, and Merck’s Manuals: What Would a Teacher’s Equivalent Be?

Abstract: The intensified discussion over knowledge bases for teaching ought to be reflected not only in teacher education programs but also in the resources directly available to teachers in the daily performance of their tasks. Microcomputers married to the interactive video disk are the likely technology. Handbooks might be developed in such domains as curriculum; instructional strategies and techniques; student, classroom, and behavior management; instructional technology; developmental stages of learning; and teacher education. Issues to be addressed and/or solved include defining the organization and content of the handbooks, avoiding the danger of premature rejection, coping with multiplistic values and frames of reference, and organizing the effort to create them.

A Comparison of Methods for Measuring Achievement in Basic Skills Program Evaluation.

Abstract: Standardized achievement tests continue to be used as the primary criterion measures in the evaluation of basic skills programs despite explicit recommendations against this practice by many testing and evaluation experts. A major issue behind this criticism is examined in this comparative study of three types of tests. These tests represent three points on a content validity continuum which references their correspondence to the instructional program being evaluated. Data are examined on these tests at two levels in eight groups of elementary school students receiving instruction on various basic skills. The results show why and how standardized tests provide different information from other tests that are more closely matched to program instruction. They also show why out-of-level standardized test results often correspond more closely to the results of instructionally referenced tests as compared to at-level tests. The implications of this research for basic skill program evaluations are discussed.