Showing papers in "Educational Assessment in 2005"

Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions.

Abstract: Student test-taking motivation in low-stakes assessment testing is examined in terms of both its relationship to test performance and the implications of low student effort for test validity. A theoretical model of test-taking motivation is presented, with a synthesis of previous research indicating that low student motivation is associated with a substantial decrease in test performance. A number of assessment practices and data analytic procedures for managing the problems posed by low student motivation are discussed.

Noncognitive Constructs and Their Assessment in Graduate Education: A Review

Abstract: We reviewed the literature on "noncognitive" predictors-specifically, personality as it pertains to graduate education. The review is divided into 3 sections. In the first section, we reviewed measures typically used in studies of graduate school outcomes, such as attrition and time to degree. We also reviewed which student qualities faculty and administrators said they desired and cultivated in graduate programs. We also noted that there are many qualities faculty ranked high in desirability but which could only imperfectly be gleaned from sources such as letters of recommendation and personal statements. In the second section, we reviewed general personality factors (e.g., the "Big Five"), specifically, definitions, measures, correlates, and the validity of those measures. We concluded with a discussion of how personality factors might be used in admissions and guidance applications for graduate education.

Monetary Incentives for Low-Stakes Tests

Abstract: This research attempted to discover the degree to which performance on assessments that have no direct consequences for individual students can be attributed to students' motivational states in addition to their underlying achievement in the domains measured. If such findings were reliable, it would allow us to correct underestimates of domain performance due to lack of motivation. Such corrections would be extremely interesting in explaining differential subgroup performance in state or national assessments that have institutional but not individual consequences. Recent information on international assessments (e.g., the Third International Mathematics and Science Study [TIMSS]) indicates that 12th-grade students in the United States are doing poorly on such assessments compared with their peers in other countries. These poor results are usually attributed to cognitive factors such as students' opportunities to learn, etc. However, a partial explanation may be motivational. Because the low-stakes tests w...

Artifact Packages for Characterizing Classroom Practice: A Pilot Study

Abstract: This article describes the development of artifact collection and scoring procedures to characterize classroom practice in mathematics and science. A data collection tool called the "Scoop Notebook" was used to gather artifacts related to key features of classroom practice, such as teachers' use of instructional materials and strategies, classroom learning activities, and students' work. Pilot studies were conducted in a small number of middle school science and mathematics classrooms to provide initial information about the reliability, validity, and feasibility of artifact collections as measures of classroom practice. The pilot studies yielded positive results, indicating that the Scoop Notebook and associated scoring guide have promise for providing accurate representations of what teachers and students do in classrooms. Due to the small sample size, results are most appropriately used formatively, to help improve artifact collection procedures before they are tested on a larger scale.

Assessing Motivational Factors in Foreign Language Learning: Cultural Variation in Key Constructs

Abstract: Assessment in 2nd and foreign language learning has often centered on proficiency, although motivation has been shown to be critical in learning in general, as well as in 2nd and foreign language acquisition specifically. One of the issues relevant to the assessment of motivational processes in this area is that of cultural differences. In much of the existing work, motivational processes and constructs are assumed to be invariant across cultural and linguistic groups. This may be due to limitations in the existing models or in the narrowness of populations sampled. The purpose of this study was to test the robustness of motivational processes in language acquisition across different ethnic subgroups for the purpose of informing efforts to create more valid and sensitive measures. Participants in the study completed a background information survey, a motivation information questionnaire, and a learning outcome questionnaire. Factor analysis yielded 7 factors, and comparative analyses confirmed the importa...

Engaged Participation: A Sociocultural Model of Motivation with Implications for Educational Assessment.

Abstract: This article considers issues in educational reform, particularly accountability-oriented policies, and student motivation. We argue that prominent theories of motivation derived from cognitive traits/states are antithetical to the assumptions underlying conventional assessment formats and accountability-oriented reforms. We advance an alternative perspective that emerges from sociocultural assumptions. In this perspective, the values and beliefs that motivate engagement in learning reside alongside the practices that characterize knowledge communities and, together, constitute a model of practice. We therefore explore the complex issue of reconciliation between the activities of individuals and social contexts and how "dialectical" reconciliation addresses tensions between classroom assessment and external testing, and between formative and summative functions of assessment. Data and conclusions from a program of research in science education illustrate and warrant these considerations.

Cross-Cultural Approaches to Measuring Motivation

Abstract: This study analyzes similarities and differences in students' motivation to learn based on nationally representative samples of 15-year-olds from 26 countries. Scales of extrinsic and intrinsic motivation proved to be equivalent across countries but there seems to be a shift in meaning at the country level. Focusing on functional relationships within countries, the results show that in most countries, extrinsic and intrinsic motivation vary independently of each other and are by no means antagonistic. On the contrary, both are important in the regulation of actual learning behavior: Whereas interest in reading mostly showed direct effects on students reading performance, instrumentally motivated students seemed to perform better only because of their frequent use of control strategies during learning.

Relationship between Opportunity to Learn and Student Performance on English and Algebra Assessments.

Abstract: The purpose of this study was to examine how various opportunity to learn (OTL) variables impact student outcomes and whether these effects are consistent across different subject areas. In this study, we examined the impact of OTL variables on student performance on English and algebra assessments. Despite the differences in the subject areas, 3 consistent findings emerged. First, we found a significant positive relationship between teacher expertise and student performance. Teacher expertise in this study is defined specifically as expertise and knowledge within the content areas covered in the standards and the district assessment, rather than overall expertise in the subject area. Our study also showed that content coverage was positively correlated with student performance in English and algebra. Finally, consistent with previous findings, our study indicated that the socioeconomic status of the classroom has a significant impact on student performance.

Teacher Candidates' Exposure to Formative Assessment in Educational Psychology Textbooks: A Content Analysis

Abstract: The purpose of this article is to define formative assessment, outline what is known about the prevalence of formative assessment implementation in the classroom, establish the importance of formative assessment with regards to student motivation and achievement, and present the results of a content analysis of current educational psychology textbooks. Several key definitions of formative assessment are examined, and various means of conducting formative assessment are outlined. Numerous studies that have examined the effects of formative assessment on motivation and achievement are summarized. The theoretical link between formative assessment and several important motivational constructs is established, and suggestions for future research are delineated.

Mutually Informative Measures of Knowledge: Concept Maps Plus Problem Sorts in Statistics

Abstract: This case study examines the validity of 1 measure of representation and employs multiple measures to examine whether they provide mutually informative or independent pieces of information. These measures include (a) concept maps, which measure how individuals represent their content knowledge of a domain as a whole; (b) problem sorts, which measure how individuals represent a specific aspect of their knowledge on word problems; and (c) structured interviews, which identify the reasons underlying sorting and concept mapping performance. Data from statistics instructors showed that the sorting task was a useful measure of representation when supplemented with instructors' explanations of their rationales. Moreover, concept maps assisted in the interpretation of performance on the sorting task. Concept maps and problem sorts were mutually informative, with the former providing a broader picture and the latter illustrating how particular concepts became salient when applied to a different context.

The Role of Anxiety on Novices' Rifle Marksmanship Performance.

Abstract: In this study we examined the role of anxiety on novices' performance on the psychomotor task of rifle marksmanship. The stages-of-skill-development model suggests that aptitude influences performance while learning a skill; however, the model does not address the role of affect. Assessments for state anxiety and state worry were administered to 42 novice shooters. Aptitude and state worry predicted shooting performance (r = .67) comparable to more direct psychomotor measures used in prior research. Incremental validity analyses showed that aptitude accounted for 11% of the variance, and state worry explained an additional 34% above and beyond aptitude. The use of noncognitive assessments can broaden the understanding of skilled performance. Although aptitude influences learning a skill, it appears that novice shooters' performance is also influenced by affective states such as anxiety.

Regression to the Mean in Average Test Scores

Abstract: A group's average test score is often used to evaluate different educational approaches, curricula, teachers, and schools. Studies of group test scores over time often try to measure "value-added" by holding constant certain student characteristics such as race, parents' education, or socioeconomic status; however, the important statistical phenomenon of regression to the mean is often ignored. There is a substantial literature on the importance of regression to the mean in a variety of contexts, including individual test scores. Here, we look at regression to the mean in group averages. If this regression is not taken into account, changes in a group's average test score over time may be misinterpreted as changes in the group's average ability rather than natural and expected fluctuations in scores about ability. California Academic Performance Index scores are used to illustrate this argument.

Improving Assessment Validity for Students With Disabilities in Large-Scale Assessment Programs.

Abstract: A test designed with built-in modifications and covering the same grade-level mathematics content provided more precise measurement of mathematics achievement for lower performing students with disabilities. Fourth-grade students with disabilities took a test based on modified state curricular standards for their mandated statewide mathematics assessment. To link the modified test with the general test, a block of items was administered to students with and without disabilities who took the general mathematics assessment. Item difficulty and student mathematics ability parameters were estimated using item response theory (IRT) methodology. Results support the conclusion that a modified test, based on the same curricular objectives but providing a more targeted measurement of expected outcomes for lower achieving students, could be developed for this special population.

Sample Exclusion and Student Attrition Effects in the Longitudinal Study of Middle School Mathematics Performance

Abstract: Achievement data from a longitudinally matched student cohort from a large school district in the southwestern United States were analyzed to investigate sample exclusion and student attrition effects on estimates of student, school, and district mathematics performance. Use of 2- and 3-level longitudinal growth models to estimate the growth trajectories of middle school students revealed that mathematics performance differed across 2 sample conditions. Relative to the achievement outcomes associated with a sample that included all students from the longitudinal cohort, district and school achievement were generally higher and student group performance more similar in the smaller, more advantaged student sample used for district accountability reporting. Further investigation of the school performance estimates showed that cross-sample changes in student achievement outcomes were closely related to the proportion of students from special student populations who were excluded from the district accountabili...

Alignment of Content and Effectiveness of Mathematics Assessment Items.

Abstract: Alignment has taken on increased importance given the current high-stakes nature of assessment. To make well-informed decisions about student learning on the basis of test results, assessment items need to be well aligned with standards. Project 2061 of the American Association for the Advancement of Science (AAAS) has developed a procedure for analyzing the content and quality of assessment items. The authors of this study used this alignment procedure to closely examine 2 mathematics assessment items. Student work on these 2 items was analyzed to determine whether the conclusions reached through the use of the alignment procedure could be validated. It was found that the Project 2061 alignment procedure was effective in providing a tool for in-depth analysis of the mathematical content of the item and a set of standards and in identifying 1 particular content standard that was most closely aligned with the standard. Through analyzing student work samples and student interviews, it was also found that st...

The Consistency of Student Proficiency Classifications Under Competing IRT Models

Abstract: States are implementing statewide assessment programs that classify students into proficiency levels that reflect state-defined performance standards. In an effort to provide support for score interpretations, this study examined the consistency of classifications based on competing item response theory (IRT) models for data from a state assessment program. Classification of students into proficiency levels was compared based on a 1-parameter vs. a 3-parameter IRT model. Despite an overall high level of agreement between classifications based on the 2 models, systematic differences were observed. Under the 1-parameter model, proficiency was underestimated for low proficiency classifications but overestimated for upper proficiency classifications. This resulted in higher "Below Basic" and "Advanced" classifications under 1-parameter vs. 3-parameter IRT applications. Implications of these differences are discussed.

Assessment of Noncognitive Influences on Learning

Abstract: (2005). Assessment of Noncognitive Influences on Learning. Educational Assessment: Vol. 10, No. 3, pp. 147-151.