Statistical Analysis with Missing Data

How can the validity of performance assessments be established? What type of logical argumentation is necessary to generalize from a particular assessment to the ability or abilities of interest?

Validating Measures of Performance

The title Assessment 2000 would have sounded like science fiction a few decades ago, an opportunity to use my imagination in making creative and wild speculations about assessment in a distant future. However, less than half a decade before the due date, this chapter entails more modest and careful speculations, based on contemporary theories and on lessons gained from current practice. Indeed, it starts by introducing the most generic term currently used in educational literature with respect to assessment, i.e., alternative assessment. It briefly explains to what and why an alternative is sought and describes the main features of this type of assessment, as it is currently viewed. Of the various devices subsumed under the alternative assessment umbrella a focus is put on the portfolio describing its various types, uses and criteria for judgment. Next, criteria for evaluating alternative assessment and lessons to be learnt from current practice are discussed, and finally, a rationale for a pluralistic approach to assessment is presented.

Assessment 2000: Towards a Pluralistic Approach to Assessment

A framework for evaluation and use of automated scoring of constructed-response tasks is provided that entails both evaluation of automated scoring as well as guidelines for implementation and maintenance in the context of constantly evolving technologies. Consideration of validity issues and challenges associated with automated scoring are discussed within the framework. The fit between the scoring capability and the assessment purpose, the agreement between human and automated scores, the consideration of associations with independent measures, the generalizability of automated scores as implemented in operational practice across different tasks and test forms, and the impact and consequences for the population and subgroups are proffered as integral evidence supporting use of automated scoring. Specific evaluation guidelines are provided for using automated scoring to complement human scoring for tests used for high-stakes purposes. These guidelines are intended to be generalizable to new automated scoring systems and as existing systems change over time.

A Framework for Evaluation and Use of Automated Scoring

Assessment instruments of the future will probably be composed of a combination of different types of questions. Even though different kinds of questions require different scoring procedures, there may be a need to have those different scores combined as a composite. In this article, we describe how mixtures of such scores may be efficaciously combined. Also, if no post hoc adjustment is desired, we provide two characterizations of measurement effectiveness to aid in making unadjusted score combinations efficient. In addition, we explore the implications for test construction of some typical findings.

Combining multiple-choice and constructed response test scores: toward a marxist theory of test construction

Using analyses based on fitting item response models to data from the College Board's Advanced Placement exams in chemistry and United States history, we found that the constructed response portion of the tests yielded little information over and above that provided by the multiple-choice sections. These tests also allow examinees to select subsets of the constructed response items; we found that scoring on the basis of the selections themselves provided almost as much information as did scoring on the basis of the answers

On the Relative Value of Multiple-Choice, Constructed Response, and Examinee-Selected Items on Two Achievement Tests

Using analyses based on fitting item response models to data from the College Board's Advanced Placement exams in chemistry and United States history, we found that the constructed response portion of the tests yielded little information over and above that provided by the multiple-choice sections These tests also allow examinees to select subsets of the constructed response items; we found that scoring on the basis of the selections themselves provided almost as much information as did scoring on the basis of the answers One goal of the Advanced Placement (AP) testing program of the College Board is to determine the proficiency attained by high school students in college level courses Such courses are offered at more than 40% of the 21,000 secondary schools in this country, including 80% of those having a junior class larger than 250 students With few exceptions, AP exams are made up of one section consisting of multiple-choice (MC) items and a second section of constructed response

/pdf/on-the-relative-value-of-multiple-choice-constructed-mgp9m4bgyk.pdf

On the relative value of multiple-choice, constructed-response, and examinee-selected items on two achievement tests1

The reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL) was estimated using a hybrid IRT model. It was found that there was very little difference between their overall reliability when the testlet items were assumed to be independent and when their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40% overestimate in reading comprehension testlets, with the longer testlets of the newest form of TOEFL showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate item response theory (IRT) to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex.

How Reliable are TOEFL Scores

Building tests out of items that individually take a substantial amount of examinee time brings with it a number of problems One major problem is that it is often too difficult and too expensive to extensively pretest such large items Thus, the sorts of screening for flaws that are pro forma for multiple-choice items are not often done for large items In addition, because there are so few large items on an operational test, not counting an entire item that is found to be flawed in an operational administration may be tantamount to aborting that administration In this article, we examine the efficacy of the alternative of continuous item weighting This alternative is illustrated on data from the 1988 administration of the College Board's Advanced Placement History Test

Robert Lukhele

Papers

On the Relative Value of Multiple-Choice, Constructed Response, and Examinee-Selected Items on Two Achievement Tests

On the relative value of multiple-choice, constructed-response, and examinee-selected items on two achievement tests1

How Reliable are TOEFL Scores

Managing the influence of dif from big items: the 1988 advanced placement history test as an example