Computerized adaptive testing (CAT) is a mode of testing which enables more efficient and accurate recovery of one or more latent traits. Traditionally, CAT is built upon Item Response Theory (IRT) models that assume unidimensionality. However, the problem of how to build CAT upon latent class models (LCM) has not been investigated until recently, when Tatsuoka (J. R. Stat. Soc., Ser. C, Appl. Stat. 51:337–350, 2002) and Tatsuoka and Ferguson (J. R. Stat., Ser. B 65:143–157, 2003) established a general theorem on the asymptotically optimal sequential selection of experiments to classify finite, partially ordered sets. Xu, Chang, and Douglas (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003) then tested two heuristics in a simulation study based on Tatsuoka’s theoretical work in the context of computerized adaptive testing. One of the heuristics was developed based on Kullback–Leibler information, and the other based on Shannon entropy. In this paper, we showcase the application of the optimal sequential selection methodology in item selection of CAT that is built upon cognitive diagnostic models. Two new heuristics are proposed, and are compared against the randomized item selection method and the two heuristics investigated in Xu et al. (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003). Finally, we show the connection between the Kullback–Leibler-information-based approaches and the Shannon-entropy-based approach, as well as the connection between algorithms built upon LCM and those built upon IRT models.

When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT

The Kullback-Leibler divergence (KLD) is a widely used method for measuring the fit of two distributions. In general, the distribution of the KLD is unknown. Under reasonable assumptions, common in psychometrics, the distribution of the KLD is shown to be asymptotically distributed as a scaled (non-central) chi-square with one degree of freedom or a scaled (doubly non-central) F. Applications of the KLD for detecting heterogeneous response data are discussed with particular emphasis on test security.

Distributions of the Kullback-Leibler divergence with applications.

SEQUENTIAL CLASSIFICATION ON PARTIALLY ORDERED SETS Curtis Tatsuoka 1 and Thomas Ferguson 2 Department of Statistics, The George Washington University, Washington, DC 20052, USA Department of Statistics, UCLA, Los Angeles, CA 90095, USA Abstract A general theorem on asymptotically optimal sequential selection of experiments is pre- sented and applied to a Bayesian classiﬁcation problem when the parameter space is a ﬁnite partially ordered set The main results include establishing conditions under which the posterior probability of the true state converges to one almost surely, and determining optimal rates of convergence Properties of various classes of experiment selection rules are explored KEY WORDS: partially ordered set, cognitive diagnosis, group testing, sequential selection of experiment, optimal rates of convergence, Kullback-Leibler information 1 Introduction: Background and Motivation Partially ordered sets are natural models for many statistical applications As an example of how partially ordered sets (posets) can be used in cognitively diagnostic analysis and computerized intelligent tutoring systems, consider the following example (eg see K Tatsuoka (1995)) Suppose a student is to be tested on a certain subject domain for which there is a known ﬁnite set of knowledge states, denoted by S It is of interest to determine the student’s knowledge state in S Responses from sequentially selected test items (ie experiments) will be used to classify the person into one of the states A natural model for S is to assume that certain states are at higher levels than others Two states i and j in S may be related to each other in the following manner If a student in state i has the knowledge to answer correctly all the test items that a student can who is in state j, we denote this by j ≤ i It is thus natural to assume that S is a partially ordered set Another example of the use of partially ordered sets in statistics with a rich body of literature, is group testing, originated by Dorfman (1943) (see also Ungar (1960), Sobel and Groll (1959) and (1966), Yao and Hwang (1990), and Gastwirth and Johnson (1994)) This is the problem of identifying all defectives in a set of ﬁnite objects by experiments that ﬁnd for a given subset if there is at least one defective in the subset Note that the classiﬁcation states, consisting of all subsets of the objects, can also be viewed as partially ordered In general, an experiment consists in observing a random variable or vector, X, whose distribution depends on the true unknown state, call it s ∈ S We assume that for each experiment e and state s ∈ S, the corresponding class conditional response distribution of X has some density f(x|e, s) We assume that the prior distribution of the true state is known, and consider the Bayesian approach The basic problem is to choose a sequence of experiments sequentially and a stopping rule to determine the true state as quickly as possible

Sequential Classification on Partially Ordered Sets

Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) is an innovative development in psychological and educational measurement. The CDCAT is designed to select items sequentially during the process of assessment to provide accurate information about examinees' cognitive strengths and weaknesses. This paper introduces the development and implementation of a web-based CD-CAT program for a large-scale English test in China, the Level 2 English Achievement. The paper is organized as follows. First, the main components of the CD-CAT design will be introduced, such as identification of attributes, Q-matrix formation, choice of CD model, item bank and calibration, item selection strategy, the parameter estimation, and design of CAT delivery system. Next, a study investigating an adequacy of model fitting for a large-scale pretest is described. This is followed by a classification accuracy study using the item parameters calibrated from the pretest. An ensuing section presents the results of a field CD-CAT test with 582 students in Beijing. A validity study of the computer-generated diagnostic results was also conducted. The final section discusses directions for future work.

The Development of Computerized Adaptive Testing with Cognitive Diagnosis for an English Achievement Test in China

This article proposes a new item selection method, namely, the modified maximum global discrimination index (MMGDI) method, for cognitive diagnostic computerized adaptive testing (CD-CAT). The new method captures two aspects of the appeal of an item: (a) the amount of contribution it can make toward adequate coverage of every attribute and (b) the amount of contribution it can make toward recovering the latent cognitive profile. A simulation study shows that the new method ensures adequate coverage of every attribute, which improves the validity of the test scores, and defensibility of the proposed uses of the test. Furthermore, compared with the original global discrimination index method, the MMGDI method improves the recovery rate of each attribute and of the entire cognitive profile, especially the latter. Therefore, the new method improves both the validity and reliability of the test scores from a CD-CAT program.

Improving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method

Nonparametric item response models have been developed as alternatives to the relatively inflexible parametric item response models. An open question is whether it is possible and practical to administer computerized adaptive testing with nonparametric models. This paper explores the possibility of computerized adaptive testing when using nonparametric item response models. A central issue is that the derivatives of item characteristic Curves may not be estimated well, which eliminates the availability of the standard maximum Fisher information criterion. As alternatives, procedures based on Shannon entropy and Kullback–Leibler information are proposed. For a long test, these procedures, which do not require the derivatives of the item characteristic eurves, become equivalent to the maximum Fisher information criterion. A simulation study is conducted to study the behavior of these two procedures, compared with random item selection. The study shows that the procedures based on Shannon entropy and Kullback–Leibler information perform similarly in terms of root mean square error, and perform much better than random item selection. The study also shows that item exposure rates need to be addressed for these methods to be practical.

Computerized adaptive testing under nonparametric IRT models

In educational testing, it is a common practice to produce test forms under a nonequivalent groups with anchor test (NEAT) design. In this design, the two test forms share a certain number of common items, while the populations who take the test forms might not be equivalent to each other. Test linking is conducted to establish the equivalency of ability scales from separate item response theory (IRT) calibrations of two test forms. Among existing linking methods, two approaches are related to the study in this paper: IRT model-based linking (Loyd & Hoover, 1980; Marco, 1977; Stocking & Lord, 1983) and equipercentile linking (Kolen & Brennan, 1995; von Davier, Holland, & Thayer, 2004b).

Xueli Xu

Papers

Computerized adaptive testing under nonparametric IRT models

Linking With Nonparametric IRT Models