Pedagosko i psiholosko testiranje i procjenjivanje spadaju među najvažnije doprinose znanosti o ponasanju nasem drustvu i pružaju temeljna i znacajna poboljsanja u odnosu na ranije postupke. Iako se ne može ustvrditi da su svi testovi dovoljno usavrseni niti da su sva testiranja razborita i korisna, postoji velika kolicina informacija koje ukazuju na ucinkovitost kvalitetnih instrumenata u onim situacijama u kojima je njihovo koristenje potkrijepljeno validacijskim podacima. Pravilna upotreba testova može dovesti do boljih odluka o pojedincima i programima nego sto bi to bio slucaj bez njihovog koristenja, a također i ukazati na put za siri i pravedniji pristup obrazovanju i zaposljavanju. Međutim, losa upotreba testova može dovesti do zamjetne stete nanesene ispitanicima i drugim sudionicima u procesu donosenja odluka na temelju testovnih podataka. Cilj Standarda je promoviranje kvalitetne i eticne upotrebe testova te uspostavljanje osnovice za ocjenu kvalitete postupaka testiranja. Svrha objavljivanja Standarda je uspostavljanje kriterija za evaluaciju testova, provedbe testiranja i posljedica upotrebe testova. Iako bi evaluacija prikladnosti testa ili njegove primjene trebala ovisiti prvenstveno o strucnim misljenjima, Standardi pružaju okvir koji osigurava obuhvacanje svih relevantnih pitanja. Bilo bi poželjno da svi autori, sponzori, nakladnici i korisnici profesionalnih testova usvoje Standarde te da poticu druge da ih također prihvate.

Standards for educational and psychological testing

Purpose. This article examines and illustrates the use and interpretation of the kappa statistic in musculoskeletal research. Summary of Key Points. The reliability of clinicians' ratings is an important consideration in areas such as diagnosis and the interpretation of examination findings. Often, these ratings lie on a nominal or an ordinal scale. For such data, the kappa coefficient is an appropriate measure of reliability. Kappa is defined, in both weighted and unweighted forms, and its use is illustrated with examples from musculoskeletal research. Factors that can influence the magnitude of kappa (prevalence, bias, and nonindependent ratings) are discussed, and ways of evaluating the magnitude of an obtained kappa are considered. The issue of statistical testing of kappa is considered, including the use of confidence intervals, and appropriate sample sizes for reliability studies using kappa are tabulated. Conclusions. The article concludes with recommendations for the use and interpretation of kappa.

The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements

In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.

/pdf/answering-the-call-for-a-standard-reliability-measure-for-15yf50gc9e.pdf

Answering the Call for a Standard Reliability Measure for Coding Data

The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models. It avoids the supposed subjectivity in the threshold selection process, when continuous probability derived scores are converted to a binary presence‐absence variable, by summarizing overall model performance over all possible thresholds. In this manuscript we review some of the features of this measure and bring into question its reliability as a comparative measure of accuracy between model results. We do not recommend using AUC for five reasons: (1) it ignores the predicted probability values and the goodness-of-fit of the model; (2) it summarises the test performance over regions of the ROC space in which one would rarely operate; (3) it weights omission and commission errors equally; (4) it does not give information about the spatial distribution of model errors; and, most importantly, (5) the total extent to which models are carried out highly influences the rate of well-predicted absences and the AUC scores.

/pdf/auc-a-misleading-measure-of-the-performance-of-predictive-qqj6tbb550.pdf

AUC: a misleading measure of the performance of predictive distribution models

As a method specifically intended for the study of messages, content analysis is fundamental to mass communication research. Intercoder reliability, more specifically termed intercoder agreement, is a measure of the extent to which independent judges make the same coding decisions in evaluating the characteristics of messages, and is at the heart of this method. Yet there are few standard and accessible guidelines available regarding the appropriate procedures to use to assess and report intercoder reliability, or software tools to calculate it. As a result, it seems likely that there is little consistency in how this critical element of content analysis is assessed and reported in published mass communication studies. Following a review of relevant concepts, indices, and tools, a content analysis of 200 studies utilizing content analysis published in the communication literature between 1994 and 1998 is used to characterize practices in the field. The results demonstrate that mass communication researchers often fail to assess (or at least report) intercoder reliability and often rely on percent agreement, an overly liberal index. Based on the review and these results, concrete guidelines are offered regarding procedures for assessment and reporting of this important aspect of content analysis.

/pdf/content-analysis-in-mass-communication-assessment-and-3l6g5t7rnd.pdf

Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

This paper considers some appropriate and inappropriate uses of coefficient kappa and alternative kappa-like statistics. Discussion is restricted to the descriptive characteristics of these statist...

Coefficient Kappa: Some Uses, Misuses, and Alternatives:

Test equating, scaling, and linking: Methods and practices, 3rd ed.

Observed score equating using the random groups design random groups - smoothing in equipercentile equating nonequivalent groups - linear methods nonequivalent groups - equipercentile methods item response theory methods standard errors of equating practical issues in equating in Scaling to achieve comparability. Appendices: answers to exercises computer programs and data sets.

Test Equating: Methods and Practices

Evidence of the uncertainty attached to school and individual scores is required to avoid over interpretation of results. Generalizability analysis provides in the standard error a suitable indicator of uncertainty. Assessments depart from traditional measurements in ways that require extensions and re interpretatons of generalizability analysis. The authors find, for example, that in many analyses the variance component interpreted as Pupil x Task interaction actually arises in part from instability in pupil performance. It is necessary to recognize in the school-level analysis that pupils are nested in classes and schools. Whether to treat the population of a school's pupils as infinite or as limited to the student body assessed also requires careful consideration.

Generalizability analysis for performance assessments of student achievement or school effectiveness

Glaser and Nitko (1971) define a criterion-referenced test as "one that is deliberately constructed to yield measurements that are directly interpretable in terms of specified performance standards" (p. 653). This is probably the best-known definition of a criterion-referenced test, but others have been proposed (e.g., Harris and Stewart, 1971; Ivens, 1970; Kriewall, 1969; and Livingston, 1972). Nothing in the Glaser and Nitko definition, or in most other definitions of "criterion-referenced test," necessitates the existence or use of a single criterion or cutting score as a "specified performance standard." However, much of the literature subsumed under the heading of criterion-referenced measurement does, in fact, postulate the existence of a single cutting score. Since this inconsistency in terminology can lead to confusion, we prefer to reserve the term mastery test for a criterion-referenced test with a single fixed mastery cutting score (see Harris, 1974). Hively (1974) and Millman (1974), among others, suggest using the descriptor "domain-referenced test" rather than "criterion-referenced test." They note that the word "criterion" is ambiguous in some contexts, and they argue that the word "domain" provides a more direct specification of the entir, set of items or tasks under consideration. If one accepts these arguments, a mastery test can be defined as a domain-referenced test with a single cutting score. In this paper we develop and discuss an index of dependability for mastery tests. For reasons discussed later, we choose not to call our index a reliability coefficient, although many of the indices previously developed for mastery tests have been proposed as indices of reliability (see Brennan, 1974, for a review of the literature). For example, Livingston (1972) proposed a reliability coefficient based upon the squared deviations of scores from the cutting score; Swaminathan, Hambleton, and Algina (1974) proposed the use of Cohen's (1960) coefficient K; Marshall and Haertel (1975) suggest using a mean split-half coefficient of agreement; and Carver (1970) suggests two other coefficients.

Robert L. Brennan

Papers

Coefficient Kappa: Some Uses, Misuses, and Alternatives:

Test equating, scaling, and linking: Methods and practices, 3rd ed.

Test Equating: Methods and Practices

Generalizability analysis for performance assessments of student achievement or school effectiveness

An index of dependability for mastery tests