Pedagosko i psiholosko testiranje i procjenjivanje spadaju među najvažnije doprinose znanosti o ponasanju nasem drustvu i pružaju temeljna i znacajna poboljsanja u odnosu na ranije postupke. Iako se ne može ustvrditi da su svi testovi dovoljno usavrseni niti da su sva testiranja razborita i korisna, postoji velika kolicina informacija koje ukazuju na ucinkovitost kvalitetnih instrumenata u onim situacijama u kojima je njihovo koristenje potkrijepljeno validacijskim podacima. Pravilna upotreba testova može dovesti do boljih odluka o pojedincima i programima nego sto bi to bio slucaj bez njihovog koristenja, a također i ukazati na put za siri i pravedniji pristup obrazovanju i zaposljavanju. Međutim, losa upotreba testova može dovesti do zamjetne stete nanesene ispitanicima i drugim sudionicima u procesu donosenja odluka na temelju testovnih podataka. Cilj Standarda je promoviranje kvalitetne i eticne upotrebe testova te uspostavljanje osnovice za ocjenu kvalitete postupaka testiranja. Svrha objavljivanja Standarda je uspostavljanje kriterija za evaluaciju testova, provedbe testiranja i posljedica upotrebe testova. Iako bi evaluacija prikladnosti testa ili njegove primjene trebala ovisiti prvenstveno o strucnim misljenjima, Standardi pružaju okvir koji osigurava obuhvacanje svih relevantnih pitanja. Bilo bi poželjno da svi autori, sponzori, nakladnici i korisnici profesionalnih testova usvoje Standarde te da poticu druge da ih također prihvate.

Standards for educational and psychological testing

Statistical Analysis with Missing Data

In meta-analysis, the usual way of assessing whether a set of single studies is homogeneous is by means of the Q test. However, the Q test only informs meta-analysts about the presence versus the absence of heterogeneity, but it does not report on the extent of such heterogeneity. Recently, the I(2) index has been proposed to quantify the degree of heterogeneity in a meta-analysis. In this article, the performances of the Q test and the confidence interval around the I(2) index are compared by means of a Monte Carlo simulation. The results show the utility of the I(2) index as a complement to the Q test, although it has the same problems of power with a small number of studies.

/pdf/assessing-heterogeneity-in-meta-analysis-q-statistic-or-i2-57zh1r444i.pdf

Assessing heterogeneity in meta-analysis: Q statistic or I2 index?

미국의 “전국 수학 교사 협의회”(National Council of Teachers of Mathematics, NCTM)는 1989년부터 〈학교 수학의 교육과정과 평가 규준〉(1989), 〈수학 가르침(교수)의 전문성 규준〉(1991), 〈학교 수학의 평가(시험) 규준〉(NCTM, 1995), 〈학교 수학의 원리와 규준〉(2000)을 출판하여 미국의 수학 교육 의 전망(목표, 나아갈 길)과 규준(실행 지침)을 제시하였다. 수학 교사들로 구성된 미국의 NCTM은 학생, 학부모, 학교 행정가 등 많은 사람들과 힘을 합하여 모든 학생들에게 수준 높은 수학 교육을 받을 수 있는 여건(환경, 기회)을 조성하는 데 구심점의 역할을 하였다. 한편 많은 관련 단체들은 여러 배경과 능력을 가진 학생들이 전문성을 지닌 교사(특수 교사를 일컫는 밀이 아니다. 수학 교과를 이해하고 수학의 전문성과 특수성을 가르칠 수 있는 일반 교사를 일컫는 말이다.)로부터 미래를 대비해 평등하고, 진취적이며, 지원이 잘 이루어지고, 공학 도구(IT)가 잘 갖춰진 환경에서 중요한 수학적 아이디어를 이해하면서 학습할 수 있는 수학 교실(미국에서는 우리나라처럼 수학 교사가 수학 시간에 학생의 방(교실: Homeroom)에 찾아가지 않고 학생들이 선생의 방(수학 교실: Classroom)을 찾아온다. 전형적인 수학 교실의 사진은 2쪽에 나와 있다.)을 만들기 위해 함께 힘썼다. NCTM에서 출간한 여러 규준들은 우리나라의 제6차와 제7차 교육과정에도 큰 영향을 미쳤다. 이 글에서는 NCTM(2000)에서 제시한 학습 원리를 간단히 살펴본 다음 이를 중심으로 현재 미국 수학 교육의 교수ㆍ학습 이론의 동향을 살펴본다.

미국 NCTM의 Principles and Standards for School Mathematics에 나타난 수학과 교수,학습의 이론

This article urges counseling psychology researchers to recognize and report how missing data are handled, because consumers of research cannot accurately interpret findings without knowing the amount and pattern of missing data or the strategies that were used to handle those data. Patterns of missing data are reviewed, and some of the common strategies for dealing with them are described. The authors provide an illustration in which data were simulated and evaluate 3 methods of handling missing data: mean substitution, multiple imputation, and full information maximum likelihood. Results suggest that mean substitution is a poor method for handling missing data, whereas both multiple imputation and full information maximum likelihood are recommended alternatives to this approach. The authors suggest that researchers fully consider and report the amount and pattern of missing data and the strategy for handling those data in counseling psychology research and that editors advise researchers of this expectation.

Best Practices for Missing Data Management in Counseling Psychology

Meta-analytic methods were used to integrate the findings of a sample of Monte Carlo studies of the robustness of the F test in the one- and two-factor fixed effects ANOVA models. Monte Carlo results for theWelch (1947) and Kruskal-Wallis (Kruskal & Wallis, 1952) tests were also analyzed. The meta-analytic results provided strong support for the robustness of the Type I error rate of the F test when certain assumptions were violated. The F test also showed excellent power properties. However, the Type I error rate of the F test was sensitive to unequal variances, even when sample sizes were equal. The error rate of the Welch test was insensitive to unequal variances when the population distribution was normal, but nonnormal distributions tended to inflate its error rate and to depress its power. Meta-analytic and exact statistical theory results were used to summarize the effects of assumption violations for the tests.

Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases.

The use of eligibility for a free lunch as a measure of a student’s socioeconomic status continues to be a fixture of quantitative education research. Despite its popularity, it is unclear that education researchers are familiar with what student eligibility for a free lunch does (and does not) represent. The authors examine the National School Lunch Program, which is responsible for certifying students as eligible for a free lunch, and conclude that free lunch eligibility is a poor measure of socioeconomic status, which suffers from important deficiencies that can bias inferences. A table characterizing key strengths and weaknesses of variables used as measures of socioeconomic status is provided to facilitate comparisons.

Student Eligibility for a Free Lunch as an SES Measure in Education Research

The impact of missing data on quantitative educational research has been a great concern to methodologists but less so for many educational researchers. In studies where missing data are present, it is common for educational researchers to use ad hoc methods such as listwise or pairwise deletion to deal with the missing data problem. Yet the recent American Psychological Association Task Force on Statistical Inference warned against the use of these methods for dealing with missing data and encouraged researchers to instead employ alternative methods that take into account the conditions under which missing data occurred. These newer methods have been implemented in statistical software (e.g., SPSS® and SAS®); thus, they are accessible to researchers. In this paper, we promote the use of principled statistical methods for treating missing data that employ singleor multipleimputation of missing values. These methods are illustrated with real-world data sets, and the results are shown to be different from those obtained from listwise deletion method. Statistical assumptions underlying these principled methods are discussed and emphasized. Quality of educational research will be enhanced if (a) researchers explicitly acknowledge missing data problems and the conditions under which they occurred, (b) principled, rather than ad hoc methods, are used to handle missing data, and (c) journal referees and editors incorporate the reporting of appropriate missing data methods in standards used in judging manuscripts for publication. Advances in Missing Data 3 Advances in Missing Data Methods and Implications for Education Research Consider a scenario in which a freshly minted Ph.D. showed off his bound dissertation to a fellow graduate student who was known among her peers to be a “number cruncher.” As she leafed through the pages of the methods section, she quickly noticed that all missing data were coded zero, instead of being declared as missing. She also saw that all analyses in this dissertation, including factor analysis, multiple regression, discriminant function analyses, etc., were based on all data including the zeros. No one on the dissertation committee caught this mistake. In another scenario, a Ph.D. candidate explaining the results of the data analyses to her dissertation committee noted that the choice of listwise deletion led to the removal of approximately one third of the data. Both the candidate and committee members failed to recognize the possibly devastating effects on inferences due to these missing data. Variations of these scenarios may occur more frequently in educational research than is commonly believed or acknowledged. Missing data in research studies are the rule rather than the exception. For example, various literatures indicate that a 20% attrition rate has been held as a benchmark for studies related to youth, school-based programs, and clinical research (Hall, 1993; Kellam, Rebok, Ialongo, & Mayer, 1994; Mason, 1999). Many reasons contribute to data missing from research projects: subject’s attrition in pretest-posttest or longitudinal designs, equipment failure or unclear instruction or intrusive questions on a survey. Whatever the reason(s) for missing data, their impact on quantitative research has been a great concern to methodologists. In survey research, missing data problems are particularly acute, with substantial amounts of missing data frequently appearing (Little & Rubin, 1987). To combat these problems, investigators spend enormous time, funding, and energy to minimize incomplete data or nonAdvances in Missing Data 4 response among respondents (Mason, 1999). Procedures have been devised by statisticians to mitigate inferential or descriptive problems caused by missing data (Dempster, Laird & Rubin, 1977; Glasser, 1964; Rubin, 1987; Schafer, 1997). The development of these principled procedures is a unique feature of the survey research literature. For the past two decades, the statistical properties of these procedures have been studied and made known. And many have been incorporated into widely accessible software since 1990 (Cool, 2000; Kim & Curry, 1977; Mason, 1999; Raymond & Roberts, 1987; Witta, 2000). Even so, an examination of articles published in 11 education journals for the past 5 years reveals that, in studies where missing data were reported and care was taken to treat missing data, an overwhelming majority of these studies used ad hoc methods, such as listwise and pairwise deletion, to handle missing data. Ad hoc methods are characterized by their failure to take into account the mechanism that led to missing data. Yet the recent APA Task Force on Statistical Inference explicitly warmed against the use of these methods: Special issues arise in modeling when we have missing data. The two popular methods for dealing with missing data that are found in basic statistics packages—listwise and pairwise deletion of missing values—are among the worst methods available for practical applications. (Wilkinson & The APA Task Force on Statistical Inference, 1999, p. 598) Newer and principled methods take into consideration conditions under which missing data occurred. In this paper, we promote the use of three such methods, namely, two singleimputation methods and one based on multiple imputation. These methods are illustrated with real-world data sets, and the results are shown to be different from those obtained from listwise deletion. Key issues in guiding the use of these methods are also discussed. Since these methods have been implemented into statistical software (e.g., SPSS® and SAS®) and should be widely Advances in Missing Data 5 available, their utility by educational researchers is expected to substantially increase. Throughout our illustration and discussion, the focus is on handling cases in which partial information is missing rather than a complete lack of information. The remainder of this paper is divided into nine sections: (1) Problems Caused by Missing Data and Missing Data Treatments, (2) Missing Data Methods Reported in 11 Education Journals, (3) Missing Data Mechanisms, (4) Overview of Five Ad Hoc Missing Data Methods, (5) Principled Single Imputation--FIML, (6) Principled Single-Imputation—SPSS® EM, (7) Multiple Imputation, (8) Importance of Statistical Assumptions, and (9) Implications and Recommendations for Educational Research. 1. Problems Caused by Missing Data and Missing Data Treatments Why are missing data a problem? The most serious concern is that missing data can introduce bias into estimates derived from a statistical model (Becker & Walstad, 1990; Becker & Powers, 2001; Holt, 1997; Rubin, 1987). For example, it is possible that non-respondents might have different response profiles compared to those who responded completely. Thus, the remaining sample is no longer random or representative of the population from which it was randomly drawn. If the researcher chose to draw their conclusions based solely on those who responded, the conclusions would be biased. Furthermore, missing data result in a loss of information and statistical power (Anderson, Basilevsky & Hum, 1983; Kim & Curry, 1977). The elimination of subjects with missing information on one or more variables from the statistical analysis in listwise deletion decreases the error degrees of freedom (df) in statistical tests such as the t. This decrease in turn leads to reduced statistical power and larger standard errors compared to those obtained from complete random samples (Cohen & Cohen, 1983; Cool, 2000). Advances in Missing Data 6 Similar loss of df and statistical power occurs with pairwise deletion, where standard deviations, correlations and covariances are calculated on the basis of ava ilable data on each variable (Glasser, 1964; Raymond & Robert, 1987). As a result, the sample composition differs from variable to variable, and the population to which the results are generalized is no longer clearly defined. Another problem with missing data is that they make common statistical methods inappropriate or difficult to apply (Rubin, 1987). For example, when missing data are present in a factorial analysis of variance the design is unbalanced. Consequently, the standard statistical analysis that is appropriate for balanced designs is no longer appropriate under this condition. Even if data are assumed to be missing in a completely random fashion, the proper analysis is complicated. Multivariate statistical methods, as they are programmed into commercial statistical software, are applicable to complete data sets by default. Finally, valuable resources are wasted as a result of missing data. Time and funding spent on subjects who subsequently leave a study and/or produce missing data represents a loss (Buu, 1999; Holt, 1997). Such loss is a particular concern in longitudinal research, large-scale assessments, high-stake studies, and/or surveys that ask sensitive information or target respondents who are not accustomed to responding to opinions surveys (such as the first generation Hmong immigrants). Efforts to achieve higher response rates and complete profiles from respondents require researchers to allocate additional time and resources to trace cases who failed to respond or those whose responses were incomplete or unusable. These efforts may not always pay off. 2. Missing Data Methods Reported in 11 Education Journals Advances in Missing Data 7 In order to understand how educational researchers currently deal with missing data, we reviewed the quantitative studies published in 11 education journals from 1998 to the summer of 2002. The 11 journals were American Educational Research Journal (AERJ ), Educational Researcher (ER), Journal of Counseling Psychology (JCP), Journal of Educational Psychology (JEP), Journal of Research in Science Teaching (JRST), Journal of Special Education (JSE), Journal of School Psychology (JSP), The Modern Language Journal (MLJ), Research in

Advances in Missing Data Methods and Implications for Educational Research

Monte carlo studies are being used in item response theory (IRT) to provide information about how validly these methods can be applied to realistic datasets (e.g., small numbers of examinees and mu...

/pdf/monte-carlo-studies-in-item-response-theory-3058bmgqkl.pdf

Monte carlo studies in item response theory

Many statistical procedures used in educational research are described as requiring that dependent variables follow a normal distribution, implying an interval scale of measurement. Despite the desirability of interval scales, many dependent variables possess an ordinal scale of measurement in which the differences among values composing the scale are unequal in terms of what is being measured, permitting only a rank ordering of scores. This means that data possessing an ordinal scale will not satisfy the assumption of normality needed in many statistical procedures and may produce biased statistical results that threaten the validity of inferences. This article shows how the measurement technique known as item response theory can be used to rescale ordinal data to an interval scale. The authors provide examples of rescaling using student performance data and argue that educational researchers should routinely consider rescaling ordinal data using item response theory.

https://www.jstor.org/stable/pdfplus/3516068.pdf

Michael R. Harwell

Papers

Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases.

Student Eligibility for a Free Lunch as an SES Measure in Education Research

Advances in Missing Data Methods and Implications for Educational Research

Monte carlo studies in item response theory

Rescaling Ordinal Data to Interval Data in Educational Research