There has been conspicuously little research concerning missing data problems in the applied psychology literature. Fortunately, other fields have begun to investigate this issue. These include survey research, marketing, statistics, economics, and biometrics. A review of this literature suggests several trends for applied psychologists. For example, listwise deletion of data is often the least accurate technique to deal with missing data. Other methods for estimating missing data scores may be more accurate and preserve more data for investigators to analyze. Further, the literature reveals that the amount of missing data and the reasons for deletion of data impact how investigators should handle the problem. Finally, there is a great need for more investigation of strategies for dealing with missing data, especially when data are missing in nonrandom or systematic patterns.

Missing data: a conceptual review for applied psychologists

The use of the Gibbs sampler with fully conditionally specified models, where the distribution of each variable given the other variables is the starting point, has become a popular method to create imputations in incomplete multivariate data. The theoretical weakness of this approach is that the specified conditional densities can be incompatible, and therefore the stationary distribution to which the Gibbs sampler attempts to converge may not exist. This study investigates practical consequences of this problem by means of simulation. Missing data are created under four different missing data mechanisms. Attention is given to the statistical behavior under compatible and incompatible models. The results indicate that multiple imputation produces essentially unbiased estimates with appropriate coverage in the simple cases investigated, even for the incompatible models. Of particular interest is that these results were produced using only five Gibbs iterations starting from a simple draw from observed marginal distributions. It thus appears that, despite the theoretical weaknesses, the actual performance of conditional model specification for multivariate imputation can be quite good, and therefore deserves further study. © 2006 Taylor & Francis.

/pdf/fully-conditional-specification-in-multivariate-imputation-3t2lfljk99.pdf

Fully conditional specification in multivariate imputation

A taxonomy of 31 multiple-choice item-writing guidelines was validated through a logical process that included two sources of evidence: the consensus achieved from reviewing what was found in 27 textbooks on educational testing and the results of 27 research studies and reviews published since 1990. This taxonomy is mainly intended for classroom assessment. Because textbooks have potential to educate teachers and future teachers, textbook writers are encouraged to consider these findings in future editions of their textbooks. This taxonomy may also have usefulness for developing test items for large-scale assessments. Finally, research on multiple-choice item writing is discussed both from substantive and methodological viewpoints.

https://www.lesn.appstate.edu/olson/RES5560/Articles%20used%20in%205560/Haladyna_Downing_Rodrequez_2002.pdf

A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment

The effects of using two methods (item mean and person mean) for replacing missing data in Likert scales were studied. The results showed that both methods were good representations of the original data when both the number of respondents with missing data and the number of items missing were 20% or less. As the numbers of missing items and of respondents with missing data increased for the person mean substitution method, a spurious increase in the inter-item correlations (and, therefore, reliability) for the scale was produced. The item mean substitution reduced the reliability estimates of the scale. These results suggest caution in the use of the person mean substitution method as the numbers of missing items and respondents increase.

Missing Data in Likert Ratings: A Comparison of Replacement Methods

For organizational research on individual change, missing data can greatly reduce longitudinal sample size and potentially bias parameter estimates. Within the structural equation modeling framework, this article compares six missing data techniques (MDTs): listwise deletion, pairwise deletion, stochastic regression imputation, the expectation-maximization (EM) algorithm, full information maximization likelihood (FIML), and multiple imputation (MI). The rationale for each technique is reviewed, followed by Monte Carlo analysis based on a threewave simulation of organizational commitment and turnover intentions. Parameter estimates and standard errors for each MDT are contrasted with complete-data estimates, under three mechanisms of missingness (completely random, random, and nonrandom) and three levels of missingness (25%, 50%, and 75%; all monotone missing). Results support maximum likelihood and MI approaches, which particularly outperform listwise deletion for parameters involving many recouped cases....

Longitudinal Modeling with Randomly and Systematically Missing Data: A Simulation of Ad Hoc, Maximum Likelihood, and Multiple Imputation Techniques

A 34 item scale entitled Statistics Attitude Survey (SAS) was developed and administered to three samples of students taking a beginning statistics course. Analyses showed that the scale was highly homogeneous and that total scale scores had moderate correlations with statistics grades.

Reliability and Validity of a Statistics Attitude Survey

Data were simulated to conform to covariance patterns taken from the personnel selection literature. Two, six, and ten percent of the values were deleted from one of three predictor variables in sample sizes of 50, 100 and 200. Incomplete data matrices were treated by four methods: (a) elimination of cases with incomplete data records; (b) substitution of missing values with the variable mean; (c) replacement of missing values with an estimate obtained from simple regression; and (d) replacement of missing values with an estimate derived from iterated multiple regression. The treated data matrices were subjected to multiple regression analyses, and the resulting regression equations were compared to the equations obtained from the original, complete data. The two regression based estimation procedures provided the most accurate regression equations, followed by the method of inserting means. Discarding cases with incomplete records was the least accurate method. Although the results supported the practice...

A Comparison of Methods for Treating Incomplete Data in Selection Research

The validity of the Statistics Attitude Survey (SAS) was further examined in the present study. Students were assessed on a number of pretest and posttest cognitive and non-cognitive variables, including the SAS. SAS scores were found to be significantly related to such cognitive variables as basic mathematics skills, statistics preknowledge, and course grades. Non-cognitive factors with which SAS was significantly correlated were sex, the degree to which students indicated that they had wanted to take the course and that they were glad they had taken the course, number of previous mathematics courses completed, the status of a course being required or elective, calculator attitudes, and course and instructor evaluations. In addition, SAS scores showed a significant positive change from the beginning to the end of the course.

Validity of a Statistics Attitude Survey: A Follow-up Study.

The present investigation compared two statistics attitude scales, the Statistics Attitude Survey (SAS) by Roberts and Bilderback (1980) and the Attitudes Toward Statistics (ATS) by Wise (1985). Wise had claimed that the SAS was partially invalid because many of the items appeared to be achievement oriented rather than attitudinal in nature. The SAS and ATS were combined in one scale and administered to 280 introductory statistics students at the beginning of the course. Course grades were obtained at the end of the course. Analyses showed high reliabilities for both SAS and ATS but a correlation of nearly .9 between the two scales. In addition, none of the correlations between the two scales with several criteria, including course grades, were significantly different. It was concluded that the ATS was essentially an alternate form of the previously developed SAS.

A Comparison of Two Scales Measuring Attitudes Towards Statistics.

The Perceptions of Cheating Scale (POCS) was developed to assess perceptions of academic cheating in examination-related situations. The final form of the POCS contained 30 items depicting cheating scenarios. Respondents were required to assign levels of punishment appropriate for the cheating described. Data were collected from both college students and faculty. As partial validation, a comparison was made between faculty and student response patterns. Results showed clear differences with faculty consistently assigning harsher punishments. Examination of the relationships between the POCS and several criterion variables yielded some small but statistically significant coefficients.

Dennis M. Roberts

Papers

Reliability and Validity of a Statistics Attitude Survey

A Comparison of Methods for Treating Incomplete Data in Selection Research

Validity of a Statistics Attitude Survey: A Follow-up Study.

A Comparison of Two Scales Measuring Attitudes Towards Statistics.

A Scale to Assess Perceptions of Cheating in Examination-Related Situations