The widespread use of standardized health surveys is predicated on the largely untested assumption that scales constructed from those surveys will satisfy minimum psychometric requirements across diverse population groups. Data from the Medical Outcomes Study (MOS) were used to evaluate data completeness and quality, test scaling assumptions, and estimate internal-consistency reliability for the eight scales constructed from the MOS SF-36 Health Survey. Analyses were conducted among 3,445 patients and were replicated across 24 subgroups differing in sociodemographic characteristics, diagnosis, and disease severity. For each scale, item-completion rates were high across all groups (88% to 95%), but tended to be somewhat lower among the elderly, those with less than a high school education, and those in poverty. On average, surveys were complete enough to compute scales scores for more than 96% of the sample. Across patient groups, all scales passed tests for item-internal consistency (97% passed) and item-discriminant validity (92% passed). Reliability coefficients ranged from a low of 0.65 to a high of 0.94 across scales (median = 0.85) and varied somewhat across patient subgroups. Floor effects were negligible except for the two role disability scales. Noteworthy ceiling effects were observed for both role disability scales and the social functioning scale. These findings support the use of the SF-36 survey across the diverse populations studied and identify population groups in which use of standardized health status measures may or may not be problematic.

https://www.jstor.org/stable/pdfplus/3766189.pdf

The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups.

This book gives an up-to-date overview of methodological and data-analytical issues of cross-cultural studies. Written by leading experts in the field, it presents the most important tools for doing cross-cultural research and outlines design considerations, methods, and analytical techniques that can improve ecological validity and help researchers to avoid pitfalls in cross-cultural psychology. By focusing on the relevant research questions that can be tackled with particular methods, it provides practical guidance on how to translate conceptual questions into decisions on study design and statistical techniques. Featuring examples from cognitive and educational assessment, personality, health, and intercultural communication and management, and illustrating key techniques in feature boxes, this concise and accessible guide is essential reading for researchers, graduate students, and professionals who work with culture-comparative data.

Methods and Data Analysis for Cross-Cultural Research

1. Engaged Scholarship in a Professional School 2. Philosophy of Science Underlying Engaged Scholarship 3. Formulating the Research Problem 4. Building a Theory 5. Process and Variance Models 6. Designing Variance Studies 7. Designing Process Studies 8. Communicating and Using Research Knowledge 9. Practicing Engaged Scholarship

Engaged Scholarship: A Guide for Organizational and Social Research

The consequences of prejudice against accepting the null hypothesis were examined through (a) a mathematical model intended to stimulate the research-publication process and (b) case studies of apparent erroneous rejections of the null hypothesis in published psychological research. The input parameters for the model characterize investigators' probabilities of selecting a problem for which the null hypothesis is true, of reporting, following up on, or abandoning research when data do or do not reject the null hypothesis, and they characterize editors' probabilities of publishing manuscripts concluding in favor of or against the null hypothesis. With estimates of the input parameters based on a questionnaire survey of a sample of social psychologists, the model output indicates a dysfunctional research-publication system. Particularly, the model indicates that there may be relatively few publications on problems for which the null hypothesis is (at least to a reasonable approximation) true, and of these, a high proportion will erroneously reject the null hypothesis. The case studies provide additional support for this conclusion. Accordingly, it is concluded that research traditions and customs of discrimination against accepting the null hypothesis may be very detrimental to research progress. Some procedures that can help eliminate this bias are prescribed.

/pdf/consequences-of-prejudice-against-the-null-hypothesis-ftww9pyl5k.pdf

Consequences of Prejudice Against the Null Hypothesis

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0140525X00011183

Peer-review practices of psychological journals: The fate of published articles, submitted again

In a recent paper, Cleary and Hilton (1968) discussed one possible interpretation of test bias. According to that definition, an item of a test is considered biased for members of a particular group if the item produces an uncommon discrepancy between the performance of that group and the performance of other groups. That is, the members of the group obtain an average score which differs from the average score of other groups by more or less than expected from their performance on other items of the same test. In terms of the analysis of variance, bias was defined as an item-group interaction. On 'the basis of their data, Cleary and Hilton concluded that the items in the Preliminary Scholastic Aptitude Test are not biased, and that, if the test is discriminatory, the discrimination is not largely attributable to particular sets of items.

Test bias: prediction of grades of negro and white students in integrated colleges

a concomitant increase in concern about the applicability of widely used tests in different cultural groups. The College Entrance Examination Board, for example, has become concerned about the appropriateness of the Scholastic Aptitude Test (SAT) and the Preliminary Scholastic Aptitude Test (PSAT) for some subgroups of the population, particularly Negro Americans. It is often difficult to determine what is meant by the word &dquo;bias&dquo; when it is used in reference to tests. Test &dquo;bias&dquo; is explored here in terms of individual test items. An item of a test is said to

/pdf/an-investigation-of-item-bias-5dxtm5iwu3.pdf

An Investigation of Item Bias

“… there's this desert prison, see, with an old prisoner, resigned to his life, and a young one just arrived. The young one talks constantly of escape, and, after a few months, he makes a break. He's gone a week, and then he's brought back by the guards. He's half dead, crazy with hunger and thirst. He describes how awful it was to the old prisoner. The endless stretches of sand, no oasis, no signs of life anywhere. The old prisoner listens for a while, then says, ‘Yep. I know. I tried to escape myself, twenty years ago.’ The young prisoner says, ‘You did? Why didn't you tell me, all these months I was planning my escape? Why didn't you let me know it was impossible?’ And the old prisoner shrugs, and says, ‘So who publishes negative results?’” (Hudson, 1968, p. 168)

A Proposal for a New Editorial Policy in the Social Sciences

Formulae are developed from the assumptions of classical test theory to demonstrate the effect of error of measurement on the power of the F test for a fixed-effects one-way analysis of variance model. A simple model specifying the fixed and variable costs of testing is adopted and equations are derived that indicate the sample size that maximizes the non-centrality parameter subject to the cost constraints. Given the sample size, the corresponding test length is then implied by the cost model. A computer program used for the estimation of power for permissible allocations of resources is described.

Error of measurement and the power of a statistical test

Seven programmed test procedures were developed and investigated using existing data on 190 verbal-type items for 4,885 students. Tests were scored as if the student had taken items in the order implied by the test. Shortened conventional tests were also scored for comparative purposes. The criteria against which these procedures were evaluated were the 190-item total test score and four outside criterion tests.



Against the criterion of reproducing the 190-item total test score in the cross-validation sample the programmed tests were found to he only slightly superior to the shortened conventional tests. However, the programmed tests had correlations with the outside criterion tests that were substantially higher than the corresponding shortened conventional tests. It was estimated that a test which was parallel to the 190-item total test would have to be 3.36 times as long as the best programmed test to have an equal median correlation with the outside criterion tests.

T. Anne Cleary

Papers

Test bias: prediction of grades of negro and white students in integrated colleges

An Investigation of Item Bias

A Proposal for a New Editorial Policy in the Social Sciences

Error of measurement and the power of a statistical test

The Development and Evaluation of Several Programmed Testing Methods.