Responding to the scarcity of suitable measures of teacher knowledge, this article reports on a novel assessment approach to measuring teacher knowledge of teaching mathematics. The new approach uses teachers' ability to analyze teaching as a proxy for their teaching knowledge. Video clips of classroom instruction, which respondents were asked to analyze in writing, were used as item prompts. Teacher responses were scored along four dimensions: mathematical content, student thinking, alternative teaching strategies, and overall quality of interpretation. A prototype assessment was developed and its reliability and validity were examined. Respondents' scores were found to be reliable. Positive, moderate correlations between teachers' scores on the video-analysis assessment, a criterion measure of mathematical content knowledge for teaching, and expert ratings provide initial evidence for the criterion-related validity of the video-analysis assessment. Results suggest that teachers' ability to analyze teach...

Using Video Clips of Mathematics Classroom Instruction as Item Prompts to Measure Teachers' Knowledge of Teaching Mathematics.

Magnitude and moderators of bias in observer ratings: A meta-analysis.

Although it appears that many assume job analysis information is accurate, there is considerable evidence from other fields to suggest that the types of subjective judgments often involved in job analysis may be subject to systematic sources of inaccuracy. Drawing from the social, cognitive, and industrial-organizational psychology literatures, this review develops a framework that delineates 16 potential sources of inaccuracy in job analysis. This includes such social sources as social influence and self-presentation processes as well as cognitive sources such as limited and biased information processing. For each source of inaccuracy, the relevant literature is first reviewed, its potential operation in the job analysis context is described, and propositions for future research are derived. In addition, the likelihood of these sources of inaccuracy across various job analysis facets are described, concluding with recommendations for research and practice. As a process of obtaining information about jobs (McCormick, 1979), job analysis is one of the most widely used organizational data collection techniques. It forms the foundation upon which virtually all other human resource management systems are built (Buffer & Harvey, 1988), including personnel selection, performance appraisal, training, career development, workforce planning, safety, and licensing requirements (Ash, 1988). Its use is mandated to meet legal requirements (Uniform Guidelines, i978), and estimated annual costs for job analyses

https://www.krannert.purdue.edu/faculty/campionm/Social_Cognitive_Sources.pdf

Social and Cognitive Sources of Potential Inaccuracy in Job Analysis

The assessment of writing ability is notoriously difficult. Different facets of the assessment seem to influence its outcome. Besides the writer’s writing proficiency, the topic of the assignment, the features or traits scored (e.g., content or language use) and even the way in which these traits are scored (e.g., holistically or analytically) may contribute to the writer’s score. In this article, the effect of these facets is estimated in a generalizability study using variance analytic techniques. A structural equation modeling (SEM) approach is used to estimate the variance components in the writing scores. Eighty-nine grade 6 students (aged 11-12 years) wrote four essays, each of which was scored by five raters using two scoring methods (i.e., holistically and analytically) for two traits (i.e., Content and Organization, and Language Use). Analyses of these ratings showed that the generalizability of writing scores and the effects of raters and topics are very much dependent on the way the essays are ...

Generalizability of writing scores: an application of structural equation modeling

Relevant aspects of generalizability theory relating to performance assessments (PAs) are reviewed. The role of various facets in assessing the generalizability of PAs is examined, and some popular estimates of reliabilityfor PAs are considered from the perspective of generalizability theory.

Performance Assessments from the Perspective of Generalizability Theory

In this article, univariate and multivariate generalizability theory is employed to examine psychometric characteristics of Listening and Writing tests developed by American College Testing (ACT) for its Work Keys program. For both of these tests, the same tape-recorded messages provide the stimuli for written responses that are evaluated by raters. Of particular concern are the numbers of messages and raters needed for adequate measurement precision. The occurrence of relatively large person-message interactions suggested that several (at least six) messages are probably required. Also, the analyses suggest that it is highly desirable to employ at least two raters. For these sample sizes, however, the multivariate results are less encouraging in part because of correlated errors.

Generalizability Analyses of Work Keys Listening and Writing Tests.

Job analysis is a critical component in evaluating the validity of many high-stakes testing programs, particularly those used for licensure or certification. The ratings of criticality and frequency of various activities that are derived from such job analyses can be combined in a number of ways. This paper develops a multiplicative model as a natural and effective way to combine ratings o f frequency and criticality in order to obtain estimates of the relative importance of different activities for practice. An example of the model's use is presented. The multiplicative model incorporates adjustments to ensure that the effective weights of frequency and criticality are appropriate.

Combining Data on Criticality and Frequency in Developing Test Plans for Licensure and Certification Examinations

In order to meet its need for a current, integrated picture of the dynamic profession of dietetics, The American Dietetic Association, with assistance from American College Testing, undertook a study focusing on the roles of entry-level dietetic technicians and registered dietitians and beyond-entry-level registered dietitians. A nationwide mail survey was conducted on random samples of 3,559 dietitians who had been registered for up to 3 years, 6,647 dietitians who had been registered for more than 3 years, and all 1,226 graduates (in the previous 3 years) from the 70 ADA-approved programs preparing dietetic technicians that agreed to participate (out of the 77 ADA-approved programs). Response rates of 77.5%, 78.7%, and 68.5% were achieved for the two samples of registered dietitians and the sample of dietetic technicians, respectively. The results show that the most commonly chosen work setting for the three respondent groups was "inpatient-care, acute-care facility." There were substantial differences among the three groups in the types of activities performed and in the level of responsibility for various kinds of activities. The beyond-entry-level registered dietitians were involved in the broadest range of activities and had the most responsibility for policy setting and other administrative activities, such as preparing budgets. The results of the study provide a detailed description of dietetic practice over a broad range of practitioner-experience levels, practice settings, and client groups.

Role delineation for dietetic practitioners: Empirical results

A strategy for examining the validity of job analysis inventory data is introduced. This strategy involves the testing of hypotheses regarding expected consistencies within the data, and between the data and independent sources of information. Although the purpose of the job analysis is to obtain an empirical description of work patterns, some relationships among the data can be predicted with a high degree of confidence, and these predictions can then be used to test the validity of the job analysis data. An example investigates the validity of data collected as part of a job analysis for nurses. Licensure examinations (e.g., teacher certification) involve high stakes decisions about new graduates seeking to enter the professions. Such examinations can therefore have a strong impact on the institutions that prepare these graduates as well as on the settings in which they subsequently work. Empirical job analyses support claims for the validity of score interpretations for licensure and certification examinations by linking the content domain of the test directly to what is done in practice. The chapter on licensure and certification examinations in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1985) states that, "a rationale should be provided to support a claim that the knowledge or skills being assessed are required for competent performance in an occupation" (p. 64) and that "Job analyses provide the primary basis for defining the content domain" (p. 64).

Dean A. Colton

Papers

Generalizability Analyses of Work Keys Listening and Writing Tests.

Combining Data on Criticality and Frequency in Developing Test Plans for Licensure and Certification Examinations

Role delineation for dietetic practitioners: Empirical results

A Strategy for Examining the Validity of Job Analysis Data