Although training evaluation is recognized as an important component of the instructional design model, there are no theoretically based models of training evaluation. This article attempts to move toward such a model by developing a classification scheme for evaluating learning outcomes. Learning constructs are derived from a variety of research domains, such as cognitive, social, and instructional psychology and human factors. Drawing from this research, we propose cognitive, skill-based, and affective learning outcomes (relevant to training) and recommend potential evaluation measures. The learning outcomes and associated evaluation measures are organized into a classification scheme. Requirements for providing construct-oriented evidence of validity for the scheme are also discussed

Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.

Background. Non-technical skills are critical for good anaesthetic practice but are not addressed explicitly in normal training. Realization of the need to train and assess these skills is growing, but these activities must be based on properly developed skills frameworks and validated measurement tools. A prototype behavioural marker system was developed using human factors research techniques. The aim of this study was to conduct an experimental evaluation to establish its basic psychometric properties and usability. Method. The Anaesthetists’ Non-Technical Skills (ANTS) system prototype comprises four skill categories (task management, team working, situation awareness, and decision making) divided into 15 elements, each with example behaviours. To investigate its experimental validity, reliably and usability, 50 consultant anaesthetists were trained to use the ANTS system. They were asked to rate the behaviour of a target anaesthetist using the prototype system in eight videos of simulated anaesthetic scenarios. Data were collected from the ratings forms and an evaluation questionnaire. Results. The results showed that the system is complete, and that the skills are observable and can be rated with acceptable levels of agreement and accuracy. The internal consistency of the system appeared sound, and responses regarding usability were very positive. Conclusions. The findings of the evaluation indicated that the ANTS system has a satisfactory level of validity, reliability and usability in an experimental setting, provided users receive adequate training. It is now ready to be tested in real training environments, so that full guidelines can be developed for its integration into the anaesthetic curriculum.

Anaesthetists’ Non‐Technical Skills (ANTS): evaluation of a behavioural marker system†

In this paper, we propose that the debate concerning the locus of attentional selection can be resolved by specifying the conditions under which early selection is possible. In the first part, we present a theoretical discussion that integrates aspects from structural and capacity approaches to attention and suggest that perceptual load is a major factor in determining the locus of selection. In the second part, we present a literature review that examines the conditions influencing the processing of irrelevant information. This review supports the conclusion that a clear physical distinction between relevant and irrelevant information is not sufficient to prevent irrelevant processing; early selection also requires that the perceptual load of the task be sufficiently high to exceed the upper limit of available attentional resources.

https://link.springer.com/content/pdf/10.3758/BF03213897.pdf

Perceptual load as a major determinant of the locus of selection in visual attention

The search for new, authentic science assessments of what students know and can do is well under way. This has unearthed measures of students' hands-on performance in carrying out science investigations, and has been expanded to discover more or less direct measures of students' knowledge structures. One potential finding is concept mapping, the focus of this review. A concept map is a graph consisting of nodes representing concepts and labeled lines denoting the relation between a pair of nodes. A student's concept map is interpreted as representing important aspects of the organization of concepts in his or her memory (cognitive structure). In this article we characterize a concept map used as an assessment tool as: (a) a task that elicits evidence bearing on a student's knowledge structure in a domain, (b) a format for the student's response, and (c) a scoring system by which the student's concept map can be evaluated accurately and consistently. Based on this definition, multiple concept-mapping techniques were found from the myriad of task, response format, and scoring system variations identified in the literature. Moreover, little attention has been paid to the reliability and validity of these variations. The review led us to arrive at the following conclusions: (a) an integrative working cognitive theory is needed to begin to limit this variation in concept-mapping techniques for assessment purposes; (b) before concept maps are used for assessment and before map scores are reported to teachers, students, the public, and policy makers, research needs to provide reliability and validity information on the effect of different mapping techniques; and (c) research on students' facility in using concept maps, on training techniques, and on the effect on teaching is needed if concept map assessments are to be used in classrooms and in large-scale accountability systems. © 1996 John Wiley & Sons, Inc.

Problems and Issues in the Use of Concept Maps in Science Assessment.

Based on an engagement perspective of reading development, we investigated the extent to which an instructional framework of combining motivation support and strategy instruction (Concept-Oriented Reading Instruction—CORI) influenced reading outcomes for third-grade children. In CORI, five motivational practices were integrated with six cognitive strategies for reading comprehension. In the first study, we compared this framework to an instructional framework emphasizing Strategy Instruction (SI), but not including motivation support. In the second study, we compared CORI to SI and to a traditional instruction group (TI), and used additional measures of major constructs. In both studies, class-level analyses showed that students in CORI classrooms were higher than SI and/or TI students on measures of reading comprehension, reading motivation, and reading strategies. A widespread goal of education in the elementary grades is reading comprehension for all students. Reading comprehension becomes especially important in the later elementary grades (Sweet & Snow, 2003) and provides the basis for a substantial amount of learning in secondary school (Kirsch et al., 2002). Without the skills of reading comprehension and the motivation for reading to learn, students’ academic progress is limited (Alvermann & Earle, 2003). In view of the prominence of reading comprehension, a vital issue for educational psychology is investigating the characteristics of effective instruction for reading comprehension (Hiebert & Raphael, 1996). The growing knowledge base about instruction for reading comprehension is rightly directed toward identifying classroom practices with known effects on specific aspects of reading, and a major focus of this research has been on identifying effective reading strategies that increase children’s comprehension (Block & Pressley, 2002; Duke & Pearson, 2002; National Reading Panel, 2000). However, the evidence rests primarily on instructional research in which single cognitive strategies, such as questioning, are taught in controlled experiments. Relatively little investigation has been conducted on how multiple strategies can be combined in long-term comprehension instruction within the classroom, and

http://cori.umd.edu/research-publications/2004-guthrie-wigfield-etal.pdf

Increasing Reading Comprehension and Engagement Through Concept-Oriented Reading Instruction

This study evaluated the validity of some new methods to assess student's cognitive representations of classroom learning. In the context of a sophomore-junior level class in statistics and design, 40 students judged the relatedness of 30 of the important concepts in the domain. Network representations of the rating data were derived using the Pathfinder scaling algorithm. The similarity between each student's and the instructor's networks was then assessed using a set-theoretic measure C

Assessing Structural Knowledge.

Network representations of student knowledge are usually evaluated by comparing them to an expert «referent» structure. This study compared referent structures produced by the instructor, other experts, averaged experts, and an average based on the best students in the class. The referents were compared in their ability to predict exam performance in 2 college level computer programming courses and to differentiate among levels of expertise. Results showed that in terms of these criteria, (a) instructor-based referents were no better than other experts; (b) there was substantial variability among experts; and (c) structures derived from both averaged experts and averaged best students provided valid referents, but the expert-based referent was superior

Structural knowledge assessment: comparison of referent structures

A structural assessment of classroom learning

Earlier T. E. Goldsmith, P. J. Johnson, and W. H. Acton (1991) reported that a Pathfinder-based representation of domain knowledge was more predictive of classroom exam performance than the underlying data from which the Pathfinder representation was derived (relatedness ratings of pairs of concepts from the domain). The question is, what affords Pathfinder this predictive advantage? Analyses reveal that whereas the relatedness ratings are based on a 7-point scale, all of the predictiveness is contained in the most highly related pairs. Moreover, Pathfinder's advantage is related to the manner in which it dichotomizes the relatedness data into directly linked and indirectly linked concept pairs

Locus of the Predictive Advantage in Pathfinder-Based Representations of Classroom Knowledge.

Evaluation of aircrew performance serves the critical functions of assessing the qualifications of individual pilots and, under newer proficiency-based training programs, providing data for modifying training programs. We apply psychometric methods to assessing and improving the quality of evaluation of aircrew performance. Quality evaluations require human judges to recognize and discriminate changes in performance levels (sensitivity) and map these observations onto the appropriate grade-scale values (accuracy). We define statistical measures for both of these properties. A distinction is made between referent-rater reliability (RRR) and traditional interrater reliability, and we argue that RRR more meaningfully measures evaluators' grading performance and has clearer training implications. We also discuss the implementation of training and calibration sessions that are intended to help improve evaluators' ratings of aircrew performance. We offer several practical guidelines for designing and conducting...

Peder J. Johnson

Papers

Assessing Structural Knowledge.

Structural knowledge assessment: comparison of referent structures

A structural assessment of classroom learning

Locus of the Predictive Advantage in Pathfinder-Based Representations of Classroom Knowledge.

Assessing and improving evaluation of aircrew performance