The search for new, authentic science assessments of what students know and can do is well under way. This has unearthed measures of students' hands-on performance in carrying out science investigations, and has been expanded to discover more or less direct measures of students' knowledge structures. One potential finding is concept mapping, the focus of this review. A concept map is a graph consisting of nodes representing concepts and labeled lines denoting the relation between a pair of nodes. A student's concept map is interpreted as representing important aspects of the organization of concepts in his or her memory (cognitive structure). In this article we characterize a concept map used as an assessment tool as: (a) a task that elicits evidence bearing on a student's knowledge structure in a domain, (b) a format for the student's response, and (c) a scoring system by which the student's concept map can be evaluated accurately and consistently. Based on this definition, multiple concept-mapping techniques were found from the myriad of task, response format, and scoring system variations identified in the literature. Moreover, little attention has been paid to the reliability and validity of these variations. The review led us to arrive at the following conclusions: (a) an integrative working cognitive theory is needed to begin to limit this variation in concept-mapping techniques for assessment purposes; (b) before concept maps are used for assessment and before map scores are reported to teachers, students, the public, and policy makers, research needs to provide reliability and validity information on the effect of different mapping techniques; and (c) research on students' facility in using concept maps, on training techniques, and on the effect on teaching is needed if concept map assessments are to be used in classrooms and in large-scale accountability systems. © 1996 John Wiley & Sons, Inc.

Problems and Issues in the Use of Concept Maps in Science Assessment.

Meta-analytictechniqueswereusedtoexaminetheeffectivenessofWebbased instruction (WBI) relative to classroom instruction (CI) and to examine moderators of the comparative effectiveness of the 2 delivery media.TheoverallresultsindicatedWBIwas6%moreeffectivethanCI for teaching declarative knowledge, the 2 delivery media were equally effective for teaching procedural knowledge, and trainees were equally satisfied with WBI and CI. However, WBI and CI were equally effective for teaching declarative knowledge when the same instructional methods were used to deliver both WBI and CI, suggesting media effects are spurious and supporting Clark’s (1983, 1994) theory. Finally, WBI was 19% more effective than CI for teaching declarative knowledge when Web-based trainees were provided with control, in long courses, and when trainees practiced the training material and received feedback during training. Study limitations and directions for future research are discussed.

The comparative effectiveness of web-based and classroom instruction:

This study updates the Baldwin and Ford (1988) review of the empirical articles published on training transfer. The updated review and analysis revolves around the four key limitations noted in the original review: (1) the criterion problem of how and when to measure training transfer, (2) the generalizability of results from training design studies, (3) the choice of which trainee characteristics to examine for their impact on transfer, and (4) the conceptualization and operationalization of work environment factors that can impact transfer. Twenty studies were found in the literature since the 1988 review on training transfer. An analysis of these studies found that progress has been made in addressing many of the limitations noted by Baldwin and Ford. This paper concludes with a discussion of future research directions for training transfer research relevant to each of the four areas of criterion measurement, training design, trainee characteristics, and work environment.

Transfer of Training: An Updated Review and Analysis

Intelligence, expertise, ability and talent, as these terms have traditionally been used in education and psychology, are socially agreed upon labels that minimize the dynamic, evolving, and contextual nature of individual-environment relations. These hypothesized constructs can instead be described as functional relations distributed across whole persons and particular contexts through which individuals appear knowledgeably skillful. The purpose of this article is to support a concept of ability and talent development that is theoretically grounded in 5 distinct, yet interrelated, notions: ecological psychology, situated cognition, distributed cognition, activity theory, and legitimate peripheral participation. Although talent may be reserved by some to describe individuals possessing exceptional ability and ability may be described as an internal trait, in our description neither ability nor talent are possessed. Instead, they are treated as equivalent terms that can be used to describe functional trans...

http://comphacker.org/pdfs/631/Barab_S.pdf

Smart People or Smart Contexts? Cognition, Ability, and Talent Development in an Age of Situated Approaches to Knowing and Learning

A cognitive item response theory model called the attribute hierarchy method (AHM) is introduced and illustrated. This method represents a variation of Tatsuoka's rule-space approach. The AHM is designed explicitly to link cognitive theory and psychometric practice to facilitate the development and analyses of educational and psychological tests. The following are described: cognitive properties of the AHM; psychometric properties of the AHM, as well as a demonstration of how the AHM differs from Tatsuoka's rule-space approach; and application of the AHM to the domain of syllogistic reasoning to illustrate how this approach can be used to evaluate the cognitive competencies required in a higher-level thinking task. Future directions for research are also outlined.

The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach

The intent of the article is to survey procedures that could be used to assess progress in instructional programs designed to enhance cognitive skills. The organizational framework is provided by J. R. Anderson’s (1982) theory of cognitive skill development and by Glaser, Lesgold, and Lajoie’s (1985) categorization of dimensions of cognitive skills. After describing Anderson’s theory, the article discusses the following types of measures of cognitive skills: (a) measures of knowledge acquisition, organization, and structure; (b) measures of depth of problem representation; (c) measures of mental models; (d) measures of metacognitive skills; (e) measures of the automaticity of performance; and (f) measures of efficiency of procedures. Each of the sections describing measurement procedures is followed by a discussion of the strengths and weaknesses of the procedures. The article closes with a general discussion of techniques for measuring cognitive skills.

Cheryl A. Cisero

Papers

Techniques and Procedures for Assessing Cognitive Skills