Showing papers in "Journal of Cognitive Engineering and Decision Making in 2008"

Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering Constructs

Abstract: Cognitive engineering needs viable constructs and principles to promote better understanding and prediction of human performance in complex systems. Three human cognition and performance constructs that have been the subjects of much attention in research and practice over the past three decades are situation awareness (SA), mental workload, and trust in automation. Recently, Dekker and Woods (2002) and Dekker and Hollnagel (2004; henceforth DWH) argued that these constructs represent "folk models" without strong empirical foundations and lacking scientific status. We counter this view by presenting a brief description of the large science base of empirical studies on these constructs. We show that the constructs can be operationalized using behavioral, physiological, and subjective measures, supplemented by computational modeling, but that the constructs are also distinct from human performance. DWH also caricatured as "abracadabra" a framework suggested by us to address the problem of the design of automated systems (Parasuraman, Sheridan, & Wickens, 2000). We point to several factual and conceptual errors in their description of our approach. Finally, we rebut DWH's view that SA, mental workload, and trust represent folk concepts that are not falsifiable. We conclude that SA, mental workload, and trust are viable constructs that are valuable in understanding and predicting human-system performance in complex systems.

The Effect of Uncertainty Visualizations on Decision Making in Weather Forecasting

Abstract: People's reasoning with uncertainty information is often flawed. Visual representations can help, but little is known about what is the best way to present such information. Two studies investigated the effect of visualizations on the understanding and use of wind speed forecast uncertainty. Participants varied in expertise from novices in weather forecasting (Experiment 1) to professional forecasters (Experiment 2). The authors investigated three visualizations: (a) a chart showing the amount of uncertainty, (b) a chart showing the worst-case scenario, and (c) a box plot of likely wind speeds. Participants were asked to determine the relative uncertainty in the forecast, predict wind speed, and decide whether to post a high-wind warning advisory. The results for novices and professional forecasters were similar. The uncertainty chart enhanced awareness of the degree of uncertainty associated with the forecast, box plots improved reading accuracy, and presenting a visualization of the worst-case scenario introduced bias in the deterministic wind speed forecast. An interactive display (e.g., a combination of an uncertainty chart with a box plot display) may be optimal to convey uncertainty information.

Staying Up to Speed: Four Design Principles for Maintaining and Recovering Situation Awareness

Abstract: Dynamic operational tasks, such as airspace monitoring and civil emergency operations, require maintaining awareness of changing situations, in part by detecting and interpreting significant changes. Poor change detection ability makes this maintenance difficult enough while monitoring a situation display uninterrupted, but multitasking and interruptions increase the difficulty because the situation can change during the interruptions. Yet little research has addressed the need for better interface tools to help users detect and interpret changes, either to maintain situation awareness or to recover it following interruptions. The objective of this article is to present a principled, theoretical basis for designing interface tools that help users recover and maintain situation awareness in dynamic operational tasks. First, we present a framework for understanding the processes of interruption recovery during dynamic tasks within the broader context of interruption recovery generally. Then we briefly review a series of experiments, primarily from our own laboratory, from which we derive four design principles. Last, we contrast a number of interface designs in terms of the principles. Consideration of the four principles should facilitate the design of more effective tools to help users get back up to speed for these important and high-risk tasks.

Improving decision making in crisis response through critical thinking support

Abstract: In this study, we describe how to use innovative techniques to improve the decision-making process in crisis response organizations The focus was on building situation awareness of a crisis and overcoming pitfalls such as tunnel vision and information bias through using critical thinking We started by observing typical difficulties in crisis management in a field study The essential elements of concern were a deficit in sharing and communicating understanding and a patchy overview of the topics communicated, within as well as between teams Communication frequently did not entail the reasoning behind a decision that was made We therefore developed a critical thinking tool that made the reasoning process more explicit and at the same time more robust by tying it to specific hypotheses We studied a candidate support tool in a controlled setting and found that people made better judgments, particularly in situations where they would be prone to decision biases We subsequently extended the critical thinking tool to a team setting We list a number of requirements that are essential for support systems that intend to limit tunnel vision and alleviate communication and coordination problems in crisis response organizat

Ecological Interface Design for Military Command and Control

Abstract: The authors use the cognitive systems engineering framework to design and evaluate an interface for military command and control. They discuss analytic tools and principles of this framework and provide concrete examples (e.g., work domain analyses for U.S. Army tactical operations at the battalion level). They also discuss principles of ecological interface design, including direct perception, direct manipulation, and the perception-action loop. The translation between work domain analyses and the specific characteristics of the interface are made explicit. The authors describe the potential for this interface to support effective decision making and problem solving, including links with naturalistic decision-making approaches. Evaluations of the interface have been positive and are described briefly. Actual or potential applications of this research include both specific interface design strategies for military command and control and general interface design principles for this category of work domain.

Type 2 Diabetes Self-Management: Controlling a Dynamic System

Abstract: People with type 2 diabetes risk disability and early death when they fail to control their blood glucose levels. Despite advances in medicine, pharmacology, human factors, and education, dangerous glucose levels remain endemic. To investigate cognitive barriers to control, we observed American Diabetes Association (ADA) certified training programs; reviewed ADA and National Institute of Health diabetes Web sites; and interviewed patients with type 2 diabetes using a critical decision method. A consistent picture emerged. The prevailing rules and procedures approaches are not preparing patients for the dynamic control task they face. Patients are often unable to understand and use the rules and procedures provided. They are unprepared to detect problems, make sense of dynamic relationships, and manage complex situations. Our results suggest that glucose self-regulation is better conceptualized as a dynamic control challenge requiring complex processes, including problem detection, sensemaking, decision ma...

Situation Awareness and the Decision-Making Process in a Dynamic Situation: Avoiding Collisions at Sea

Abstract: In this study, the authors aim to determine the impact of situation awareness (SA) in the decision-making process of "young" watch officers of a Merchant Marine training facility. The trainees were shown an ambiguous interaction situation in which they could choose among several actions. The results show that Level 1 SA (perception of the elements in the environment) tends to be of secondary importance in decision making. The major variables of the decision-making process are the interpretation of the rules and anticipation of the other vessel's intentions. Moreover, four different trainee "profiles" emerged. The main difference between them lies in the distance at which they decided to change course, the direction of this maneuver (port or starboard), the way in which they interpreted the other vessel's intentions (is it going to change course?), and whether the trainees referred to the rules. Of the trainees, 55% performed a maneuver that was against regulations, and 34% did so in an unsafe manner. This result provides an incentive to rethink the training course to put more stress on recognizing prototypical situations and choosing which actions to take in situations such as the one presented here.

Human Views for MODAF as a Bridge Between Human Factors Integration and Systems Engineering

Abstract: The paper describes the human views for the UK Ministry of Defence Architectural Framework (MODAF) as essential concepts to enable a common modeling approach between the fields of systems engineering and human factors integration. Human views express high-level human factors design decisions through a systems engineering modeling approach to better communicate human-related design concerns to engineers and to enable early application of human factors methods in the systems engineering process. Integration barriers attributable to differing approaches between human factors integration and systems engineering are outlined. Cognitive systems engineering is considered under the umbrella of human factors integration. The paper discusses how selected cognitive systems engineering approaches and methods provide an invaluable resource in supporting the underlying philosophy and application of the human views as part of MODAF.

Integrating Cognitive Systems Engineering Throughout the Systems Engineering Process

Abstract: In order for cognitive systems engineering (CSE) to deliver the benefits of recent theoretical advances in actual systems being developed, the insights of CSE must be transformed into pragmatic engineering practices. The CSE engineering practices described in this article (using the applied cognitive systems engineering methodology as an exemplar) are typical of just such an engineering adaptation of rev- olutionary science and represent engineering practices that are dedicated to building effective systems. CSE research insights can have a significant impact on their corre- sponding systems engineering (SE) practices by expanding SE's concept of a system from just the technology components to a joint cognitive system (Hollnagel & Woods, 2005) and expanding the associated SE practices appropriately. This article uses the Department of Defense system life cycle and the SE V-model (Forsberg & Mooz, 1992) as SE process exemplars to illustrate how CSE engineering practices can be integrated into the SE process. Specifically, we propose four key integration points where CSE can contribute significantly to the SE process: concept refinement, software development, testing, and postsystem development (i.e., operations support, training, and mainte- nance). Our approach shows that the practice of CSE does not compete with SE but, instead, augments current SE practices to ensure that the technology components are engineered with the users' cognitive needs in mind.

Polychronicity and information acquisition in pilot decision making

Abstract: This study was designed to test an explanation of the individual differences in the preference for heuristic-based information acquisition interfaces that were reported by Wiggins and Bollwerk (2006) in the context of a simulated in-flight decision task. The information acquisition interfaces were developed on the basis of three decision heuristics: elimination by aspects, frequency comparison, and majority of confirming dimensions. Forty-three pilots, with a mean age of 30, interacted with three information acquisition interfaces during familiarization scenarios and were subsequently asked to select their preferred interface to complete a fourth test scenario. The results indicated that the selection of a particular interface in the test scenario was related to factors other than polychronicity. However, polychronicity was associated with perceptions of the information acquisition interfaces and performance during the familiarization scenarios. The outcomes of the study are discussed in terms of their impact on the development of decision support systems.

Some New Perspectives for Introducing Human-Systems Integration into the System Development Process:

Abstract: As further background for this Special Issue, this paper introduces a sampling of the findings of a National Research Council Committee on Human-Systems Design Support for Changing Technology. The ...

Capturing the Context of Use to Inform System Design

Abstract: New system design often focuses on opportunities afforded by technology without careful analysis of the needs of the people and work involved. A strong consensus has been found across diverse design traditions regarding the need for in-depth analysis of the context of use to inform system design. Analysis of context of use encompasses identifying and characterizing the range of situations that can arise, the demands they impose, and the motivations and activities of the people involved. This paper presents three case studies in which context of use analysis was a key part of successful system design. The three case studies represent work-centered design, emerging from cognitive engineering and cognitive systems engineering; interaction design, emerging from industrial and communication design; and participatory design, emerging from the Scandinavian labor movement. The case studies highlight commonalities in experience and core tenets across traditions. They argue for the importance of incorporating context of use analyses as part of the systems engineering process and illustrate a variety of approaches for context of use analysis.

The Impact of Time Pressure and Experience on Information Search and Decision-Making Processes

Abstract: The goal of this research was to examine the diagnosis and decision processes of pilots when dealing with conflicting information. The loose coupling between decision processes and an event outcome makes it hard for researchers to use data such as accident reports as reliable indicators of the quality of decisions, and one problem faced by researchers is determining the extent to which pilot decision processes during events are faulty. We examined pilot strategies in a Web-based automated cockpit scenario as a function of time pressure condition (present vs. absent) and experience with automated systems. Time pressure had an impact on time to diagnosis and information accessed for more experienced pilots only. Pilots who accessed more information were more likely to make accurate diagnoses but were less confident in them. Neither time pressure nor experience had a main effect on diagnosis accuracy or confidence. More experienced pilots tended to conduct a thorough information search when under no time pressure, but time pressure caused pilots of all experience levels to accelerate and filter their information search. Results highlight the importance of training for coherent information use under time pressure given the complexities of automated aircraft.

Effects of Automatic Detection on Dynamic Decision Making

Abstract: The usefulness of the dual-process theory of automaticity to dynamic decision-making tasks is unclear. Dynamic decision making is characterized by multiple, diverse, and interrelated decisions that are often constrained by time limitations and workload. We investigated the relevance of this theory in a compound task consisting of multiple, dynamic components. In the first experiment, we reproduced the original findings that shaped the theory in a dynamic visual search task. In the second experiment, we added a decision-making component. The results replicated the original findings and extended them to decision-making components. Working under consistent-mapping conditions in the visual search component led to more accurate decision making despite variability in the decision-making conditions. Likewise, working under varied-mapping conditions in the visual search component led to poorer decision making, particularly under high workload. The implications of these results to real-world dynamic tasks are discussed.

Recognizing Potential: A Naturalistic Investigation of Employee Promotion Decisions

Abstract: Over the past two decades, substantial progress has been made in describing how frontline staff, such as nurses and military commanders, make rapid tactical decisions. Managerial decisions, by contrast, are often made relatively slowly. One important class of managerial decisions relates to staffing. Managers often make decisions regarding the hiring and promotion of staff that have significant consequences for both the company and staff. Relatively little is known about how people make decisions in these contexts. In this study, the authors examine how managers make employee promotion decisions. The participants in the study were 16 senior managers from a multinational beverage company. The critical decision method was used to elicit knowledge from the participants. The analyses examined the role of decision strategy, situation assessment, and decision evaluation. The findings suggest that senior managers make promotion decisions in a manner consistent with recognition-primed decision-making theory. Implications for training, design, and decision support are discussed.

Cognitive Engineering in Radiation Screening for Homeland Security

Abstract: Cognitive engineering methods in a large-scale homeland security program are described. Screening cargo containers at seaports for illicit radioactive material involves substantial staff, technology, and human operator decision making. The systems engineering model used by the radiation portal monitor (RPM) program entails elements similar to the incremental commitment model, which provides a risk-based focus for applications of cognitive engineering. This model includes methods for defining opportunities and context of use, methods for requirements and design solutions, and methods for evaluation. The application examples illustrate each of these general areas through work domain analysis, signal detection modeling, design of displays for radiation threat likelihood, and evaluation of the design concepts with field test data. The signal detection analysis shows how system detection sensitivity requirements for design can be established for low-probability, high-consequence events such as threats. The thr...

Providing Battlespace Information to Reduce Uncertainty: Will More Information Lead to Better Plans?:

Abstract: One basic assumption behind network-centric warfare (NCW) is that increased supply of information will reduce battlespace uncertainty and increase decision quality. The purpose of this study was to ...

Integrating Cognitive Engineering in the Systems Engineering Process: Opportunities, Challenges, and Emerging Approaches Introduction to the Special Issue

Abstract: TRADITIONAL SYSTEMS ENGINEERING AND SOFTWARE ENGINEERING PRACTICES OFTEN DO not expressly consider human-centered objectives and constraints in defining system requirements, or they are considered too late in the design cycle to have any meaningful impact in shaping the system design. Too often this results in systems that are frustrating to use, prone to human errors that can have serious negative consequences, or never implemented because they are simply unusable (Hoffman & Elm, 2006). Cognitive engineering and related human-centered disciplines offer the potential to contribute to the development of systems that support individuals, teams, and organizations more effectively in achieving their objectives. Cognitive engineering and its related disciplines hold the promise to create systems that have improved performance (increased efficiency, reduced error, and greater resilience in the face of unanticipated conditions) and that provide a more pleasant user experience. Although there have been significant advances in the development of cognitive engineering methods and tools, these methods have yet to be widely adopted into large system procurement and development programs. Both the cognitive engineering and systems engineering communities show a growing interest in trying to understand the barriers that have prevented cognitive engineering methods from being more widely integrated into large system development programs and in developing new approaches (new tools, training, collaborative design frameworks, and outreach programs) to more effectively bridge the gap between cognitive engineering and the larger systems and software engineering enterprise (Hoffman & Elm, 2006; Muralidhar, 2008). For example, the International Council on Systems Engineering (INCOSE) has an active interdisciplinary human-systems integration (HSI) working group whose mission is to more effectively embed HSI into system engineering practices. The April 2008 issue of the INCOSE newsletter, INSIGHT, is dedicated to championing the need for “integrating

Immediacy, Trust, and Remote Command and Control

Abstract: In this article, I discuss an experiment that tests the prediction of social impact theory that immediacy matters and the claim that trust partially mediates its impact in command and control settings Active-duty soldiers completed oral commands more quickly when collocated with an unfamiliar leader than when that leader was sheltered at a remote location A questionnaire on trust in the leader revealed greater levels of trust in the collocated condition The best-fit linear regression function reveals a significant positive association between self-reported levels of trust and response time Additional regressions reveal that trust mediates the influence of immediacy on the speed with which soldiers executed direct orders These findings support arguments against plans proposed by both the US and Swedish armed forces that would extract platoon leaders from the field and replace them with remote command and control of the dismounted infantry