Cognitive engineering

Clinical decision support systems (CDSS) are increasingly used by healthcare professionals for evidence-based diagnosis and treatment support. However, research has suggested that users often over-rely on system suggestions -- even if the suggestions are wrong. Providing explanations could potentially mitigate misplaced trust in the system and over-reliance. In this paper, we explore how explanations are related to user trust and reliance, as well as what information users would find helpful to better understand the reliability of a system's decision-making. We investigated these questions through an exploratory user study in which healthcare professionals were observed using a CDSS prototype to diagnose hypothetic cases using fictional patients suffering from a balance-related disorder. Our results show that the amount of system confidence had only a slight effect on trust and reliance. More importantly, giving a fuller explanation of the facts used in making a diagnosis had a positive effect on trust but also led to over-reliance issues, whereas less detailed explanations made participants question the system's reliability and led to self-reliance problems. To help them in their assessment of the reliability of the system's decisions, study participants wanted better explanations to help them interpret the system's confidence, to verify that the disorder fit the suggestion, to better understand the reasoning chain of the decision model, and to make differential diagnoses. Our work is a first step toward improved CDSS design that better supports clinicians in making correct diagnoses.

/pdf/the-role-of-explanations-on-trust-and-reliance-in-clinical-3ijtfv3rtx.pdf

The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems

Introduction Expertise, by definition, refers to the manifestation of skills and understanding resulting from the accumulation of a large body of knowledge. This implies that in order to understand how experts perform and why they are more capable than non-experts, we must understand the representation of their knowledge, that is, how their knowledge is organized or structured, and how their representations might differ from those of novices. For example, if a child who is fascinated with dinosaurs and has learned a lot about them correctly infers attributes about some dinosaurs that was new to them by reasoning analogically to some known dinosaurs (e.g., the shape of teeth for carnivores versus vegetarians), we would not conclude that the “expert” child has a more powerful analogical reasoning strategy. Instead, we would conclude that such a global or domain-general reasoning strategy is available to all children, but that novice children might reason analogically to some other familiar domain, such as animals (rather than dinosaurs), as our data have shown (Chi, Hutchinson, & Robin, 1989). Thus, the analogies of domain-novice are less powerful not necessarily because they lack adequate analogical reasoning strategies, although they may, but because they lack the appropriate domain knowledge from which analogies can be drawn. Thus, in this framework, a critical locus of proficiency lies in the representation of their domain knowledge.

/pdf/laboratory-methods-for-assessing-experts-and-novices-20a4tm9054.pdf

Laboratory Methods for Assessing Experts' and Novices' Knowledge

Background: Order communication systems (OCS) are computer applications used to enter diagnostic and therapeutic patient care orders and to view test results. Many potential benefits of OCS have been identified including improvements in clinician ordering patterns, optimisation of clinical time, and aiding communication processes between clinicians and different departments. Many OCS now include computerised decision support systems (CDSS), which are information systems designed to improve clinical decision-making. CDSS match individual patient characteristics to a computerised knowledge base, and software algorithms generate patient-specific recommendations. 

Objectives: To investigate which CDSS in OCS are in use within the UK and the impact of CDSS in OCS for diagnostic, screening or monitoring test ordering compared to OCS without CDSS. To determine what features of CDSS are associated with clinician or patient acceptance of CDSS in OCS and what is known about the cost-effectiveness of CDSS in diagnostic, screening or monitoring test OCS compared to OCS without CDSS. 

Data sources: A generic search to identify potentially relevant studies for inclusion was conducted using MEDLINE, EMBASE, Cochrane Controlled Trials Register (CCTR), CINAHL (Cumulative Index to Nursing and Allied Health Literature), DARE (Database of Abstracts of Reviews of Effects), Health Technology Assessment (HTA) database, IEEE (Institute of Electrical and Electronic Engineers) Xplore digital library, NHS Economic Evaluation Database (NHS EED) and EconLit, searched between 1974 and 2009 with a total of 22,109 titles and abstracts screened for inclusion. 

Review methods: CDSS for diagnostic, screening and monitoring test ordering OCS in use in the UK were identified through contact with the 24 manufacturers/suppliers currently contracted by the National Project for Information Technology (NpflT) to provide either national or specialist decision support. A generic search to identify potentially relevant studies for inclusion in the review was conducted on a range of medical, social science and economic databases. The review was undertaken using standard systematic review methods, with studies being screened for inclusion, data extracted and quality assessed by two reviewers. 

Results were broadly grouped according to the type of CDSS intervention and study design where possible. These were then combined using a narrative synthesis with relevant quantitative results tabulated. Results: Results of the studies included in review were highly mixed and equivocal, often both within and between studies, but broadly showed a beneficial impact of the use of CDSS in conjunction with OCS over and above OCS alone. Overall, if the findings of both primary and secondary outcomes are taken into account, then CDSS significantly improved practitioner performance in 15 out of 24 studies ((sic) 62.5%). Only two studies covered the cost-effectiveness of CDSS: a Dutch study reported a mean cost decrease of 3% for blood tests orders ((sic)639) in each of the intervention clinics compared with a 2% ((sic) 208) increase in control clinics in test costs; and a Spanish study reported a significant increase in the cost of laboratory tests from (sic)41.8 per patient per annum to (sic)47.2 after implementation of the system. 

Limitations: The response rate from the survey of manufacturers and suppliers was extremely low at only 17% and much of the feedback was classified as being commercial-in-confidence (CC). No studies were identified which assessed the features of CDSS that are associated with clinician or patient acceptance of CDSS in OCS in the test ordering process and only limited data was available on the cost-effectiveness of CDSS plus OCS compared with OCS alone and the findings highly specific. Although CDSS appears to have a potentially small positive impact on diagnostic, screening or monitoring test ordering, the majority of studies come from a limited number of institutions in the USA. 

Conclusions: If the findings of both primary and secondary outcomes are taken into account then CDSS showed a statistically significant benefit on either process or practitioner performance outcomes in nearly two-thirds of the studies. Furthermore, in four studies that assessed adverse effects of either test cancellation or delay, no significant detrimental effects in terms of additional utilisation of health-care resources or adverse events were observed. We believe the key current need is for a well designed and comprehensive survey, and on the basis of the results of this potentially for evaluation studies in the form of cluster randomised controlled trials or randomised controlled trials which incorporate process, and patient outcomes, as well as full economic evaluations alongside the trials to assess the impact of CDSS in conjunction with OCS versus OCS alone for diagnostic, screening or monitoring test ordering in the NHS. The economic evaluation should incorporate the full costs of potentially developing, testing, and installing the system, including staff training costs. 

Study registration: Study registration 61.

https://njl-admin.nihr.ac.uk/document/download/2001962

Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems.

Contemporary Neonatal Intensive Care Units collect vast amounts of patient data in various formats, making efficient processing of information by medical professionals difficult. Moreover, different stakeholders in the neonatal scenario, which include parents as well as staff occupying different roles, have different information requirements. This paper describes recent and ongoing work on building systems that automatically generate textual summaries of neonatal data. Our evaluation results show that the technology is viable and comparable in its effectiveness for decision support to existing presentation modalities. We discuss the lessons learned so far, as well as the major challenges involved in extending current technology to deal with a broader range of data types, and to improve the textual output in the form of more coherent summaries.

/pdf/from-data-to-text-in-the-neonatal-intensive-care-unit-using-2h6sxldjud.pdf

From data to text in the Neonatal Intensive Care Unit: Using NLG technology for decision support and information management

This paper presents the outcomes from a cognitive engineering project addressing the design problems of computerized monitoring in neonatal intensive care. Cognitive engineering is viewed, in this project, as a symbiosis between cognitive science and design practice. A range of methodologies has been used: interviews with neonatal staff, ward observations and experimental techniques. The results of these investigations are reported, focusing specifically on the differences between junior and senior physicians in their interpretation of monitored physiological data. It was found that the senior doctors made better use of the different knowledge sources available than the junior doctors. The senior doctors were able to identify more relevant physiological patterns and generated more and better inferences than did their junior colleagues. Expertise differences are discussed in the context of previous psychological research in medical expertise. Finally, the paper discusses the potential utility of these outcomes to inform the design of computerized decision support in neonatal intensive care.

/pdf/expertise-and-the-interpretation-of-computerized-3s7ldive41.pdf

Expertise and the interpretation of computerized physiological data: implications for the design of computerized monitoring in neonatal intensive care

This paper reports results from a cognitive engineering study that looked at the role of computerised monitoring in neonatal intensive care. A range of methodologies was used: interviews with neonatal staff, ward observations, and experimental techniques. The purpose was to investigate the sources of information used by clinicians when making decisions in the neonatal ICU. It was found that, although it was welcomed by staff, computerised monitoring played a secondary role in the clinicians' decision making (especially for junior and nursing staff) and that staff used the computer less often than indicated by self-reports. Factors that seemed to affect staff use of the computer were the lack (or shortage) of training on the system, the specific clinical conditions involved, and the availability of alternative sources of information. These findings have relevant repercussions for the design of computerised decision support in intensive care and suggest ways in which computerised monitoring can be enhanced, namely: by systematic staff training, by making available online certain types of clinical information, by adapting the user interface, and by developing intelligent algorithms.

/pdf/computerisation-and-decision-making-in-neonatal-intensive-43dxlup9p9.pdf

Andy Lyon

Papers

Expertise and the interpretation of computerized physiological data: implications for the design of computerized monitoring in neonatal intensive care

Computerisation and decision making in neonatal intensive care: a cognitive engineering investigation.

Decision support in the neonatal intensive care unit: Expertise differences in the interpretation of monitored physiological data.

Expertise and the interpretation of computerized physiological data: Implications problems by experts and novices