A Comparison of the Ability of the Physiologic Components of Medical Emergency Team Criteria and the U.K. National Early Warning Score to Discriminate Patients at Risk of a Range of Adverse Clinical Outcomes

TL;DR: Comparing the ability of medical emergency team criteria and the National Early Warning Score to discriminate cardiac arrest, unanticipated ICU admission and death within 24 hours of a vital signs measurement and to quantify the associated workloads found some medical emergencyteam systems have a lower specificity and would generate greater workloads.

Abstract: To compare the ability of medical emergency team criteria and the National Early Warning Score to discriminate cardiac arrest, unanticipated ICU admission and death within 24 hours of a vital signs measurement, and to quantify the associated workload. Retrospective cohort study. A large U.K. National Health Service District General Hospital. Adults hospitalized from May 25, 2011, to December 31, 2013. None. We applied the National Early Warning Score and 44 sets of medical emergency team criteria to a database of 2,245,778 vital signs sets (103,998 admissions). The National Early Warning Score's performance was assessed using the area under the receiver-operating characteristic curve and compared with sensitivity/specificity for different medical emergency team criteria. Area under the receiver-operating characteristic curve (95% CI) for the National Early Warning Score for the combined outcome (i.e., death, cardiac arrest, or unanticipated ICU admission) was 0.88 (0.88-0.88). A National Early Warning Score value of 7 had sensitivity/specificity values of 44.5% and 97.4%, respectively. For the 44 sets of medical emergency team criteria studied, sensitivity ranged from 19.6% to 71.2% and specificity from 71.5% to 98.5%. For all outcomes, the position of the National Early Warning Score receiver-operating characteristic curve was above and to the left of all medical emergency team criteria points, indicating better discrimination. Similarly, the positions of all medical emergency team criteria points were above and to the left of the National Early Warning Score efficiency curve, indicating higher workloads (trigger rates). When medical emergency team systems are compared to a National Early Warning Score value of greater than or equal to 7, some medical emergency team systems have a higher sensitivity than National Early Warning Score values of greater than or equal to 7. However, all of these medical emergency team systems have a lower specificity and would generate greater workloads.

Summary

INTRODUCTION

• Staff failures in recognising and responding to patient deterioration have led hospitals to use early warning scoring systems (EWSS) (1) or Medical Emergency Team (MET) calling criteria (2) to improve vital signs monitoring and facilitate the calling of expert help to a patient’s bedside.
• The EWS is used to direct subsequent care, e.g. changes to vital signs monitoring frequency; involvement of more experienced ward staff; or calling a rapid response team (RRT).
• Many EWSS are in use, with marked variation in measured physiological variables, assigned weightings and outcome discrimination (3-8).
• To produce NEWS, the RCPL used clinical opinion to make minor adjustments to the VitalPAC Early Warning Score (5).

Setting and study population

• Portsmouth Hospitals NHS Trust (PHT) is a single site NHS District General Hospital with ~1000 inpatient beds and ~5500 staff.
• It provides all acute services except burns, spinal injury, neurosurgical and cardiothoracic surgery to a local population of ~540,000.
• Data from patients discharged alive from the hospital before midnight on the day of admission were excluded.
• Where oxygen was used, VitalPAC estimated its fractional inspired concentration (FiO2) using the mask type +/- flow rate (or in the case of a Venturi mask, the concentration), which were recorded during each vital signs collection.

Evaluation of NEWS and MET criteria

• The vital signs database was used to evaluate the performance of NEWS and 44 different MET criteria (identified from two previous publications (16, 17) - see Supplementary Digital Content 3).
• As the subjective component of MET criteria - staff concern (14) – is also used to escalate care when using NEWS, the authors made an a priori decision to evaluate only the following physiological components of the MET criteria: 7 high/low pulse rate, high/low breathing rate, high/low systolic blood pressure, high/low temperature, SpO2 and reduced consciousness.
• For the same reason, the authors did not evaluate criteria such as threatened airway or repeated/prolonged seizures.
• The remainder require only an SpO2 value or simply whether supplemental O2 was being administered.
• The authors removed hospital episodes where FiO2 could not be estimated.

Outcomes

• Deaths, cardiac arrests and unanticipated ICU admission data were identified from the hospital’s patient administration system (PAS), cardiac arrest database and ICU admission database, respectively.
• The authors excluded episodes of care where (i) the episode had a first outcome before the first observation set and (ii) the episode did not have an observation set within the last 24 h before the outcome.

Statistical analysis

• All data manipulation was performed using Microsoft® Visual FoxPro 9.0.
• The authors used IBM SPSS Statistics v22 and R v3.02. (22) to calculate the AUROC; R was also used to generate the figures.
• An ROC curve plots sensitivity against 1-specificity, and each point on it represents a sensitivity/specificity pairing corresponding to a particular decision threshold for NEWS.
• For each set of MET criteria, and for each outcome, the authors calculated the sensitivity and specificity.
• To compare the efficiency of NEWS and the MET criteria, the authors superimposed the sensitivity/trigger rate points for the 44 sets of MET criteria on the NEWS efficiency curves.

Additional analyses

• The authors have previously shown that the use of multiple observation sets from a single episode does not bias the ranking of EWSs when assessing the performance of these systems (24).
• This has not previously been done for sets of MET criteria.
• Therefore, the authors repeated the above analyses using 10,000 samples of observation sets, each sample being constructed by selecting one observation set at random from every care episode (i.e., so each observation set in an episode had an equal chance of being selected in each sample).

RESULTS

• For some of these 5809 episodes there were other observation sets where FiO2 could be estimated, so the episode itself remained in the analysis (with fewer observation sets).
• Figures 2a-d and Supplementary Digital Content 5 show the sensitivity and specificity (plotted as 1 - specificity) points for NEWS (i.e., the NEWS ROC curve) and the MET criteria for the outcomes studied.
• The relative positions of the MET criteria and NEWS were essentially unchanged when using the 10,000 random sample sets (see Supplementary Digital Content 9 and 10).

DISCUSSION

• The selection of an RRT triggering system can be based upon several criteria, including the balance between its sensitivity and the workload it generates.
• Depending upon their specific criteria, all sets of MET calling criteria have fixed relationships between sensitivity and workload, and the resulting clinical response can only ever be of an ‘all or none’ nature.
• That EWSs, such as NEWS, are better discriminators of outcomes than MET criteria is perhaps not surprising.
• It considers all completed admissions over 31 months.
• The current study differs markedly from the NEWS development work, using a larger database, a different study period, medical and surgical patients (compared to only acute medicine) and vital signs from the whole patient admission rather than merely from the patient’s stay in the Medical Assessment Unit.

CONCLUSIONS

• When MET systems are compared to a NEWS value of >7 (i.e., the recommended triggers for RRT intervention for each system), some MET systems have a higher sensitivity than NEWS.
• All of these MET systems have a lower specificity and would generate greater workloads.
• NEWS also provides the opportunity to titrate the trigger value against available resources, and permits a graduated, multi-tiered clinical response, whereas the clinical response resulting from a MET call can only ever be of an ‘all or none’ nature.

Figures

Table 1: Patient demographics, number of vital signs observations, vital sign values and observations followed by an adverse outcome for the study group (after exclusions).

Table 2: The sensitivity and specificity of 44 MET calling criteria, and NEWS values of 3-7, for cardiac arrest, unanticipated ICU admission, death, and any of these outcomes, each within 24 h of a vital signs dataset.

Full text

This is a repository copy of A Comparison of the Ability of the Physiologic Components of
Medical Emergency Team Criteria and the U.K. National Early Warning Score to
Discriminate Patients at Risk of a Range of Adverse Clinical Outcomes.
White Rose Research Online URL for this paper:
https://eprints.whiterose.ac.uk/104932/
Version: Accepted Version
Article:
Smith, Gary B., Prytherch, David R, Jarvis, Stuart William orcid.org/0000-0001-8447-0306
et al. (4 more authors) (2016) A Comparison of the Ability of the Physiologic Components
of Medical Emergency Team Criteria and the U.K. National Early Warning Score to
Discriminate Patients at Risk of a Range of Adverse Clinical Outcomes. Critical Care
Medicine. 2171–2181. ISSN 0090-3493
https://doi.org/10.1097/CCM.0000000000002000
eprints@whiterose.ac.uk
https://eprints.whiterose.ac.uk/
Reuse
Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless
indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by
national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of
the full text version. This is indicated by the licence information on the White Rose Research Online record
for the item.
Takedown
If you consider content in White Rose Research Online to be in breach of UK law, please notify us by
emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request.

1
A comparison of the ability of the physiological components of Medical Emergency Team criteria and
the UK National Early Warning Score (NEWS) to discriminate patients at risk of a range of adverse
clinical outcomes
Professor Gary B Smith, FRCA, FRCP
1
Professor David R Prytherch, PhD, MIPEM, CSci
2,3
Dr. Stuart Jarvis PhD.
3,4
Mrs. Caroline Kovacs, BSc
3
Dr. Paul Meredith, PhD
2
Dr. Paul E Schmidt, MRCP, B.Med.Sc, MBA
2
Dr. Jim Briggs, BA, DPhil
3
1
Faculty of Health and Social Sciences, University of Bournemouth, Bournemouth, UK
2
Portsmouth Hospitals NHS Trust, Portsmouth, UK
3
Centre for Healthcare Modelling & Informatics, University of Portsmouth, Portsmouth, UK
4
Department of Health Sciences, University of York, York, UK
Work performed at Portsmouth Hospitals NHS Trust & University of Portsmouth
No reprints will be available
Correspondence from:
Professor G B Smith, FRCA, FRCP,
Centre of Postgraduate Medical Research & Education (CoPMRE),
Faculty of Health and Social Sciences,
Bournemouth University, Royal London House,
Christchurch Road, Bournemouth,
Dorset BH1 3LT, United Kingdom
Tel: +44 (0) 1202 962782; Fax: +44 (0) 1202 962218
Email: gbsresearch@virginmedia.com
Funding: Nil
Key words: Hospital rapid response team; Monitoring, Physiologic; Quality improvement; Vital Signs,
Medical Emergency Team.
Word count = 2998. Abstract word count = 248
Number of references = 37; Figures = 3; Tables = 2; Supplementary Digital Content = 10

2
ABSTRACT
Objective: To compare the ability of Medical Emergency Team (MET) criteria and the National Early
Warning Score (NEWS) to discriminate cardiac arrest, unanticipated ICU admission and death within 24 h of
a vital signs measurement, and to quantify the associated workload.
Design: Retrospective cohort study.
Setting: A large UK NHS District General Hospital.
Patients: Adults hospitalized from 25/05/2011 to 31/12/2013.
Interventions: None
Measurements and Main Results: We applied NEWS and 44 sets of MET criteria to a database of 2,245,778
vital signs sets (103,998 admissions). NEWS’ performance was assessed using the area under the receiver-
operating characteristic (ROC) curve (AUROC) and compared with sensitivity/specificity for the different MET
criteria. AUROC (95% CI) for NEWS for the combined outcome (i.e., death, cardiac arrest or unanticipated
ICU admission) was 0.88 (0.88 - 0.88). A NEWS value of 7 had sensitivity/specificity values of 44.5%/97.4%.
For the 44 sets of MET criteria studied, sensitivity ranged from 19.6% to 71.2%, and specificity from 71.5% to
98.5%. For all outcomes, the position of the NEWS ROC curve was above and to the left of all MET criteria
points, indicating better discrimination. Similarly, the positions of all MET criteria points were above and to the
left of the NEWS efficiency curve, indicating higher workloads (trigger rates).
Conclusions: When MET systems are compared to a NEWS value of >7, some MET systems have a higher
sensitivity than NEWS values of >7. However, all of these MET systems have a lower specificity and would
generate greater workloads.

3
CONFLICT OF INTERESTS STATEMENT
VitalPAC is a collaborative development of The Learning Clinic Ltd (TLC) and Portsmouth Hospitals NHS
Trust (PHT). At the time of the research, PHT had a royalty agreement with TLC to pay for the use of PHT
intellectual property within the VitalPAC product. Professor Prytherch, Dr Schmidt, and Dr Meredith are
employed by PHT. Professor Smith was an employee of PHT until 31/03/2011. Professors Smith and
Prytherch, and Dr Schmidt, are unpaid research advisors to TLC and have received reimbursement of travel
expenses from TLC for attending symposia in the UK. Dr Briggs' research has previously received funding
from TLC through a Knowledge Transfer Partnership. Professor Smith acted as expert advisor to the
National Institute for Health and Clinical Excellence during the development of the NICE clinical guideline 50:
'Acutely ill patients in hospital: recognition of and response to acute illness in adults in hospital'. He was also
a member of the National Patient Safety Agency committee that wrote the two reports: 'Recognising and
responding appropriately to early signs of deterioration in hospitalised patients' and ' Safer care for the
acutely ill patient: learning from serious incidents'. He was a member of the Royal College of Physicians
of
London’s National Early Warning Score Development and Implementation Group (NEWSDIG). Professor
Prytherch assisted the Royal College of Physicians of London in the analysis of data validating NEWS. Dr
Jarvis and Mrs Kovacs have no conflicts of interest.

4
INTRODUCTION
Staff failures in recognising and responding to patient deterioration have led hospitals to use early
warning scoring systems (EWSS) (1) or Medical Emergency Team (MET) calling criteria (2) to improve vital
signs monitoring and facilitate the calling of expert help to a patient’s bedside.
EWSS allocate points in a weighted manner, based on the derangement of a patient’s measured
vital signs from arbitrarily agreed “normal” ranges - the sum of these is termed the early warning score
(EWS). The EWS is used to direct subsequent care, e.g. changes to vital signs monitoring frequency;
involvement of more experienced ward staff; or calling a rapid response team (RRT). Many EWSS are in
use, with marked variation in measured physiological variables, assigned weightings and outcome
discrimination (3-8). In 2012, the Royal College of Physicians of London (RCPL) recommended the use of a
standardised EWSS in the National Health Service (NHS) - the National EWS (NEWS) (Supplementary
Digital Content 1) (9)
. To produce NEWS, the RCPL used clinical opinion to make minor adjustments to the
VitalPAC Early Warning Score (ViEWS) (5). The RCPL recommends that NEWS values of >7 should prompt
assessment by an RRT (9). NEWS demonstrates better ability than other published EWSS to discriminate
patients at risk of a range of clinical outcomes (6) and has been validated outside its development site (10-
13).
Some hospitals, especially in the USA and Australia, use MET calling criteria in preference to EWSS.
Most MET criteria are based on extreme values of specific objective physiological parameters (e.g., pulse
rate <40 or >120 beats.min
−1
) (2) (Supplementary Digital Content 2). When one or more objective MET
criteria occurs, or staff are ‘worried’ about a patient, a MET or other RRT is called to provide expert
assistance (14). As with EWSS, a wide range of MET criteria is in use, with varied abilities to discriminate
patients at risk of adverse events (3, 15-17).
Ideally, hospitals should use an RRT triggering system that provides the highest discrimination of
patient outcome and the lowest trigger rate, thereby minimising both the risk of missing serious outcomes
and of excessive staff workload. A recent study comparing the performances of NEWS and one set of MET
criteria suggests that NEWS is a better (and earlier) detector of patient deterioration (13). Therefore, we used
a large database of vital sign measurements to (a) compare the abilities of NEWS and 44 different MET

Frequently asked questions

Q1. What have the authors contributed in "This is a repository copy of a comparison of the ability of the physiologic components of medical emergency team criteria and the u.k. national early warning score to discriminate patients at risk of a range of adverse clinical outcomes. white rose research online url for this paper: https://eprints.whiterose.ac.uk/104932/" ?

This is indicated by the licence information on the White Rose Research Online record for the item.