Using aggregate patient data at the bedside via an on-demand consultation service

TLDR: The design and implementation of an on-demand consultation service to derive evidence from patient data to answer clinician questions and support their bedside decision making and the tools and methods developed are made publicly available to facilitate the broad adoption of such services by health systems and academic medical centers.

Abstract: Using evidence derived from previously collected medical records to guide patient care has been a long standing vision of clinicians and informaticians, and one with the potential to transform medical practice. As a result of advances in technical infrastructure, statistical analysis methods, and the availability of patient data at scale, an implementation of this vision is now possible. Motivated by these advances, and the information needs of clinicians in our academic medical center, we offered an on-demand consultation service to derive evidence from patient data to answer clinician questions and support their bedside decision making. We describe the design and implementation of the service as well as a summary of our experience in responding to the first 100 requests. Consultation results informed individual patient care, resulted in changes to institutional practices, and motivated further clinical research. We make the tools and methods developed to implement the service publicly available to facilitate the broad adoption of such services by health systems and academic medical centers.

Figures

Table 2. Clinical motivations and subsequent actions taken by requestors for the 83 completed consultations. Each request was assigned a single motivation category and one or more follow-up action categories.

Figure 2. Sankey plot illustrates the flow (horizontal colored lines) of completed consultations in terms of the clinical motivation (left) to the analysis type (center) to the subsequent action (right). The thickness of each flow is proportional to the number of consultations. For example, consultations motivated by evaluating institutional patient management required mostly descriptive analyses, and resulted in all three categories of subsequent action.

Figure 1. The workflow for fulfilling a consultation request, illustrating the order of each step, the time required, and the personnel responsible.

Table 1. Summary of completed consultations by specialty, showing the number of consultations of each analysis type.

Full text

Using&aggregate&patient&data&at&the&
bedside&via&an&on-demand&consultation&
service&
Alison'Callahan
1§
*,'Saurabh'Gombar
1,2§
,'Eli'M.'Cahan
1,4
,'Kenneth'Jung
1
,'Ethan'Steinberg
1
,'Vladimir'
Polony
1
,'Keith'Morse
1
,'Robert'Tibshirani
3
,'Trevor'Hastie
3
,'Robert'Harrington
1
,'Nigam'H.'Shah
1'
'
1'
Department'of'Medicine,'School'of'Medicine,'Stanford'University,'CA'
2'
Department'of'Pathology,'School'of'Medicine,'Stanford'University,'CA'
3'
Department'of'Statistics,'School'of'Humanities'&'Sciences,'Stanford'University,'CA'
4'
Department'of'Medicine,'School'of'Medicine,'New'York'University,'NY'
'
§
Equal'contributors'
*'Corresponding'author:'Alison'Callahan'-'acallaha@stanford.edu'
Abstract(
Using'evidence'derived'from'previously'collected'medical'records'to'guide'patient'care'has'been'a'
long'standing'vision'of'clinicians'and'informaticians,'and'one'with'the'potential'to'transform'
medical'practice.'As'a'result'of'advances'in'technical'infrastructure,'statistical'analysis'methods,'
and'the'availability'of'patient'data'at'scale,'an'implementation'of'this'vision'is'now'possible.'
Motivated'by'these'advances,'and'the'information'needs'of'clinicians'in'our'academic'medical'
center,'we'offered'an'on-demand'consultation'service'to'derive'evidence'from'patient'data'to'
answer'clinician'questions'and'support'their'bedside'decision'making.'We'describe'the'design'and'
implementation'of'the'service'as'well'as'a'summary'of'our'experience'in'responding'to'the'first'100'
requests.'Consultation'results'informed'individual'patient'care,'resulted'in'changes'to'institutional'
practices,'and'motivated'further'clinical'research.'We'make'the'tools'and'methods'developed'to'
implement'the'service'publicly'available'to'facilitate'the'broad'adoption'of'such'services'by'health'
systems'and'academic'medical'centers.'''
. CC-BY-ND 4.0 International licenseIt is made available under a
is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)
The copyright holder for this preprint this version posted June 24, 2021. ; https://doi.org/10.1101/2021.06.16.21259043doi: medRxiv preprint
NOTE: This preprint reports new research that has not been certified by peer review and should not be used to guide clinical practice.

The(need(for(on-demand(evidence(
Evidence-based'medicine'emphasizes'the'“conscientious,'explicit'and'judicious'use'of'current'best'
evidence”
1
'when'making'treatment'decisions
2,3
.'Randomized'controlled'trials'(RCTs)'are'
considered'the'highest'quality'source'of'evidence'about'treatment'efficacy'and'safety.'Evidence'
derived'from'RCTs,'however,'often'does'not'generalize'to'the'vast'majority'of'patients,'who'tend'to'
have'multiple'comorbidities,'take'many'medications,'and'differ'from'individuals'enrolled'in'RCTs'
on'many'characteristics
4
,'resulting'in'an'inferential'gap'between'the'evidence'that'is'available'and'
that'which'is'needed
5,6
.'Therefore,'it'is'necessary'to'transform'the'evidence'generation'process
7
'and'
to'incorporate'the'use'of'aggregate'patient'data'at'the'point'of'care
8
'in'order'to'create'a'successful'
learning'health'system
9
.'
'
Electronic'medical'records'(EMRs)'are'a'source'of'rich'longitudinal'data'about'millions'of'real'
world'patients.'Since'the'1970s,'clinicians'and'scientists'have'envisioned'using'the'medical'records'
of'previously'treated'patients'to'inform'the'care'of'current'and'future'patients
10,11
.'As'a'recent'
example,'in'2011'the'New'England'Journal'of'Medicine'published'an'article'by'Frankovich'et'al.
12
'
describing'the'use'of'EMR'data'to'support'management'of'an'adolescent'female'with'systemic'lupus'
erythematosus.'At'the'time,'incorporating'data'from'EMRs'into'clinical'decision'making'required'
significant'manual'effort,'rendering'it'infeasible'for'use'in'routine'patient'care.''
'
A'decade'later,'the'adoption'of'EMRs'across'the'United'States'and'internationally,'the'increasing'
ease'of'use'of'advanced'statistical'methods,'and'the'ability'to'compute'with'large'patient'cohorts'
has'enabled'a'core'tenet'of'the'learning'health'system:'deriving'on-demand'evidence'for'diverse'
clinical'scenarios'from'the'EMR
7,13
.'
'
Using'these'advances'as'a'foundation,'we'designed,'developed,'and'offered'a'consultation'service'
that'used'EMR'and'medical'insurance'claims'data'at'Stanford'Medicine'to'provide'on-demand'
evidence'for'questions'arising'during'clinical'care
14
.'Here,'we'report'our'findings'from'responding'
to'the'first'100'requests'to'the'service:'we'summarize'requests'by'medical'specialty,'the'types'of'
analyses'required'to'fulfill'their'requests,'and'clinicians’'responses'to'the'evidence'returned.'
. CC-BY-ND 4.0 International licenseIt is made available under a
is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)
The copyright holder for this preprint this version posted June 24, 2021. ; https://doi.org/10.1101/2021.06.16.21259043doi: medRxiv preprint

The(setup(of(the(consultation(service(
Beginning'in'2017,'with'approval'from'the'Stanford'Institutional'Review'Board,'we'offered'a'
consultation'service'to'provide'on-demand'evidence'to'clinicians'at'Stanford'Medicine,'staffed'by'a'
team'of'four'(described'below).'As'part'of'offering'the'service,'we'collected'data'on'the'motivations'
for'consultation'requests,'and'the'subsequent'actions'taken'in'light'of'the'evidence'returned.'At'the'
conclusion'of'the'study'in'August'2019,'we'analyzed'the'consultation'request'motivations'and'
resulting'actions,'and'assessed'the'concordance'of'consultation'results'across'clinical'data'sources'
as'a'measure'of'reliability'of'consultation'analysis'methods.'
'
In'designing'the'service,'we'leveraged'best'practices
15
,'methods
16
,'and'tools
17,18
'to'derive'evidence'
from'EMRs.'Callahan'et'al
15
'summarizes'recommendations'for'conducting'and'reporting'
observational'studies'done'using'EMRs'derived'from'a'large'body'of'our'team’s'prior'work.'For'
example,'we'have'used'EMR'data'for'vigilance,'such'as'monitoring'adverse'drug'events
19,20
'and'
surveilling'implantable'devices
21
;'for'answering.clinical.questions'such'as'whether'there'is'an'
association'between'androgen'deprivation'therapy'and'dementia
22,23
;'and'for'elucidating.quality.of.
care,'by'profiling'unplanned'ED'visits
24
,'surfacing'patient'reported'outcomes
25
'and'quantifying'
treatment'variability'in'metastatic'breast'cancer
26
.'We'have'also'learned'from'leading'collaborative'
studies
27
,'developing'methods'for'electronic'phenotyping
28–31
,'and'from'participating'in'multiple'
OHDSI'network'studies
32–38
.'
'
Gombar'et'al.
14
'describes'the'consultation'service'setup'to'receive'questions'from'clinicians,'
retrieve'the'appropriate'patient'data'using'a'specialized'search'engine
18
,'perform'the'analyses'
required'for'the'question,'and'return'a'report'summarizing'the'results.'Schuler'et'al.
16
'describes'the'
methods'for'data'extraction,'processing,'and'analysis'used'in'the'consultation'service.'Datta'et'al
17
'
describes'the'platform'for'clinical'data'science'at'Stanford'Medicine'that'supported'the'operation'of'
the'service.'
The(workflow(for(fulfilling(a(consultation(request(
A'consultation'began'with'an'email'from'a'requestor,'detailing'a'clinical'question.'Upon'receiving'
the'request,'the'team’s'informatics'clinician'scheduled'an'intake'discussion'with'the'requesting'
. CC-BY-ND 4.0 International licenseIt is made available under a
is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)
The copyright holder for this preprint this version posted June 24, 2021. ; https://doi.org/10.1101/2021.06.16.21259043doi: medRxiv preprint

clinician'to'specify'the'population,'intervention,'comparator,'outcome'and'timeframe'(PICOT)'for'
their'particular'question
14
.''
'
Based'on'the'PICOT'formulation'of'the'question,'the'EMR'data'specialist'constructed'patient'
cohorts'using'the'Advanced'Cohort'Engine'(ACE)
18
'to'search'one'of'three'data'sources:'EMRs'of'3.1'
million'individuals'from'Stanford'Medicine;'IBM'MarketScan®'insurance'claims'for'124'million'
individuals;'or'Optum'Clinformatics'Data'Mart®'insurance'claims'for'53'million'individuals.'The'
data'scientist'then'conducted'the'necessary'statistical'analyses'and'worked'with'the'informatics'
clinician'to'write'a'report'summarizing'the'analyses'and'their'results.'The'report'was'then'shared'
with'the'requestor'and'explained'during'an'in-person'debrief'session.'Each'report'consisted'of'the'
original'question'as'posed,'the'PICOT're-formulation,'and'sections'summarizing'the'cohort'
demographics,'the'interpretation'of'the'analyses,'and'a'detailed'walkthrough'of'the'analyses.'Three'
example'reports'are'provided'in'the'Supplement.'The'interaction'was'designed'to'be'similar'to'
obtaining'a'second'opinion'from'a'colleague.''
'
Our'workflow'evolved'to'incorporate'real-time'searches'of'the'EMR'as'the'informatics'clinician'
collected'PICOT'details.'For'example,'if'a'given'cohort'criterion'returned'very'few'patients,'then'the'
informatics'clinician'could'relay'this'information'during'the'intake'interview'in'order'to'elicit'
modifications'to'the'cohort'definition'from'the'requestor.'Clarifications'needed'during'debrief'
interviews'were'also'incorporated'into'subsequent'reports'and'debriefs'to'better'contextualize'
analysis'results'for'requestors.'The'majority'of'this'evolution'occurred'during'the'first'3'months'of'
offering'the'service.'
'
Based'on'the'time'required'to'respond'to'the'first'100'consultations'received'(see'Findings.from.the.
first.100.consultations),'we'believe'a'team'compose d'of'one'full-time'clinical'informatics'fellow,'two'
full-time'EMR'data'specialists,'and'a'20%'part-time'data'scientist'would'be'able'to'complete'up'to'
20'such'consultations'in'one'week.'The'personnel'costs'for'our'geographic'area'(San'Francisco'Bay)'
for'this'team'are'estimated'at'$505,000/year.'Yearly'data'access'infrastructure,'cloud'compute,'
licensing,'and'professional'service'expenses'come'to'an'additional'$70,000/year.'With'these'
assumptions,'the'cost'of'running'such'a'service'would'be'approximately'$550'per'consultation.'
'
. CC-BY-ND 4.0 International licenseIt is made available under a
is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)
The copyright holder for this preprint this version posted June 24, 2021. ; https://doi.org/10.1101/2021.06.16.21259043doi: medRxiv preprint

Figure'1'illustrates'the'process'of'fulfilling'a'consultation'request.'The'datasets,'cohort'building'and'
analysis'methods'used'in'completing'consultations,'and'our'assessments'of'their'performance,'are'
further'described'in'the'following'subsections.''
!
Figure!1.'The'workflow'for'fulfilling'a'consultation'request,'illustrating'the'order'of'each'step,'the'
time'required,'and'the'personnel'responsible.'
Datasets&and&Cohort&Building&
The'service'used'demographics,'diagnoses,'procedures,'medications,'laboratory'values,'clinical'
notes,'length'of'stay,'and'mortality'information'for'millions'of'patients'from'three'data'sources:'
EMRs'from'3.1'million'Stanford'Medicine'(Stanford)'patients'(54%'female,'spanning'1995-2019)
39'
consisting'of'diagnosis,'procedure,'medication,'and'laboratory'test'records,'as'well'as'clinical'notes'
processed'using'a'previously'developed'and'evaluated'text-processing'pipeline
40,41
;'IBM'
MarketScan®'(MarketScan)'which'contains'employer'and'Medicare'insurance'claims'for'124'
million'lives'(53%'female,'spanning'2007-2015);'and'Optum'Clinformatics'Data'Mart®'(Optum)'
. CC-BY-ND 4.0 International licenseIt is made available under a
is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)
The copyright holder for this preprint this version posted June 24, 2021. ; https://doi.org/10.1101/2021.06.16.21259043doi: medRxiv preprint

Frequently asked questions

Q1. What are the contributions in "Using aggregate patient data at the bedside via an on-demand consultation service" ?

The authors describe the design and implementation of the service as well as a summary of their experience in responding to the first 100 requests. It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. This preprint reports new research that has not been certified by peer review and should not be used to guide clinical practice. As a recent example, in 2011 the New England Journal of Medicine published an article by Frankovich et al. 12 describing the use of EMR data to support management of an adolescent female with systemic lupus erythematosus. Using these advances as a foundation, the authors designed, developed, and offered a consultation service that used EMR and medical insurance claims data at Stanford Medicine to provide on-demand evidence for questions arising during clinical care14. Here, the authors report their findings from responding to the first 100 requests to the service: they summarize requests by medical specialty, the types of analyses required to fulfill their requests, and clinicians ’ responses to the evidence returned. It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Consultation results informed individual patient care, resulted in changes to institutional practices, and motivated further clinical research. 

Q2. What is the expected false positive rate for consultations?

Because the OMOP and EU-ADR reference sets were constructed to evaluate methods for treatment comparisons, the 17-20% expected false positive rate is applicable to consultations requesting a comparison of the hazard ratio of an outcome between treatments. 

Q3. What is the net benefit of providing on-demand evidence to inform clinical decision making?

On-demand evidence generation to inform clinical decision making is an achievable goal, given the confluence of scalable technology for data analysis, a growing data science workforce, the training of increasingly data savvy clinicians, and the availability of large amounts of patient data from EMRs and claims 8,14. 

Q4. How many consultations were completed in the first 100?

As the service workflow matured, by the end of the study, 19 consultation reports were returned in 48 hours or less by reusing cohort definitions, experience in PICOT formulation of the request, and analysis code optimization. 

Q5. How many non-associated pairs were found in the OMOP community?

Of the 234 non-associated pairs from the OMOP community, there were 137 drug-outcome pairs for which a minimum 100 patients exposed to the drug were present in Stanford data. 

Q6. What were the common types of debriefs?

Debriefs where the requester used the results from the consultation report directly as the basis of a publication, poster, abstract, grant submission, or presentation were categorized as follow-up analyses. 

Q7. How often did the patient receive a positive blood culture?

By analyzing approximately 16,000 procalcitonin test results and 29,000 blood culture results, the authors calculated how often a positive blood culture was obtained within 48 hours of one cut-off value for a procalcitonin result, how frequently antibiotic therapy was discontinued at different cut-offs of procalcitonin values, and how often antibiotics were restarted within 72 hours of discontinuation. 

Q8. Why did 83 consultations not be completed?

17 consultations could not be completed due to missing data elements, available data sources having too few patients meeting the specified cohort criteria, inability to define a cohort, or requiring an unsupported study design. 

Q9. What is the core tenet of the learning health system?

A decade later, the adoption of EMRs across the United States and internationally, the increasing ease of use of advanced statistical methods, and the ability to compute with large patient cohorts has enabled a core tenet of the learning health system: deriving on-demand evidence for diverse clinical scenarios from the EMR7,13. 

Q10. What is the definition of evidence-based medicine?

Evidence-based medicine emphasizes the “conscientious, explicit and judicious use of current best evidence”1 when making treatment decisions2,3. 

Q11. What is the rate of agreement between the RCTs and RCTs?

the authors found 68-74% concordance of consultation results across multiple datasets, a rate of agreement comparable both to the rate at which results from randomized controlled trials (RCTs) agree with each other (67-87%)69 and to the rate at which results derived from observational claims data agree with RCTs (60-80%)70. 

Q12. How many consultations did the team complete in one week?

Based on the time required to respond to the first 100 consultations received (see Findings from the first 100 consultations), the authors believe a team composed of one full-time clinical informatics fellow, two full-time EMR data specialists, and a 20% part-time data scientist would be able to complete up to 20 such consultations in one week. 

Q13. What were the frequent diagnoses of a melanoma patient?

The authors provided the requestor with summaries of the most frequent diagnoses preceding and following mononeuritis multiplex diagnosis in 118 similarly aged patients, which included bacterial and viral infections as well as psychosomatic disorders. 

Q14. What were the motivations for the consultation requests?

Consultation requests had diverse motivationsConsultation requests were driven by a variety of motivations, including evaluating patient management strategies for a given disease or patient presentation, identifying comparatively effective treatments for patients with typically understudied characteristics, and quantifying associations between diseases. 

Q15. How many outcomes were concordant across datasets?

When comparing results obtained using different data sources for the same consultation request, 68% to 74% of results were concordant across datasets. 

Q16. What was the first time that EMRs were used in clinical care?

At the time, incorporating data from EMRs into clinical decision making required significant manual effort, rendering it infeasible for use in routine patient care. 

Q17. What were the reasons for the longer turnaround times?

Longer turnaround times occurred when additional data elements were needed, there were delays in scheduling conversations with the requestor, or when matching required substantial time for large cohorts.