Who Is an Efficient and Effective Physician? Evidence From Emergence Medicine

TL;DR: It is found that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads, and peer influence on a focal physician's effectiveness and efficiency is found, suggesting an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

Abstract: Improving the performance of the healthcare sector requires an understanding of the effectiveness and efficiency of care delivered by providers. Although this topic is of great interest to policymakers, researchers, and hospital managers, rigorous methods of measuring effectiveness and efficiency of care delivery have proven elusive. Through Data Envelopment Analysis (DEA), we make use of evidence from care delivered by emergency physicians, and develop scores that gauge physicians' performance in terms of effectiveness and efficiency. In order to validate our DEA scores, we independently use various Machine Learning (ML) algorithms, including Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Classification and Regression Trees (CART), Random Forest (RF), a Generalized Linear Model (GLM), and Least Absolute Shrinkage and Selection Operator (LASSO). After validating our DEA scores via comparison with predictions made by these algorithms, we make use of them to identify the distinguishing behaviors of highly effective and efficient physicians. We find that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads. We also observe that highly efficient physicians order less tests on average and become even more efficient during high-volume shifts. Importantly, our results indicate a statistically significant positive relationship between a physician's effectiveness and efficiency scores suggesting that, contrary to conventional wisdom, effectiveness and efficiency in care delivery should be viewed as compliments not substitutes. In addition, we find that effectiveness is lower among physicians who have higher job tenure or average test order count. Efficiency, however, is lower among physicians with less experience (measured in number of years after graduation from medical school) or high average test order count. Furthermore, our results indicate an increase in a physician's average efficiency and a decrease in his/her average effectiveness when faced with high workloads. Finally, we find evidence of peer influence on a focal physician's effectiveness and efficiency, which suggests an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

Summary

Introduction

• Healthcare spending is projected to rise to 19.9% of the GDP by 2025 (Keehan et al. 2017), spurring interest in finding new ways to increase both the efficiency and effectiveness of care delivery.
• Specifically, the authors consider how frequently a physician admits patients who are subsequently discharged after a brief period, which suggests that the physician potentially overcalled the patients’ illness severity (i.e., patients are admitted when they could have potentially been discharged).
• In particular, the authors consider peer physician characteristics such as relative effectiveness, efficiency, experience, gender, and type of medical degree (MD vs. DO), and examine how they affect another physician’s efficiency and effectiveness.
• Contrary to the conventional wisdom that efficiency may come at the price of effectiveness (and vice-versa), their findings demonstrate a statistically significant positive association between efficiency and effectiveness (P = 0.0209).

Data

• The authors data consist of detailed care delivery information in a leading U.S. hospital with 32 ED physicians.
• The patients are algorithmically assigned to physicians upon arrival through an automated rotational patient assignment process (Traub et al. 2016).
• The authors included all patient visits from July 12, 2012, to July 31, 2016 who were identified in the electronic health record system as having been seen by an ED physician.
• Encounter-level data included laboratory tests, chief complaint, Emergency Severity Index (ESI), day of the ED visit, time of day, etc., totaling over 70 variables.
• To avoid distortion of the results by outliers, 6 physicians with relatively low patient volumes (fewer than 250 visits over the 4-year period) were excluded from the analysis.

DEA Models

• DEA, first introduced by Charnes, Cooper, and Rhodes in (Charnes et al. 1978), is a methodology useful in evaluating the relative performance of a set of decision making units (DMUs) in a multiple input, multiple output setting.
• The original DEA model was based on a constant returns to scale (CRS) methodology.
• The efficiency DEA model considers the relative use of hospital resources by a physician to his/her throughput.
• The authors have chosen herein to define the models’ input and output variables in terms of parameters that (a) best reflect a physician’s performance, and (b) for which there is at least face validity and some level of agreement among physicians.
• These variables are then used along with the optimization program (1) to create efficiency and effectiveness scores.

Effectiveness DEA Model

• Outputs: Rate of discharged patients who do not return within 72-hours :.
• Since this would be considered an undesirable output, the authors use the 6-hour non-upgrade patient admission rate as an output variable.
• The choice of threshold numbers (72, 17, and 6) is made based on observations made in the literature (see, e.g., Keith et al.
• Average number of the physician’s lab orders per patient visit; Plain Radiograph Order Count : 8 Electronic copy available at: https://ssrn.com/abstract=3227873.

Efficiency DEA Model

• Average number of patients seen by the physician per shift; Inputs: Lab Order Count :.
• Average number of the physician’s lab orders per patient visit; Ultrasound Order Count :.
• It is important to note that the authors compared the physicians’ average number of hours worked per shift using the paired-observation t-test method and after removing two physicians from their analysis, they found no significant differences between the remaining physicians’ average hours worked per shift.
• The authors did not include the plain radiograph order count as an input variable in the efficiency model because of its negative correlation with the model’s output variable.
• The authors choice of variables for both models was validated using the stepwise variable selection method mentioned earlier.

Peer-Effect DEA Model

• In order to also examine the effects of peer presence on a physician’s effectiveness and efficiency, the authors use a variation of the proposed DEA models in which each DMU, denoted by 9 Electronic copy available at: https://ssrn.com/abstract=3227873 jk, comprise a physician j who has worked alongside his/her peer physician k for at least 5 hours.
• The authors choose the 5-hour criterion to be able to capture any meaningful peer physician influence.
• This assumption was tested by Simar and Wilson’s (Simar et al. 2002, Simar et al. 2011) returns-to-scale tests for input-oriented DEA models.
• For this reason, an input-oriented approach was used to test whether a DMU under evaluation can reduce its inputs while keeping the outputs at their current levels.

Statistical Methodology

• With the goal of learning about the practice of physicians who have a better performance than the rest, the authors regress the generated DEA scores of physician i (θi), on a set of explanatory variables related to patient, physician, and peer physician j characteristics which they denote by Ui, Wi, and Zij, respectively.
• The regression model takes the following general form θi = a+.
• This is because this standard regression technique assumes a normal and homoscedastic distribution of the noise.
• Following the normalization approach of Greene (1993) which assumes a censoring point at zero, the authors transform the DEA scores to: yi = (1/θi)− 1, where θi is the DEA measure of physician i’s performance.

Discussion and Results

• The authors begin their analysis by examining the relationship between physicians’ scores on measures related to effectiveness and efficiency.
• This is a counterintuitive result, which questions the traditional belief that experience enables physicians to use hospital resources more frugally.
• The regression results of peer physician efficiency analysis (displayed in Table 4) show that the presence of a more effective peer is associated with (on average) an increase in physician efficiency.
• Nevertheless, the natural experiment setting discussed earlier removes potential biases, and hence, many of the associations the authors find may be causal.

Conclusions

• Using evidence from emergency medicine, the authors develop and analyze metrics for physicians’ effectiveness and efficiency.
• Unlike what the conventional wisdom suggests, their findings show that a physician’s effectiveness is positively (and not negatively) associated with his/her efficiency.
• The authors also find that highly efficient physicians have on average lower MRI orders per patient visit.
• The authors believe that their analysis serves as an early step to explore issues of physician effectiveness and efficiency.
• Importantly, the authors do not believe that the scores they develop are the 19 Electronic copy available at: https://ssrn.com/abstract=3227873 only ways to measure effectiveness or efficiency.

Figures

Figure 4: Spectral Clustering - Silhouette coefficient = 0.6650, AUC = 0.8233

Table 2: Regression Results - Efficiency Model - Individual Physician

Figure 3: Random Forest Clustering - Silhouette coefficient = 0.6790, AUC = 0.8487

Figure 2: K-means Clustering - Silhouette coefficient = 0.6890, AUC = 0.8533

Table 3: Regression Results - Effectiveness Model - Peer Physician

Table 1: Regression Results - Effectiveness Model - Individual Physician

Figure 1: Physicians’ Performance Scores

Table 4: Regression Results - Efficiency Model - Peer Physician

Full text

www.hks.harvard.edu
Who is an Efficient and Effective
Physician? Evidence from Emergence
Medicine
Faculty Research Working Paper Series
Soroush Saghafian
Harvard Kennedy School
Raha Imanirad
Harvard Business School
Stephen J. Traub
Mayo Clinic Arizona
September 2018
RWP18-029
Visit the HKS Faculty Research Working Paper Series at:
https://www.hks.harvard.edu/research-insights/publications?f%5B0%5D=publication_types%3A121
The views expressed in the HKS Faculty Research Working Paper Series are those of the author(s) and do not
necessarily reflect those of the John F. Kennedy School of Government or of Harvard University. Faculty Research
Working Papers have not undergone formal review and approval. Such papers are included in this series to elicit
feedback and to encourage debate on important public policy challenges. Copyright belongs to the author(s). Papers
may be downloaded for personal use only.

Electronic copy available at: https://ssrn.com/abstract=3227873
Who is an Efficient and Effective Physician?
Evidence from Emergence Medicine
Soroush Saghafian,
1
Raha Imanirad,
2
Stephen J. Traub
3
1
Harvard Kennedy School, Harvard University, Cambridge, MA
2
Technology and Operations Management, Harvard Business School, Cambridge, MA
3
Department of Emergency Medicine, Mayo Clinic Arizona, Phoenix, AZ
Abstract
Improving the performance of the healthcare sector requires an understanding of the
efficiency and effectiveness of care delivered by providers. Although this topic is of great
interest to policymakers, researchers, and hospital managers, fair and scientific methods
of measuring efficiency and effectiveness of care delivery have proven elusive. Through
Data Envelopment Analysis (DEA), we make use of evidence from care delivered by
emergency physicians, and shed light on scientific metrics that can gauge performance
in terms of efficiency and effectiveness. We use these metrics along with Machine
Learning techniques and Tobit analyses to identify the distinguishing behaviors of
physicians who perform highly on these metrics. Our findings indicate a statistically
significant positive relationship between a physician’s effectiveness and efficiency scores
suggesting that, contrary to conventional wisdom, high levels of effectiveness are not
necessarily associated with low efficiency levels. In addition, we find that a physician’s
effectiveness is positively associated with his/her average contact-to-disposition time
and negatively associated with his/her years of experience. We also find a statistically
significant negative relationship between a physician’s efficiency and his/her average
MRI orders per patient visit. Furthermore, we find evidence of a peer effect of one
1

Electronic copy available at: https://ssrn.com/abstract=3227873
physician upon another, which suggests an opportunity to improve system performance
by taking physician characteristics into account when determining the set of physicians
that should be scheduled during same shifts.
Introduction
Healthcare spending is projected to rise to 19.9% of the GDP by 2025 (Keehan et al. 2017),
spurring interest in finding new ways to increase both the efficiency and effectiveness of
care delivery. As most decisions regarding utilization of healthcare services are ultimately
made by frontline clinicians (Tsugawa et al. 2017), understanding and evaluating provider
performance in a fair and scientific manner could help identify sources of waste in healthcare
spending and help to determine optimal incentives for healthcare reimbursement. Although
care delivery performance measurement initiatives have proliferated in recent years, there
are few rigorous scientific methods to evaluate the efficiency and effectiveness of physicians.
We employ Data Envelopment Analysis (DEA)—a linear programming (LP) optimization
technique that provides a multi-dimensional evaluation tool—to develop and evaluate metrics
related to both efficiency and effectiveness. Efficiency in the Emergency Department (ED)
can be measured in multiple ways, but throughput—the average number of patients seen
by a provider per unit of time—per resources used by the provider possesses significant face
validity as an output-to-input ratio for exploratory analysis. For a given level of resources
used by the provider, a higher throughput means that more patients can be moved through
the ED per unit of time. Given that ED crowding has reached epidemic proportions in the
last several years (Salway et al. 2017), improving throughput without adding resources has
become even more important.
Similarly, effectiveness of care delivery in the ED can be measured in different ways.
However, compared to efficiency, it is a more inherently difficult outcome to measure, and
hence, we suggest using a composite of three metrics. The first pertains only to discharged
patients. Specifically, for a fixed level of resources used by a provider, we consider the
2

Electronic copy available at: https://ssrn.com/abstract=3227873
percentage of discharged patients who do not return to the ED within 72 hours. Returns to
the ED within 72 hours of discharge may result from a sub-optimal (i.e., ineffective) first visit,
in which not all medical issues were sufficiently identified or addressed. The 72-hour rate of
return has been even proposed as a measure of quality in the Emergency Medicine literature,
although using it for measuring quality is controversial (Abualenain et al. 2013, Pham et al.
2011, Klasco et al. 2015). The second and third metrics pertain only to admitted patients
(i.e., those admitted to the hospital after their ED visit). Specifically, we consider how
frequently a physician admits patients who are subsequently discharged after a brief period,
which suggests that the physician potentially overcalled the patients’ illness severity (i.e.,
patients are admitted when they could have potentially been discharged). We also consider
how frequently a physician admits patients whose care is escalated from a low-acuity bed
(ward/floor) to a high-acuity bed (Intermediate Unit or Intensive Care Unit), which suggests
that the physician potentially undercalled the patients’ illness severity. Although none of
these three metrics is ideal in isolation in measuring effectiveness of care delivery, their use
in the aggregate (per resources used by the provider) possesses face validity, and covers the
performance of a physician among both discharged and admitted patients.
We apply our proposed DEA approach (which considers the above measures as outputs
and the level of resources used by the physician as input) to a large data set of care delivered
by ED physicians that includes more than 115,000 patient visits. To learn about what
the high-performing physicians do differently from other physicians, and thereby generate
insights into best practices, we first make use of some Machine Learning (ML) algorithms
(including k-means, spectral, and random forest) to (a) segment physicians based on their
effectiveness and efficiency scores, and (b) identify physicians who have a high performance.
We then conduct a second-stage analysis in which we use a Tobit framework to identify
factors (e.g., test order count, physician contact-to-disposition time, etc.) associated with
higher levels of performance. We also use our framework to study peer effect: the effect
of the characteristics of another physician who is scheduled to work side-by-side with the
3

Electronic copy available at: https://ssrn.com/abstract=3227873
first physician. In particular, we consider peer physician characteristics such as relative
effectiveness, efficiency, experience, gender, and type of medical degree (MD vs. DO), and
examine how they affect another physician’s efficiency and effectiveness.
Our results offer important insights into physician performance. Contrary to the con-
ventional wisdom that efficiency may come at the price of effectiveness (and vice-versa), our
findings demonstrate a statistically significant positive association between efficiency and
effectiveness (P = 0.0209). We also find that a physician’s efficiency is negatively associated
with his/her average number of MRI orders per patient visit (P = 0.0496). In addition, we
observe a statistically significant negative relationship between a physician’s effectiveness and
his/her experience level measured in years since graduation from medical school (P = 0.0301),
which can be due to the recent changes in medical training programs that further emphasize
the efficient use of hospital resources. We also identify a statistically significant positive
relationship between a physician’s effectiveness and his/her average contact-to-disposition
time (P = 0.0290), suggesting that effectiveness can be enhanced via training programs that
enable physicians to avoid speed-up activities that reduce the contact-to-disposition time
but negatively affect the effectiveness of the care delivered (e.g., cutting necessary tests or
reducing value added direct or indirect care time).
Finally, our findings with regards to physician peer effects suggest a statistically signif-
icant impact of peer effect on each individual physician’s performance, a fact that can be
utilized by hospital administrators to improve performance via scheduling the correct set of
physicians during same shifts. In particular, our findings suggest that working alongside a
more effective peer is positively associated with improving both a physician’s efficiency (P
= 0.0044) and effectiveness (P = 0.0147).
4