Who Is an Efficient and Effective Physician? Evidence From Emergence Medicine

TLDR: It is found that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads, and peer influence on a focal physician's effectiveness and efficiency is found, suggesting an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

Abstract: Improving the performance of the healthcare sector requires an understanding of the effectiveness and efficiency of care delivered by providers. Although this topic is of great interest to policymakers, researchers, and hospital managers, rigorous methods of measuring effectiveness and efficiency of care delivery have proven elusive. Through Data Envelopment Analysis (DEA), we make use of evidence from care delivered by emergency physicians, and develop scores that gauge physicians' performance in terms of effectiveness and efficiency. In order to validate our DEA scores, we independently use various Machine Learning (ML) algorithms, including Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Classification and Regression Trees (CART), Random Forest (RF), a Generalized Linear Model (GLM), and Least Absolute Shrinkage and Selection Operator (LASSO). After validating our DEA scores via comparison with predictions made by these algorithms, we make use of them to identify the distinguishing behaviors of highly effective and efficient physicians. We find that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads. We also observe that highly efficient physicians order less tests on average and become even more efficient during high-volume shifts. Importantly, our results indicate a statistically significant positive relationship between a physician's effectiveness and efficiency scores suggesting that, contrary to conventional wisdom, effectiveness and efficiency in care delivery should be viewed as compliments not substitutes. In addition, we find that effectiveness is lower among physicians who have higher job tenure or average test order count. Efficiency, however, is lower among physicians with less experience (measured in number of years after graduation from medical school) or high average test order count. Furthermore, our results indicate an increase in a physician's average efficiency and a decrease in his/her average effectiveness when faced with high workloads. Finally, we find evidence of peer influence on a focal physician's effectiveness and efficiency, which suggests an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

Figures

Figure 4: Spectral Clustering - Silhouette coefficient = 0.6650, AUC = 0.8233

Table 2: Regression Results - Efficiency Model - Individual Physician

Figure 3: Random Forest Clustering - Silhouette coefficient = 0.6790, AUC = 0.8487

Figure 2: K-means Clustering - Silhouette coefficient = 0.6890, AUC = 0.8533

Table 3: Regression Results - Effectiveness Model - Peer Physician

Table 1: Regression Results - Effectiveness Model - Individual Physician

Full text

www.hks.harvard.edu
Who is an Efficient and Effective
Physician? Evidence from Emergence
Medicine
Faculty Research Working Paper Series
Soroush Saghafian
Harvard Kennedy School
Raha Imanirad
Harvard Business School
Stephen J. Traub
Mayo Clinic Arizona
September 2018
RWP18-029
Visit the HKS Faculty Research Working Paper Series at:
https://www.hks.harvard.edu/research-insights/publications?f%5B0%5D=publication_types%3A121
The views expressed in the HKS Faculty Research Working Paper Series are those of the author(s) and do not
necessarily reflect those of the John F. Kennedy School of Government or of Harvard University. Faculty Research
Working Papers have not undergone formal review and approval. Such papers are included in this series to elicit
feedback and to encourage debate on important public policy challenges. Copyright belongs to the author(s). Papers
may be downloaded for personal use only.

Electronic copy available at: https://ssrn.com/abstract=3227873
Who is an Efficient and Effective Physician?
Evidence from Emergence Medicine
Soroush Saghafian,
1
Raha Imanirad,
2
Stephen J. Traub
3
1
Harvard Kennedy School, Harvard University, Cambridge, MA
2
Technology and Operations Management, Harvard Business School, Cambridge, MA
3
Department of Emergency Medicine, Mayo Clinic Arizona, Phoenix, AZ
Abstract
Improving the performance of the healthcare sector requires an understanding of the
efficiency and effectiveness of care delivered by providers. Although this topic is of great
interest to policymakers, researchers, and hospital managers, fair and scientific methods
of measuring efficiency and effectiveness of care delivery have proven elusive. Through
Data Envelopment Analysis (DEA), we make use of evidence from care delivered by
emergency physicians, and shed light on scientific metrics that can gauge performance
in terms of efficiency and effectiveness. We use these metrics along with Machine
Learning techniques and Tobit analyses to identify the distinguishing behaviors of
physicians who perform highly on these metrics. Our findings indicate a statistically
significant positive relationship between a physician’s effectiveness and efficiency scores
suggesting that, contrary to conventional wisdom, high levels of effectiveness are not
necessarily associated with low efficiency levels. In addition, we find that a physician’s
effectiveness is positively associated with his/her average contact-to-disposition time
and negatively associated with his/her years of experience. We also find a statistically
significant negative relationship between a physician’s efficiency and his/her average
MRI orders per patient visit. Furthermore, we find evidence of a peer effect of one
1

Electronic copy available at: https://ssrn.com/abstract=3227873
physician upon another, which suggests an opportunity to improve system performance
by taking physician characteristics into account when determining the set of physicians
that should be scheduled during same shifts.
Introduction
Healthcare spending is projected to rise to 19.9% of the GDP by 2025 (Keehan et al. 2017),
spurring interest in finding new ways to increase both the efficiency and effectiveness of
care delivery. As most decisions regarding utilization of healthcare services are ultimately
made by frontline clinicians (Tsugawa et al. 2017), understanding and evaluating provider
performance in a fair and scientific manner could help identify sources of waste in healthcare
spending and help to determine optimal incentives for healthcare reimbursement. Although
care delivery performance measurement initiatives have proliferated in recent years, there
are few rigorous scientific methods to evaluate the efficiency and effectiveness of physicians.
We employ Data Envelopment Analysis (DEA)—a linear programming (LP) optimization
technique that provides a multi-dimensional evaluation tool—to develop and evaluate metrics
related to both efficiency and effectiveness. Efficiency in the Emergency Department (ED)
can be measured in multiple ways, but throughput—the average number of patients seen
by a provider per unit of time—per resources used by the provider possesses significant face
validity as an output-to-input ratio for exploratory analysis. For a given level of resources
used by the provider, a higher throughput means that more patients can be moved through
the ED per unit of time. Given that ED crowding has reached epidemic proportions in the
last several years (Salway et al. 2017), improving throughput without adding resources has
become even more important.
Similarly, effectiveness of care delivery in the ED can be measured in different ways.
However, compared to efficiency, it is a more inherently difficult outcome to measure, and
hence, we suggest using a composite of three metrics. The first pertains only to discharged
patients. Specifically, for a fixed level of resources used by a provider, we consider the
2

Electronic copy available at: https://ssrn.com/abstract=3227873
percentage of discharged patients who do not return to the ED within 72 hours. Returns to
the ED within 72 hours of discharge may result from a sub-optimal (i.e., ineffective) first visit,
in which not all medical issues were sufficiently identified or addressed. The 72-hour rate of
return has been even proposed as a measure of quality in the Emergency Medicine literature,
although using it for measuring quality is controversial (Abualenain et al. 2013, Pham et al.
2011, Klasco et al. 2015). The second and third metrics pertain only to admitted patients
(i.e., those admitted to the hospital after their ED visit). Specifically, we consider how
frequently a physician admits patients who are subsequently discharged after a brief period,
which suggests that the physician potentially overcalled the patients’ illness severity (i.e.,
patients are admitted when they could have potentially been discharged). We also consider
how frequently a physician admits patients whose care is escalated from a low-acuity bed
(ward/floor) to a high-acuity bed (Intermediate Unit or Intensive Care Unit), which suggests
that the physician potentially undercalled the patients’ illness severity. Although none of
these three metrics is ideal in isolation in measuring effectiveness of care delivery, their use
in the aggregate (per resources used by the provider) possesses face validity, and covers the
performance of a physician among both discharged and admitted patients.
We apply our proposed DEA approach (which considers the above measures as outputs
and the level of resources used by the physician as input) to a large data set of care delivered
by ED physicians that includes more than 115,000 patient visits. To learn about what
the high-performing physicians do differently from other physicians, and thereby generate
insights into best practices, we first make use of some Machine Learning (ML) algorithms
(including k-means, spectral, and random forest) to (a) segment physicians based on their
effectiveness and efficiency scores, and (b) identify physicians who have a high performance.
We then conduct a second-stage analysis in which we use a Tobit framework to identify
factors (e.g., test order count, physician contact-to-disposition time, etc.) associated with
higher levels of performance. We also use our framework to study peer effect: the effect
of the characteristics of another physician who is scheduled to work side-by-side with the
3

Electronic copy available at: https://ssrn.com/abstract=3227873
first physician. In particular, we consider peer physician characteristics such as relative
effectiveness, efficiency, experience, gender, and type of medical degree (MD vs. DO), and
examine how they affect another physician’s efficiency and effectiveness.
Our results offer important insights into physician performance. Contrary to the con-
ventional wisdom that efficiency may come at the price of effectiveness (and vice-versa), our
findings demonstrate a statistically significant positive association between efficiency and
effectiveness (P = 0.0209). We also find that a physician’s efficiency is negatively associated
with his/her average number of MRI orders per patient visit (P = 0.0496). In addition, we
observe a statistically significant negative relationship between a physician’s effectiveness and
his/her experience level measured in years since graduation from medical school (P = 0.0301),
which can be due to the recent changes in medical training programs that further emphasize
the efficient use of hospital resources. We also identify a statistically significant positive
relationship between a physician’s effectiveness and his/her average contact-to-disposition
time (P = 0.0290), suggesting that effectiveness can be enhanced via training programs that
enable physicians to avoid speed-up activities that reduce the contact-to-disposition time
but negatively affect the effectiveness of the care delivered (e.g., cutting necessary tests or
reducing value added direct or indirect care time).
Finally, our findings with regards to physician peer effects suggest a statistically signif-
icant impact of peer effect on each individual physician’s performance, a fact that can be
utilized by hospital administrators to improve performance via scheduling the correct set of
physicians during same shifts. In particular, our findings suggest that working alongside a
more effective peer is positively associated with improving both a physician’s efficiency (P
= 0.0044) and effectiveness (P = 0.0147).
4