scispace - formally typeset

Journal ArticleDOI

Who Is an Efficient and Effective Physician? Evidence From Emergence Medicine

TL;DR: It is found that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads, and peer influence on a focal physician's effectiveness and efficiency is found, suggesting an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

AbstractImproving the performance of the healthcare sector requires an understanding of the effectiveness and efficiency of care delivered by providers. Although this topic is of great interest to policymakers, researchers, and hospital managers, rigorous methods of measuring effectiveness and efficiency of care delivery have proven elusive. Through Data Envelopment Analysis (DEA), we make use of evidence from care delivered by emergency physicians, and develop scores that gauge physicians' performance in terms of effectiveness and efficiency. In order to validate our DEA scores, we independently use various Machine Learning (ML) algorithms, including Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Classification and Regression Trees (CART), Random Forest (RF), a Generalized Linear Model (GLM), and Least Absolute Shrinkage and Selection Operator (LASSO). After validating our DEA scores via comparison with predictions made by these algorithms, we make use of them to identify the distinguishing behaviors of highly effective and efficient physicians. We find that highly effective physicians order less tests compared to their peers and maintain their effectiveness when working under high workloads. We also observe that highly efficient physicians order less tests on average and become even more efficient during high-volume shifts. Importantly, our results indicate a statistically significant positive relationship between a physician's effectiveness and efficiency scores suggesting that, contrary to conventional wisdom, effectiveness and efficiency in care delivery should be viewed as compliments not substitutes. In addition, we find that effectiveness is lower among physicians who have higher job tenure or average test order count. Efficiency, however, is lower among physicians with less experience (measured in number of years after graduation from medical school) or high average test order count. Furthermore, our results indicate an increase in a physician's average efficiency and a decrease in his/her average effectiveness when faced with high workloads. Finally, we find evidence of peer influence on a focal physician's effectiveness and efficiency, which suggests an opportunity to improve system performance by taking physician characteristics into account when determining the set of physicians that should be scheduled during the same shifts.

Summary (3 min read)

Introduction

  • Healthcare spending is projected to rise to 19.9% of the GDP by 2025 (Keehan et al. 2017), spurring interest in finding new ways to increase both the efficiency and effectiveness of care delivery.
  • Specifically, the authors consider how frequently a physician admits patients who are subsequently discharged after a brief period, which suggests that the physician potentially overcalled the patients’ illness severity (i.e., patients are admitted when they could have potentially been discharged).
  • In particular, the authors consider peer physician characteristics such as relative effectiveness, efficiency, experience, gender, and type of medical degree (MD vs. DO), and examine how they affect another physician’s efficiency and effectiveness.
  • Contrary to the conventional wisdom that efficiency may come at the price of effectiveness (and vice-versa), their findings demonstrate a statistically significant positive association between efficiency and effectiveness (P = 0.0209).

Data

  • The authors data consist of detailed care delivery information in a leading U.S. hospital with 32 ED physicians.
  • The patients are algorithmically assigned to physicians upon arrival through an automated rotational patient assignment process (Traub et al. 2016).
  • The authors included all patient visits from July 12, 2012, to July 31, 2016 who were identified in the electronic health record system as having been seen by an ED physician.
  • Encounter-level data included laboratory tests, chief complaint, Emergency Severity Index (ESI), day of the ED visit, time of day, etc., totaling over 70 variables.
  • To avoid distortion of the results by outliers, 6 physicians with relatively low patient volumes (fewer than 250 visits over the 4-year period) were excluded from the analysis.

DEA Models

  • DEA, first introduced by Charnes, Cooper, and Rhodes in (Charnes et al. 1978), is a methodology useful in evaluating the relative performance of a set of decision making units (DMUs) in a multiple input, multiple output setting.
  • The original DEA model was based on a constant returns to scale (CRS) methodology.
  • The efficiency DEA model considers the relative use of hospital resources by a physician to his/her throughput.
  • The authors have chosen herein to define the models’ input and output variables in terms of parameters that (a) best reflect a physician’s performance, and (b) for which there is at least face validity and some level of agreement among physicians.
  • These variables are then used along with the optimization program (1) to create efficiency and effectiveness scores.

Effectiveness DEA Model

  • Outputs: Rate of discharged patients who do not return within 72-hours :.
  • Since this would be considered an undesirable output, the authors use the 6-hour non-upgrade patient admission rate as an output variable.
  • The choice of threshold numbers (72, 17, and 6) is made based on observations made in the literature (see, e.g., Keith et al.
  • Average number of the physician’s lab orders per patient visit; Plain Radiograph Order Count : 8 Electronic copy available at: https://ssrn.com/abstract=3227873.

Efficiency DEA Model

  • Average number of patients seen by the physician per shift; Inputs: Lab Order Count :.
  • Average number of the physician’s lab orders per patient visit; Ultrasound Order Count :.
  • It is important to note that the authors compared the physicians’ average number of hours worked per shift using the paired-observation t-test method and after removing two physicians from their analysis, they found no significant differences between the remaining physicians’ average hours worked per shift.
  • The authors did not include the plain radiograph order count as an input variable in the efficiency model because of its negative correlation with the model’s output variable.
  • The authors choice of variables for both models was validated using the stepwise variable selection method mentioned earlier.

Peer-Effect DEA Model

  • In order to also examine the effects of peer presence on a physician’s effectiveness and efficiency, the authors use a variation of the proposed DEA models in which each DMU, denoted by 9 Electronic copy available at: https://ssrn.com/abstract=3227873 jk, comprise a physician j who has worked alongside his/her peer physician k for at least 5 hours.
  • The authors choose the 5-hour criterion to be able to capture any meaningful peer physician influence.
  • This assumption was tested by Simar and Wilson’s (Simar et al. 2002, Simar et al. 2011) returns-to-scale tests for input-oriented DEA models.
  • For this reason, an input-oriented approach was used to test whether a DMU under evaluation can reduce its inputs while keeping the outputs at their current levels.

Statistical Methodology

  • With the goal of learning about the practice of physicians who have a better performance than the rest, the authors regress the generated DEA scores of physician i (θi), on a set of explanatory variables related to patient, physician, and peer physician j characteristics which they denote by Ui, Wi, and Zij, respectively.
  • The regression model takes the following general form θi = a+.
  • This is because this standard regression technique assumes a normal and homoscedastic distribution of the noise.
  • Following the normalization approach of Greene (1993) which assumes a censoring point at zero, the authors transform the DEA scores to: yi = (1/θi)− 1, where θi is the DEA measure of physician i’s performance.

Discussion and Results

  • The authors begin their analysis by examining the relationship between physicians’ scores on measures related to effectiveness and efficiency.
  • This is a counterintuitive result, which questions the traditional belief that experience enables physicians to use hospital resources more frugally.
  • The regression results of peer physician efficiency analysis (displayed in Table 4) show that the presence of a more effective peer is associated with (on average) an increase in physician efficiency.
  • Nevertheless, the natural experiment setting discussed earlier removes potential biases, and hence, many of the associations the authors find may be causal.

Conclusions

  • Using evidence from emergency medicine, the authors develop and analyze metrics for physicians’ effectiveness and efficiency.
  • Unlike what the conventional wisdom suggests, their findings show that a physician’s effectiveness is positively (and not negatively) associated with his/her efficiency.
  • The authors also find that highly efficient physicians have on average lower MRI orders per patient visit.
  • The authors believe that their analysis serves as an early step to explore issues of physician effectiveness and efficiency.
  • Importantly, the authors do not believe that the scores they develop are the 19 Electronic copy available at: https://ssrn.com/abstract=3227873 only ways to measure effectiveness or efficiency.

Did you find this useful? Give us your feedback

...read more

Content maybe subject to copyright    Report

www.hks.harvard.edu
Who is an Efficient and Effective
Physician? Evidence from Emergence
Medicine
Faculty Research Working Paper Series
Soroush Saghafian
Harvard Kennedy School
Raha Imanirad
Harvard Business School
Stephen J. Traub
Mayo Clinic Arizona
September 2018
RWP18-029
Visit the HKS Faculty Research Working Paper Series at:
https://www.hks.harvard.edu/research-insights/publications?f%5B0%5D=publication_types%3A121
The views expressed in the HKS Faculty Research Working Paper Series are those of the author(s) and do not
necessarily reflect those of the John F. Kennedy School of Government or of Harvard University. Faculty Research
Working Papers have not undergone formal review and approval. Such papers are included in this series to elicit
feedback and to encourage debate on important public policy challenges. Copyright belongs to the author(s). Papers
may be downloaded for personal use only.

Electronic copy available at: https://ssrn.com/abstract=3227873
Who is an Efficient and Effective Physician?
Evidence from Emergence Medicine
Soroush Saghafian,
1
Raha Imanirad,
2
Stephen J. Traub
3
1
Harvard Kennedy School, Harvard University, Cambridge, MA
2
Technology and Operations Management, Harvard Business School, Cambridge, MA
3
Department of Emergency Medicine, Mayo Clinic Arizona, Phoenix, AZ
Abstract
Improving the performance of the healthcare sector requires an understanding of the
efficiency and effectiveness of care delivered by providers. Although this topic is of great
interest to policymakers, researchers, and hospital managers, fair and scientific methods
of measuring efficiency and effectiveness of care delivery have proven elusive. Through
Data Envelopment Analysis (DEA), we make use of evidence from care delivered by
emergency physicians, and shed light on scientific metrics that can gauge performance
in terms of efficiency and effectiveness. We use these metrics along with Machine
Learning techniques and Tobit analyses to identify the distinguishing behaviors of
physicians who perform highly on these metrics. Our findings indicate a statistically
significant positive relationship between a physician’s effectiveness and efficiency scores
suggesting that, contrary to conventional wisdom, high levels of effectiveness are not
necessarily associated with low efficiency levels. In addition, we find that a physician’s
effectiveness is positively associated with his/her average contact-to-disposition time
and negatively associated with his/her years of experience. We also find a statistically
significant negative relationship between a physician’s efficiency and his/her average
MRI orders per patient visit. Furthermore, we find evidence of a peer effect of one
1

Electronic copy available at: https://ssrn.com/abstract=3227873
physician upon another, which suggests an opportunity to improve system performance
by taking physician characteristics into account when determining the set of physicians
that should be scheduled during same shifts.
Introduction
Healthcare spending is projected to rise to 19.9% of the GDP by 2025 (Keehan et al. 2017),
spurring interest in finding new ways to increase both the efficiency and effectiveness of
care delivery. As most decisions regarding utilization of healthcare services are ultimately
made by frontline clinicians (Tsugawa et al. 2017), understanding and evaluating provider
performance in a fair and scientific manner could help identify sources of waste in healthcare
spending and help to determine optimal incentives for healthcare reimbursement. Although
care delivery performance measurement initiatives have proliferated in recent years, there
are few rigorous scientific methods to evaluate the efficiency and effectiveness of physicians.
We employ Data Envelopment Analysis (DEA)—a linear programming (LP) optimization
technique that provides a multi-dimensional evaluation tool—to develop and evaluate metrics
related to both efficiency and effectiveness. Efficiency in the Emergency Department (ED)
can be measured in multiple ways, but throughput—the average number of patients seen
by a provider per unit of time—per resources used by the provider possesses significant face
validity as an output-to-input ratio for exploratory analysis. For a given level of resources
used by the provider, a higher throughput means that more patients can be moved through
the ED per unit of time. Given that ED crowding has reached epidemic proportions in the
last several years (Salway et al. 2017), improving throughput without adding resources has
become even more important.
Similarly, effectiveness of care delivery in the ED can be measured in different ways.
However, compared to efficiency, it is a more inherently difficult outcome to measure, and
hence, we suggest using a composite of three metrics. The first pertains only to discharged
patients. Specifically, for a fixed level of resources used by a provider, we consider the
2

Electronic copy available at: https://ssrn.com/abstract=3227873
percentage of discharged patients who do not return to the ED within 72 hours. Returns to
the ED within 72 hours of discharge may result from a sub-optimal (i.e., ineffective) first visit,
in which not all medical issues were sufficiently identified or addressed. The 72-hour rate of
return has been even proposed as a measure of quality in the Emergency Medicine literature,
although using it for measuring quality is controversial (Abualenain et al. 2013, Pham et al.
2011, Klasco et al. 2015). The second and third metrics pertain only to admitted patients
(i.e., those admitted to the hospital after their ED visit). Specifically, we consider how
frequently a physician admits patients who are subsequently discharged after a brief period,
which suggests that the physician potentially overcalled the patients’ illness severity (i.e.,
patients are admitted when they could have potentially been discharged). We also consider
how frequently a physician admits patients whose care is escalated from a low-acuity bed
(ward/floor) to a high-acuity bed (Intermediate Unit or Intensive Care Unit), which suggests
that the physician potentially undercalled the patients’ illness severity. Although none of
these three metrics is ideal in isolation in measuring effectiveness of care delivery, their use
in the aggregate (per resources used by the provider) possesses face validity, and covers the
performance of a physician among both discharged and admitted patients.
We apply our proposed DEA approach (which considers the above measures as outputs
and the level of resources used by the physician as input) to a large data set of care delivered
by ED physicians that includes more than 115,000 patient visits. To learn about what
the high-performing physicians do differently from other physicians, and thereby generate
insights into best practices, we first make use of some Machine Learning (ML) algorithms
(including k-means, spectral, and random forest) to (a) segment physicians based on their
effectiveness and efficiency scores, and (b) identify physicians who have a high performance.
We then conduct a second-stage analysis in which we use a Tobit framework to identify
factors (e.g., test order count, physician contact-to-disposition time, etc.) associated with
higher levels of performance. We also use our framework to study peer effect: the effect
of the characteristics of another physician who is scheduled to work side-by-side with the
3

Electronic copy available at: https://ssrn.com/abstract=3227873
first physician. In particular, we consider peer physician characteristics such as relative
effectiveness, efficiency, experience, gender, and type of medical degree (MD vs. DO), and
examine how they affect another physician’s efficiency and effectiveness.
Our results offer important insights into physician performance. Contrary to the con-
ventional wisdom that efficiency may come at the price of effectiveness (and vice-versa), our
findings demonstrate a statistically significant positive association between efficiency and
effectiveness (P = 0.0209). We also find that a physician’s efficiency is negatively associated
with his/her average number of MRI orders per patient visit (P = 0.0496). In addition, we
observe a statistically significant negative relationship between a physician’s effectiveness and
his/her experience level measured in years since graduation from medical school (P = 0.0301),
which can be due to the recent changes in medical training programs that further emphasize
the efficient use of hospital resources. We also identify a statistically significant positive
relationship between a physician’s effectiveness and his/her average contact-to-disposition
time (P = 0.0290), suggesting that effectiveness can be enhanced via training programs that
enable physicians to avoid speed-up activities that reduce the contact-to-disposition time
but negatively affect the effectiveness of the care delivered (e.g., cutting necessary tests or
reducing value added direct or indirect care time).
Finally, our findings with regards to physician peer effects suggest a statistically signif-
icant impact of peer effect on each individual physician’s performance, a fact that can be
utilized by hospital administrators to improve performance via scheduling the correct set of
physicians during same shifts. In particular, our findings suggest that working alongside a
more effective peer is positively associated with improving both a physician’s efficiency (P
= 0.0044) and effectiveness (P = 0.0147).
4

Citations
More filters

Book
01 Jan 2009

8,216 citations


Posted Content
TL;DR: The results show that newly-hired and/or high-performing physicians are typically more influenced than others by their peers, and can be utilized by hospital administrators to improve the overall performance of physicians via better scheduling patterns and/ or training programs that require physicians to work during same shifts.
Abstract: Understanding potential ways through which physicians impact each other's performance can yield new insights into better management of hospitals' operations. We use evidence from Emergency Medicine to study whether and how physicians who work alongside each other during same shifts affect each other's performance. We find strong empirical evidence that physicians affect each other's speed and quality, and scheduling diverse peers during the same shift could have a positive net impact on the operations of a hospital Emergency Department (ED). Specifically, our results show that a faster (slower) peer decreases (increases) the average speed of a focal physician compared to a same-speed peer. Similarly, a higher- (lower-) quality peer decreases (increases) a focal physician's average quality. Furthermore, the presence of a less-experienced peer improves a focal physician's average speed. However, in contrast to the conventional wisdom, we do not find any evidence that more-experienced physicians can affect the performance of their less-experienced peers. We investigate various mechanisms that might be the driving force behind our findings, including psychological channels such as learning, social influence, and homophily as well as resource spillover. We identify resource spillover as the main driver of the effects we observe and show that, under high ED volumes (i.e., when the shared resources are most constrained), the magnitude of the observed effects increases. While some of these observed effects tend to be long-lived, we find that their magnitudes are fairly heterogeneous among physicians. In particular, our results show that newly-hired and/or high-performing physicians are typically more influenced than others by their peers. Finally, we draw conclusions from our results and discuss how they can be utilized by hospital administrators to improve the overall performance of physicians via better scheduling patterns and/or training programs that require physicians to work during same shifts.

4 citations


References
More filters

Journal ArticleDOI
Abstract: In management contexts, mathematical programming is usually used to evaluate a collection of possible alternative courses of action en route to selecting one which is best. In this capacity, mathematical programming serves as a planning aid to management. Data Envelopment Analysis reverses this role and employs mathematical programming to obtain ex post facto evaluations of the relative efficiency of management accomplishments, however they may have been planned or executed. Mathematical programming is thereby extended for use as a tool for control and evaluation of past accomplishments as well as a tool to aid in planning future activities. The CCR ratio form introduced by Charnes, Cooper and Rhodes, as part of their Data Envelopment Analysis approach, comprehends both technical and scale inefficiencies via the optimal value of the ratio form, as obtained directly from the data without requiring a priori specification of weights and/or explicit delineation of assumed functional forms of relations between inputs and outputs. A separation into technical and scale efficiencies is accomplished by the methods developed in this paper without altering the latter conditions for use of DEA directly on observational data. Technical inefficiencies are identified with failures to achieve best possible output levels and/or usage of excessive amounts of inputs. Methods for identifying and correcting the magnitudes of these inefficiencies, as supplied in prior work, are illustrated. In the present paper, a new separate variable is introduced which makes it possible to determine whether operations were conducted in regions of increasing, constant or decreasing returns to scale in multiple input and multiple output situations. The results are discussed and related not only to classical single output economics but also to more modern versions of economics which are identified with "contestable market theories."

13,542 citations


"Who Is an Efficient and Effective P..." refers methods in this paper

  • ...Furthermore, since there is no reason to believe that an increase in inputs results in a proportional change in outputs (and vice-versa) in our effectiveness and efficiency DEA models, we have used the Banker- Charnes-Cooper (BCC) model (Banker et al. 1984) which extends the CRS model to allow for variable returns to scale....

    [...]

  • ...…no reason to believe that an increase in inputs results in a proportional change in outputs (and vice-versa) in our effective- ness and efficiency DEA models, we have used the Banker- Charnes-Cooper (BCC) model (Banker et al. 1984) which extends the CRS model to allow for variable returns to scale....

    [...]


Journal ArticleDOI
TL;DR: A new graphical display is proposed for partitioning techniques, where each cluster is represented by a so-called silhouette, which is based on the comparison of its tightness and separation, and provides an evaluation of clustering validity.
Abstract: A new graphical display is proposed for partitioning techniques. Each cluster is represented by a so-called silhouette, which is based on the comparison of its tightness and separation. This silhouette shows which objects lie well within their cluster, and which ones are merely somewhere in between clusters. The entire clustering is displayed by combining the silhouettes into a single plot, allowing an appreciation of the relative quality of the clusters and an overview of the data configuration. The average silhouette width provides an evaluation of clustering validity, and might be used to select an ‘appropriate’ number of clusters.

10,821 citations


Book
01 Jan 2009

8,216 citations


"Who Is an Efficient and Effective P..." refers methods in this paper

  • ...Following the normalization approach of Greene (1993), which assumes 17 a censoring point at zero, we transform the DEA scores to: yit = (1/θit)− 1, where θit is the DEA measure of physician i’s performance in year t....

    [...]


Book
30 Nov 1999
Abstract: List of Tables. List of Figures. Preface. 1. General Discussion. 2. The Basic CCR Model. 3. The CCR Model and Production Correspondence. 4. Alternative DEA Models. 5. Returns to Scale. 6. Models with Restricted Multipliers. 7. Discretionary, Non-Discretionary and Categorical Variables. 8. Allocation Models. 9. Data Variations. Appendices. Index.

4,264 citations


"Who Is an Efficient and Effective P..." refers background in this paper

  • ...A DMU can be viewed as an entity responsible for converting a number of inputs into a set of outputs and whose performance is to be evaluated relative to its peers (Cooper et al. 2007)....

    [...]

  • ...7 A DMU can be viewed as an entity responsible for converting a number of inputs into a set of outputs and whose performance is to be evaluated relative to its peers (Cooper et al. 2007)....

    [...]


Journal ArticleDOI
Abstract: Efficiency scores of production units are generally measured relative to an estimated pro-duction frontier. Nonparametric estimators (DEA, FDH,···) are based on a finite sample of observed production units. The bootstrap is one easy way to analyze the sensitivity of efficiency scores relative to the sampling variations of the estimated frontier. The main point in order to validate the bootstrap is to define a reasonable data-generating process in this complex framework and to propose a reasonable estimator of it. This paper provides a general methodology of bootstrapping in nonparametric frontier models. Some adapted methods are illustrated in analyzing the bootstrap sampling variations of input efficiency measures of electricity plants.

1,842 citations


"Who Is an Efficient and Effective P..." refers methods in this paper

  • ...To account for unobserved serial correlation in the DEA scores, which might arise as a result of calculating a DMU’s DEA score by incorporating all other DMUs in the dataset, we use Simar and Wilson’s bootstrap procedure (Simar and Wilson 1998) for bias-correction of the scores....

    [...]