What’s trending in difference-in-differences? A synthesis of the recent econometrics literature

Existing research shows that distrust of the police is widespread and consequential for public safety. However, there is a shortage of interventions that demonstrably reduce negative police interactions with the communities they serve. A training program in Chicago attempted to encourage 8,480 officers to adopt procedural justice policing strategies. These strategies emphasize respect, neutrality, and transparency in the exercise of authority, while providing opportunities for civilians to explain their side of events. We find that training reduced complaints against the police by 10.0% and reduced the use of force against civilians by 6.4% over 2 y. These findings affirm the feasibility of changing the command and control style of policing which has been associated with popular distrust and the use of force, through a broad training program built around the concept of procedurally just policing.

https://www.pnas.org/content/pnas/117/18/9815.full.pdf

Procedural justice training reduces police use of force and complaints against officers.

Increasingly, modern Artificial Intelligence (AI) research has become more computationally intensive. However, a growing concern is that due to unequal access to computing power, only certain firms and elite universities have advantages in modern AI research. Using a novel dataset of 171394 papers from 57 prestigious computer science conferences, we document that firms, in particular, large technology firms and elite universities have increased participation in major AI conferences since deep learning's unanticipated rise in 2012. The effect is concentrated among elite universities, which are ranked 1-50 in the QS World University Rankings. Further, we find two strategies through which firms increased their presence in AI research: first, they have increased firm-only publications; and second, firms are collaborating primarily with elite universities. Consequently, this increased presence of firms and elite universities in AI research has crowded out mid-tier (QS ranked 201-300) and lower-tier (QS ranked 301-500) universities. To provide causal evidence that deep learning's unanticipated rise resulted in this divergence, we leverage the generalized synthetic control method, a data-driven counterfactual estimator. Using machine learning based text analysis methods, we provide additional evidence that the divergence between these two groups - large firms and non-elite universities - is driven by access to computing power or compute, which we term as the "compute divide". This compute divide between large firms and non-elite universities increases concerns around bias and fairness within AI technology, and presents an obstacle towards "democratizing" AI. These results suggest that a lack of access to specialized equipment such as compute can de-democratize knowledge production.

The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research.

Recent emphasis on credible causal designs has led to the expectation that scholars justify their research designs by testing the plausibility of their causal identification assumptions, often through balance and placebo tests. Yet current practice is to use statistical tests with an inappropriate null hypothesis of no difference, which can result in equating nonsignificant differences with significant homogeneity. Instead, we argue that researchers should begin with the initial hypothesis that the data are inconsistent with a valid research design, and provide sufficient statistical evidence in favor of a valid design. When tests are correctly specified so that difference is the null and equivalence is the alternative, the problems afflicting traditional tests are alleviated. We argue that equivalence tests are better able to incorporate substantive considerations about what constitutes good balance on covariates and placebo outcomes than traditional tests. We demonstrate these advantages with applications to natural experiments. Replication Materials: The data, code, and any additional materials required to replicate all analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at: https://doi.org/10.7910/DVN/RYNSDG. Recent debates over the difficulties of causal inference, and the rise of causal empiricism, in the social sciences have spurred a growing literature on how to judge the quality of causal research designs (Austin 2008; Hansen 2008; Dunning 2010; Samii 2016) and a growing expectation that scholars defend the merits of their research designs with tests of empirically refutable implications of the assumptions justifying their inferences (Sekhon 2009, 503). For example, as evidence in favor of their designs, observational researchers are expected to provide evidence of covariate balance, and experimental researchers run randomization checks for balance on pretreatment covariates. The procedures used to check the assumptions justifying a design are just as important as those used to estimate causal effects (Rubin 2008). In this article, we argue that “tests of design,” such as balance and placebo tests, discussed in the next section, should be structured so that the responsibility lies Erin Hartman is Assistant Professor, Departments of Political Science and Statistics, UCLA, 4289 Bunche Hall, 405 Hilgard Avenue/315 Portola Plaza, Los Angeles, CA 90095-1472 (ekhartman@ucla.edu). F. Daniel Hidalgo is Associate Professor, Department of Political Science, Massachusetts Institute of Technology, 77 Massachusetts Avenue, E53-470, Cambridge, MA 02142 (dhidalgo@mit.edu). Thanks to Santiago Olivella, Jasjeet Sekhon, Philip Stark, and Hans Noel for their comments and encouragement and to Kosuke Imai’s Research Group, and SLAMM! 2016 Conference, and the International Methods Colloquium participants for valuable feedback. We also thank Shiyao Liu for research assistance. We are grateful for support for the R package development provided by the EGAP 2018 Standards Grant. Author order reflects contribution. Identification assumptions are assumptions about the data-generating process that allow for identification of causal effects, and which are usually inherently untestable, but often have testable, observable implications. with researchers to positively demonstrate that the data are consistent with their identification assumptions or theory.1 This means that researchers should begin with the initial hypothesis that the data are inconsistent with a valid research design, and only reject this hypothesis if they provide sufficient statistical evidence in favor of data consistent with a valid design. The conceptual distinction between beginning with a null hypothesis of no difference, as is standard in current practice, versus beginning with a null hypothesis of a difference, as we advocate, may seem small, but the practical implications are substantial. To implement our tests of design, we rely on the large body of literature in biostatistics on equivalence testing (Wellek 2010; Westlake 1976). We show how to apply these procedures to tests of design, discussed in the Mechanics of an Equivalence Test section. We pay particular attention to the selection of an equivalence American Journal of Political Science, Vol. 00, No. 0, xxxx 2018, Pp. 1–14 C ©2018, Midwest Political Science Association DOI: 10.1111/ajps.12387

An Equivalence Approach to Balance and Placebo Tests

This paper proposes a Bayesian alternative to the synthetic control method for comparative case studies with a single or multiple treated units. We adopt a Bayesian posterior predictive approach to Rubin's causal model, which allows researchers to make inferences about both individual and average treatment effects on treated observations based on the empirical posterior distributions of their counterfactuals. The prediction model we develop is a dynamic multilevel model with a latent factor term to correct biases induced by unit-specific time trends. It also considers heterogeneous and dynamic relationships between covariates and the outcome, thus improving precision of the causal estimates. To reduce model dependency, we adopt a Bayesian shrinkage method for model searching and factor selection. Monte Carlo exercises demonstrate that our method produces more precise causal estimates than existing approaches and achieves correct frequentist coverage rates even when sample sizes are small and rich heterogeneities are present in the data. We illustrate the method with two empirical examples from political economy.

A Bayesian Alternative to Synthetic Control for Comparative Case Studies

This paper introduces a unified framework of counterfactual estimation for time-series cross-sectional data, which estimates the average treatment effect on the treated by directly imputing treated counterfactuals. Its special cases include several newly developed methods, such as the fixed effects counterfactual estimator, interactive fixed effects counterfactual estimator, and matrix completion estimator. These estimators provide more reliable causal estimates than conventional two-way fixed effects models when the treatment effects are heterogeneous or unobserved time-varying confounders exist. Under this framework, we propose two sets of diagnostic tests, tests for (no) pre-trend and placebo tests, accompanied by visualization tools, to help researchers gauge the validity of the no-time-varying-confounder assumption. We illustrate these methods with two political economy examples and develop an open-source package, fect, in both R and Stata to facilitate implementation.

A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data

This paper introduces a unified framework of counterfactual estimation for time-series cross-sectional data, which estimates the average treatment effect on the treated by directly imputing treated counterfactuals. Examples include the fixed effects counterfactual estimator, interactive fixed effects counterfactual estimator, and matrix completion estimator. These estimators provide more reliable causal estimates than conventional twoway fixed effects models when treatment effects are heterogeneous or unobserved time-varying confounders exist. Under this framework, we propose a new dynamic treatment effects plot, as well as several diagnostic tests, to help researchers gauge the validity of the identifying assumptions. We illustrate these methods with two political economy examples and develop an open-source package, fect, in both R and Stata to facilitate implementation.

Licheng Liu

Papers

A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data

A Bayesian Alternative to Synthetic Control for Comparative Case Studies

A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data

A Bayesian Alternative to Synthetic Control for Comparative Case Studies

Monetary policy and corporate financing: Evidence from different industries