REVIEW 3 major objections 3 minor 28 references
Good Controls Gone Bad: Difference-in-Differences with Covariates
T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that standard DiD estimators require covariate effects to be constant across groups and time, and proposes DID-INT to identify the ATT when that fails.
desk verdict The paper's central claim about heterogeneous treatment effects isn't proven—its own proof uses homogeneous effects—but the CCC taxonomy and simulation evidence make it worth a careful look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the group-by-time intersection dummy $I(s,t)$ and the associated regression $Y_{i,s,t} = \sum_s \sum_t \lambda_{s,t} I(s,t) + f(X_{i,s,t}) + \epsilon$, run without a constant. The four versions of DID-INT differ in $f(X)$: homogeneous, state-varying (covariates interacted with $I(s)$ only), time-varying (interacted with $I(t)$ only), and two-way (interacted with both). The ATT for a treated group and period is the difference between the treated and control cells of the long difference $\lambda_{s,t} - \lambda_{s,t^{-}_{s}}$, and the overall ATT is a weighted average that omits forbidden comparisons. This construction is what makes the covariate effects cancel even when $\gamma_{s,t}$ varies across cells.
What would settle it
Run the paper's Monte Carlo design with two-way CCC violations and treatment effects that depend on a covariate, such as $\tau_i = X_{1i}$, while keeping Assumptions 1-4; if the two-way DID-INT estimates do not center on the true ATT across many replications, the identification claim as stated fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a covariate that is a 'good control' in one pair of groups and periods can become a source of bias in another: the effect of the covariate on the untreated outcome is written $\gamma^k_{s,t}$, and unless $\gamma$ is constant across states and years the conventional estimators misestimate the ATT. DID-INT removes this problem by allowing the covariate effect to be fully cell-specific: it runs $Y_{i,s,t} = \sum_s \sum_t \lambda_{s,t} I(s,t) + f(X_{i,s,t}) + \epsilon$, with $f(X)$ chosen as covariate-by-group, covariate-by-time, or covariate-by-group-by-time interactions, and estimates the ATT for a treated cell as the long difference of the $\lambda$'s in the treated cell minus the same long difference for a not-yet-treated control cell. Because the covariate terms are absorbed at the group-by-time level, the bias terms from CCC violations cancel before the DiD comparison is made. The paper states this identification result as Theorem 2, requiring only binary treatment, overlap, conditional parallel trends, and no anticipation, and the abstract extends the claim to heterogeneous treatment effects and staggered rollout.
Load-bearing premise
The proof of Theorem 2 assumes homogeneous treatment effects when simplifying the bias terms, and the practical guarantee assumes the model-selection algorithm finds the correct covariate functional form; if either fails, the bias terms need not cancel.
Editorial extensions
If this is right
- Researchers can include time-varying covariates without assuming their effects are constant across groups and periods, provided they use the fully interacted two-way DID-INT.
- TWFE, CS-DID, imputation, and FLEX are all biased when the two-way CCC is violated, and the bias grows with the degree of violation in the paper's simulations.
- The two-way DID-INT is unbiased across all simulated DGPs but is the least efficient estimator, so there is a bias-variance tradeoff.
- A modified TWFE with fully interacted covariates also removes the CCC bias under common treatment timing, but it remains vulnerable to staggered-adoption negative weights, while DID-INT is not.
- The model-selection algorithm can turn non-parallel pre-trend figures into parallel ones by residualizing outcomes with group-by-time interacted covariates.
Reading between the lines
- The appendix proof of Theorem 2 appears to import Assumption 13 (homogeneous treatment effects) when it replaces $E[\tau_{s,t} | s \in S_T, t, f(X)]$ with $\tau$, so the advertised robustness to heterogeneous treatment effects is not demonstrated by that proof.
- The model-selection algorithm chooses the covariate functional form by visual inspection of pre-trend plots; identification therefore carries an informal, data-dependent step that would need a formal test to be fully automatic.
- An immediate testable extension is to run the paper's Monte Carlo with treatment effects that vary with a covariate, which directly checks whether the Theorem 2 claim holds beyond homogeneous effects.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces the 'common causal covariates' (CCC) assumption for difference-in-differences with covariates, distinguishing state-invariant, time-invariant, and two-way CCC violations. It argues that existing estimators (TWFE, CS-DID, imputation, FLEX) are biased when the two-way CCC is violated, and proposes a new estimator, DID-INT, which residualizes outcomes after interacting covariates with group and time dummies. The paper claims that DID-INT identifies the ATT under Assumptions (1)–(4) alone, without any CCC assumptions, and also under heterogeneous treatment effects and staggered adoption. The theoretical results are supplemented by Monte Carlo simulations and an empirical illustration using Canadian Labour Force Survey data.
Significance. The paper identifies a real and important gap in the DiD-with-covariates literature: the implicit assumption that covariate effects are constant across groups and time. The CCC taxonomy (state-invariant, time-invariant, two-way) is a useful conceptual contribution, and the Monte Carlo comparisons across estimators are informative. The paper also provides open-source software (Julia and Stata) for the proposed estimator, which is a practical strength. However, the central identification claim for DID-INT is not supported by the proof as written: the proof of Theorem 2 relies on a homogeneous treatment effect assumption that is neither stated in the theorem nor reflected in the abstract's heterogeneous-treatment-effects claim. Consequently, the paper's headline contribution—that DID-INT identifies the ATT under the standard DiD assumptions without extra covariate restrictions—is not established.
major comments (3)
- [Section 10 / Theorem 2 / Appendix B] The proof of Theorem 2 relies on Assumption 13 (homogeneous treatment effects), but the theorem and the abstract claim identification under heterogeneous treatment effects. Appendix B's Eq. (9) assumes E[Y_{i,s,t}|s in ST, t, f(X)] = lambda_{s,t} + sum gamma^k_{s,t} E[X^k|...] for the treated post-period cell. This holds only if the OLS coefficient on X in that cell equals the untreated-potential-outcome coefficient gamma_{s,t}. When treatment effects vary with X, the slope instead converges to gamma_{s,t} + Cov(X, tau(X))/Var(X), so the cancellation in Eq. (17) fails and theta_{s,t} is generally not tau. Therefore the central claim that DID-INT identifies the ATT without any CCC assumptions is unproven.
- [Appendix E, Eq. (43)] Appendix E derives the DR-DID estimand under time-varying covariates and two-way CCC violations, obtaining an explicit nonzero bias term in Eq. (42), but then concludes with Eq. (43): theta_DRDID = E[hat theta_DRDID] = tau. This contradicts Theorem 9, which states theta_DRDID != tau in that setting. The displayed equation should be theta_DRDID != tau. As printed, the appendix's conclusion is internally inconsistent with the stated theorem and with the preceding derivation.
- [Section 9 / Assumption 3] The identification result in Theorem 2 depends on Assumption 3, which conditions on the 'correct functional form of covariates identified from the model selection algorithm defined in Section 9.' The model selection algorithm is based on visual inspection of parallel-trends plots and is not shown to recover the true f(X) under any formal criterion. If the algorithm selects an incorrect functional form, conditional parallel trends need not hold, and the bias terms in Eq. (17) that the proof cancels would not cancel. The paper provides no formal guarantee for this step, so the identification argument is incomplete.
minor comments (3)
- [General] There are numerous typographical errors, including 'SUTV A' for SUTVA, 'accross' for across, 'the the' instances, and inconsistent equation numbering between the main text and appendices. A careful proofreading pass is needed.
- [Section 13.1] The text says Panel (d) uses a DGP 'Y^4' where 'Assumption (2) is violated'; this appears to be a typo for 'the two-way CCC is violated' (or similar). Also, the statement that Assumption (2) is violated for Y^2 in the main Monte Carlo seems inconsistent with the earlier description that Assumption (2) holds for Y^1 and is violated for Y^2; the text should clarify which assumptions are actually violated in each DGP.
- [Section 7 / Appendix A] Theorem 1 states identification under Assumptions (1)-(4), (8), (9), and (10), but the proof in Appendix A uses Assumption 13 (homogeneous treatment effects) to write E[tau_{s,t}|...] = tau. The theorem statement should either include Assumption 13 or the proof should handle heterogeneous treatment effects explicitly.
Circularity Check
Theorem 2's proof cancels the covariate bias only after substituting Appendix A Eq. (6), which was derived under homogeneous-treatment-effects Assumption 13, so the abstract's heterogeneous-effects claim reduces by construction to a constant-τ input.
-
other
[Theorem 2 (Section 10); Appendix B, Equations (9)-(18); Appendix A, Equation (6); Assumption 13]
"Imposing Assumptions (1) to (4), we can replace ... in Equation (15) with two-way version of the term shown in Equation (6) to get: τ + ... Canceling out the relevant terms in Equation (17), we can show that: θs,t = E[θ̂s,t] = τ. Equation (6) uses Assumption (13), which implies E[τ_{s,t}|s∈ST,t∈T^s,f(X̃)] = τ."
The proof of Theorem 2 does not derive τ from Assumptions (1)-(4); it substitutes the 'two-way version' of Appendix A Eq. (6), and that expression is obtained only after imposing Assumption 13 (homogeneous treatment effects). Appendix B Eq. (9)-(10) also writes the treated-post cell conditional mean as λs,t + Σ γ_{s,t}E[X|...], omitting any E[τ|...] term, which is valid only if τ is constant within the cell or mean-independent of X. Substituting Eq. (6) makes the bias terms cancel by construction, so θ=τ is an algebraic restatement of the homogeneous-τ input, not identification of the ATT under heterogeneous treatment effects.
-
fitted input called prediction
[Section 9 (Model Selection Algorithm); Assumption 3, Section 5; Section 4 step 1]
"The algorithm relies on plotting a sequence of parallel trends figures, conditional on the appropriate functional form of covariates, and stopping where pre-trends seem plausible through visual inspection. ... Assumption (3) in this setup holds conditional on the correct functional form of covariates identified from the model selection algorithm defined in Section 9."
The 'correct' functional form f(X) is selected by inspecting residualized outcome trends in the same dataset later used for estimation, and the stopping rule is 'pre-trends seem plausible'. Hence the selected specification is guaranteed by the selection criterion to make the pre-trend figure look parallel, so the paper's claim that covariate residualization 'can recover parallel trends' is in part a property manufactured by the search rather than independent evidence. Subsequent inference also ignores this outcome-based specification search, so unbiasedness of the chosen DID-INT variant is conditional on a data-dependent event that the formal theorem does not account for.
full rationale
The paper's negative results about TWFE, CS-DID, imputation, and FLEX under CCC violations are derived under the explicitly stated homogeneous-treatment-effects Assumption 13 and are not the circular part. The central circular step is Theorem 2: the proof substitutes the 'two-way version' of Appendix A Eq. (6) into the DID-INT estimand, and Eq. (6) is derived only after invoking Assumption 13 (E[τ|s,t,f(X)] = τ). Thus the cancellation that yields θ=τ is produced by assuming the treatment effect is constant in covariates, not by Assumptions (1)-(4). For heterogeneous τ(X), the treated-post OLS slope on X equals γ_{s,t} + Cov(X,τ)/Var(X), so the residual λ_{s,t} carries an extra mean-zero term and the Eq. (17) cancellation fails; the abstract's heterogeneous-effects claim is therefore an assumption restated as a conclusion. A secondary practical circularity is the model selection algorithm: f(X) is chosen by inspecting residualized outcome trends from the estimation sample and stopping when pre-trends 'seem plausible', so the 'recovered parallel trends' are in part manufactured by the selection criterion, and inference ignores this selection. The self-citations (Karim et al. 2024, 2025) concern software, inference routines, and a related estimator, and are not load-bearing for the identification argument, so they do not raise the score further. The Monte Carlo exercises are internal consistency checks with DGPs matching the estimator's assumed linear structure; they do not constitute external validation, but that is a simulation-design limitation rather than a separate circularity. Overall, one central prediction reduces by construction to a hidden homogeneity input; the rest of the identification algebra is coherent once that input is admitted.
Assumptions & free parameters
free parameters (2)
- Monte Carlo degree of CCC violation increments =
10, 50, 100, 250, 500
- Weights w_{s,t} in ATT aggregation =
not specified
assumptions (4)
- standard math SUTVA (observed outcome equals treated or untreated potential outcome)
- domain assumption Untreated outcome is linear in f(X) with cell-specific coefficients (Eq. 6)
- domain assumption Homogeneous treatment effects (Assumption 13)
- ad hoc to paper Model selection algorithm recovers the true f(X)
Cite this review
Pith. "Pith review of Good Controls Gone Bad: Difference-in-Differences with Covariates." pith.science (2026). https://pith.science/paper/ZOJITA4B
@misc{pith2026241214447,
author = {Pith},
title = {Pith review of: Good Controls Gone Bad: Difference-in-Differences with Covariates},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZOJITA4B}},
note = {Machine review of arXiv:2412.14447}
}
read the original abstract
This paper introduces the two-way common causal covariates (CCC) assumption, which is necessary to get an unbiased estimate of the ATT when using time-varying covariates in existing Difference-in-Differences methods. The two-way CCC assumption implies that the effect of the covariates remain the same between groups and across time periods. This assumption has been implied in previous literature, but has not been explicitly addressed. Through theoretical proofs and a Monte Carlo simulation study, we show that the standard TWFE and the CS-DID estimators are biased when the two-way CCC assumption is violated. We propose a new estimator called the Intersection Difference-in-differences (DID-INT) which can provide an unbiased estimate of the ATT under two-way CCC violations. DID-INT can also identify the ATT under heterogeneous treatment effects and with staggered treatment rollout. The estimator relies on parallel trends of the residuals of the outcome variable, after appropriately adjusting for covariates. This covariate residualization can recover parallel trends that are hidden with conventional estimators.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Abadie, A. (2005) Semiparametric difference-in-differences estimators, The review of economic studies 72(1), 1--19
work page 2005
-
[2]
Abadie, A., A. Diamond, and J. Hainmueller (2010) Synthetic control methods for comparative case studies: Estimating the effect of california’s tobacco control program, Journal of the American statistical Association 105(490), 493--505
work page 2010
-
[3]
Bertrand, M., E. Duflo, and S. Mullainathan (2004) How much should we trust differences-in-differences estimates?, The Quarterly journal of economics 119(1), 249--275
work page 2004
-
[4]
Jaravel (2017) Revisiting event study designs, Available at SSRN 2826228
Borusyak, K., and X. Jaravel (2017) Revisiting event study designs, Available at SSRN 2826228
work page 2017
-
[5]
Borusyak, K., X. Jaravel, and J. Spiess (2024) Revisiting event-study designs: robust and efficient estimation, Review of Economic Studies 91(6), 3253--3285
work page 2024
-
[6]
Difference-in-Differences when Parallel Trends Holds Conditional on Covariates
Caetano, C., and B. Callaway (2024) Difference-in-differences when parallel trends holds conditional on covariates, arXiv preprint arXiv:2406.15288
work page Pith review arXiv 2024
-
[7]
Caetano, C., B. Callaway, S. Payne, and H. S. Rodrigues (2022) Difference in differences with time-varying covariates, arXiv preprint arXiv:2202.02903
arXiv 2022
-
[8]
Callaway, B., and P. H. Sant’Anna (2021) Difference-in-differences with multiple time periods, Journal of Econometrics 225(2), 200--230
work page 2021
Show all 28 references
-
[9]
Rothstein, and M
Card, D., J. Rothstein, and M. Yi (2024) Industry wage differentials: A firm-based approach, Journal of Labor Economics 42(S1), S11--S59
2024
-
[10]
d’Haultfoeuille (2020a) Two-way fixed effects estimators with heterogeneous treatment effects, American Economic Review 110(9), 2964--2996
De Chaisemartin, C., and X. d’Haultfoeuille (2020a) Two-way fixed effects estimators with heterogeneous treatment effects, American Economic Review 110(9), 2964--2996
2020
-
[11]
--- --- --- (2023) Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: A survey, The Econometrics Journal 26(3), C1--C30
2023
-
[12]
Deb, P., E. C. Norton, J. M. Wooldridge, and J. E. Zabel (2024) A flexible, heterogeneous treatment effects difference-in-differences estimator for repeated cross-sections, Technical report, National Bureau of Economic Research
2024
-
[13]
Denning, J. T., E. R. Eide, K. J. Mumford, R. W. Patterson, and M. Warnick (2022) Why have college completion rates increased?, American Economic Journal: Applied Economics 14(3), 1--29
2022
-
[14]
Fortin, N. M. (2006) Higher-education policies and the college wage premium: Cross-state evidence from the 1990s, American Economic Review 96(4), 959--987
2006
-
[15]
(2021) Difference-in-differences with variation in treatment timing, Journal of Econometrics 225(2), 254--277
Goodman-Bacon, A. (2021) Difference-in-differences with variation in treatment timing, Journal of Econometrics 225(2), 254--277
2021
-
[16]
Heckman, J. J., H. Ichimura, J. A. Smith, and P. E. Todd (1998) Characterizing selection bias using experimental data,
1998
-
[17]
Heckman, J. J., H. Ichimura, and P. E. Todd (1997) Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme, The review of economic studies 64(4), 605--654
1997
-
[18]
(2018) Relative education and the advantage of a college degree, American Sociological Review 83(4), 771--801
Horowitz, J. (2018) Relative education and the advantage of a college degree, American Sociological Review 83(4), 771--801
2018
-
[19]
Strumpf, N
Karim, S., E. Strumpf, N. Austin, and M. D. Webb (2025) Which policy works and where? estimation and inference of state level treatment effects using difference-in-differences, Slides presented at the Canadian Economics Association Annual Conference, accessed July 11, 2025
2025
-
[20]
Karim, S., M. D. Webb, N. Austin, and E. Strumpf (2024) Difference-in-differences with unpoolable data, arXiv preprint arXiv:2403.15910
2024 arXiv
-
[21]
(1958) Investment in human capital and personal income distribution, Journal of political economy 66(4), 281--302
Mincer, J. (1958) Investment in human capital and personal income distribution, Journal of political economy 66(4), 281--302
1958
-
[22]
Kreif, R
O’Neill, S., N. Kreif, R. Grieve, M. Sutton, and J. S. Sekhon (2016) Estimating causal effects: considering three alternatives to difference-in-differences estimation, Health Services and Outcomes Research Methodology 16, 1--21
2016
-
[23]
Roth (2023) A more credible approach to parallel trends, Review of Economic Studies 90(5), 2555--2591
Rambachan, A., and J. Roth (2023) A more credible approach to parallel trends, Review of Economic Studies 90(5), 2555--2591
2023
-
[24]
Rios-Avila, F., P. H. Sant'Anna, and B. Callaway (2021) CSDID: Stata module for the estimation of Difference-in-Difference models with multiple time periods , Statistical Software Components, Boston College Department of Economics
2021
-
[25]
(2025) Julia as a universal platform for statistical software development, The Stata Journal 25(2), 255--284
Roodman, D. (2025) Julia as a universal platform for statistical software development, The Stata Journal 25(2), 255--284
2025
-
[26]
Roth, J., P. H. Sant'Anna, A. Bilinski, and J. Poe (2022) What's trending in difference-in-differences? a synthesis of the recent econometrics literature, arXiv preprint arXiv:2201.01194
2022 arXiv
-
[27]
H., and J
Sant’Anna, P. H., and J. Zhao (2020) Doubly robust difference-in-differences estimators, Journal of Econometrics 219(1), 101--122
2020
-
[28]
Abraham (2021) Estimating dynamic treatment effects in event studies with heterogeneous treatment effects, Journal of Econometrics 225(2), 175--199
Sun, L., and S. Abraham (2021) Estimating dynamic treatment effects in event studies with heterogeneous treatment effects, Journal of Econometrics 225(2), 175--199
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.