Identifying Treatment and Spillover Effects with Control-Based and Forecast-Based Counterfactuals

Augusto Cerqua; Guido Pellegrini; Viviana Celli

REVIEW 1 major objections 5 minor 1 cited by

Identifying Treatment and Spillover Effects with Control-Based and Forecast-Based Counterfactuals

T0 review · 1 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Causal effects can be identified without clean control units by replacing cross-sectional comparisons with forecast-based counterfactuals, even when spillovers reach every unit.

desk verdict A genuinely useful identification framework for spillovers, but the practical claim that FBCMs can handle pervasive interference is too strong because the load-bearing stability assumption is untestable, and the simulations average away the common-shock case where the method collapses. read the letter →

arxiv 2607.20156 v1 pith:HCSBL6U5 submitted 2026-07-22 econ.EM

Viviana Celli , Augusto Cerqua , Guido Pellegrini This is my paper · ORCID

classification econ.EM

keywords potentialoutcomesspilloversinterferenceforecast-basedcounterfactualsdifference-in-differencesmachinelearningcontrolmethodtreatmenteffectspolicyevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when a policy's effects spill over to every unit—so that no untouched control group exists—causal effects can still be identified by forecasting each unit's no-policy outcome from its pre-treatment history and comparing observed outcomes to those forecasts. The authors show that control-based methods such as difference-in-differences require clean controls that are both present and observable, and collapse when spillovers are widespread or exposure is unobserved. Forecast-based methods, including interrupted time series and machine-learning control methods, instead identify aggregate policy effects such as the Average Total Effect on the Treated and the Population Average Policy Effect, without modelling how spillovers propagate. This comes at the price of assuming that the no-policy outcome process is structurally stable and sufficiently predictable, an assumption that may be harder to sustain over long horizons. Simulations and a police-protection natural experiment illustrate the trade-off.

What carries the argument

The central object is the forecast-based counterfactual: an estimate of the no-treatment/no-exposure potential outcome Y_it(0,0) obtained by extrapolating each unit's pre-treatment outcome and covariate dynamics into the post-treatment period (the Machine Learning Control Method is the leading example). The three assumptions—no anticipation, structural stability of the baseline outcome process, and sufficient predictability of that process—are what make the forecast a valid counterfactual. The forecast-based gap Y_it − Ŷ_it(0,0) then carries the causal content, identifying total effects on treated units and spillover effects on untreated units, and aggregating to the population average poli

What would settle it

Apply a forecast-based method to a panel with a known structural break at a date unrelated to any policy; a systematically nonzero forecast-based gap for a placebo treatment at that date shows the stability/predictability assumption is violated. In the Buenos Aires data, forecast car theft for blocks farthest from protected institutions and check whether the positive gap observed after August 1994 persists in placebo pre-treatment periods; if it does, the gap reflects a baseline shift rather than displacement.

Watch

Extended reading notes

Core claim

The paper's central claim is that forecast-based counterfactual methods (FBCMs) can point-identify the aggregate causal estimands ATOT and PAPE even under pervasive spillovers where every unit is exposed and no clean control exists, a setting in which control-based counterfactual methods (CBCMs) cannot identify these estimands without additional strong assumptions. The identifying device is the forecast-based gap: for each unit, forecast the no-treatment/no-exposure potential outcome Y_it(0,0) from pre-treatment dynamics, then compare the observed outcome to that forecast. Under no anticipation, structural stability of the baseline process, and sufficient predictability, the average of these

Load-bearing premise

The entire edifice rests on the assumption that the no-policy outcome process is stable and sufficiently predictable after the treatment begins: if a structural break or non-forecastable shock hits the baseline outcome right after T0, the forecast-based gap is biased, and with only five pre-treatment periods in the empirical application this convergence premise cannot be credibly verified.

Editorial extensions

If this is right

Policies that operate through general-equilibrium or network channels can be evaluated even when all units are indirectly treated, provided credible forecasts of the no-policy path can be constructed.
Exposure-specific spillover estimates still require observing exposure; forecast-based methods do not remove that need.
The credibility of the design shifts from assumptions about the spillover structure to assumptions about the stability and predictability of baseline outcomes, which are more plausible for short-term horizons.
In the Buenos Aires application, forecast-based estimates find a positive population-wide effect (0.029 and 0.022 car thefts per block) even though protected blocks themselves show fewer thefts, because spillover effects on more distant blocks outweigh local deterrence.
In the simulations, misspecifying the exposure mapping biases control-based estimates through contaminated controls, while forecast-based ATOT and PAPE remain unchanged because they do not depend on the assumed mapping.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's logic suggests a testable extension: in a randomized experiment with a known spillover structure, compare forecast-based estimates with the true aggregate effect under different pre-treatment lengths to map how the convergence assumption degrades with short panels.
The framework implies that when exposure is latent, direct and indirect effects are not separately recoverable, but total and population effects are—so policy evaluation can proceed without solving the often-impossible exposure-mapping problem, at least for aggregate questions.
Because forecast-based identification relies on time-series stability, long-run policy evaluation would require external validation of the baseline model (e.g., placebo forecasts in pre-treatment periods), not just cross-sectional balance checks.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper gives applied researchers a clear map of what control-based (CBCM) and forecast-based (FBCM) counterfactuals can and cannot identify under different spillover regimes. The novelty is the FBCM identification results for latent selective and pervasive interference: aggregate estimands like ATOT and PAPE can be recovered even when no clean control exists, provided the baseline process is stable and forecastable. The propositions are simple algebra, and they are correct under the stated assumptions. The simulation design is genuinely thoughtful, comparing a true queen-contiguity mapping with misspecified distance bands, and the authors properly flag when forecast-based invariance across mappings is mechanical.

The soft spots are in the gap between the identification logic and the practical claim. The load-bearing assumptions are FBCM2 (structural stability) and FBCM3 (sufficient predictability). With five pre-treatment periods, FBCM3 cannot be credibly verified. More specific problem: the simulation DGP draws a fresh common time shock each Monte Carlo replication and averages bias over replications. A realized post-treatment common shock therefore cancels out in the reported near-zero MLCM/FAT biases. In any single sample, a citywide shock is indistinguishable from the policy effect and biases every FBCM estimand, because there are no controls to difference it out. The stress-test note about this is right, and the paper does not address it. The Buenos Aires application is exactly a setting where citywide crime shocks are plausible, and the placebo check uses only the last pre-treatment month, low power.

Also: Appendix F has two incompatible descriptions of the MLCM standard errors (bootstrap vs cross-sectional dispersion), and no replication code or data are provided.

None of this sinks the paper. The identification framework is coherent and novel, and the authors are honest about the core fragility of FBCMs. But the abstract's claim that FBCMs 'can more credibly identify' aggregate effects under pervasive spillovers is not established for the common-shock case. That needs to be reframed.

Who benefits: applied researchers choosing between DiD and forecast-based designs, and methodologists working on interference. It deserves a serious referee. I would send it to review, with the practical claims trimmed and a common-shock analysis added.

Referee Report

1 major / 5 minor

Summary. The paper develops a unified potential-outcomes framework for treatment and spillover effects, comparing control-based counterfactual methods (CBCMs), such as difference-in-differences, with forecast-based counterfactual methods (FBCMs), such as MLCM and FAT. It defines aggregate estimands (ATOT, ASEU, PAPE), states identifying assumptions (CBCM1–3, FBCM1–3), and proves which estimands are identified under four interference regimes ranging from no spillovers to pervasive spillovers with unobserved exposure. The theoretical results are complemented by Monte Carlo simulations based on U.S. county geography under true and misspecified exposure mappings, and by an empirical application to police protection and car theft in Buenos Aires. The central claim is that FBCMs can identify aggregate policy effects even when all units are exposed to spillovers and no clean control group exists, a setting where CBCMs generally fail.

Significance. If the claims hold, the paper makes a useful contribution by clarifying the identifying content of forecast-based counterfactuals in interference settings. The distinction between the existence and observability of clean controls (Cases ii.a vs. ii.b) is valuable, and the explicit treatment of estimands defined relative to the no-treatment/no-exposure baseline is careful. The simulation design is thoughtful: using true and misspecified exposure mappings and reporting target-group composition effects is informative. The paper also honestly notes several limitations, including the dependence of FBCMs on structural stability and predictability. The formal propositions are algebraically correct under the stated assumptions. The main open question is whether the practical conditions under which FBCMs work are sufficiently credible in realistic settings with common shocks and short pre-treatment histories.

major comments (1)

[Section 3.2.2, Propositions FBCM4–FBCM5] These propositions claim identification of ASEU when all units are exposed but exposure intensity is unobserved. In the distance-decay DGP, exposure is continuous and every untreated unit has strictly positive exposure, so 'exposed untreated' coincides with all untreated units. This is a special case of Case (iii). If exposure is unobserved but some untreated units have zero exposure, ASEU is not identified by FBCMs. The propositions should be stated with the conditioning set explicitly defined: for universal exposure, the average over untreated units is the ASEU only because the exposed set is the entire untreated population. Otherwise, the latent-selective-interference case (Proposition FBCM3) already covers the unobserved-strata scenario. This is a clarity issue rather than a mathematical error, but it affects the interpretation of Table 4.

minor comments (5)

[Introduction] The sentence 'Third, we formally demonstrates that FBCMs...' contains a subject-verb agreement error ('demonstrates' should be 'demonstrate').
[Section 4.1] The text says 'standard deviation 1' for common time shocks and later 'common time shocks are independently drawn from a normal distribution with standard deviation 1'; this is fine, but the AR(1) coefficient is given as 0.60 in the text while Appendix E uses 0.6. Please standardize notation.
[Section 5, Table 5] The notes for Table 5 refer to 'ATOT under clean-control assumption' in the row label, but the row itself is labeled 'ATOT under clean-control assumption: treated blocks vs controls more than two blocks away.' Consider making the label shorter and moving the assumption descriptor to the notes for readability.
[Appendix F, Table F1] The placebo table is an important validity check, but it covers only the first 17 days of July, which is a very short window. Please state explicitly that this is a low-power test, and if possible, report a placebo for earlier months (e.g., June) using the same recursive forecasting scheme.
[References] The reference list includes 'Botosaru, I., Giacomini, R., & Weidner, M. (2026)' which is an arXiv preprint; if a published version exists, please update. Also, 'Callaway, B., & Sant’Anna, P. H. C. (2021)' is cited in text but not listed in the reference list; it appears to be missing from the references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FBCM identification results are conditional on explicit forecast-validity assumptions, and forecasts are built from pre-treatment data only.

full rationale

The paper's central identification results are not circular by construction. The FBCM propositions are derived under explicit assumptions—FBCM1 (no anticipation), FBCM2 (structural stability), and FBCM3 (sufficient predictability, formally Y_hat_it(0,0) - Y_it(0,0) -> 0). FBCM3 is an identifying assumption about the forecasting model and the baseline outcome process, not a restatement of the estimand or a fit to the treatment effect. The forecast-based counterfactual is constructed using only pre-treatment information, as stated in Section 3.2: 'Using only pre-treatment observations, the method estimates a flexible prediction rule for the evolution of outcomes under no treatment and no exposure.' The estimands ATOT and PAPE are defined as contrasts with Y_it(0,0), and the proofs in Appendix D show that, under the stated assumptions, the mean forecast gap equals these contrasts. This is a standard conditional identification argument, not a tautology. The simulations evaluate forecasts out-of-sample against a truth generated by the specified treatment and spillover coefficients, not by the forecast itself. The self-citation to Cerqua et al. (2024) for the MLCM is used as a leading example, but the paper's identification framework does not depend on the validity of that citation; the assumptions are stated in full. The concern that common post-treatment shocks may violate FBCM2/FBCM3 is a substantive robustness critique about assumption credibility, not a circularity in the derivation.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The identification theorems rest on standard potential-outcomes consistency and on domain assumptions about no anticipation, forecast stability and predictability, and, for CBCMs, clean controls and parallel trends. No new entities are invented. Simulation DGP constants are hand-chosen calibration values rather than fitted causal parameters.

free parameters (6)

Simulation direct-effect scale tau/(1-rho) = 5
Hand-chosen so the realized direct effect in the first post-treatment period is 5; determines true ATOT in simulations (Section 4.1).
Simulation spillover scale (1-rho)*gamma0*delta = 1.2
Hand-chosen so the realized spillover per queen-contiguous treated neighbor is 1.2; determines true SE, ASEU, and PAPE.
AR(1) persistence rho = 0.60
Chosen outcome persistence; drives forecastability of the baseline process and therefore FBCM performance.
Number of treated counties = 200
Choice of treatment saturation affects the exposure distribution and the values of the simulation estimands.
Researcher exposure thresholds c = 30 km, 60 km
Chosen misspecified mappings; define the contamination and target-group mismatch analyzed in Table 3.
MLCM/FAT tuning parameters = RF 500 trees, 3 candidate predictors; bagging 1,000 trees; FAT linear trend
Method hyperparameters chosen by the authors; affect finite-sample forecast accuracy and the reported standard errors.

assumptions (7)

standard math Consistency: Y_it = Y_it(D_i, S_it).
Standard potential-outcomes consistency relation invoked throughout (Section 2 and Appendix C).
domain assumption No anticipation: Y_it = Y_it(0,0) for t < T0.
Assumptions CBCM1 and FBCM1; excludes behavioral or equilibrium responses before the treatment date.
domain assumption FBCM2: the law of motion of Y_it(0,0) is stable over time.
Rules out structural breaks independent of the policy; critical for forecast-based counterfactuals.
domain assumption FBCM3: forecast error converges to zero.
Core premise for all FBCM identification results; requires persistence and forecastability of the baseline outcome process.
domain assumption Existence and observability of clean controls (CBCM2_v2).
Required for CBCM identification results in the selective and widespread interference cases.
domain assumption Parallel trends CBCM3(a)-CBCM3(c).
Required for DiD-style identification; stated in Section 3.1.1 and used in the CBCM proofs.
domain assumption Exposure mapping S_i summarizes the true interference structure when observed.
Used in Case ii.a and in the simulation benchmark; misspecification of this mapping is the focus of the simulation study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Identifying Treatment and Spillover Effects with Control-Based and Forecast-Based Counterfactuals." pith.science (2026). https://pith.science/paper/HCSBL6U5

@misc{pith2026260720156,
  author       = {Pith},
  title        = {Pith review of: Identifying Treatment and Spillover Effects with Control-Based and Forecast-Based Counterfactuals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HCSBL6U5}},
  note         = {Machine review of arXiv:2607.20156}
}

read the original abstract

Spillovers and interference pose fundamental challenges for causal inference, as treatment assigned to one unit may affect the outcome of others, violating the no-interference assumption underlying most empirical strategies. Existing approaches, based on partial interference, exposure mapping, spatial, network, or structural frameworks, typically rely on strong assumptions about interaction structures or require the existence of uncontaminated control units to estimate relevant causal parameters. We revisit this identification challenge within the potential outcomes framework and compare the conditions under which causal effects can be identified using two broad classes of counterfactual methods: control-based counterfactual methods (CBCMs), such as matching and difference-in-differences designs, and forecast-based counterfactual methods (FBCMs), including interrupted time-series and machine learning control methods. We show under which circumstances CBCMs and FBCMs identify average direct and spillover effects. Through simulations and an empirical application, we illustrate the main advantages and limitations of each approach. We show that, in the presence of pervasive or ill-defined spillover effects, CBCMs either cannot be used or entail severe identification concerns, whereas FBCMs can more credibly identify some of the causal parameters of interest, at least in the short term.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Learning about Treatment Effects in Panels under Unknown Interference
econ.EM 2026-08 conditional novelty 6.0 of 10

Under unknown interference, the sharp identified set for a panel treatment effect is characterized exactly by feasibility of a finite linear system, enabling uniform candidatewise bootstrap inference.

Reference graph

Works this paper leans on

2 extracted references · cited by 1 Pith paper

[1]

All estimands listed in Proposition CBCM7 are defined relative to 𝑌𝑖𝑡 0, either directly or through averages involving this baseline counterfactual

Hence, untreated outcomes are contaminated by spillovers and cannot be used as observations of the no-treatment/no-exposure potential outcome. All estimands listed in Proposition CBCM7 are defined relative to 𝑌𝑖𝑡 0, either directly or through averages involving this baseline counterfactual. Since CBCMs recover counterfactuals from observed cross-unit comp...
[60]

Spillovers therefore decline smoothly with distance but never become exactly zero

, 𝑤𝑖𝑖 = 0, and the true exposure score of county 𝑖 is 𝑆𝑖 = ∑ 𝑤𝑖𝑗 𝑗≠𝑖 𝐷𝑗. Spillovers therefore decline smoothly with distance but never become exactly zero. Since the exponential weights are strictly positive, every county receives some exposure from the treated counties, although the resulting effect may be extremely small for geographically distant units...

1994

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

All estimands listed in Proposition CBCM7 are defined relative to 𝑌𝑖𝑡 0, either directly or through averages involving this baseline counterfactual

Hence, untreated outcomes are contaminated by spillovers and cannot be used as observations of the no-treatment/no-exposure potential outcome. All estimands listed in Proposition CBCM7 are defined relative to 𝑌𝑖𝑡 0, either directly or through averages involving this baseline counterfactual. Since CBCMs recover counterfactuals from observed cross-unit comp...

[2] [60]

Spillovers therefore decline smoothly with distance but never become exactly zero

, 𝑤𝑖𝑖 = 0, and the true exposure score of county 𝑖 is 𝑆𝑖 = ∑ 𝑤𝑖𝑗 𝑗≠𝑖 𝐷𝑗. Spillovers therefore decline smoothly with distance but never become exactly zero. Since the exponential weights are strictly positive, every county receives some exposure from the treated counties, although the resulting effect may be extremely small for geographically distant units...

1994