REVIEW 3 major objections 5 minor 27 references
Averaging over candidate shapes for a nuisance interruption's effect—rather than committing to one—yields less biased counterfactual forecasts in interrupted time series, and places the Texas abortion ban's excess documented pregnancies bet
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 12:33 UTC pith:D2CTQZRA
load-bearing objection A useful stacking extension for ITS with nuisance interruptions, but a possible temporal leakage in the stacking weights needs to be resolved before the method can be trusted. the 3 major comments →
Interrupted Time Series Analysis Of Count Data With Nuisance Interruptions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that stacking over a range of functional forms for a nuisance interruption's effect—instead of selecting or assuming one—makes counterfactual forecasts in interrupted time series less biased and better calibrated when a prior interruption overlaps the intervention period. The paper demonstrates this in simulations where the true nuisance effect is either inside or outside the candidate model space, and shows the stacked semi-parametric and fully parametric Poisson models reach nominal coverage in most settings, unlike the comparison methods. It further claims this holds for count outcomes modelled by Bayesian autoregressive Poisson/negative-binomial GLMs, and that the ap
What carries the argument
The load-bearing object is the Bayesian stack over a model space of nuisance-effect functional forms. Each sub-model is an autoregressive Poisson (or negative binomial) GLM for the count outcome; the 14 candidate forms run from 'no impact' and step/slope changes to exponential decay, pulse, triangle, and direct COVID-19 case/death covariates. The models are combined with weights maximizing the stacked log predictive density, estimated with Pareto-smoothed importance-sampling leave-future-out (PSIS-LFO) cross-validation on the pre-intervention series. In the application, the 14 nuisance forms cross a set of 7 trend/covariate specifications, giving 98 stacked sub-models; the cumulative interve
Load-bearing premise
The method assumes that at least one of the candidate functional forms for the nuisance interruption's effect is close to the true effect and that the weight learned on the pre-intervention data still describes the nuisance's effect after the intervention starts; if the pandemic's effect changes shape once the ban is in place, the counterfactual forecast is biased and the estimated intervention effect is invalid.
What would settle it
Re-run the simulation with a nuisance effect that is a temporary spike-and-recovery or a change point occurring after the intervention—shapes outside the 14 used here. If the stacked estimator's bias exceeds that of a correctly specified single model, or if its credible interval coverage drops well below nominal when the shift is unannounced, the central claim that stacking performs well in terms of bias and coverage collapses. A second, data-based check: fit the stacked model only up to a known post-pandemic recovery period, forecast forward, and compare the forecast to observed counts; syste
If this is right
- Policy evaluations in periods with overlapping disruptions (pandemics, natural disasters, prior reforms) can replace the 'pick one shape' step with a principled average over shapes, removing a source of bias.
- The stacking framework transfers to other outcome models (Gaussian, zero-inflated, negative binomial) and to the intervention effect itself, not just the nuisance effect.
- Counterfactual uncertainty is propagated honestly: in the Texas analysis the credible intervals widen as uncertainty accumulates, rather than being artificially narrowed by a single model.
- The simulation results imply that standard structural time-series forecasting can be biased and overconfident when a nuisance interruption is present, cautioning against its uncritical use in such settings.
- The Texas estimates give quantitative bounds—35,000–43,000 excess documented pregnancies and a 76–90% posterior probability of increase—that can inform debates about the ban's fertility impact, with the caveat of wide intervals.
Where Pith is reading between the lines
- A natural stress test would be to run the stacking procedure on data from a period where the pandemic effect is known to have changed shape (e.g., after vaccine rollout or a new variant wave) and check whether the pre-tuned weights still track the counterfactual; the paper's assumption that the nuisance shape persists is testable in this way.
- The method's reliance on the candidate model space suggests a practical rule for users: include deliberately wrong shapes, since the stacking weights will down-weight them, but a shape outside the space cannot be down-weighted because it was never considered.
- One could extend the stacking idea to average over autoregressive orders and over the intervention effect's shape (step vs. slope vs. pulse), turning most model-selection steps in interrupted time series into model-averaging steps.
- For reproductive-health policy, the wide credible intervals indicate that even the best counterfactual may not distinguish the ban's effect from pandemic-era noise; this suggests that claims of a precise excess-pregnancy count should be read as ranges with substantial uncertainty.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an interrupted time series (ITS) method for count outcomes when a nuisance interruption (e.g., the COVID-19 pandemic) precedes the intervention of interest. The authors introduce Bayesian stacking over a set of functional forms for the nuisance interruption, combined with semi-parametric or fully parametric autoregressive Poisson/negative binomial outcome models, and use posterior predictive counterfactuals to estimate cumulative intervention effects. The method is evaluated in a simulation study and applied to estimate excess documented pregnancies in Texas following the 2021 six-week abortion ban, using Epic Cosmos EHR data.
Significance. The problem is important and timely: nuisance interruptions with unknown shape can bias single-series ITS analyses, and existing practice often ignores them or relies on a single assumed functional form. Bayesian stacking over nuisance-effect shapes is a sensible and flexible approach. The simulation study uses relevant data-generating settings and reports frequentist criteria with Monte Carlo uncertainty, which is good practice. The application addresses a high-stakes policy question. However, the manuscript does not currently establish that the stacking weights are estimated only from pre-intervention information—a point that is load-bearing for the validity of the counterfactual. The coverage failure in one key simulation setting also tempers the broad claim of good coverage. Reproducibility is further limited because the code is only 'available upon request.'
major comments (3)
- [Section 3.4, Eq. (6)] The stacking objective in Eq. (6) is written as a sum over i = L to N - M, where N is the full series length, with no restriction excluding post-intervention periods. This directly conflicts with the Introduction's claim that stacking 'optimize[s] the out-of-sample predictiveness on pre-intervention data.' If the weights are optimized on post-intervention outcomes, then the semi-parametric counterfactual in Section 3.5 (Eq. 8) is not conditional only on pre-intervention information, and the simulation results do not validate the proposed forecasting procedure. Please either restrict the LFO-CV sum to i + M <= t_A (or i <= t_A - M) and restate Eq. (6), or verify in the code that the implementation truncates the sum at t_A and clarify the text accordingly.
- [Section 4, Table 2] For the 'complex' confounding scenario with beta_2 = 0.01 and alpha = 0, FP Stacking coverage is 0.424 (SE 0.022) and SP Stacking coverage is 0.355 (SE 0.022), far below nominal 95%. This is precisely the setting where the nuisance effect is outside the model space and the intervention is absent—arguably the scenario most relevant to the method's raison d'être. The paper acknowledges this as an exception, but the Discussion's statement that stacking 'reached nominal coverage across simulation scenarios' is not supported. Please present this failure more prominently and investigate its cause (e.g., weight instability, recursive AR counterfactual, or weak identifiability).
- [Section 3.3, Eq. (4), functional forms 5-7] The semi-parametric model is described as independent of the intervention, but the nuisance functional forms 'Step Ends' (5), 'Slope Ends' (6), and 'Step and Slope Ends' (7) include the intervention indicator A_t. When used in Eq. (4), the counterfactual forecast in the post-period depends on A_t, contradicting the definition of Y_t(0). If these forms are intended to model a nuisance effect that dissipates at t_A for reasons unrelated to the intervention, this should be stated explicitly; otherwise the semi-parametric counterfactual is misspecified and the estimated effect can be contaminated. The application uses these forms, so this is not merely a theoretical point.
minor comments (5)
- [Section 5.2 / Reproducibility] The paper states that R code is 'available upon request.' For a methods paper, a public repository with code and data (or simulated data) would substantially strengthen reproducibility and allow readers to verify the critical pre-intervention-weight assumption discussed above.
- [Section 4, paragraph 1] The simulation setup sentence 'with parameters = (2.2, 0.01, 2), , and 1 = 0' appears garbled; the parameter notation should be defined explicitly.
- [Section 3.4] The paper mentions both stacking (Eq. 6) and pseudo-BMA (Eq. 7) as weighting schemes, but it is not always clear which is used in the simulation and application. Please state the choice explicitly and justify it.
- [Figure 2] The forest plots are dense, especially with seven methods and six scenarios per panel. Larger fonts, explicit panel labels, or separate panels per criterion would improve readability.
- [Section 5.1] The discussion of address misclassification and changing Cosmos site composition is candid, but no sensitivity analysis is provided. A brief analysis using alternative address definitions or excluding possibly mobile patients would strengthen the application.
Circularity Check
Eq. (6)'s stacking objective runs over the full series (N-M), so post-intervention outcomes can enter the weights used for the semi-parametric counterfactual in Eq. (8), contaminating the 'prediction' with the intervention effect it is supposed to estimate.
specific steps
-
fitted input called prediction
[Section 3.4, Eq. (6); Section 3.5, Eq. (8)]
"For the stacking rule, the weights w = (w1, ..., wK) are the solution to the optimization problem, w = argmax_{w in S_1^K} 1/(N-M) sum_{i=L}^{N-M} log sum_{k=1}^K w_k p_k(y_{i+1:M}|y_{1:i}, M_k), (6) where S_1^K = {w in [0,1]^K : sum w_k = 1}, N is the full series length, M is the number of steps ahead to predict, L is the minimum series length to evaluate."
Eq. (6) sums i = L, ..., N-M over the full observed series, with no restriction to pre-intervention periods. For i with i+1 >= tA, the predictive term p_k(y_{i+1:M}|y_{1:i}) is scored against observed post-intervention outcomes Y_t(1). These stacking weights are used to select/average the nuisance-interruption models whose posterior predictive draws define the semi-parametric counterfactual Y_t^(q)(0) in Eq. (8). The counterfactual is claimed to be conditional only on pre-intervention data and covariates, but through w it can depend on post-intervention outcomes. The paper never truncates the LFO sum at i+M < tA. Thus the 'counterfactual forecast' is partly fitted to Y_t(1), the very intervention effect being estimated, so the simulation and application results do not demonstrate genuine p
full rationale
The remainder of the derivation is self-contained: the causal estimands follow Papadogeorgou et al. (external, no author overlap); stacking follows Yao et al. and Bürkner et al. (external); the simulation DGP is explicit and includes a complex confounding function outside the model space; priors are stated; there is no self-citation uniqueness argument. The only load-bearing circularity is the unresolved full-series LFO objective in Eq. (6). Because the paper's own equation defines the stacking weights over N-M (the full series) rather than tA-1-M, and because the semi-parametric counterfactual in Eq. (8) is described as 'given the pre-intervention outcome,' the manuscript as written permits post-intervention outcomes to enter the nuisance-model weights. If the implementation instead restricted the LFO sum to pre-intervention data, Eq. (6) and the surrounding text would be misleading; if not, the counterfactual prediction is contaminated by the intervention effect. This is a partial, conditional circularity of the fitted-input-called-prediction type; it does not reduce the whole method to a definition, so the score is 6 rather than higher.
Axiom & Free-Parameter Ledger
free parameters (2)
- Prior scale hyperparameter for overdispersion (θ) =
E(θ)=10 (Inverse-Gamma prior with hyper-prior)
- Autoregressive order P =
1 (assumed in application)
axioms (4)
- domain assumption Potential outcomes pre-intervention are unaffected by the intervention
- domain assumption Covariates are unaffected by the intervention, and conditional on covariates, the outcome series is stationary
- ad hoc to paper The nuisance interruption occurs strictly prior to the intervention and its effect is captured by one of the specified functional forms
- domain assumption Outcome follows an autoregressive Poisson or negative binomial GLM with log link and offset
read the original abstract
Interrupted time series analysis has been used to model the effect of policy and other interventions on public health by forecasting a counterfactual time series during the intervention period using data from prior to the intervention. However, due to typically relying on a single study unit, this approach risks not adjusting for other interruptions that precede and co-occur during the intervention period. The COVID-19 pandemic is a prominent example of this phenomenon of nuisance interruptions in contemporary public health research. To address this complication, we propose using Bayesian stacking over a range of functional forms for the impact of the nuisance interruption in order to make counterfactual forecasts for the intervention period. We used our proposed methods to estimate the impact of the 2021 Texas six-week abortion ban on documented pregnancies among women in Texas while adjusting for the impact of the COVID-19 pandemic.
Figures
Reference graph
Works this paper leans on
-
[1]
The prior scale hyperparameter’s expected value E() = is varied along the x-axis
while the bottom row sets r Exp (). The prior scale hyperparameter’s expected value E() = is varied along the x-axis. From Figure A1, we can see that while there is considerable variation in the posterior of r for each sub-model, the posteriors stay relatively consistent for the Inverse-Gamma prior. However, for the exponential priors, the posterior of r ...
2026
-
[2]
ISSN 0098-7484. doi: 10.1001/jama.2023.12034. Suzanne O. Bell, Alexander M. Franks, David Arbour, Selena Anjur-Dietrich, Elizabeth A. Stuart, Eli Ben-Michael, Avi Feller, and Alison Gemmill. US Abortion Bans and Fertility. JAMA, February
arXiv 2023
-
[6]
URL https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30120-1/fulltext
doi: 10.1016/S1473-3099(20)30120-1. URL https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30120-1/fulltext . Publisher: Elsevier PMID: 32087114. Jonah Gabry, Rok ˇCešnovar, and Andrew Johnson. cmdstanr: R Inferfacet to ’CmdStan’
-
[7]
20 A P REPRINT - 2026-07-15 Alison Gemmill, Claire E
URL https://mc- stan.org/cmdstanr/,https://discourse.mc-stan.org. 20 A P REPRINT - 2026-07-15 Alison Gemmill, Claire E. Margerison, Elizabeth A. Stuart, and Suzanne O. Bell. Infant Deaths After Texas’ 2021 Ban on Abortion in Early Pregnancy. JAMA Pediatrics, 178(8):784–791, August
2026
-
[9]
ISSN 0098-7484. doi: 10.1001/jama.2024.28517. Alison Gemmill, Alexander Franks, Avi Feller, Elizabeth A Stuart, Eli Ben-Michael, and Suzanne O Bell. Method- ological considerations for investigating the impact of abortion restrictions on outcomes using aggregate panel data. American Journal of Epidemiology , page kwag032, March
arXiv 2024
-
[11]
ISSN 0895-4356. doi: 10.1016/j.jclinepi.2023. 10.003. Rob J. Hyndman and Bahman Rostami-Tabar. Forecasting interrupted time series. Journal of the Operational Research Society, 0(0):1–14, September
-
[12]
doi: 10.1080/01605682.2024.2395315
ISSN 0160-5682. doi: 10.1080/01605682.2024.2395315. Kaitlyn Jackson, Deborah Karasek, Alison Gemmill, Daniel F. Collin, and Rita Hamad. Maternal Health During the COVID-19 Pandemic in the United States: An Interrupted Time-series Analysis. Epidemiology, 35(6):823, November
arXiv 2024
-
[13]
doi: 10.1097/EDE.0000000000001779
ISSN 1044-3983. doi: 10.1097/EDE.0000000000001779. Huan Jiang, Xinyang Feng, Shannon Lange, Alexander Tran, Jakob Manthey, and Jürgen Rehm. Estimating effects of health policy interventions using interrupted time-series analyses: A simulation study. BMC Medical Research Methodology, 22(1):235, August
-
[18]
URL https://doi.org/10.1007/s10742-022-00294-8
doi: 10.1007/s1 0742-022-00294-8. URL https://doi.org/10.1007/s10742-022-00294-8 . Taylor Riley, Elizabeth Nethery, Esther K. Chung, and Vivienne Souter. Impact of the COVID-19 pandemic on perinatal care and outcomes in the United States: An interrupted time series analysis. Birth, 49(2):298–309,
-
[19]
ISSN 1523-536X. doi: 10.1111/birt.12606. Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Edu- cational Psychology, 66(5):688–701,
-
[21]
doi: 10.1186/s12874-021-01235-8
ISSN 1471-2288. doi: 10.1186/s12874-021-01235-8. Amanda Jean Stevenson and Leslie Root. Trends in Maternal Death Post-Dobbs v Jackson Women’s Health. JAMA Network Open, 7(8):e2430035, August
-
[22]
doi: 10.1001/jamanetworkopen.2024.30035
ISSN 2574-3805. doi: 10.1001/jamanetworkopen.2024.30035. Y asir Tarabichi, Adam Frees, Steven Honeywell, Courtney Huang, Andrew M. Naidech, Jason H. Moore, and David C. Kaelber. The Cosmos Collaborative: A V endor-Facilitated Electronic Health Record Data Aggregation Platform. ACI Open, 05(01):e36–e46, January
arXiv 2024
-
[23]
ISSN 2566-9346. doi: 10.1055/s-0041-1731004. Stan Development Team. Stan Modeling Language Users Guide and Reference Manual
-
[24]
ISSN 0098-7484. doi: 10.1001/jama.2023.25599. Y uling Y ao, Aki V ehtari, Daniel Simpson, and Andrew Gelman. Using Stacking to Average Bayesian Predictive Distributions (with Discussion). Bayesian Analysis , 13(3):917–1007, September
arXiv 2023
-
[1974]
Place: US Publisher: American Psychological Association
doi: 10.1037/h0037350. Place: US Publisher: American Psychological Association. Andrea L. Schaffer, Timothy A. Dobbins, and Sallie-Anne Pearson. Interrupted time series analysis using autoregres- sive integrated moving average (ARIMA) models: A guide for evaluating large-scale health interventions. BMC Medical Research Methodology, 21(1):58, March
-
[2015]
ISSN 1932- 6157, 1941-7330. doi: 10.1214/14-AOAS788. Paul-Christian Bürkner, Jonah Gabry, and Aki V ehtari. Approximate leave-future-out cross-validation for Bayesian time series models. Journal of Statistical Computation and Simulation , 90(14):2499–2523, September
-
[2017]
ISSN 0300-5771, 1464-3685. doi: 10.1093/ije/dyw098. Fiammetta Menchetti, Fabrizio Cipollini, and Fabrizia Mealli. Estimating the causal effect of an intervention in a time series setting: The C-ARIMA approach, September
-
[2018]
ISSN 1936-0975, 1931-
1936
-
[2019]
URL https://onlinelibrary.wiley.com/doi/ abs/10.1002/sim.8086
doi: 10.1002/sim.8086. URL https://onlinelibrary.wiley.com/doi/ abs/10.1002/sim.8086. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.8086. Georgia Papadogeorgou, Fiammetta Menchetti, Christine Choirat, Jason H. Wasfy, Corwin M. Zigler, and Fabrizia Mealli. Evaluating federal policies using bayesian time series models: estimating the causal i...
doi:10.1002/sim.8086 2026
-
[2020]
doi: 10.1080/00949655.2020.1783262
ISSN 0094-9655. doi: 10.1080/00949655.2020.1783262. Paul-Christian Bürkner, Jonah Gabry, and Aki V ehtari. Approximate leave-future-out cross-validation for bayesian time series models, 07
arXiv 2020
-
[2021]
Morris, Ian R
Tim P . Morris, Ian R. White, and Michael J. Crowther. Using simulation studies to evaluate statistical methods. Statistics in Medicine , 38(11):2074–2102,
2074
-
[2022]
doi: 10.1186/s12874-022-01716-4
ISSN 1471-2288. doi: 10.1186/s12874-022-01716-4. James Lopez Bernal, Steven Cummins, and Antonio Gasparrini. Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology , 46(1):348–355,
-
[2023]
doi: 10.1016/j.ijforecast.2022.01.011
ISSN 0169-2070. doi: 10.1016/j.ijforecast.2022.01.011. Suzanne O. Bell, Elizabeth A. Stuart, and Alison Gemmill. Texas’ 2021 Ban on Abortion in Early Pregnancy and Changes in Live Births. JAMA, 330(3):281–282, July
-
[2024]
doi: 10.1001/jamapediatrics.2024.0885
ISSN 2168-6203. doi: 10.1001/jamapediatrics.2024.0885. Alison Gemmill, Alexander M. Franks, Selena Anjur-Dietrich, Amy Ozinsky, David Arbour, Elizabeth A. Stuart, Eli Ben-Michael, Avi Feller, and Suzanne O. Bell. US Abortion Bans and Infant Mortality. JAMA, February
arXiv 2024
-
[2025]
ISSN 0098-7484. doi: 10.1001/jama.2024.28527. Kay H. Brodersen, Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L. Scott. Inferring causal impact using Bayesian structural time-series models. The Annals of Applied Statistics , 9(1):247–274, March
arXiv 2024
-
[2026]
ISSN 0002-9262. doi: 10.1093/aje/kwag032. Christopher Gianacas, Bette Liu, Martyn Kirk, Gian Luca Di Tanna, Josephine Belcher, Suzanne Blogg, and David J. Muscatello. Bayesian structural time series, an alternative to interrupted time series in the right circumstances. Journal of Clinical Epidemiology , 163:102–110, November
-
[6690]
doi: 10.1214/17-BA1091. 22 A P REPRINT - 2026-07-15 A Appendix A.1 Asymptotic Confidence Intervals for Simulation Evaluation Criteria Let the true cumulative difference be T with posterior mean estimate m T ,i for estimation method m { BSTS, SP No Conf, FP No Conf, SP Step-Slope, FP Step-Slope, SP Stacking, FP Stacking } in the i-th simulation iteration. A...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.