Comment on "An implementation of neural simulation-based inference for parameter estimation in ATLAS''

Prasanth Shyamsundar

Comment on "An implementation of neural simulation-based inference for parameter estimation in ATLAS''

T0 review · 0 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Double bootstrap in the ATLAS NSBI analysis misses the Monte Carlo statistical uncertainty entirely.

desk verdict This comment is right: the ATLAS double-bootstrap only captures variance conditional on the original simulated dataset D, and the toy proof plus numerical check make the case cleanly. read the letter →

arxiv 2505.19156 v1 pith:2H625XKY submitted 2025-05-25 stat.ME hep-exhep-ph

Prasanth Shyamsundar This is my paper · ORCID

classification stat.MEhep-exhep-ph MSC 62F40

keywords doublebootstrapMonteCarlostatisticaluncertaintyneuralsimulation-basedinferencequantificationconditionalindependencetoymodelATLASmeasurement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This comment targets a specific uncertainty estimate in the ATLAS implementation of neural simulation-based inference: a double bootstrapping defined in Ref. [2] and used in the off-shell Higgs measurement Ref. [1]. The author argues that the second-level bootstrap, which resamples already-trained network estimators with replacement, only reproduces randomness that is conditional on one fixed simulated dataset, and therefore captures none of the Monte Carlo statistical uncertainty that comes from that dataset being finite. The supporting toy model, where the training noise is represented by independent normal perturbations added to bootstrap sample means, gives exact variance formulas showing the double-bootstrap error misses the $\sigma_X^2/N$ term; numerically it reports $3.2\times10^{-3}$ instead of the true $1.0\times10^{-1}$ standard deviation. A reader should care because the stated error bars in a published physics measurement would be too small if the claim is right.

What carries the argument

The load-bearing object is the two-level bootstrap built on a single fixed training set $D$: first $D$ is resampled $M$ times, one neural estimator is trained per resample, and then those $M$ estimators are resampled with replacement to form $K$ pseudo-ensembles. The argument turns on equations (6)-(8), which distinguish the joint distribution of the bootstrapped estimators, dependent through $D$, from the product distribution assumed by the second-level resampling; this is the point where the tangible bootstrap assumption of repeating the whole experiment fails. In the toy model the machinery is explicit: each $\hat\theta^{(boot,m)}$ equals a bootstrap sample mean plus an independent normal $\epsilon_m$, so the exact expected variances in (17a)-(17d) can be derived and the missing $\sigma_X^2/N$ term located.

What would settle it

Generate many independent copies of the training set $D$ (or of a faithful surrogate), run the full two-level bootstrap on each copy, and compare the across-copy standard deviation of the merged estimator with the median within-copy double-bootstrap uncertainty; the claim predicts a large factor, about $0.1/3.2\times10^{-3}\approx 30$ in the toy, while a correct procedure would show agreement. If the reported uncertainty instead tracks the across-copy spread, or does not shrink toward zero as $M$ grows with $N$ fixed, the comment's central claim would be refuted.

Watch

Extended reading notes

Core claim

The central claim, stated in Section 3, is that the double-bootstrapping uncertainty estimation in Ref. [2] "does not capture any of the mc-stat uncertainties" in the merged likelihood-ratio estimator and in the derived signal-strength parameter. The mechanism is that the $M$ first-level networks are trained on bootstrap resamplings of the same training set $D$, so they are dependent through $D$; the second-level bootstrap treats the resulting ensemble as independent and identically distributed, which is only correct conditional on $D$. The comment demonstrates the failure in a toy normal example: $\hat\theta^{(boot,m)}$ is a bootstrap sample mean plus independent training noise $\epsilon_m$, and the exact variance decomposition shows the expected double-bootstrap squared error equals the conditional variance and omits $\sigma_X^2/N$, the contribution from the finiteness of $D$. The same structural argument is claimed to apply to the full ATLAS pipeline because the merged averaging procedure is not repeated independently inside each pseudo-experiment.

Load-bearing premise

The conclusion that the same flaw affects the full ATLAS pipeline assumes that the toy model captures the structure of the real training: the trained networks are conditionally independent given the fixed simulated dataset, and the step from the merged likelihood-ratio estimate to the physics parameter introduces no further dependence on the dataset beyond what additive training noise represents.

Editorial extensions

If this is right

The double-bootstrap uncertainty scales to zero as the number of merged networks $M$ increases with $N$ fixed, so adding more networks cannot cure the missing Monte Carlo statistical uncertainty.
The alternative formula that averages the $M$ first-level estimates with a standard error also omits the $\sigma_X^2/N$ term, so the flaw is not specific to the second-level resampling choice.
If the ATLAS analysis has other uncertainties that dominate, the final measurement is likely unaffected, but the interpretation that mc-stat was estimated by the double bootstrap would be inaccurate.
Replacing the merging with a plain single-network-per-bootstrap procedure, or repeating the full merge inside every pseudo-experiment, would restore the missing term at a computational cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The conditional-versus-unconditional independence distinction gives a cheap diagnostic: any ensemble-based uncertainty estimate should be checked by resampling the training data at the outermost level and comparing the resulting spread with the within-ensemble spread.
Deep ensembles that vary only weight initialisation share the same training data, so by this same logic they understate data-sampling uncertainty even though they capture model stochasticity.
A natural next test is to rerun the toy model with correlated training noise across the $M$ networks; if the double-bootstrap error then fails to match even the conditional variance, that would strengthen the paper's caveat that the surrogate distribution itself must be checked.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This comment identifies a real methodological hole in the ATLAS NSBI implementation, and it does so with a clean toy proof rather than hand-waving. The new piece is the exact variance decomposition in Eqs. (17a)-(17d): the double-bootstrapped estimator's expected squared error equals the conditional variance given D, and misses the σ_X^2/N term that comes from the finiteness of the original simulated dataset. The numerical experiment confirms the algebra—about 3.2e-3 versus the true 1.0e-1—and the code is linked. That is reproducible evidence.

The structural argument is the stronger part. Once the ensemble E_boot is fixed, resampling it with replacement only produces variability among already-trained networks; it cannot produce a new draw of D. So the second bootstrap is estimating Var(·|D), not the full variance. The first bootstrap has the same issue in this procedure because the bootstrapped datasets are all derived from the same D and the merging step averages over them without repeating the whole experiment. I think the paper is correct on the central claim.

Soft spots are in proportion. The transfer from the toy to the real NSBI pipeline is argued by analogy, not by rerunning ATLAS's training or by a theorem that covers arbitrary training procedures. The toy assumes additive, D-independent training noise, and the paper acknowledges that the exact claim that mc-stat is completely missed would need qualification if, say, the μ-estimation step has D-dependent stochasticity beyond what the toy represents. But the conditioning argument does not actually rest on the additive-noise form; it rests on the fact that resampling E_boot is conditional on D. So I see this as a minor caveat rather than a load-bearing flaw.

The paper is appropriately modest about practical impact: the author notes ATLAS reported mc-stat as a small component, so the measurement is probably unaffected. That keeps the significance at methodological-precedent level rather than crisis level. For anyone building ensemble or bootstrap-based uncertainty estimates for simulation-based inference, this is exactly the kind of warning that should be on record.

I would take this paper seriously. The math is checkable, the code is public, the claim is clear, and it identifies a failure mode that future analyses need to design around. Send it to a capable referee; it deserves a proper review rather than a desk rejection.

Referee Report

0 major / 4 minor

Summary. The manuscript is a comment on the ATLAS NSBI implementation paper (arXiv:2412.01600). It argues that the double-bootstrapping procedure used there to estimate uncertainties arising from the finiteness of the simulated dataset (mc-stat) is flawed: the second-level bootstrap resamples the fixed ensemble of trained networks, so it can only quantify the dispersion of the merged estimator conditional on the original dataset D, and cannot capture the component of variance that arises from the randomness of D itself. The argument is developed in three layers: a conceptual discussion of the assumptions behind bootstrapping (Section 3), an exact toy-model variance decomposition (Section 4 and Appendix A), and a numerical simulation. The toy model exhibits exactly the claimed behavior: the double-bootstrap and naive standard-error estimates are about 3.2e-3, while the true standard deviation of the merged estimator is about 1.0e-1.

Significance. If correct, the comment identifies a genuine and important methodological gap in a flagship NSBI application. The flaw is not specific to the normal toy model: the structural reason—resampling a fixed ensemble cannot reintroduce the sampling variance of D—is independent of the noise model. The paper is appropriately careful in scope: it notes that the ATLAS analysis may be unaffected because the reported mc-stat contribution is small, and it suggests several concrete fixes. Strengths include the closed-form variance formulas in Appendix A, which are derived carefully and match the numerical experiment, and the availability of the code. The main limitation is that the transfer to the full ATLAS pipeline is argued structurally rather than by re-running that pipeline; this is a reasonable level of evidence for a comment, and the structural argument is convincing.

minor comments (4)

[Section 2] Equation (6) introduces the distribution p(·|Eboot) for the second-level bootstrap; the notation would be clearer if it explicitly stated that the draws are independent and identically distributed from the empirical distribution over the M trained functions in Eboot, rather than from a generic surrogate distribution, since this point is central to the subsequent comparison with Eq. (7).
[Section 2] There is a typo: "subscipt" should be "subscript". In addition, the spacing in expressions such as "Eboot-boot ,k" and "r(boot-boot ,k,m)" is irregular and should be cleaned up before publication.
[Section 4 and Appendix A] The exact variance decomposition in Eq. (17a) separates the D-sampling term sigma_X^2/N from the resampling and training-noise terms. The discussion of "mc-stat" would be sharper if the text explicitly identified the first term with the finiteness-of-D uncertainty and clarified that the middle term in Eq. (17a), although it originates from bootstrapping D, is a conditional-on-D resampling effect rather than a statistical uncertainty from the finiteness of D.
[Section 5] The sentence "if sigma_epsilon^2 / M >> sigma_X^2 / N" states a sufficient condition without defining a threshold; the qualitative conclusion is clear, but a more precise inequality (or a statement that the comparison is meant only heuristically) would avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the comment's claim is established by an independent variance decomposition and a fresh numerical simulation, not by importing the target paper's conclusions.

full rationale

The paper's central claim—that the double-bootstrapping procedure of Ref. [2] misses mc-stat uncertainties—is derived from first principles in Appendix A and demonstrated with a standalone toy simulation. The derivation starts from explicit assumptions (iid Xi, independent training errors, resampling with replacement) and computes the relevant variances via Bienaymé identities and the law of total variance; the resulting formulas (17a)–(17d) are internal to the comment. The numerical experiment uses independently specified parameters θ = 5.0, σX = 100, σϵ = 0.01, N = 10^6, M = 10^3, K = 10^4 and compares against the analytic formulas, not against any output of the ATLAS paper. The only soft point, the transfer from the toy model to the full ATLAS pipeline, is explicitly acknowledged in the concluding remarks and is a matter of modeling fidelity, not circularity. No fitted parameter is renamed as a prediction, no load-bearing self-citation occurs, and no uniqueness theorem is imported from the authors' prior work. The analysis is therefore self-contained with respect to the claim it makes.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on an analytical variance decomposition for a constructed toy model and on a structural analogy to the NSBI training pipeline. The key unverified inputs are the additive-noise model for training randomness and the conditional-independence assumption for real networks; neither is fitted to ATLAS data, and the paper's conclusion does not depend on the exact toy parameter values.

free parameters (1)

sigma_epsilon (training-noise scale) = 0.01 (illustrative, not fitted)
Introduced in Eq. (10) to mimic random neural-network training errors as iid additive Gaussian noise. The analytical conclusion that the double-bootstrap drops the sigma_X^2/N term does not require any particular value; the value is used only in the numerical illustration.

assumptions (5)

domain assumption The original training data D is an iid sample of size N from a fixed distribution with finite variance.
Section 4 defines the toy model this way; this is what makes sigma_X^2/N the dataset-level variance that the double-bootstrap misses.
domain assumption Pseudo-experiments in a bootstrap must repeat the original experiment in full, including the drawing of a new primary dataset (the paper's Assumption I).
Section 3 uses this assumption to argue that second-level resampling of the ensemble does not replicate the original data-generation step.
ad hoc to paper Neural-network training randomness can be represented as independent additive noise, independent of D and of the bootstrap resampling.
Eq. (10) models training error as epsilon_m; the paper does not prove this holds for actual deep-network training, but the structural argument only needs training randomness to be independent across runs given D.
domain assumption The trained likelihood-ratio estimators are mutually independent conditional on D.
Eq. (8) states this independence for the estimators given D; it is plausible when bootstrap samples and training random seeds are independent, but is asserted rather than verified for the actual ATLAS networks.
standard math Standard identities of probability: Bienayme's covariance identities and the law of total variance/covariance.
Used throughout Appendix A to derive Eqs. (17a)-(17d).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Comment on "An implementation of neural simulation-based inference for parameter estimation in ATLAS''." pith.science (2026). https://pith.science/paper/2H625XKY

@misc{pith2026250519156,
  author       = {Pith},
  title        = {Pith review of: Comment on "An implementation of neural simulation-based inference for parameter estimation in ATLAS''},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2H625XKY}},
  note         = {Machine review of arXiv:2505.19156}
}

read the original abstract

The paper titled "An implementation of neural simulation-based inference for parameter estimation in ATLAS" by the ATLAS collaboration (arXiv:2412.01600v1 [hep-ex]) describes the implementation of neural simulation-based inference for a measurement analysis performed by ATLAS. The uncertainties in the analysis arising from the finiteness of the simulated datasets are estimated using a novel double-bootstrapping technique described in that work. In the present comment, it is claimed and demonstrated, using a toy example, that the double-bootstrapping technique does not actually capture the aforementioned uncertainties.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages

[1]

ATLAS Collaboration, “Measurement of off-shell Higgs boson production in the H ∗ → ZZ → 4ℓ decay channel using a neural simulation-based inference technique in 13 TeV pp collisions with the ATLAS detector,” Rept. Prog. Phys.88, no.5, 057803 (2025) doi:10.1088/1361-6633/adcd9a [arXiv:2412.01548v2 [hep-ex]]

arXiv 2025
[2]

An implementation of neural simulation-based inference for parameter estimation in ATLAS,

ATLAS Collaboration, “An implementation of neural simulation-based inference for parameter estimation in ATLAS,” arXiv:2412.01600v1 [hep-ex]

arXiv
[3]

Recommendations for the Modeling of Smooth Backgrounds,

ATLAS Collaboration, “Recommendations for the Modeling of Smooth Backgrounds,” ATL- PHYS-PUB-2020-028, https://cds.cern.ch/record/2743717/

arXiv 2020
[4]

Dark Energy Survey Year 3 results: likelihood-free, simulation-based wCDM inference with neural compression of weak-lensing map statistics,

N. Jeffrey et al. [DES], “Dark Energy Survey Year 3 results: likelihood-free, simulation-based wCDM inference with neural compression of weak-lensing map statistics,” Mon. Not. Roy. Astron. Soc. 536, no.2, 1303-1322 (2025) doi:10.1093/mnras/stae2629 [arXiv:2403.02314 [astro- ph.CO]]

arXiv 2025
[5]

Python reference manual,

G. Van Rossum and F. L. Drake Jr, “Python reference manual,” Centrum voor Wiskunde en Informatica Amsterdam (1995)

work page 1995
[6]

Array programming with NumPy,

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus et al., “Array programming with NumPy,” Nature 585 (7825), 357 (2020) doi:10.1038/s41586-020-2649-2 [arXiv:2006.10256 [cs.MS]]. A Derivations for the toy example As a preface, Bienaym´ e’s identities, given...

arXiv 2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

ATLAS Collaboration, “Measurement of off-shell Higgs boson production in the H ∗ → ZZ → 4ℓ decay channel using a neural simulation-based inference technique in 13 TeV pp collisions with the ATLAS detector,” Rept. Prog. Phys.88, no.5, 057803 (2025) doi:10.1088/1361-6633/adcd9a [arXiv:2412.01548v2 [hep-ex]]

arXiv 2025

[2] [2]

An implementation of neural simulation-based inference for parameter estimation in ATLAS,

ATLAS Collaboration, “An implementation of neural simulation-based inference for parameter estimation in ATLAS,” arXiv:2412.01600v1 [hep-ex]

arXiv

[3] [3]

Recommendations for the Modeling of Smooth Backgrounds,

ATLAS Collaboration, “Recommendations for the Modeling of Smooth Backgrounds,” ATL- PHYS-PUB-2020-028, https://cds.cern.ch/record/2743717/

arXiv 2020

[4] [4]

Dark Energy Survey Year 3 results: likelihood-free, simulation-based wCDM inference with neural compression of weak-lensing map statistics,

N. Jeffrey et al. [DES], “Dark Energy Survey Year 3 results: likelihood-free, simulation-based wCDM inference with neural compression of weak-lensing map statistics,” Mon. Not. Roy. Astron. Soc. 536, no.2, 1303-1322 (2025) doi:10.1093/mnras/stae2629 [arXiv:2403.02314 [astro- ph.CO]]

arXiv 2025

[5] [5]

Python reference manual,

G. Van Rossum and F. L. Drake Jr, “Python reference manual,” Centrum voor Wiskunde en Informatica Amsterdam (1995)

work page 1995

[6] [6]

Array programming with NumPy,

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus et al., “Array programming with NumPy,” Nature 585 (7825), 357 (2020) doi:10.1038/s41586-020-2649-2 [arXiv:2006.10256 [cs.MS]]. A Derivations for the toy example As a preface, Bienaym´ e’s identities, given...

arXiv 2020