REVIEW 4 major objections 5 minor 32 references
Machine learning method for enforcing variable independence in background estimation with LHC data: ABCDisCoTEC
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read ABCDisCoTEC adds a differentiable closure term to the ABCDisCo loss, so neural-network training directly minimizes the ABCD background-estimation error, yielding decorrelated discriminants with improved sensitivity in a stealth…
desk verdict Useful extension of ABCDisCo that adds a differentiable closure loss and MDMM; the main idea works for the demonstrated case, but the surrogate's fidelity is under-validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing piece is the nonclosure loss term, $L_{\mathrm{nonclosure}} = \left(\frac{N_A N_D - N_B N_C}{N_A N_D + N_B N_C}\right)^2$, which measures how far the ABCD prediction $N_B N_C/N_D$ is from the observed count $N_A$. Because event counts are discrete, the paper replaces hard counting with a two-dimensional sigmoid, $\sigma(S_1,S_2,b_1,b_2)=1/[(1+e^{-a(S_1-b_1)})(1+e^{-a(S_2-b_2)})]$ with scale $a=100$ and boundaries chosen randomly per batch, so that gradients can flow through the ABCD geometry. The full loss combines binary cross-entropy for classification, distance correlation for independence, and this closure term; the modified differential method of multipliers then treats the decorrelation and closure losses as constraints with learnable multipliers, which stabilizes the training and gives the hyperparameters a physical meaning.
What would settle it
Train ABCDisCoTEC on a simulated sample with known signal and background, then measure the hard-boundary nonclosure on an independent test set across a fine grid of ABCD boundary choices and compare it with the smoothed nonclosure loss evaluated at the same boundaries; a weak or reversed correlation between the two would show that the smooth surrogate is not controlling the quantity it claims to control.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that one can train a neural network to minimize the ABCD nonclosure directly, rather than hoping that minimizing distance correlation between two outputs is enough. The nonclosure loss is $L_{\mathrm{nonclosure}} = \left(\frac{N_A N_D - N_B N_C}{N_A N_D + N_B N_C}\right)^2$, and it becomes differentiable when hard event counts are replaced by a two-dimensional sigmoid weighting with scale $a=100$ and randomly chosen boundaries per batch. Adding this term to the binary cross-entropy and distance-correlation losses yields two decorrelated discriminants with strong signal-background separation. In the paper's stealth supersymmetry case, the combined loss gives lower average nonclosure and higher normalized significance than either the distance-correlation or the closure term alone, and the resulting background estimates show good agreement between simulation and observed data. The accompanying use of MDMM turns the subordinate losses into constrained objectives with learnable multipliers, which stabilizes training and lets the analyst set physically meaningful targets such as a 10% nonclosure.
Load-bearing premise
The load-bearing premise is that minimizing the smooth, sigmoid-weighted version of the nonclosure during training actually reduces the true discrete nonclosure at the boundaries used later; if that surrogate is unfaithful, the claimed background accuracy does not follow.
Editorial extensions
If this is right
- Any analysis that relies on the ABCD method can in principle replace hand-selected independent variables with two learned discriminants trained to control nonclosure directly.
- Smaller nonclosure translates into smaller systematic uncertainty in the background prediction, which directly improves discovery significance for searches where signal and background look similar.
- MDMM converts loss weights into interpretable constraints, so an analyst can target a specific nonclosure value rather than scanning dimensionless hyperparameters.
- The sub-ABCD validation procedure (VR I, VR II, VR III) provides a way to test the method in observed data even when no orthogonal validation region exists.
- The same sigmoid-relaxation trick can be applied to extended ABCD formulations with additional control regions, as the paper notes.
Reading between the lines
- Beyond the paper: because the smooth nonclosure is only a surrogate, the method's success for a new analysis should be checked by validating the correlation between the smoothed loss and the final hard-boundary nonclosure at the chosen boundaries.
- Beyond the paper: random boundary sampling during training effectively averages the closure constraint over many possible ABCD partitions, which may make the learned discriminants more uniformly decorrelated and could be studied as an implicit regularizer.
- Beyond the paper: the differentiable-counting trick is not specific to high-energy physics and could be reused wherever a ratio of region counts is optimized with gradient descent, for example in anomaly detection or survey analyses.
- Beyond the paper: when the Pareto front is strongly nonconvex, the advantage of MDMM over grid search should be larger than in the convex stealth-supersymmetry example, so a synthetic benchmark with a known nonconvex front would quantify the benefit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ABCDisCoTEC, an extension of the ABCDisCo method for background estimation in LHC searches. The key addition is a differentiable nonclosure loss term that replaces hard ABCD region boundaries with a two-dimensional sigmoid relaxation and samples boundaries randomly per training batch. The method is applied to a stealth supersymmetry search using CMS simulation and data, and is compared with variants using only the distance-correlation loss, only the closure loss, both, and with a lambda hyperparameter scan versus the modified differential method of multipliers (MDMM). The validation uses held-out test samples, simulation control regions, and data-driven validation regions constructed inside the ABCD plane.
Significance. If the method works as claimed, it provides a practical way to train classifiers that satisfy the ABCD background-estimation relation across a broad range of boundaries, which is directly useful for many LHC searches. The paper includes strong validation elements: separate test samples, comparisons of loss variants, and a data-based validation-region study in Section 6. The MDMM comparison, if reproducible, is a useful contribution to multiobjective training in this context. The main limitation is that the central claim of 'directly minimizing the nonclosure' relies on a heuristic sigmoid surrogate whose fidelity is not demonstrated, and the reproducibility is limited by missing hyperparameter details. The algebraic issue in Eq. (6) also needs correction.
major comments (4)
- [2.2, Eq. (6)] The statement that Eq. (6) is the nonclosure loss 'in terms of the explicit nonclosure definition' is algebraically incorrect for r = N_B N_C/(N_A N_D) > 1. When r > 1, C/ = r-1, but Eq. (6) gives ((r-1)/(3-r))^2, which differs from Eq. (5) and is non-monotonic; for example at r=2, Eq. (6) equals 1 while Eq. (5) equals 1/9. The equality holds only for r <= 1. Since Eq. (5) is the loss actually used, this does not invalidate the training, but the equivalence claim in the text should be corrected or restricted.
- [2.2, Eqs. (5)-(8)] The central claim that minimizing the sigmoid-relaxed L_nonclosure directly minimizes the hard-boundary nonclosure of Eq. (2) is not established. The soft loss weights every event fractionally in all four regions and is minimized over randomly sampled boundaries, so it is a different functional from the hard-boundary nonclosure used in validation. The paper states a=100 was 'found to provide the best closure performance in general' but shows no fidelity check, no sensitivity scan over a, and no characterization of the boundary sampling distribution. I request a direct comparison: on a fixed test set, plot the soft loss against the hard nonclosure at several training checkpoints, and report the hard nonclosure for a = 10, 30, 100, 300. Without this, the phrase 'directly minimizes the nonclosure' overstates what is demonstrated.
- [4.1] It is not specified whether L_nonclosure and L_DisCo are evaluated on background events only, on signal events only, or on the full batch. This matters because the ABCD relation in Eq. (1) is a statement about background events and does not hold for signal, and Fig. 6 shows signal deliberately concentrated in region A. If the nonclosure loss were computed on the full batch including signal, it would penalize the desired signal topology. The training description should state unambiguously which event classes enter each loss term.
- [5.2, Eqs. (9)-(11)] The MDMM implementation is not described with enough detail to be reproducible or to support the claimed advantages. The paper does not quote the values or update schedules of the Lagrange multipliers alpha_i, the damping factors c_i, or the constraint targets epsilon_DisCo, nor the number of trainings used in Fig. 14. Since the MDMM comparison is a principal result highlighted in the abstract, these implementation details should be provided in a table or appendix.
minor comments (5)
- [2.2] The statement 'a=100 was found to provide the best closure performance in general' should be supported by a sensitivity study; as written it is an unexplained empirical choice.
- [4.1] Final training hyperparameters, including the lambda values, learning-rate schedule, number of epochs, and early-stopping criterion, are not listed; add a table with the values used for the final models.
- [4.2] The sentence 'All other hyperparameters are set to optimal values for each training configuration' is vague; specify the performance criteria used to determine optimality.
- [6, Fig. 16] The quoted 3-15% systematic uncertainty from Ref. [8] is mentioned but no uncertainty band is shown in Fig. 16; indicate the band or refer the reader to the companion paper for the exact procedure.
- [4.2, Eq. (13)] The significance formula is a rough approximation; clarify that N_bkg is the background event count and that the nonclosure term is intended to be added in quadrature as a systematic uncertainty.
Circularity Check
No circular derivation: the nonclosure loss is a training objective evaluated out-of-sample; the only self-citation is a non-load-bearing companion reference.
full rationale
The derivation chain is not circular. The claimed contribution is a differentiable relaxation of the ABCD nonclosure metric: Eq. (5) defines the loss from the same four-region counts as Eq. (2), and Eq. (8) replaces hard Heaviside counting with a sigmoid weight. This is a surrogate objective, not a relabeled prediction. The reported nonclosure is measured on the held-out test sample (Fig. 12) and in data validation regions defined inside the B, C, and D control regions (Section 6, Fig. 16), so it is an out-of-sample check rather than the training loss itself. No fitted parameter is renamed as a physics prediction; the background estimate Npred_A = N_B N_C / N_D is evaluated on held-out counts. The only internal reference is the companion CMS paper [8], cited for the physics results (3-15% systematic uncertainty, improved limits) and not as evidence for the methodological claim; the method validation is self-contained in this paper. The skeptic's concern that the sigmoid surrogate (Eq. 7 with a=100 and random boundaries) may not faithfully track hard nonclosure is a robustness/correctness risk, not a circularity, because the paper does not assume the equivalence; it empirically demonstrates low hard nonclosure on data not used for training. Similarly, Eq. (6) is only algebraically equivalent to Eq. (5) when N_A N_D > N_B N_C, and the mapping from epsilon to 10% nonclosure in Section 2.3 is questionable; these are technical caveats, not input-output circularity.
Assumptions & free parameters
free parameters (7)
- Sigmoid scale a =
100
- Nonclosure loss weight lambda_nonclosure =
Scanned 1-1000; optimal value not quoted for final models
- DisCo loss weight lambda_DisCo =
Scanned 5000-50000 (DisCo-only) and 10-100 (combined); not quoted for final models
- MDMM constraint epsilon_nonclosure =
0.1 in the example
- MDMM damping factor c_i =
Not specified
- Batch size =
4096
- Learning rate =
1e-4 to 1e-5
assumptions (4)
- standard math Theorem 3 of Ref. [5]: a distance correlation of zero implies statistical independence.
- domain assumption The ABCD relation N_A_pred = N_B N_C / N_D holds when the two variables are independent for background events.
- domain assumption The simulated event samples (POWHEG, MadGraph, PYTHIA, Geant4) accurately represent the background and signal processes.
- domain assumption The constrained optimization reformulation (Eq. 9-11) with BCE as objective and DisCo/nonclosure as constraints is a valid way to navigate the Pareto front.
Cite this review
Pith. "Pith review of Machine learning method for enforcing variable independence in background estimation with LHC data: ABCDisCoTEC." pith.science (2026). https://pith.science/paper/IJPAVLDA
@misc{pith2026250608826,
author = {Pith},
title = {Pith review of: Machine learning method for enforcing variable independence in background estimation with LHC data: ABCDisCoTEC},
year = {2026},
howpublished = {\url{https://pith.science/paper/IJPAVLDA}},
note = {Machine review of arXiv:2506.08826}
}
read the original abstract
A novel solution is presented for the problem of estimating the backgrounds of a signal search using observed data while simultaneously maximizing the sensitivity of the search to the signal. The ``ABCD method'' provides a reliable framework for background estimation by partitioning events into one signal-enhanced region (A) and three background-enhanced control regions (B, C, and D) via two statistically independent variables. In practice, even slight correlations between the two variables can significantly undermine the method's performance. Thus, choosing appropriate variables by hand can present a formidable challenge, especially when background and signal differ only subtly. To address this issue, the ABCD with distance correlation (ABCDisCo) method was developed to construct two artificial variables from the output scores of a neural network trained to maximize signal-background discrimination while minimizing correlations using the distance correlation measure. However, relying solely on minimizing the distance correlation can yield undesirable characteristics in the resulting distributions, which may compromise the validity of the background prediction obtained using this method. The ABCDisCo training enhanced with closure (ABCDisCoTEC) method is introduced to solve this issue by directly minimizing the nonclosure, expressed as a dedicated differentiable loss term. This extended method is applied to a data set of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the CMS detector at the CERN LHC. Additionally, given the complexity of the minimization problem with constraints on multiple loss terms, the modified differential method of multipliers is applied and shown to greatly improve the stability and robustness of the ABCDisCoTEC method, compared to grid search hyperparameter optimization procedures.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[1]
Search for High Mass Top Quark Production in p anti-p Collisions at S**(1/2) = 1.8 TeV
D0 Collaboration, “Search for high mass top quark production inp pcollisions at√s=1.8 TeV”,Phys. Rev. Lett.74(1995) 2422, doi:10.1103/PhysRevLett.74.2422,arXiv:hep-ex/9411001
work page Pith review arXiv 1995
-
[2]
Data analysis in high energy physics: a practical guide to statistical methods
O. Behnke, K. Kr ¨oninger, G. Schott, and T. Sch¨orner-Sadenius, eds., “Data analysis in high energy physics: a practical guide to statistical methods”. Wiley-VCH, Weinheim, Germany, 2013.doi:10.1002/9783527653416
-
[3]
Background estimation with the ABCD method featuring the TRooFit toolkit
W. Buttinger, “Background estimation with the ABCD method featuring the TRooFit toolkit”, 2018.https://api.semanticscholar.org/CorpusID:235806829
work page 2018
-
[4]
Automating the ABCD method with machine learning
G. Kasieczka, B. Nachman, M. D. Schwartz, and D. Shih, “Automating the ABCD method with machine learning”,Phys. Rev. D103(2021) 035021, doi:10.1103/PhysRevD.103.035021,arXiv:2007.14400
arXiv 2021
-
[5]
Measuring and testing dependence by correlation of distances
G. J. Sz ´ekely, M. L. Rizzo, and N. K. Bakirov, “Measuring and testing dependence by correlation of distances”,Ann. Stat.35(2007) 2769, doi:10.1214/009053607000000505,arXiv:0803.4101
arXiv 2007
-
[6]
The CMS experiment at the CERN LHC
CMS Collaboration, “The CMS experiment at the CERN LHC”,JINST3(2008) S08004, doi:10.1088/1748-0221/3/08/S08004
-
[7]
L. Evans and P . Bryant, “LHC Machine”,JINST3(2008) S08001, doi:10.1088/1748-0221/3/08/S08001
-
[8]
CMS Collaboration, “Search for top squarks in final states with many light-flavor jets and 0, 1, or 2 charged leptons in proton-proton collisions at √s=13 TeV”, 2025. Submitted to JHEP
work page 2025
Show all 32 references
-
[9]
Constrained differential optimization
J. Platt and A. Barr, “Constrained differential optimization”, inNIPS’87: Proceedings of the 1st International Conference on Neural Information Processing Systems. MIT Press, Cambridge, MA, USA, 1987
1987
-
[10]
Note on regression and inheritance in the case of two parents
K. Pearson, “Note on regression and inheritance in the case of two parents”,Proceedings of the Royal Society of London58(1895) 240
-
[11]
Visualizing the Pareto frontier
A. V . Lotov and K. Miettinen, “Visualizing the Pareto frontier”, p. 213. Springer Berlin Heidelberg, 2008.doi:10.1007/978-3-540-88908-3_9
2008 doi
-
[12]
Stealth supersymmetry
J. Fan, M. Reece, and J. T. Ruderman, “Stealth supersymmetry”,JHEP11(2011) 012, doi:10.1007/JHEP11(2011)012,arXiv:1105.5135. References 27
2011 arXiv
-
[13]
A stealth supersymmetry sampler
J. Fan, M. Reece, and J. T. Ruderman, “A stealth supersymmetry sampler”,JHEP07 (2012) 196,doi:10.1007/JHEP07(2012)196,arXiv:1201.4875
2012 arXiv
-
[14]
Stealth supersymmetry simplified
J. Fan et al., “Stealth supersymmetry simplified”,JHEP07(2016) 016, doi:10.1007/JHEP07(2016)016,arXiv:1512.05781
2016 arXiv
-
[15]
Search for top squarks in final states with two top quarks and several light-flavor jets in proton-proton collisions at √s=13 TeV
CMS Collaboration, “Search for top squarks in final states with two top quarks and several light-flavor jets in proton-proton collisions at √s=13 TeV”,Phys. Rev. D104 (2021) 032006,doi:10.1103/PhysRevD.104.032006,arXiv:2102.06976
2021 arXiv
-
[16]
Chollet et al., “Keras”, 2015.https://keras.io
F. Chollet et al., “Keras”, 2015.https://keras.io
2015
-
[17]
TensorFlow: A system for large-scale machine learning
M. Abadi et al., “TensorFlow: A system for large-scale machine learning”, inProceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI’16, p. 265. USENIX Association, Savannah, GA, USA, 2016.arXiv:1605.08695
2016 arXiv
-
[18]
PyTorch: an imperative style, high-performance deep learning library
A. Paszke et al., “PyTorch: an imperative style, high-performance deep learning library”, inNIPS’19: 33rd International Conference on Neural Information, p. 721. Curran Associates Inc., 2019.arXiv:1912.01703
2019 arXiv
-
[19]
Adam: A method for stochastic optimization
D. P . Kingma and J. Ba, “Adam: A method for stochastic optimization”, 2014. arXiv:1412.6980
2014 arXiv
-
[20]
A new method for combining NLO QCD with shower Monte Carlo algorithms
P . Nason, “A new method for combining NLO QCD with shower Monte Carlo algorithms”,JHEP11(2004) 040,doi:10.1088/1126-6708/2004/11/040, arXiv:hep-ph/0409146
2004 arXiv
-
[21]
Matching NLO QCD computations with parton shower simulations: the POWHEG method
S. Frixione, P . Nason, and C. Oleari, “Matching NLO QCD computations with parton shower simulations: the POWHEG method”,JHEP11(2007) 070, doi:10.1088/1126-6708/2007/11/070,arXiv:0709.2092
2007 arXiv
-
[22]
A general framework for implementing NLO calculations in shower Monte Carlo programs: the POWHEG BOX
S. Alioli, P . Nason, C. Oleari, and E. Re, “A general framework for implementing NLO calculations in shower Monte Carlo programs: the POWHEG BOX”,JHEP06(2010) 043, doi:10.1007/JHEP06(2010)043,arXiv:1002.2581
2010 arXiv
-
[23]
A positive-weight next-to-leading-order Monte Carlo for heavy flavour hadroproduction
S. Frixione, P . Nason, and G. Ridolfi, “A positive-weight next-to-leading-order Monte Carlo for heavy flavour hadroproduction”,JHEP09(2007) 126, doi:10.1088/1126-6708/2007/09/126,arXiv:0707.3088
2007 arXiv
-
[24]
Top++: A program for the calculation of the top-pair cross-section at hadron colliders
M. Czakon and A. Mitov, “Top++: A program for the calculation of the top-pair cross-section at hadron colliders”,Comput. Phys. Commun.185(2014) 2930, doi:10.1016/j.cpc.2014.06.021,arXiv:1112.5675
2014 arXiv
-
[25]
The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations
J. Alwall et al., “The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations”,JHEP07 (2014) 079,doi:10.1007/JHEP07(2014)079,arXiv:1405.0301
2014 arXiv
-
[26]
Squark and gluino production cross sections in pp collisions at√s=13, 14, 33 and 100 TeV
C. Borschensky et al., “Squark and gluino production cross sections in pp collisions at√s=13, 14, 33 and 100 TeV”,Eur. Phys. J. C74(2014) 3174, doi:10.1140/epjc/s10052-014-3174-y,arXiv:1407.5066
2014 arXiv
-
[27]
NNLL-fast: predictions for coloured supersymmetric particle production at the LHC with threshold and Coulomb resummation
W. Beenakker et al., “NNLL-fast: predictions for coloured supersymmetric particle production at the LHC with threshold and Coulomb resummation”,JHEP12(2016) 133, doi:10.1007/JHEP12(2016)133,arXiv:1607.07741. 28
2016 arXiv
-
[28]
An introduction to PYTHIA 8.2
T. Sj ¨ostrand et al., “An introduction to PYTHIA 8.2”,Comput. Phys. Commun.191(2015) 159,doi:10.1016/j.cpc.2015.01.024,arXiv:1410.3012
2015 arXiv
-
[29]
Parton distributions from high-precision collider data
NNPDF Collaboration, “Parton distributions from high-precision collider data”,Eur. Phys. J. C77(2017) 663,doi:10.1140/epjc/s10052-017-5199-5, arXiv:1706.00428
2017 arXiv
-
[30]
Extraction and validation of a new set of CMS PYTHIA8 tunes from underlying-event measurements
CMS Collaboration, “Extraction and validation of a new set of CMS PYTHIA8 tunes from underlying-event measurements”,Eur. Phys. J. C80(2020) 4, doi:10.1140/epjc/s10052-019-7499-4,arXiv:1903.12179
2020 arXiv
-
[31]
GEANT4—a simulation toolkit
GEANT4 Collaboration, “GEANT4—a simulation toolkit”,Nucl. Instrum. Meth. A506 (2003) 250,doi:10.1016/S0168-9002(03)01368-8
2003 doi
-
[32]
Improved extrapolation methods of data-driven background estimations in high energy physics
S. Choi and H. Oh, “Improved extrapolation methods of data-driven background estimations in high energy physics”,Eur. Phys. J. C81(2021) 643, doi:10.1140/epjc/s10052-021-09404-1,arXiv:1906.10831. 29 A The CMS Collaboration Yerevan Physics Institute, Yerevan, Armenia A. Hayrape...
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.