REVIEW 2 major objections 5 minor 37 references
Improving the performance of weak supervision searches using data augmentation
T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Data augmentation halves the signal data needed for weak-supervision searches.
desk verdict Useful empirical result, but the missing augmented sculpting check could undermine the headline threshold reduction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The working mechanism is label-preserving data augmentation applied to jet images before pixelation. pT smearing resamples each jet constituent's transverse momentum from a normal distribution whose width is the detector energy-resolution function, $\sqrt{0.05^2 p_T^2 + 1.50^2 p_T}$ (with $p_T$ in GeV), so the network sees realistic detector fluctuations; jet rotation rotates the (η, φ) coordinates of each jet about its center by an independent random angle in [−π, π], exposing the network to the rotational diversity of jet substructure. These transformations enlarge the training set and force the classifier to learn signal/background differences that are robust to detector effects and orientation. The paper also relies on event normalization of the jet images to remove $m_{jj}$ dependence from the inputs, which prevents the classifier from sculpting a fake signal by learning the region definitions.
What would settle it
Train the augmented CWoLa classifier on background-only data across the same signal and sideband regions, applying the same pT-smearing and jet-rotation augmentation. If the NN cut efficiency shows a statistically significant dependence on mjj beyond Monte-Carlo fluctuations, the augmentation breaks the mjj-independence assumption and the reported sensitivity gains may be partly spurious.
Extended reading notes
Core claim
The central claim is that data augmentation converts a weak-supervision classifier from a data-hungry method into a practical one. In the CWoLa framework, a neural network is trained on dijet events from a signal region and a sideband defined by the dijet invariant mass, each containing a mixture of signal and QCD background. The paper shows that by augmenting the training jet images with pT smearing (resampling constituent momenta with the detector resolution function) and jet rotation (rotating each jet by an independent random angle in [−π, π]), the network reaches the same post-cut sensitivity with roughly half the pre-cut signal sensitivity. Specifically, the learning threshold drops from around 6σ to around 3σ for both the indirect-decay and direct-decay Hidden Valley scenarios, and the standard deviation of the sensitivity across 10 retrainings is reduced to about half. The combination of both augmentations outperforms either alone, and the benefit persists when a 1% relative background systematic uncertainty is included.
Load-bearing premise
The method relies on the premise that, after event normalization, the jet-image features are statistically independent of the dijet invariant mass mjj; if residual mjj dependence survives or is introduced by augmentation, the classifier could fake a signal by learning the region definitions.
Editorial extensions
If this is right
- Weak-supervision searches can be carried out with roughly half the integrated luminosity or signal yield previously needed, making them viable earlier in an LHC run.
- Physics-informed augmentation that respects detector resolution and rotational symmetry is a general lever for improving classifier generalization in collider anomaly detection.
- The reduction in training variance means a single trained network is more reliable, so searches can trust the sensitivity estimate from one training run.
- Because performance saturates after about +30 augmented copies, practitioners can choose a modest augmentation factor and avoid diminishing returns from larger datasets.
- The method's robustness to a 1% background systematic suggests it can be combined with standard background-estimation uncertainties in a real search.
Reading between the lines
- The same augmentation recipe could likely be adapted to other weak-supervision or anomaly-detection setups that use jet images, such as autoencoder-based outlier detection, as long as the transformation preserves the label.
- The rotation augmentation effectively builds rotational invariance into the network without adding a custom architecture, implying that other known symmetries of jet images might be injected similarly.
- Since the paper's benchmark is a single resonance topology, a natural testable extension is whether the threshold reduction holds for multi-pronged or boosted-object signals, where rotation diversity may matter even more.
- Combining data augmentation with pre-training or transfer-learning approaches could push the learning threshold even lower; this stacking is not tested here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes physics-inspired data augmentation as a way to reduce the amount of signal data needed for weakly supervised (CWoLa) searches. Using a simulated Hidden Valley dijet resonance benchmark at 13 TeV, the authors compare pT smearing, jet rotation, and their combination against the un-augmented pipeline in terms of ROC-based sensitivity curves with error bars from ten retrainings. They report that the learning threshold is reduced from roughly 6σ to 3σ, that the combined method performs best, that performance saturates as the number of augmented copies grows, and that the gains survive a 1% background systematic uncertainty.
Significance. If valid, the result would be a simple and practical improvement for resonant anomaly searches: it lowers the data requirement of CWoLa without changing the search strategy or relying on external simulation labels. The paper has several strengths: it uses a concrete benchmark, reports mean and standard deviations from repeated training, compares with full supervision, and explicitly discusses limiting behavior. The central technical risk is that the augmentation may break the mjj-independence assumption on which CWoLa rests, and the manuscript does not currently rule this out; the headline threshold reduction is therefore not yet fully established.
major comments (2)
- [Sec. 4.2 and Sec. 5.1 (Eq. 5.1)] The sculpting check in Sec. 4.2 is performed only on the un-augmented jet-image pipeline. The pT smearing of Eq. (5.1) is applied to constituents before preprocessing and event normalization, and the smearing width f(pT) depends on pT. Since SR and SB events have different mjj distributions and hence different jet pT spectra, the augmented feature distributions can in principle retain a residual mjj dependence even when the un-augmented images are mjj-independent. If the classifier exploits that dependence, it can separate the SR and SB training sets and inflate the ROC-based sensitivities in Figs. 5–7. Because the best-performing method (pT-rot) includes pT smearing, the central claim rests on this untested assumption. Please repeat the background-only sculpting test of Sec. 4.2 for the augmented pipelines (at least +5 and +20 pT-rot), and report whether the mjj distributions and selection efficiencies are flat; if sculpting appears, quantify how much of the threshold reduction survives after removing it.
- [Sec. 4.3 and Figs. 3, 5–8] The headline claim of a reduction from roughly 6σ to 3σ relies on the notion of a learning threshold, but the paper gives only a qualitative definition in the Introduction and appears to read thresholds from the crossing of curves in the figures. Please specify the exact extraction rule used to define the threshold, including whether the crossing is evaluated on the mean curve or on individual retrainings, how interpolation is performed, and how the standard deviation is propagated. Without such a rule, the central quantitative claim is not precisely reproducible.
minor comments (5)
- [Sec. 5.1, Eq. (5.2)] Equation (5.2) appears to contain a typo: the second coordinate should presumably read η′ sin θ + φ′ cos θ rather than η′ sin θ + φ′ sin θ.
- [Sec. 5.4, Fig. 8] The systematic-uncertainty study uses a relative background uncertainty of 1%, while the text notes that the typical value is 5%; since the compression is stronger at larger uncertainty, the statement that augmentation remains beneficial under systematic uncertainty would be better supported by also showing the 5% case.
- [Sec. 3.2] The statistical treatment of the repeated trainings could be clarified: it is not stated whether the 10 retrainings reuse the same test sample, and how much of the quoted standard deviation is due to training variability versus finite test statistics.
- [Sec. 5.1] For pT smearing, the manuscript does not state how negative resampled pT values are handled; a brief sentence on clipping or truncation would remove ambiguity.
- [General] No code or data release is mentioned; providing the training and preprocessing pipeline would strengthen reproducibility of the numerical claims.
Circularity Check
No significant circularity: the CWoLa data-augmentation study is empirical, with training/test separation and no parameter fitted to the reported sensitivity curves.
full rationale
The paper's central claim is that physics-inspired data augmentation (pT smearing and jet rotation) lowers the CWoLa learning threshold from roughly 6σ to 3σ. This claim is supported by independent testing data, multiple retrainings, and comparisons against the un-augmented CWoLa baseline within the same simulation setup; it does not reduce to the inputs by construction. The only self-citation is ref. [11] (Beauchesne, Chen, Chiang), used to fix Hidden Valley benchmark parameters: 'As in reference [11], we consider two scenarios to study the effect of data augmentation. In both scenarios, Λ_D is set to 10 GeV, and other Hidden Valley module parameters in Pythia are set to be the same as in table 1(a) of reference [11].' This is a choice of benchmark example, not a load-bearing theorem or a mechanism that forces the reported improvement. The Neyman-Pearson/CWoLa justification is cited to external work [5,6,7], and the augmentation methods are presented as inspired by, not derived from, the current authors' prior results. The possible sculpting effect from pT smearing (Eq. 5.1) introducing residual mjj dependence after EN normalization is a validity/correctness concern, not a circularity: the paper does not fit the augmented model to the test outputs, and the skeptical mechanism is not an equation-level reduction of the claimed result to its inputs. No circular step is therefore identified.
Assumptions & free parameters
free parameters (3)
- Jet rotation angle range =
[-pi, pi]
- Number of augmented copies =
+5 for main result; also +10, +20
- Relative background systematic uncertainty =
1%
assumptions (6)
- domain assumption Hidden Valley model parameters (Z' mass 5.5 TeV, width 10 GeV, dark meson masses) are taken from prior literature.
- domain assumption Input jet-image features are independent of mjj after normalization.
- domain assumption Delphes CMS detector simulation and the pT smearing function in Eq. (5.1) accurately model detector resolution.
- domain assumption Simulated samples faithfully represent real LHC data.
- standard math Neyman-Pearson lemma and the CWoLa theory from ref. [5] are valid.
- domain assumption The chosen neural network architecture and hyperparameters are sufficient to learn the discriminating features.
Cite this review
Pith. "Pith review of Improving the performance of weak supervision searches using data augmentation." pith.science (2026). https://pith.science/paper/WGBXM6EQ
@misc{pith2026241200198,
author = {Pith},
title = {Pith review of: Improving the performance of weak supervision searches using data augmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WGBXM6EQ}},
note = {Machine review of arXiv:2412.00198}
}
abstract
Weak supervision combines the advantages of training on real data with the ability to exploit signal properties. However, training a neural network using weak supervision often requires an excessive amount of signal data, which severely limits its practical applicability. In this study, we propose addressing this limitation through data augmentation, increasing the training data's size and diversity. Specifically, we focus on physics-inspired data augmentation methods, such as $p_{\text{T}}$ smearing and jet rotation. Our results demonstrate that data augmentation can significantly enhance the performance of weak supervision, enabling neural networks to learn efficiently from substantially less data.
Reference graph
Works this paper leans on
-
[1]
Topological Obstructions to Autoencoding,
J. Batson, C. G. Haaf, Y. Kahn, and D. A. Roberts, “Topological Obstructions to Autoencoding,” JHEP 04 (2021) 280, arXiv:2102.08380 [hep-ph]
arXiv 2021
-
[2]
Searching for New Physics with Deep Autoencoders,
M. Farina, Y. Nakai, and D. Shih, “Searching for New Physics with Deep Autoencoders,” Phys. Rev. D101 no. 7, (2020) 075021, arXiv:1808.08992 [hep-ph]
arXiv 2020
-
[3]
Dijet resonance search with weak supervision using√s = 13 TeV pp collisions in the ATLAS detector,
ATLAS Collaboration, G. Aad et al., “Dijet resonance search with weak supervision using√s = 13 TeV pp collisions in the ATLAS detector,” Phys. Rev. Lett.125 no. 13, (2020) 131801, arXiv:2005.02983 [hep-ex]
arXiv 2020
-
[4]
CMS Collaboration, “Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at √s= 13 TeV,” tech. rep., CERN, Geneva, 2024. https://cds.cern.ch/record/2892677
-
[5]
Classification without labels: Learning from mixed samples in high energy physics,
E. M. Metodiev, B. Nachman, and J. Thaler, “Classification without labels: Learning from mixed samples in high energy physics,” JHEP 10 (2017) 174, arXiv:1708.02949 [hep-ph]
arXiv 2017
-
[6]
On the Problem of the Most Efficient Tests of Statistical Hypotheses,
J. Neyman and E. S. Pearson, “On the Problem of the Most Efficient Tests of Statistical Hypotheses,” Phil. Trans. Roy. Soc. Lond. A231 no. 694-706, (1933) 289–337
work page 1933
-
[7]
Anomaly Detection for Resonant New Physics with Machine Learning,
J. H. Collins, K. Howe, and B. Nachman, “Anomaly Detection for Resonant New Physics with Machine Learning,” Phys. Rev. Lett.121 no. 24, (2018) 241803, arXiv:1805.02664 [hep-ph]
arXiv 2018
-
[8]
Comparing weak- and unsupervised methods for resonant anomaly detection,
J. H. Collins, P. Mart ´ ın-Ramiro, B. Nachman, and D. Shih, “Comparing weak- and unsupervised methods for resonant anomaly detection,” Eur. Phys. J. C81 no. 7, (2021) 617, arXiv:2104.02092 [hep-ph]
arXiv 2021
Show all 37 references
-
[9]
Tree-based algorithms for weakly supervised anomaly detection,
T. Finke, M. Hein, G. Kasieczka, M. Kr¨ amer, A. M¨ uck, P. Prangchaikul, T. Quadfasel, D. Shih, and M. Sommerhalder, “Tree-based algorithms for weakly supervised anomaly detection,” Phys. Rev. D109 no. 3, (2024) 034033, arXiv:2309.13111 [hep-ph]
2024 arXiv
-
[10]
Anomaly detection in the presence of irrelevant features,
M. Freytsis, M. Perelstein, and Y. C. San, “Anomaly detection in the presence of irrelevant features,” JHEP 02 (2024) 220, arXiv:2310.13057 [hep-ph]. – 16 –
2024 arXiv
-
[11]
Improving the performance of weak supervision searches using transfer and meta-learning,
H. Beauchesne, Z.-E. Chen, and C.-W. Chiang, “Improving the performance of weak supervision searches using transfer and meta-learning,” JHEP 02 (2024) 138, arXiv:2312.06152 [hep-ph]
2024 arXiv
-
[12]
Incorporating Physical Priors into Weakly-Supervised Anomaly Detection,
C. L. Cheng, G. Singh, and B. Nachman, “Incorporating Physical Priors into Weakly-Supervised Anomaly Detection,” arXiv:2405.08889 [hep-ph]
-
[13]
Accelerating Resonance Searches via Signature-Oriented Pre-training,
C. Li et al., “Accelerating Resonance Searches via Signature-Oriented Pre-training,” arXiv:2405.12972 [hep-ph]
-
[14]
Data Augmentation at the LHC through Analysis-specific Fast Simulation with Deep Learning,
C. Chen, O. Cerri, T. Q. Nguyen, J.-R. Vlimant, and M. Pierini, “Data Augmentation at the LHC through Analysis-specific Fast Simulation with Deep Learning,” arXiv:2010.01835 [physics.comp-ph]
2010 arXiv
-
[15]
Metalearning and data augmentation for mass-generalized jet taggers,
M. J. Dolan and A. Ore, “Metalearning and data augmentation for mass-generalized jet taggers,” Phys. Rev. D105 no. 9, (2022) 094030, arXiv:2111.06047 [hep-ph]
2022 arXiv
-
[16]
Mass Agnostic Jet Taggers,
L. Bradshaw, R. K. Mishra, A. Mitridate, and B. Ostdiek, “Mass Agnostic Jet Taggers,” SciPost Phys. 8 no. 1, (2020) 011, arXiv:1908.08959 [hep-ph]
2020 arXiv
-
[17]
Extensive Studies of the Neutron Star Equation of State from the Deep Learning Inference with the Observational Data Augmentation,
Y. Fujimoto, K. Fukushima, and K. Murase, “Extensive Studies of the Neutron Star Equation of State from the Deep Learning Inference with the Observational Data Augmentation,” JHEP 03 (2021) 273, arXiv:2101.08156 [nucl-th]
2021 arXiv
-
[18]
Improving Photometric Redshift Estimates with Training Sample Augmentation,
LSST Dark Energy Science Collaboration, I. Moskowitz, E. Gawiser, J. F. Crenshaw, B. H. Andrews, A. I. Malz, and S. Schmidt, “Improving Photometric Redshift Estimates with Training Sample Augmentation,” Astrophys. J. Lett.967 no. 1, (2024) L6, arXiv:2402.15551 [astro-ph.IM]
2024 arXiv
-
[19]
Discerning Secluded Sector gauge structures,
L. Carloni, J. Rathsman, and T. Sjostrand, “Discerning Secluded Sector gauge structures,” JHEP 04 (2011) 091, arXiv:1102.3795 [hep-ph]
2011 arXiv
-
[20]
Visible Effects of Invisible Hidden Valley Radiation,
L. Carloni and T. Sjostrand, “Visible Effects of Invisible Hidden Valley Radiation,” JHEP 09 (2010) 105, arXiv:1006.2911 [hep-ph]
2010 arXiv
-
[21]
Dark matter in Hidden Valley models with stable and unstable light dark mesons,
H. Beauchesne, E. Bertuzzo, and G. Grilli Di Cortona, “Dark matter in Hidden Valley models with stable and unstable light dark mesons,” JHEP 04 (2019) 118, arXiv:1809.10152 [hep-ph]
2019 arXiv
-
[22]
Theory, phenomenology, and experimental avenues for dark showers: a Snowmass 2021 report,
G. Albouy et al., “Theory, phenomenology, and experimental avenues for dark showers: a Snowmass 2021 report,” Eur. Phys. J. C82 no. 12, (2022) 1132, arXiv:2203.09503 [hep-ph]
2022 arXiv
-
[23]
An introduction to PYTHIA 8.2,
T. Sj¨ ostrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, “An introduction to PYTHIA 8.2,” Comput. Phys. Commun. 191 (2015) 159–177, arXiv:1410.3012 [hep-ph]
2015 arXiv
-
[24]
Parton distributions with LHC data,
R. D. Ball et al., “Parton distributions with LHC data,” Nucl. Phys. B867 (2013) 244–289, arXiv:1207.1303 [hep-ph]
2013 arXiv
-
[25]
The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations,
J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, “The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations,” JHEP ...
2014 arXiv
-
[26]
DELPHES 3, A modular framework for fast – 17 – simulation of a generic collider experiment,
DELPHES 3 Collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lema ˆ ıtre, A. Mertens, and M. Selvaggi, “DELPHES 3, A modular framework for fast – 17 – simulation of a generic collider experiment,” JHEP 02 (2014) 057, arXiv:1307.6346 [hep-ex]
2014 arXiv
-
[27]
FastJet User Manual,
M. Cacciari, G. P. Salam, and G. Soyez, “FastJet User Manual,” Eur. Phys. J. C72 (2012) 1896, arXiv:1111.6097 [hep-ph]
2012 arXiv
-
[28]
The anti- kt jet clustering algorithm,
M. Cacciari, G. P. Salam, and G. Soyez, “The anti- kt jet clustering algorithm,” JHEP 04 (2008) 063, arXiv:0802.1189 [hep-ph]
2008 arXiv
-
[29]
The Machine Learning landscape of top taggers,
A. Butter et al., “The Machine Learning landscape of top taggers,” SciPost Phys. 7 (2019) 014, arXiv:1902.09914 [hep-ph]
2019 arXiv
-
[30]
Jet-images — deep learning edition,
L. de Oliveira, M. Kagan, L. Mackey, B. Nachman, and A. Schwartzman, “Jet-images — deep learning edition,” JHEP 07 (2016) 069, arXiv:1511.05190 [hep-ph]
2016 arXiv
-
[31]
Deep-learning Top Taggers or The End of QCD?,
G. Kasieczka, T. Plehn, M. Russell, and T. Schell, “Deep-learning Top Taggers or The End of QCD?,” JHEP 05 (2017) 006, arXiv:1701.08784 [hep-ph]
2017 arXiv
-
[32]
Chollet et al., “Keras.” https://keras.io, 2015
F. Chollet et al., “Keras.” https://keras.io, 2015
2015
-
[33]
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,
M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” arXiv:1603.04467 [cs.DC]
-
[34]
Formulae for Estimating Significance,
ATLAS Collaboration, “Formulae for Estimating Significance,” 2020
2020
-
[35]
A comprehensive survey on data augmentation,
Z. Wang, P. Wang, K. Liu, P. Wang, Y. Fu, C.-T. Lu, C. C. Aggarwal, J. Pei, and Y. Zhou, “A comprehensive survey on data augmentation,” 2024. https://arxiv.org/abs/2405.09591
2024
-
[36]
Anomalies, representations, and self-supervision,
B. M. Dillon, L. Favaro, F. Feiden, T. Modak, and T. Plehn, “Anomalies, representations, and self-supervision,” SciPost Phys. Core7 (2024) 056, arXiv:2301.04660 [hep-ph]
2024 arXiv
-
[37]
Search for high mass dijet resonances with a new background prediction method in proton-proton collisions at √s = 13 TeV,
CMS Collaboration, A. M. Sirunyan et al., “Search for high mass dijet resonances with a new background prediction method in proton-proton collisions at √s = 13 TeV,” JHEP 05 (2020) 033, arXiv:1911.03947 [hep-ex]. – 18 –
2020 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.