Pith. sign in

REVIEW 2 major objections 5 minor 37 references

Improving the performance of weak supervision searches using data augmentation

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Data augmentation halves the signal data needed for weak-supervision searches.

desk verdict Useful empirical result, but the missing augmented sculpting check could undermine the headline threshold reduction. read the letter →

arxiv 2412.00198 v1 pith:WGBXM6EQ submitted 2024-11-29 hep-ph cs.LGhep-ex

classification hep-phcs.LGhep-ex
keywords weaksupervisiondataaugmentationCWoLajetimagesphysics-inspiredHiddenValleymodellearningthresholddijetresonancesearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Weak supervision trains a classifier on mixed signal-enriched and signal-depleted samples, but it typically needs so many signal events that it is not competitive with traditional searches. This paper asks whether physics-inspired data augmentation—smearing jet transverse momenta to mimic detector resolution and rotating jet images by random angles—can lower that requirement. Using a Hidden Valley benchmark with jet-image inputs, the authors report that adding five augmented copies per event reduces the learning threshold from about 6σ to about 3σ in both benchmark scenarios, and cuts the training-to-training variance in half. The combined augmentation method works best, and the improvement survives a 1% background systematic uncertainty.

What carries the argument

The working mechanism is label-preserving data augmentation applied to jet images before pixelation. pT smearing resamples each jet constituent's transverse momentum from a normal distribution whose width is the detector energy-resolution function, $\sqrt{0.05^2 p_T^2 + 1.50^2 p_T}$ (with $p_T$ in GeV), so the network sees realistic detector fluctuations; jet rotation rotates the (η, φ) coordinates of each jet about its center by an independent random angle in [−π, π], exposing the network to the rotational diversity of jet substructure. These transformations enlarge the training set and force the classifier to learn signal/background differences that are robust to detector effects and orientation. The paper also relies on event normalization of the jet images to remove $m_{jj}$ dependence from the inputs, which prevents the classifier from sculpting a fake signal by learning the region definitions.

What would settle it

Train the augmented CWoLa classifier on background-only data across the same signal and sideband regions, applying the same pT-smearing and jet-rotation augmentation. If the NN cut efficiency shows a statistically significant dependence on mjj beyond Monte-Carlo fluctuations, the augmentation breaks the mjj-independence assumption and the reported sensitivity gains may be partly spurious.

Watch

Extended reading notes

Core claim

The central claim is that data augmentation converts a weak-supervision classifier from a data-hungry method into a practical one. In the CWoLa framework, a neural network is trained on dijet events from a signal region and a sideband defined by the dijet invariant mass, each containing a mixture of signal and QCD background. The paper shows that by augmenting the training jet images with pT smearing (resampling constituent momenta with the detector resolution function) and jet rotation (rotating each jet by an independent random angle in [−π, π]), the network reaches the same post-cut sensitivity with roughly half the pre-cut signal sensitivity. Specifically, the learning threshold drops from around 6σ to around 3σ for both the indirect-decay and direct-decay Hidden Valley scenarios, and the standard deviation of the sensitivity across 10 retrainings is reduced to about half. The combination of both augmentations outperforms either alone, and the benefit persists when a 1% relative background systematic uncertainty is included.

Load-bearing premise

The method relies on the premise that, after event normalization, the jet-image features are statistically independent of the dijet invariant mass mjj; if residual mjj dependence survives or is introduced by augmentation, the classifier could fake a signal by learning the region definitions.

Editorial extensions

If this is right

  • Weak-supervision searches can be carried out with roughly half the integrated luminosity or signal yield previously needed, making them viable earlier in an LHC run.
  • Physics-informed augmentation that respects detector resolution and rotational symmetry is a general lever for improving classifier generalization in collider anomaly detection.
  • The reduction in training variance means a single trained network is more reliable, so searches can trust the sensitivity estimate from one training run.
  • Because performance saturates after about +30 augmented copies, practitioners can choose a modest augmentation factor and avoid diminishing returns from larger datasets.
  • The method's robustness to a 1% background systematic suggests it can be combined with standard background-estimation uncertainties in a real search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same augmentation recipe could likely be adapted to other weak-supervision or anomaly-detection setups that use jet images, such as autoencoder-based outlier detection, as long as the transformation preserves the label.
  • The rotation augmentation effectively builds rotational invariance into the network without adding a custom architecture, implying that other known symmetries of jet images might be injected similarly.
  • Since the paper's benchmark is a single resonance topology, a natural testable extension is whether the threshold reduction holds for multi-pronged or boosted-object signals, where rotation diversity may matter even more.
  • Combining data augmentation with pre-training or transfer-learning approaches could push the learning threshold even lower; this stacking is not tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The manuscript proposes physics-inspired data augmentation as a way to reduce the amount of signal data needed for weakly supervised (CWoLa) searches. Using a simulated Hidden Valley dijet resonance benchmark at 13 TeV, the authors compare pT smearing, jet rotation, and their combination against the un-augmented pipeline in terms of ROC-based sensitivity curves with error bars from ten retrainings. They report that the learning threshold is reduced from roughly 6σ to 3σ, that the combined method performs best, that performance saturates as the number of augmented copies grows, and that the gains survive a 1% background systematic uncertainty.

Significance. If valid, the result would be a simple and practical improvement for resonant anomaly searches: it lowers the data requirement of CWoLa without changing the search strategy or relying on external simulation labels. The paper has several strengths: it uses a concrete benchmark, reports mean and standard deviations from repeated training, compares with full supervision, and explicitly discusses limiting behavior. The central technical risk is that the augmentation may break the mjj-independence assumption on which CWoLa rests, and the manuscript does not currently rule this out; the headline threshold reduction is therefore not yet fully established.

major comments (2)
  1. [Sec. 4.2 and Sec. 5.1 (Eq. 5.1)] The sculpting check in Sec. 4.2 is performed only on the un-augmented jet-image pipeline. The pT smearing of Eq. (5.1) is applied to constituents before preprocessing and event normalization, and the smearing width f(pT) depends on pT. Since SR and SB events have different mjj distributions and hence different jet pT spectra, the augmented feature distributions can in principle retain a residual mjj dependence even when the un-augmented images are mjj-independent. If the classifier exploits that dependence, it can separate the SR and SB training sets and inflate the ROC-based sensitivities in Figs. 5–7. Because the best-performing method (pT-rot) includes pT smearing, the central claim rests on this untested assumption. Please repeat the background-only sculpting test of Sec. 4.2 for the augmented pipelines (at least +5 and +20 pT-rot), and report whether the mjj distributions and selection efficiencies are flat; if sculpting appears, quantify how much of the threshold reduction survives after removing it.
  2. [Sec. 4.3 and Figs. 3, 5–8] The headline claim of a reduction from roughly 6σ to 3σ relies on the notion of a learning threshold, but the paper gives only a qualitative definition in the Introduction and appears to read thresholds from the crossing of curves in the figures. Please specify the exact extraction rule used to define the threshold, including whether the crossing is evaluated on the mean curve or on individual retrainings, how interpolation is performed, and how the standard deviation is propagated. Without such a rule, the central quantitative claim is not precisely reproducible.
minor comments (5)
  1. [Sec. 5.1, Eq. (5.2)] Equation (5.2) appears to contain a typo: the second coordinate should presumably read η′ sin θ + φ′ cos θ rather than η′ sin θ + φ′ sin θ.
  2. [Sec. 5.4, Fig. 8] The systematic-uncertainty study uses a relative background uncertainty of 1%, while the text notes that the typical value is 5%; since the compression is stronger at larger uncertainty, the statement that augmentation remains beneficial under systematic uncertainty would be better supported by also showing the 5% case.
  3. [Sec. 3.2] The statistical treatment of the repeated trainings could be clarified: it is not stated whether the 10 retrainings reuse the same test sample, and how much of the quoted standard deviation is due to training variability versus finite test statistics.
  4. [Sec. 5.1] For pT smearing, the manuscript does not state how negative resampled pT values are handled; a brief sentence on clipping or truncation would remove ambiguity.
  5. [General] No code or data release is mentioned; providing the training and preprocessing pipeline would strengthen reproducibility of the numerical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CWoLa data-augmentation study is empirical, with training/test separation and no parameter fitted to the reported sensitivity curves.

full rationale

The paper's central claim is that physics-inspired data augmentation (pT smearing and jet rotation) lowers the CWoLa learning threshold from roughly 6σ to 3σ. This claim is supported by independent testing data, multiple retrainings, and comparisons against the un-augmented CWoLa baseline within the same simulation setup; it does not reduce to the inputs by construction. The only self-citation is ref. [11] (Beauchesne, Chen, Chiang), used to fix Hidden Valley benchmark parameters: 'As in reference [11], we consider two scenarios to study the effect of data augmentation. In both scenarios, Λ_D is set to 10 GeV, and other Hidden Valley module parameters in Pythia are set to be the same as in table 1(a) of reference [11].' This is a choice of benchmark example, not a load-bearing theorem or a mechanism that forces the reported improvement. The Neyman-Pearson/CWoLa justification is cited to external work [5,6,7], and the augmentation methods are presented as inspired by, not derived from, the current authors' prior results. The possible sculpting effect from pT smearing (Eq. 5.1) introducing residual mjj dependence after EN normalization is a validity/correctness concern, not a circularity: the paper does not fit the augmented model to the test outputs, and the skeptical mechanism is not an equation-level reduction of the claimed result to its inputs. No circular step is therefore identified.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the CWoLa framework, a simulated benchmark, and hand-chosen augmentation parameters; no new entities are introduced. The free parameters are the augmentation settings and the illustrative uncertainty level.

free parameters (3)
  • Jet rotation angle range = [-pi, pi]
    Chosen by hand after testing smaller ranges; larger ranges improve performance, so this choice influences the reported gains.
  • Number of augmented copies = +5 for main result; also +10, +20
    The main claim uses +5 augmentation; the paper explores +10 and +20 in the asymptotic study and shows saturation around +30.
  • Relative background systematic uncertainty = 1%
    Used in Section 5.4; the text notes the typical uncertainty is 5%. This is a favorable choice that understates the impact of systematics.
assumptions (6)
  • domain assumption Hidden Valley model parameters (Z' mass 5.5 TeV, width 10 GeV, dark meson masses) are taken from prior literature.
    Section 2 sets the benchmark according to references [11,22]; the conclusions about augmentation are specific to this model, though the method is model-agnostic.
  • domain assumption Input jet-image features are independent of mjj after normalization.
    Section 3.3 states this assumption; the paper tests it on background-only data (Section 4.2), but residual mjj dependence would bias the classifier.
  • domain assumption Delphes CMS detector simulation and the pT smearing function in Eq. (5.1) accurately model detector resolution.
    The augmentation uses the smearing function from Delphes; if the real detector resolution differs, the augmented samples could misrepresent true events.
  • domain assumption Simulated samples faithfully represent real LHC data.
    The entire study is simulation-based; real detector data may contain additional artifacts not captured by the simulation.
  • standard math Neyman-Pearson lemma and the CWoLa theory from ref. [5] are valid.
    Used to justify that a classifier trained on mixed samples is optimal for signal/background discrimination.
  • domain assumption The chosen neural network architecture and hyperparameters are sufficient to learn the discriminating features.
    This is an empirical choice; a different architecture might change the learning threshold, though the qualitative effect of augmentation likely persists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the performance of weak supervision searches using data augmentation." pith.science (2026). https://pith.science/paper/WGBXM6EQ

@misc{pith2026241200198,
  author       = {Pith},
  title        = {Pith review of: Improving the performance of weak supervision searches using data augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGBXM6EQ}},
  note         = {Machine review of arXiv:2412.00198}
}
abstract

Weak supervision combines the advantages of training on real data with the ability to exploit signal properties. However, training a neural network using weak supervision often requires an excessive amount of signal data, which severely limits its practical applicability. In this study, we propose addressing this limitation through data augmentation, increasing the training data's size and diversity. Specifically, we focus on physics-inspired data augmentation methods, such as $p_{\text{T}}$ smearing and jet rotation. Our results demonstrate that data augmentation can significantly enhance the performance of weak supervision, enabling neural networks to learn efficiently from substantially less data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 5 canonical work pages

  1. [1]

    Topological Obstructions to Autoencoding,

    J. Batson, C. G. Haaf, Y. Kahn, and D. A. Roberts, “Topological Obstructions to Autoencoding,” JHEP 04 (2021) 280, arXiv:2102.08380 [hep-ph]

  2. [2]

    Searching for New Physics with Deep Autoencoders,

    M. Farina, Y. Nakai, and D. Shih, “Searching for New Physics with Deep Autoencoders,” Phys. Rev. D101 no. 7, (2020) 075021, arXiv:1808.08992 [hep-ph]

  3. [3]

    Dijet resonance search with weak supervision using√s = 13 TeV pp collisions in the ATLAS detector,

    ATLAS Collaboration, G. Aad et al., “Dijet resonance search with weak supervision using√s = 13 TeV pp collisions in the ATLAS detector,” Phys. Rev. Lett.125 no. 13, (2020) 131801, arXiv:2005.02983 [hep-ex]

  4. [4]

    Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at √s= 13 TeV,

    CMS Collaboration, “Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at √s= 13 TeV,” tech. rep., CERN, Geneva, 2024. https://cds.cern.ch/record/2892677

  5. [5]

    Classification without labels: Learning from mixed samples in high energy physics,

    E. M. Metodiev, B. Nachman, and J. Thaler, “Classification without labels: Learning from mixed samples in high energy physics,” JHEP 10 (2017) 174, arXiv:1708.02949 [hep-ph]

  6. [6]

    On the Problem of the Most Efficient Tests of Statistical Hypotheses,

    J. Neyman and E. S. Pearson, “On the Problem of the Most Efficient Tests of Statistical Hypotheses,” Phil. Trans. Roy. Soc. Lond. A231 no. 694-706, (1933) 289–337

  7. [7]

    Anomaly Detection for Resonant New Physics with Machine Learning,

    J. H. Collins, K. Howe, and B. Nachman, “Anomaly Detection for Resonant New Physics with Machine Learning,” Phys. Rev. Lett.121 no. 24, (2018) 241803, arXiv:1805.02664 [hep-ph]

  8. [8]

    Comparing weak- and unsupervised methods for resonant anomaly detection,

    J. H. Collins, P. Mart ´ ın-Ramiro, B. Nachman, and D. Shih, “Comparing weak- and unsupervised methods for resonant anomaly detection,” Eur. Phys. J. C81 no. 7, (2021) 617, arXiv:2104.02092 [hep-ph]

Show all 37 references
  1. [9]

    Tree-based algorithms for weakly supervised anomaly detection,

    T. Finke, M. Hein, G. Kasieczka, M. Kr¨ amer, A. M¨ uck, P. Prangchaikul, T. Quadfasel, D. Shih, and M. Sommerhalder, “Tree-based algorithms for weakly supervised anomaly detection,” Phys. Rev. D109 no. 3, (2024) 034033, arXiv:2309.13111 [hep-ph]

  2. [10]

    Anomaly detection in the presence of irrelevant features,

    M. Freytsis, M. Perelstein, and Y. C. San, “Anomaly detection in the presence of irrelevant features,” JHEP 02 (2024) 220, arXiv:2310.13057 [hep-ph]. – 16 –

  3. [11]

    Improving the performance of weak supervision searches using transfer and meta-learning,

    H. Beauchesne, Z.-E. Chen, and C.-W. Chiang, “Improving the performance of weak supervision searches using transfer and meta-learning,” JHEP 02 (2024) 138, arXiv:2312.06152 [hep-ph]

  4. [12]

    Incorporating Physical Priors into Weakly-Supervised Anomaly Detection,

    C. L. Cheng, G. Singh, and B. Nachman, “Incorporating Physical Priors into Weakly-Supervised Anomaly Detection,” arXiv:2405.08889 [hep-ph]

  5. [13]

    Accelerating Resonance Searches via Signature-Oriented Pre-training,

    C. Li et al., “Accelerating Resonance Searches via Signature-Oriented Pre-training,” arXiv:2405.12972 [hep-ph]

  6. [14]

    Data Augmentation at the LHC through Analysis-specific Fast Simulation with Deep Learning,

    C. Chen, O. Cerri, T. Q. Nguyen, J.-R. Vlimant, and M. Pierini, “Data Augmentation at the LHC through Analysis-specific Fast Simulation with Deep Learning,” arXiv:2010.01835 [physics.comp-ph]

  7. [15]

    Metalearning and data augmentation for mass-generalized jet taggers,

    M. J. Dolan and A. Ore, “Metalearning and data augmentation for mass-generalized jet taggers,” Phys. Rev. D105 no. 9, (2022) 094030, arXiv:2111.06047 [hep-ph]

  8. [16]

    Mass Agnostic Jet Taggers,

    L. Bradshaw, R. K. Mishra, A. Mitridate, and B. Ostdiek, “Mass Agnostic Jet Taggers,” SciPost Phys. 8 no. 1, (2020) 011, arXiv:1908.08959 [hep-ph]

  9. [17]

    Extensive Studies of the Neutron Star Equation of State from the Deep Learning Inference with the Observational Data Augmentation,

    Y. Fujimoto, K. Fukushima, and K. Murase, “Extensive Studies of the Neutron Star Equation of State from the Deep Learning Inference with the Observational Data Augmentation,” JHEP 03 (2021) 273, arXiv:2101.08156 [nucl-th]

  10. [18]

    Improving Photometric Redshift Estimates with Training Sample Augmentation,

    LSST Dark Energy Science Collaboration, I. Moskowitz, E. Gawiser, J. F. Crenshaw, B. H. Andrews, A. I. Malz, and S. Schmidt, “Improving Photometric Redshift Estimates with Training Sample Augmentation,” Astrophys. J. Lett.967 no. 1, (2024) L6, arXiv:2402.15551 [astro-ph.IM]

  11. [19]

    Discerning Secluded Sector gauge structures,

    L. Carloni, J. Rathsman, and T. Sjostrand, “Discerning Secluded Sector gauge structures,” JHEP 04 (2011) 091, arXiv:1102.3795 [hep-ph]

  12. [20]

    Visible Effects of Invisible Hidden Valley Radiation,

    L. Carloni and T. Sjostrand, “Visible Effects of Invisible Hidden Valley Radiation,” JHEP 09 (2010) 105, arXiv:1006.2911 [hep-ph]

  13. [21]

    Dark matter in Hidden Valley models with stable and unstable light dark mesons,

    H. Beauchesne, E. Bertuzzo, and G. Grilli Di Cortona, “Dark matter in Hidden Valley models with stable and unstable light dark mesons,” JHEP 04 (2019) 118, arXiv:1809.10152 [hep-ph]

  14. [22]

    Theory, phenomenology, and experimental avenues for dark showers: a Snowmass 2021 report,

    G. Albouy et al., “Theory, phenomenology, and experimental avenues for dark showers: a Snowmass 2021 report,” Eur. Phys. J. C82 no. 12, (2022) 1132, arXiv:2203.09503 [hep-ph]

  15. [23]

    An introduction to PYTHIA 8.2,

    T. Sj¨ ostrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, “An introduction to PYTHIA 8.2,” Comput. Phys. Commun. 191 (2015) 159–177, arXiv:1410.3012 [hep-ph]

  16. [24]

    Parton distributions with LHC data,

    R. D. Ball et al., “Parton distributions with LHC data,” Nucl. Phys. B867 (2013) 244–289, arXiv:1207.1303 [hep-ph]

  17. [25]

    The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations,

    J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, “The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations,” JHEP ...

  18. [26]

    DELPHES 3, A modular framework for fast – 17 – simulation of a generic collider experiment,

    DELPHES 3 Collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lema ˆ ıtre, A. Mertens, and M. Selvaggi, “DELPHES 3, A modular framework for fast – 17 – simulation of a generic collider experiment,” JHEP 02 (2014) 057, arXiv:1307.6346 [hep-ex]

  19. [27]

    FastJet User Manual,

    M. Cacciari, G. P. Salam, and G. Soyez, “FastJet User Manual,” Eur. Phys. J. C72 (2012) 1896, arXiv:1111.6097 [hep-ph]

  20. [28]

    The anti- kt jet clustering algorithm,

    M. Cacciari, G. P. Salam, and G. Soyez, “The anti- kt jet clustering algorithm,” JHEP 04 (2008) 063, arXiv:0802.1189 [hep-ph]

  21. [29]

    The Machine Learning landscape of top taggers,

    A. Butter et al., “The Machine Learning landscape of top taggers,” SciPost Phys. 7 (2019) 014, arXiv:1902.09914 [hep-ph]

  22. [30]

    Jet-images — deep learning edition,

    L. de Oliveira, M. Kagan, L. Mackey, B. Nachman, and A. Schwartzman, “Jet-images — deep learning edition,” JHEP 07 (2016) 069, arXiv:1511.05190 [hep-ph]

  23. [31]

    Deep-learning Top Taggers or The End of QCD?,

    G. Kasieczka, T. Plehn, M. Russell, and T. Schell, “Deep-learning Top Taggers or The End of QCD?,” JHEP 05 (2017) 006, arXiv:1701.08784 [hep-ph]

  24. [32]

    Chollet et al., “Keras.” https://keras.io, 2015

    F. Chollet et al., “Keras.” https://keras.io, 2015

  25. [33]

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,

    M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” arXiv:1603.04467 [cs.DC]

  26. [34]

    Formulae for Estimating Significance,

    ATLAS Collaboration, “Formulae for Estimating Significance,” 2020

  27. [35]

    A comprehensive survey on data augmentation,

    Z. Wang, P. Wang, K. Liu, P. Wang, Y. Fu, C.-T. Lu, C. C. Aggarwal, J. Pei, and Y. Zhou, “A comprehensive survey on data augmentation,” 2024. https://arxiv.org/abs/2405.09591

  28. [36]

    Anomalies, representations, and self-supervision,

    B. M. Dillon, L. Favaro, F. Feiden, T. Modak, and T. Plehn, “Anomalies, representations, and self-supervision,” SciPost Phys. Core7 (2024) 056, arXiv:2301.04660 [hep-ph]

  29. [37]

    Search for high mass dijet resonances with a new background prediction method in proton-proton collisions at √s = 13 TeV,

    CMS Collaboration, A. M. Sirunyan et al., “Search for high mass dijet resonances with a new background prediction method in proton-proton collisions at √s = 13 TeV,” JHEP 05 (2020) 033, arXiv:1911.03947 [hep-ex]. – 18 –

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.