Searching for Anomalies with Foundation Models

Benjamin Nachman; Vinicius Mikuni

Searching for Anomalies with Foundation Models

T0 review · 2 major / 2 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A foundation-model selection of CMS events yields a signal region the background estimate cannot model.

desk verdict Abstract-only CMS anomaly report with OmniLearned: interesting application, but the load-bearing background-transfer claim is uncheckable from what we have. read the letter →

arxiv 2603.23593 v2 pith:BE25PGD2 submitted 2026-03-24 hep-ex hep-phphysics.data-an

Vinicius Mikuni , Benjamin Nachman This is my paper · ORCID

classification hep-exhep-phphysics.data-an

keywords anomalydetectionfoundationmodelsOmniLearnedCMSbackgroundestimationsignalregioncolliderdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Foundation models trained across broad collider datasets may surface rare or unexpected event patterns that ordinary anomaly searches miss. This paper takes the large OmniLearned model, lets it pick a phase space on real CMS data where a mass sideband already looked anomalous, and then runs a full analysis with a complete background estimate on that selected region. The background description matches data in the validation regions, yet fails to describe the signal region. The authors stop short of claiming a discovery and instead open the selected events and methods to further scrutiny. If the mismatch survives independent checks, foundation-model selection would have delivered a concrete, data-driven anomaly for the community to resolve.

What carries the argument

The OmniLearned foundation model used as a data-driven selector of phase space, followed by a full background-estimation procedure whose transfer from validation regions to the model-selected signal region is the load-bearing step.

What would settle it

An independent background estimate, or a re-analysis of the same CMS events with a different selection or modeling method, that either restores agreement in the signal region or isolates a residual mismodeling that fully accounts for the observed excess.

Watch

Extended reading notes

Core claim

After applying the OmniLearned foundation model to CMS data and performing a complete background estimate on the phase space it selects, the estimate describes data well in validation regions but cannot accurately model the signal region.

Load-bearing premise

That the background-estimation procedure, once validated in control regions, should transfer cleanly into the foundation-model-selected signal region without large residual mismodeling caused by the selection itself.

Editorial extensions

If this is right

Foundation-model selections can define concrete, data-driven signal regions that standard background methods must then confront.
Validation-region agreement is not by itself sufficient to guarantee signal-region modeling once a large model has reshaped the phase space.
The selected CMS events become a public target for alternative background estimates and independent scrutiny.
Anomaly searches gain a practical workflow: model-driven region selection followed by a full analysis rather than a pure unsupervised score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the residual mismatch is real and not modeling artifact, it supplies a concrete candidate region for targeted resonance or multi-object searches.
The same foundation-model selection plus full-analysis template could be applied to other CMS or ATLAS final states to test whether similar sideband failures recur.
Disagreement between model-selected signal regions and standard background methods may become a diagnostic for both new physics and for limitations of current simulation or estimation techniques.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you need to know is that this is an abstract-only note. Mikuni and Nachman report that OmniLearned, applied to CMS data, carves out a phase space where a full background estimate matches validation regions but fails in the signal region, and they explicitly invite scrutiny rather than claim a discovery. That is the whole empirical payload.

What is new is the application: take a large foundation model, let it define the region of interest, then run a classical complete background analysis on that region. That is a legitimate step beyond the usual supervised or weakly supervised HEP anomaly searches, and the authors are careful not to oversell it. The abstract is honest about the mismatch and about wanting independent eyes on the events and methods. Credit for that restraint.

The soft spot is structural and large given the material: we cannot see the background method, the validation vs signal definitions, the selection function, event counts, fit quality, or systematics. The load-bearing assumption is that the background procedure should transfer under the model-induced selection without large residual mismodeling. From the abstract alone that assumption is untestable; it is not a demonstrated flaw, just an open condition. Circularity risk is real but moderate—the model defines the region under test—and the authors do not hide that they are studying the model’s own selection. No equations or figures exist here to confirm or refute anything stronger.

This is for people who work on ML-for-HEP anomaly detection and for experimentalists who care about foundation-model phase-space carving. A serious referee should see the full paper if it exists with methods, control regions, and quantitative residuals; the abstract alone does not support a desk accept, but the claim is important enough and framed carefully enough that it deserves referee time rather than dismissal. I would not cite it yet or bring the abstract to reading group. If the full analysis holds up under scrutiny, that changes. For now: engage if the full text appears; do not treat the abstract as settled evidence.

Referee Report

2 major / 2 minor

Summary. The manuscript reports an anomaly-detection study applying the large OmniLearned foundation model to CMS collision data. Unexpected behavior is observed in a mass sideband. The authors then perform a full analysis, including a complete background estimate, on the phase space selected by the model. They report that the background estimate describes the data well in validation regions but is unable to accurately model the signal region, and they invite further scrutiny of these events and of their methods.

Significance. If the residual in the OmniLearned-selected signal region survives a carefully validated background estimate, the result would be of high interest both as a potential new-physics hint and as a concrete stress test of foundation-model-driven anomaly searches in collider data. Framing the work as a complete analysis with an explicit invitation for external scrutiny is a strength of the claim. Because only the abstract is available, significance beyond that statement of intent cannot yet be assessed.

major comments (2)

[Abstract] The central empirical claim—that a complete background estimate matches data in validation regions yet fails in the OmniLearned-selected signal region—cannot be verified from the abstract alone. No definition of validation versus signal regions, no description of the background method (transfer factors, sideband fits, simulation templates, etc.), no event counts, fit-quality metrics, or systematic table are supplied. The load-bearing assumption that the background procedure is expected to transfer under the foundation-model selection is therefore untestable here.
[Abstract] The phase space under study is defined by the OmniLearned foundation model itself, and the background estimate is then judged on that same selected region. Without a quantitative characterization of how the model selection correlates with the background-estimation observables (e.g., the mass variable used for sidebands), it is not possible to assess whether residual mismodeling is induced by the selection rather than by new physics or by an independent background failure. This transferability condition is the weakest assumption of the claimed result and requires explicit control in the full analysis.

minor comments (2)

[Abstract] The abstract does not name the CMS dataset (run period, integrated luminosity, trigger path) or the precise mass sideband in which the unexpected behavior was first observed; these details would help place the claim.
[Abstract] The phrase 'unable to accurately model the signal region' is purely qualitative; even a brief quantitative statement of residual significance or goodness-of-fit would strengthen the abstract.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identifiable; abstract reports an empirical background-estimate mismatch without definitional reduction or load-bearing self-citation.

full rationale

Only the abstract is available, containing no equations, no fitted parameters renamed as predictions, no uniqueness theorems, no ansatz citations, and no self-citations at all. The central claim is an observational statement that a complete background estimate matches data in validation regions yet fails to model the OmniLearned-selected signal region. This is presented as an empirical finding inviting further scrutiny, not as a first-principles derivation that reduces to its own inputs by construction. Because no specific reduction (e.g., Eq. X forced equal to Eq. Y, or a fit re-labeled a prediction) can be exhibited from the supplied text, the honest finding under the hard rules is zero circularity. The phase-space selection by the foundation model and subsequent background evaluation constitute a standard analysis workflow whose transferability assumptions cannot be audited here, but that absence does not manufacture circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

Abstract-only: free parameters of the background fit, OmniLearned architecture choices, and any ad-hoc region definitions are not stated. The ledger therefore records the domain assumptions required for the central claim to be meaningful, plus the absence of inspectable fitted numbers. No new physical entities (particles, forces) are invented; the ‘anomaly’ is an unexplained residual, not a postulated mediator.

free parameters (2)

background-estimate nuisance/fit parameters
Any complete background estimate on a selected phase space typically involves fitted normalizations, shape parameters, or transfer factors. None are given in the abstract, so they remain uninspectable free parameters the claim depends on.
OmniLearned selection threshold / phase-space cut
The signal region is ‘the phase space picked out by the large model.’ The cut or score threshold that defines that region is a choice that can move the residual; it is not reported in the abstract.

assumptions (3)

domain assumption Foundation-model selection of events is a valid way to define a signal region for a classical background estimate without fatally biasing the estimate.
The entire analysis rests on treating OmniLearned’s picked phase space as a legitimate search region rather than a region whose definition correlates with background mismodeling.
domain assumption Validation regions are sufficiently similar to the signal region that agreement there is evidence the background method should work in the signal region under the null.
Standard HEP sideband/validation logic; if the regions differ in features the model selected on, the transfer assumption fails.
domain assumption CMS data and the OmniLearned model outputs used are correctly calibrated and free of unaccounted detector or training artifacts in the selected region.
Required for interpreting a residual as physically interesting rather than instrumental or ML-systematic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Searching for Anomalies with Foundation Models." pith.science (2026). https://pith.science/paper/BE25PGD2

@misc{pith2026260323593,
  author       = {Pith},
  title        = {Pith review of: Searching for Anomalies with Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BE25PGD2}},
  note         = {Machine review of arXiv:2603.23593}
}

read the original abstract

Foundation models have the potential to extend the discovery reach for anomaly detection searches. When studying the large OmniLearned foundation model on data from the CMS experiment, unexpected behavior was observed in a mass sideband. The purpose of this paper is to perform a full analysis, including a complete background estimate, on the phase space picked out by the large model. We find that the background estimation describes the data well in validation regions, but is unable to accurately model the signal region. We invite further scrutiny of these events and our methods.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.