{"id":"106dc328-8655-4523-89b3-765f9ae84f23","arxiv_id":"2603.23593","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"OmniLearned on CMS data picks a phase space where background estimates work in validation regions but cannot model the signal region.","lead":"A large foundation model applied to CMS collider data selected a phase space with unexpected mass-sideband behavior. Full background estimation matches validation regions but fails in the signal region, so the authors invite further scrutiny.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Abstract-only review leaves the transferability of the background estimate under foundation-model selection untestable; no load-bearing flaw can be confirmed or refuted from available material.","rationale":"The Reader's verdict of UNVERDICTED with LOW confidence is the only defensible outcome for an abstract-only review. The strongest claim is cautious and non-discovery; the weakest assumption (transfer of the background estimate under foundation-model selection) is correctly identified and is indeed the place where the argument is least secure. Because the full text, methods, code, and data are unavailable, no concrete technical flaw can be verified or refuted, so the stress-test cannot move the verdict. The recommended concrete test simply operationalizes the missing check that would settle the transferability question once the paper becomes available. No ad-hominem, no theatrical language, and no manufactured concern beyond what the abstract itself invites.","tokens_in":1871,"tokens_out":486,"duration_ms":5583,"concrete_test":"Obtain the full paper (or public note) and recompute the background prediction in the OmniLearned signal region after explicitly re-deriving the transfer factors from the same validation regions used by the authors, holding the model selection fixed; if the residual remains statistically significant after that independent re-derivation, the claim of mismodeling stands; if it disappears, the original residual was method-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a complete background estimate matches data in validation regions yet fails in the OmniLearned-selected signal region. Because only the abstract is available, the concrete background method, the precise definition of validation vs. signal regions, the selection function of OmniLearned, and any quantitative residual are all unspecified. The load-bearing condition is therefore that the background procedure is expected to transfer under the model-induced selection without large residual mismodeling. That condition cannot be examined: no equations, control-region definitions, transfer-factor construction, or figures exist in the supplied material. The Reader correctly flags this as the weakest assumption, but the absence of the full text means the concern remains an untestable possibility rather than a demonstrated internal inconsistency. No stronger objection can be raised without manufacturing content that is not present.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript reports an anomaly-detection study applying the large OmniLearned foundation model to CMS collision data. Unexpected behavior is observed in a mass sideband. The authors then perform a full analysis, including a complete background estimate, on the phase space selected by the model. They report that the background estimate describes the data well in validation regions but is unable to accurately model the signal region, and they invite further scrutiny of these events and of their methods.","tokens_in":2002,"tokens_out":692,"duration_ms":18555,"significance":"If the residual in the OmniLearned-selected signal region survives a carefully validated background estimate, the result would be of high interest both as a potential new-physics hint and as a concrete stress test of foundation-model-driven anomaly searches in collider data. Framing the work as a complete analysis with an explicit invitation for external scrutiny is a strength of the claim. Because only the abstract is available, significance beyond that statement of intent cannot yet be assessed.","major_comments":[{"comment":"The central empirical claim—that a complete background estimate matches data in validation regions yet fails in the OmniLearned-selected signal region—cannot be verified from the abstract alone. No definition of validation versus signal regions, no description of the background method (transfer factors, sideband fits, simulation templates, etc.), no event counts, fit-quality metrics, or systematic table are supplied. The load-bearing assumption that the background procedure is expected to transfer under the foundation-model selection is therefore untestable here.","section":"Abstract"},{"comment":"The phase space under study is defined by the OmniLearned foundation model itself, and the background estimate is then judged on that same selected region. Without a quantitative characterization of how the model selection correlates with the background-estimation observables (e.g., the mass variable used for sidebands), it is not possible to assess whether residual mismodeling is induced by the selection rather than by new physics or by an independent background failure. This transferability condition is the weakest assumption of the claimed result and requires explicit control in the full analysis.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract does not name the CMS dataset (run period, integrated luminosity, trigger path) or the precise mass sideband in which the unexpected behavior was first observed; these details would help place the claim.","section":"Abstract"},{"comment":"The phrase 'unable to accurately model the signal region' is purely qualitative; even a brief quantitative statement of residual significance or goodness-of-fit would strengthen the abstract.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"Only the abstract was available for this review (full text not supplied). A proper technical assessment of soundness is not possible until the complete manuscript—including background method, region definitions, fit quality metrics, and systematic tables—is provided. I recommend the editor obtain the full text and reassign for a standard review before any accept/reject decision. The abstract's invitation for scrutiny is appropriate given the claim; the circularity/transferability concern flagged by the stress test remains an untestable possibility rather than a demonstrated internal inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing you need to know is that this is an abstract-only note. Mikuni and Nachman report that OmniLearned, applied to CMS data, carves out a phase space where a full background estimate matches validation regions but fails in the signal region, and they explicitly invite scrutiny rather than claim a discovery. That is the whole empirical payload.\n\nWhat is new is the application: take a large foundation model, let it define the region of interest, then run a classical complete background analysis on that region. That is a legitimate step beyond the usual supervised or weakly supervised HEP anomaly searches, and the authors are careful not to oversell it. The abstract is honest about the mismatch and about wanting independent eyes on the events and methods. Credit for that restraint.\n\nThe soft spot is structural and large given the material: we cannot see the background method, the validation vs signal definitions, the selection function, event counts, fit quality, or systematics. The load-bearing assumption is that the background procedure should transfer under the model-induced selection without large residual mismodeling. From the abstract alone that assumption is untestable; it is not a demonstrated flaw, just an open condition. Circularity risk is real but moderate—the model defines the region under test—and the authors do not hide that they are studying the model’s own selection. No equations or figures exist here to confirm or refute anything stronger.\n\nThis is for people who work on ML-for-HEP anomaly detection and for experimentalists who care about foundation-model phase-space carving. A serious referee should see the full paper if it exists with methods, control regions, and quantitative residuals; the abstract alone does not support a desk accept, but the claim is important enough and framed carefully enough that it deserves referee time rather than dismissal. I would not cite it yet or bring the abstract to reading group. If the full analysis holds up under scrutiny, that changes. For now: engage if the full text appears; do not treat the abstract as settled evidence.","headline":"Abstract-only CMS anomaly report with OmniLearned: interesting application, but the load-bearing background-transfer claim is uncheckable from what we have.","tokens_in":2651,"tokens_out":504,"would_cite":false,"duration_ms":5709,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A foundation-model selection of CMS events yields a signal region the background estimate cannot model.","keywords":["anomaly detection","foundation models","OmniLearned","CMS","background estimation","signal region","collider data"],"falsifier":"An independent background estimate, or a re-analysis of the same CMS events with a different selection or modeling method, that either restores agreement in the signal region or isolates a residual mismodeling that fully accounts for the observed excess.","tokens_in":2697,"feed_emoji":"🔍","tokens_out":629,"duration_ms":9869,"temperature":0.7,"pith_summary":"Foundation models trained across broad collider datasets may surface rare or unexpected event patterns that ordinary anomaly searches miss. This paper takes the large OmniLearned model, lets it pick a phase space on real CMS data where a mass sideband already looked anomalous, and then runs a full analysis with a complete background estimate on that selected region. The background description matches data in the validation regions, yet fails to describe the signal region. The authors stop short of claiming a discovery and instead open the selected events and methods to further scrutiny. If the mismatch survives independent checks, foundation-model selection would have delivered a concrete, data-driven anomaly for the community to resolve.","feed_headline":"Foundation model flags CMS events a background estimate cannot fit","feed_subtitle":"Validation regions agree; the model-selected signal region does not. Authors invite scrutiny.","key_machinery":"The OmniLearned foundation model used as a data-driven selector of phase space, followed by a full background-estimation procedure whose transfer from validation regions to the model-selected signal region is the load-bearing step.","core_discovery":"After applying the OmniLearned foundation model to CMS data and performing a complete background estimate on the phase space it selects, the estimate describes data well in validation regions but cannot accurately model the signal region.","pith_inferences":["If the residual mismatch is real and not modeling artifact, it supplies a concrete candidate region for targeted resonance or multi-object searches.","The same foundation-model selection plus full-analysis template could be applied to other CMS or ATLAS final states to test whether similar sideband failures recur.","Disagreement between model-selected signal regions and standard background methods may become a diagnostic for both new physics and for limitations of current simulation or estimation techniques."],"forward_implications":["Foundation-model selections can define concrete, data-driven signal regions that standard background methods must then confront.","Validation-region agreement is not by itself sufficient to guarantee signal-region modeling once a large model has reshaped the phase space.","The selected CMS events become a public target for alternative background estimates and independent scrutiny.","Anomaly searches gain a practical workflow: model-driven region selection followed by a full analysis rather than a pure unsupervised score."],"fun_headline_variants":["OmniLearned selects CMS region background estimate cannot fit","Foundation model flags CMS phase space that resists background modeling","CMS validation fits; model-picked signal region does not","Background estimate fails on OmniLearned-selected CMS events","Foundation model on CMS yields unmodeled signal region"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That the background-estimation procedure, once validated in control regions, should transfer cleanly into the foundation-model-selected signal region without large residual mismodeling caused by the selection itself.","fun_headline_variants_meta":{"raw":{"variants":["OmniLearned selects CMS region background estimate cannot fit","Foundation model flags CMS phase space that resists background modeling","CMS validation fits; model-picked signal region does not","Background estimate fails on OmniLearned-selected CMS events","Foundation model on CMS yields unmodeled signal region"]},"model":"grok-4.5","effort":"low","cost_usd":0.005584,"raw_usage":{"total_tokens":1318,"prompt_tokens":581,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":55840000,"prompt_tokens_details":{"text_tokens":581,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":675,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":581,"tokens_out":62,"duration_ms":7386,"temperature":1.0,"reasoning_tokens":675,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T19:31:42.902837+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"An independent background estimate, or a re-analysis of the same CMS events with a different selection or modeling method, that either restores agreement in the signal region or isolates a residual mismodeling that fully accounts for the observed excess.","supporting_citations":[],"review_version":1}