{"id":"8c6a65a0-e5c9-4031-9a3f-4eb22400a9ba","arxiv_id":"2411.19389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A TabNet-style sequential attention model tags quenched jets with AUC 0.95 on heavy-ion simulations that include thermal background, pileup, and detector effects.","lead":"This paper applies an interpretable attention-based neural network to tell quenched jets (modified by quark-gluon plasma) from ordinary vacuum jets in simulated heavy-ion collisions that include background, pileup, and detector smearing. It reports an AUC of 0.95, claims to beat earlier methods, and uses feature masks to hint at which jet properties drive the decision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.95 AUC and model-agnostic claim rest on an untested assumption that the tagger learns quenching, not generator ID; no cross-generator or same-dataset validation is reported.","rationale":"The reader's weakest assumption is that AUC values from different papers are not directly comparable and that no same-dataset baselines or cross-generator transfer tests are provided. I agree that this is a serious gap, but I would sharpen it: the more load-bearing issue is that the model may be learning generator identity rather than quenching physics. Because signal and background are each composed of two generators, and because the input includes global jet observables and particle ID information that are known to vary across generators independent of quenching, the reported 0.95 AUC on a random split of the mixed dataset does not demonstrate that the tagger generalizes to unseen generator conditions. The paper's own framing as 'model-agnostic' makes this the central claim to test. The reader's concern about benchmark comparability is related but secondary; even a same-dataset comparison with a PFN baseline would not establish model-agnosticism. The missing cross-generator validation is the single decisive experiment. The correct verdict remains CONDITIONAL rather than REJECT because the issue is addressable with additional experiments and does not invalidate the in-pipeline classification result if the cross-generator test passes. I therefore recommend no change to the reader's verdict, while noting that the condition should explicitly require the leave-one-generator-out and same-dataset baseline tests.","tokens_in":9773,"tokens_out":4462,"duration_ms":43201,"concrete_test":"Run leave-one-generator-out or pairwise cross-generator validation: train on (Jewel, Pythia) and evaluate on (CoLBT-Hydro, Herwig), then train on (CoLBT-Hydro, Herwig) and evaluate on (Jewel, Pythia), with all events embedded under identical background, pileup, and detector-smearing conditions. Report per-test-generator AUC and confusion matrices. In parallel, retrain a simple baseline (e.g., Particle Flow Network or DeepSets) on the same train/test split with the same input features and report its AUC. If the cross-generator AUC drops substantially below 0.95, or if the baseline matches the attention model on this dataset, the central claims of model-agnosticism and benchmark superiority are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims—AUC 0.95, outperforming previous methods, and model-agnosticism—depend on the classifier learning medium-induced quenching rather than incidental differences between the generator sets used as signal (Jewel, CoLBT-Hydro) and background (Pythia, Herwig). The evaluation in Section III uses a random 90-10 split of this mixed dataset, so a model can achieve high AUC by exploiting any systematic difference between the two generator groups—e.g., hadronization model, particle ID fractions, multiplicities, or the soft-drop observables zg, Rg, kT, mg that are explicitly included as input features. No leave-one-generator-out test, per-generator AUC breakdown, or cross-generator training/evaluation is reported. Additionally, Table II compares AUC values from different papers with different generators, background treatments, and detector simulations; none of the baseline models are retrained on the present dataset. The claim of a new benchmark is therefore not established. The model-agnostic claim is the weakest link: if the tagger mostly separates generators rather than quenched from unquenched physics, the headline result does not transfer to real heavy-ion data where the generator mixture is unknown.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a machine learning framework based on a sequential attention mechanism (TabNet) to tag quenched jets in simulated heavy-ion collisions. The training sample combines JEWEL and CoLBT-Hydro jets as signal (label 1) and PYTHIA8 and Herwig7 jets as background (label 0), and the pipeline includes thermal background embedding, pileup, and detector smearing. The authors report an AUC of 0.95, claim that this significantly outperforms previous methods (CNNs, DeepSets, LSTM, PFN, etc.), and advertise the framework as model-agnostic and interpretable via sparse feature-selection masks.","tokens_in":10138,"tokens_out":4500,"duration_ms":40456,"significance":"If the reported performance and generator-agnostic behavior were rigorously established, the framework would be a practically useful tagger for heavy-ion jet substructure studies, and the inclusion of pileup, thermal background, and detector smearing is a step toward experimental realism. The public availability of code and data is a strength. However, the central comparative and model-agnostic claims are currently not supported by the evidence presented: the AUC comparison in Table II mixes results from different simulations and conditions, no baseline is retrained on the authors' dataset, and no cross-generator validation is reported. These gaps are load-bearing for the headline conclusions.","major_comments":[{"comment":"The performance comparison is not apples-to-apples. The AUC values for the existing methods are taken from papers that use different generators, background treatments, detector simulations, jet definitions, and selection criteria, while the sequential attention model is evaluated on the specific dataset described in Section II A. No baseline (EFN, PFN, LSTM, CNN, MLP, etc.) is retrained on the same embedded, smeared samples with the same train/test split. Therefore the claim that the sequential attention framework 'significantly outperforms' prior methods is not established. The authors should retrain at least one or two representative baseline models on their own data and report the resulting AUCs under identical conditions.","section":"Section III, Table II"},{"comment":"The 90-10 train/test split is described only at the level of jets, not events. Because the samples are generated by embedding hard-scattering dijet events into thermal background and pileup events, jets from the same underlying event can appear in both the training and test sets. The shared soft background, pileup composition, and detector smearing realizations can then leak information and inflate the reported AUC. The paper should specify whether an event-level split was used, and if not, it should be performed and the AUC recomputed. Reporting per-generator AUCs (e.g., JEWEL vs PYTHIA, JEWEL vs HERWIG, CoLBT-Hydro vs PYTHIA, CoLBT-Hydro vs HERWIG) would also clarify what the model actually separates.","section":"Section II B (train/test split)"},{"comment":"The paper labels JEWEL and CoLBT-Hydro as signal and PYTHIA and Herwig as background, but it never tests whether the trained model transfers across generators. Since the input includes global soft-drop observables (zg, Rg, kT, mg), particle multiplicities, and PID information, a classifier could achieve a high AUC by separating the four generators according to hadronization or shower-model differences rather than by learning medium-induced quenching. A leave-one-generator-out test (for example, train on JEWEL vs PYTHIA and evaluate on CoLBT-Hydro vs HERWIG, plus the reverse) is necessary to support the claim of 'reduced model dependence' and the descriptor 'model-agnostic.' Without such a test, the central physics interpretation of the 0.95 AUC is not established.","section":"Section II A / Section IV (model-agnostic claim)"},{"comment":"The interpretability claim is supported only by qualitative heatmaps for one JEWEL jet and one PYTHIA jet. There is no quantitative analysis of the aggregate feature masks over many jets, no stability check across generators, and no numerical demonstration that 'features corresponding to the highest and lowest pT jet constituents play the biggest roles.' Since interpretability is advertised as a unique advantage, this analysis should either be made quantitative (e.g., distributions of mask values, feature-ranking statistics, uncertainty estimates) or the claim should be softened accordingly.","section":"Section III, Fig. 4 and surrounding text"}],"minor_comments":[{"comment":"The phrase 'with bp⊥ > 100 GeV' appears to contain a typographical or notation error; presumably the hard-scattering transverse-momentum cutoff is meant, but it should be written unambiguously (e.g., p̂T or pT,hard).","section":"Section II A"},{"comment":"Encoding particle types as values 0, 0.1, 0.2, ... imposes an arbitrary ordinal relationship on categorical labels. A one-hot or learned embedding would avoid this unnecessary inductive bias.","section":"Section II A, particle ID encoding"},{"comment":"The notation in Eq. (1) is ambiguous: the global observables and the per-constituent list are not clearly separated, and the index 'nM i=1' is missing a product or set symbol. Please rewrite to make the concatenation explicit.","section":"Equation (1)"},{"comment":"The ROC curve in Fig. 2 reports a single AUC value without an uncertainty estimate. Given the sample size, a bootstrap confidence interval would make the result more informative and comparable.","section":"Section III, ROC curve"},{"comment":"The heatmaps lack a color scale, and the x-axis feature index is not mapped to specific features; adding a legend would allow the reader to interpret the sparsity claims.","section":"Section III, Fig. 4"},{"comment":"The sentence 'This not only aids in improving the classification performance but also opens new avenues...' appears to be missing a clause after 'while irrelevant information'; please revise for clarity.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the simulation setup is more realistic than much of the prior work. The main weaknesses are fixable in revision: retraining baselines on the same data, performing an event-level split, adding leave-one-generator-out validation, and quantifying the interpretability analysis. If the authors provide those results, the claims of a new benchmark and model-agnostic behavior may become supportable. The current manuscript, however, does not yet establish them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful engineering demonstration—applying TabNet to quenched jet tagging with thermal background, pileup, and detector smearing folded in—but the headline comparisons to earlier AUCs don't survive scrutiny. The 0.95 AUC is probably correct for this dataset and pipeline; the problem is what it's compared against.\n\nWhat's genuinely new is the combination of experimental realism (embedded thermal background, 150 pileup interactions, detector smearing) with an interpretable attention-based architecture, which prior quenched-jet ML papers did not do all together. The paper also ships code and data on GitHub, which is more than most Letters do. The feature-mask analysis is a reasonable interpretability step, though it is descriptive rather than physics-proving.\n\nThe main soft spot is Table II. AUC values come from different papers, different generators, different background treatments, and different detector simulations; no baseline is retrained on this paper's own samples. So \"significantly outperforming\" is not established. A bigger concern: signal comes from Jewel and CoLBT-Hydro, background from Pythia and Herwig. A classifier can get a high AUC by separating generator sets—hadronization, PID fractions, multiplicities—rather than quenching physics. The paper reports no leave-one-generator-out test, no per-generator AUC breakdown, and no cross-generator training/evaluation. That missing check is load-bearing for the \"model-agnostic\" claim. Also, the train/test split is described only as 90-10; the paper doesn't say whether the split is by event or by jet. If jets from the same event appear on both sides, the AUC is inflated. That's a minor omission but worth fixing.\n\nOn circularity: the reader note worries that attention masks are explanations of the trained model. I think that's a non-issue. The masks are post-hoc attributions, not outputs derived from their own input. Standard interpretability setup.\n\nBottom line: the tagger works on this simulation suite, but the benchmark and model-agnostic claims are under-supported. The paper deserves peer review because the core idea is useful and the issues are addressable with same-dataset baselines and a cross-generator test. I'd recommend major revision, not desk rejection.","headline":"Useful realistic-simulation tagger; benchmark comparisons need same-dataset baselines and a leave-one-generator-out test.","tokens_in":10536,"tokens_out":2817,"would_cite":false,"duration_ms":22302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sequential attention network achieves AUC 0.95 for tagging quenched jets under experimentally realistic heavy-ion conditions.","keywords":["quenched jet tagging","heavy-ion collisions","jet quenching","sequential attention network","machine learning","pileup","quark-gluon plasma","jet substructure"],"falsifier":"Retrain the CNN, DeepSets, LSTM, and PFN baselines on the exact mixed-event samples used here and compare AUCs; if the gap to 0.95 largely disappears, the performance claim is not architecture-specific. Separately, train on Jewel/Pythia jets and evaluate on CoLBT-Hydro/Herwig jets: a large drop in AUC would falsify the model-agnostic claim.","tokens_in":9592,"feed_emoji":"","tokens_out":4373,"duration_ms":35516,"temperature":0.7,"pith_summary":"The paper aims to show that a machine-learning tagger can separate jets that have been quenched by the quark-gluon plasma from vacuum jets even when the data include pileup, an uncorrelated soft thermal background, and detector smearing—conditions that prior tagging studies mostly left out. It trains a sequential attention network on jets from two quenching generators (Jewel and CoLBT-Hydro) against jets from two vacuum generators (Pythia and Herwig), reporting an AUC of 0.95. The same model produces sparse per-jet feature masks that point to which jet observables drive the decision. If these results hold, the framework would give experimental analyses a practical, interpretable tagger and a route from data-driven tagging to physics-informed observable design.","feed_headline":"Attention network tags quenched jets with 0.95 AUC","feed_subtitle":"A model-agnostic tagger that works under pileup, thermal background, and detector smearing.","key_machinery":"The carrying object is a sequential attention network of the TabNet type. At each of three decision steps, a sparsemax mask selects a sparse subset of input features; a transformer processes the masked features into a decision vector and a state passed to the next step, while a prior-scale term $\\gamma = 1.3$ controls feature reuse. The input representation concatenates global jet observables—multiplicity, inclusive mass, transverse momentum, soft-drop groomed $z_g$, $R_g$, groomed mass, and $k_\\perp$—with up to 35 leading constituents ordered by $p_\\perp$, each carrying $p_\\perp$, $\\eta$, $\\phi$, and a smeared particle-ID class. The final decision sums ReLU-transformed step outputs, and the normalized aggregate mask (Eq. 8) yields per-jet feature importance. This mechanism is what lets the model both classify and say which features it used.","core_discovery":"We introduce an interpretable, sequential attention-based framework that identifies quenched jets in heavy-ion collisions while accounting for pileup, uncorrelated soft background, and detector effects. The model reaches an AUC of 0.95 on mixed events built by embedding dijet events into central PbPb background with 150 pileup interactions and smeared constituents, outperforming published AUC values of 0.67–0.86 from CNNs, DeepSets, LSTM, MLP, and autoencoder-based taggers. By training and evaluating on two quenching models and two vacuum models, we argue the tagger is model-agnostic; by inspecting its sparse feature-selection masks, we show which global and per-constituent features carry the signal. This, we claim, is the first model-agnostic and interpretable quenched-jet tagger under experimentally realistic conditions.","pith_inferences":["A direct cross-generator test—training on one quenching model and testing on another—would be needed to confirm that the reported AUC generalizes beyond the specific samples used here; the paper does not report one.","Because the baselines in Table II were not retrained on the same dataset, part of the AUC gap may reflect differences in generators, background handling, or detector simulation rather than architecture alone.","The per-jet feature masks could be converted into a hand-crafted observable, for instance a weighted combination of the highest- and lowest-$p_\\perp$ constituent $p_\\perp$ with groomed jet mass, which would let the tagger's signal be reproduced without a network.","Applying the same pipeline to experimental pp and PbPb jets, with the network trained on matched simulations, would test whether the 0.95 separation persists after full reconstruction."],"forward_implications":["An experimental analysis could deploy the tagger on PbPb data with pileup and detector smearing and expect quenched/vacuum separation at the level reported here.","Because the mask is sparse and sample-specific, it can point to the jet constituents and global observables that most distinguish quenched from unquenched jets.","Training on two quenching and two vacuum generators is intended to make the tagger robust to modeling assumptions, reducing the model dependence of extraction of quenching signals.","The AUC comparison suggests the sequential attention architecture outperforms CNNs, LSTMs, MLPs, and autoencoder-based taggers on this task.","The framework could be extended to other jet substructure discrimination problems in heavy-ion collisions."],"supporting_citations":[{"why":"Supplies the sequential attention architecture with sparsemax masks and prior-scale feature reuse.","marker":"[50]"},{"why":"Generates the Jewel quenched-jet samples used as signal.","marker":"[35]"},{"why":"Generates the CoLBT-Hydro quenched-jet samples used as a second signal model.","marker":"[36]"},{"why":"Generates the Pythia8 vacuum samples used as background.","marker":"[37]"},{"why":"Generates the Herwig7 vacuum samples used as a second background model.","marker":"[39]"},{"why":"Provides an LSTM baseline AUC (0.76) that this work compares against.","marker":"[30]"},{"why":"Provides CNN, MLP, and LSTM baseline AUCs (0.73–0.76) under thermal background.","marker":"[31]"},{"why":"Provides Energy Flow Network and Particle Flow Network baseline AUCs (0.67–0.86).","marker":"[32]"},{"why":"Supplies the constituent-subtraction algorithm used to remove thermal background and pileup before clustering.","marker":"[45]"},{"why":"Supplies the measured PbPb pseudorapidity density used to set the thermal background multiplicity.","marker":"[44]"}],"fun_headline_variants":["Quenched-jet tagger reaches 0.95 AUC, model-agnostic","Attention net tags quenched jets with 0.95 AUC under pileup","Interpretable attention tagger sets 0.95 AUC for quenched jets","Model-agnostic tagger: 0.95 AUC under heavy-ion realism"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the assumption that the AUC values quoted from other papers were computed on comparable tasks, so the reported 0.95 is directly better than their 0.67–0.86; the paper does not retrain those baselines on its own samples.","fun_headline_variants_meta":{"raw":{"variants":["Quenched-jet tagger reaches 0.95 AUC, model-agnostic","Attention net tags quenched jets with 0.95 AUC under pileup","Interpretable attention tagger sets 0.95 AUC for quenched jets","Model-agnostic tagger: 0.95 AUC under heavy-ion realism"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3637,"prompt_tokens":826,"completion_tokens":2811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":2725}},"tokens_in":442,"tokens_out":2811,"duration_ms":18020,"temperature":1.0,"reasoning_tokens":2725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:12:45.278449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the CNN, DeepSets, LSTM, and PFN baselines on the exact mixed-event samples used here and compare AUCs; if the gap to 0.95 largely disappears, the performance claim is not architecture-specific. Separately, train on Jewel/Pythia jets and evaluate on CoLBT-Hydro/Herwig jets: a large drop in AUC would falsify the model-agnostic claim.","supporting_citations":[{"cited_title":"Identifying quenched jets in heavy ion collisions with machine learning","cited_arxiv_id":"2206.01628","evidence_quote":"Provides an LSTM baseline AUC (0.76) that this work compares against."}],"review_version":1}