{"id":"a94d5069-1d53-4320-b27b-2293cc9cee78","arxiv_id":"2507.11774","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GNN and transformer models improve atmospheric muon rejection and directional reconstruction for IceCube's extremely high energy neutrino selection.","lead":"A machine learning upgrade to IceCube's neutrino selection uses a graph network to reject muon bundles from cosmic-ray showers and a transformer network to reconstruct event directions. In simulations, the new methods improve background rejection and deliver more well-reconstructed neutrino events at wide opening angles.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Both networks appear to be evaluated on the same Monte Carlo events they were trained on, with no held-out test set described; this can inflate the reported background rejection and angular resolution gains.","rationale":"The reader's weakest assumption is about Monte Carlo fidelity, which is a real concern for any simulation-trained network. However, the most load-bearing issue in the paper as written is the apparent absence of a held-out test set. The text explicitly ties the training charge thresholds to the analysis sample: the classifier is trained above 27,500 PE and evaluated on the same threshold; the MTNN is trained above 1,000 PE, and the analysis sample at 27,500 PE is a strict subset of that. No mention of a validation split or cross-validation appears anywhere. This makes the quantitative claims in Table 1 and the ROC/angular-resolution figures potentially optimistic, and it is a standard methodological requirement that can be verified from the manuscript. The Monte Carlo fidelity concern, while important, is less immediately decisive because it requires external data to resolve; the in-sample issue can be settled by inspecting event IDs or retraining on a split. Therefore I keep the reader's CONDITIONAL verdict, but add a sharper condition: the authors must demonstrate that reported performance is on held-out Monte Carlo events. This is a partial agreement because I accept the reader's simulation-fidelity concern as a secondary issue, but I identify a different primary vulnerability. The proposed concrete test is a single, focused retraining/fold check that directly addresses the load-bearing concern.","tokens_in":10945,"tokens_out":11113,"duration_ms":130092,"concrete_test":"Ask the authors to provide the event-level train/validation split (e.g., unique event IDs) and recompute Figures 3, 7, and Table 1 using only the held-out validation events. If the validation-set ROC curve, angular resolution distributions, and 15-year event rates differ materially from the reported values (for example, the Table 1 counts shift by more than 20% or the MTNN advantage over SplineMPE shrinks), the in-sample evaluation is the cause. A cleaner check is to retrain both networks on an 80% random split of the Monte Carlo and evaluate on the disjoint 20%, then compare the performance metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper never states that a train/validation split was used, and the text implies the evaluation is in-sample. Section 2.2 says the GNN classifier is trained on events passing a minimum charge cut of 27,500 PE; Section 2.3 evaluates it on the same weighted analysis sample. Section 3.1 says the MTNN is trained on events exceeding 1,000 PE homogenized charge, while Section 3.2 evaluates on the analysis sample requiring 27,500 PE, which is a subset of the training set. If the ROC curves (Fig. 3), angular resolution histograms (Fig. 7), and the 15-year event rates (Table 1) are computed on training data, they can be optimistically biased due to overfitting, independent of whether the simulation is accurate. This directly weakens the central claim of improved background rejection and directional reconstruction, because the quantitative gains could be artifacts of memorization rather than genuine generalization. Even a perfect detector simulation would not rescue the claim if the evaluation is in-sample. The reader's concern about Monte Carlo fidelity is valid, but the missing held-out evaluation is a more immediate and testable flaw.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This proceedings paper describes two machine-learning tools for IceCube extremely-high-energy (EHE) neutrino searches. A graph neural network (GNN), built with the DYNEDGE architecture in GraphNeT, classifies neutrino-induced events against atmospheric muon bundles by exploiting the lateral spread of energy deposits. A multitask transformer (MTNN) performs directional reconstruction without assuming a single-track topology. The authors report improved muon background rejection in ROC curves, better angular resolution than the SplineMPE algorithm for most events, and expected 15-year neutrino event rates below various opening-angle thresholds (Table 1), weighted to astrophysical and atmospheric flux models. The work is framed as a proof-of-concept for future EHE selections.","tokens_in":11135,"tokens_out":6840,"duration_ms":79400,"significance":"If the reported gains survive a proper held-out evaluation, this is a useful methodological contribution for the next generation of IceCube EHE analyses. The paper has concrete strengths: it relies on publicly available software (GraphNeT/DYNEDGE), gives detailed descriptions of training weights and loss choices, compares against an existing IceCube algorithm, and provides falsifiable 15-year rate predictions with bootstrap uncertainties. However, the central claims currently rest on two load-bearing points that need work: (i) no held-out test set is described, so the reported improvements may be in-sample artifacts; and (ii) the SplineMPE baseline is used across all event topologies even though it is designed for single muon tracks, so part of the reported improvement may be an artifact of an unsuitable baseline. These issues are fixable and do not by themselves invalidate the approach, but they are essential for the paper's conclusions.","major_comments":[{"comment":"No held-out test set is described anywhere in the paper. The GNN is trained on events passing a minimum charge cut of 27,500 PE (§2.2) and then evaluated on 'the analysis sample' with the same cut (§2.3); the MTNN is trained on events above 1,000 PE homogenized charge (§3.1) and evaluated on a subset with ≥27,500 PE (§3.2). The ROC curves (Fig. 3a), angular resolution distributions (Fig. 7), and Table 1 rates therefore appear to be in-sample. Hyperparameter choices such as the focal-loss γ=1.5 are also made from the same events. This can inflate all reported gains through memorization rather than generalization. Please state the exact train/validation/test split used for each network and recompute all headline numbers on a held-out sample; if a split was used but omitted, it must be documented.","section":"Sections 2.2–2.3 and 3.1–3.2"},{"comment":"The comparison with SplineMPE is not apples-to-apples. SplineMPE is a single-track likelihood reconstruction, while the MTNN is explicitly designed to handle cascades and other topologies. The paper states that 'a large part of the difference is driven by cascade-like events,' so the factor-of-several gains in Table 1 and the per-flavor medians in Fig. 8 may largely reflect SplineMPE's known poor performance on non-track topologies rather than genuinely better direction reconstruction. Please report the comparison separately for track-like and cascade-like events, and/or include a non-track-capable reference algorithm (e.g., a cascade reconstruction or a second ML baseline) so the claimed improvement is demonstrated on comparable footing.","section":"Section 3.2, Figure 8, Table 1"},{"comment":"The central performance curves are shown without uncertainty bands. Figure 3a presents ROC curves for different focal-loss parameters with no confidence intervals, and Figure 7 presents opening-angle histograms without statistical errors. Since Table 1 does provide bootstrap uncertainties, the omission is not a question of means; the reader cannot tell whether the differences between γ values or between MTNN and SplineMPE are statistically significant. Please add bootstrap or equivalent uncertainties to these figures, or provide the relevant intervals in the text.","section":"Figure 3a and Figure 7"},{"comment":"All results are derived exclusively from CORSIKA/SIBYLL/H4a Monte Carlo. For a selection that is intended to be used on real data, the absence of any data/MC closure check is a concern. At minimum, the paper should state explicitly that the results are simulation-only, and ideally show a comparison of the classifier score distribution or the MTNN-vs-SplineMPE agreement on a high-charge atmospheric-muon data sample (e.g., through-going muons) where the simulation can be validated. This would make the central 'improved background rejection' claim robust to the assumption that the simulated lateral light distribution is accurate.","section":"Sections 2.3 and 3.2 (data/MC validation)"}],"minor_comments":[{"comment":"The text is full of missing spaces (e.g., 'KM3NeThasrecentlyreported' in the abstract). This makes the paper hard to read and should be fixed in the published version.","section":"Abstract and throughout"},{"comment":"The term 'homogenized charge' is used in the body (§2.2, §3.2) but only defined in the caption to Fig. 1b; please define it at first use in the main text.","section":"Section 2.2 / Figure 1b"},{"comment":"The column headers use 'NN' without specifying that this denotes the MTNN. Since the paper presents both a GNN and an MTNN, clarify the notation.","section":"Table 1"},{"comment":"The model notation is inconsistent: 'GaisserH4a' appears alongside 'H4a' and 'GaisserH4a model'. Please unify the naming of the hadronic interaction model.","section":"Section 2.3"},{"comment":"The figure includes the focal-loss formula in the inset, but the caption does not identify the meaning of the inset or define the variables; please annotate the equation clearly.","section":"Figure 3a"},{"comment":"The paper describes applying the GNN and MTNN 'in conjunction' with existing selection methods but does not specify how the classifier score and the MTNN direction are combined into a single event selection; a flow diagram or a concise algorithm description would help.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"This is an ICRC proceedings contribution, so the completeness bar is lower than for a full journal article. Still, the missing held-out evaluation is a standard ML-methods requirement that cannot be waived, and the SplineMPE baseline comparison needs to be made fair across event topologies. Both issues are fixable within the scope of the paper, hence major revision rather than rejection. The simulation-only scope should be stated explicitly, but I do not consider the lack of real-data validation alone to be fatal for a methods paper at this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely interesting methods contribution for IceCube EHE analyses, but I would not trust the quoted gains until they show a held-out evaluation.\n\nThe stress-test concern lands: the evaluation appears to be in-sample. Section 2.2 trains the GNN on events with at least 27,500 PE; Section 2.3 then evaluates the ROC on the same weighted analysis sample, which by construction contains those events. The MTNN is trained on everything above 1,000 PE and then evaluated on the >27,500 PE analysis sample, again a subset of training data. If there is a held-out split, it is not described. This means the ROC curves, the angular resolution histograms, and the 15-year event rates in Table 1 could all be positively biased from memorization. That is a more immediate issue than the Monte Carlo fidelity question, and it is easily testable. The bootstrap uncertainties in Table 1 only cover statistical re-sampling; they say nothing about overfitting.\n\nWhat is actually new: using the lateral spread of muon bundles as a GNN classification feature is a smart, physically motivated move. The multitask transformer is a reasonable way to get topology-agnostic directional reconstruction, and the supporting-task design is thoughtful. The paper is clearly written and gives enough detail to understand the approach. For a proceedings paper, that is a solid contribution.\n\nOther soft spots are secondary: no error bars on the ROC curves, no real-data validation or closure test, and several cuts (charge threshold, focal loss gamma, class composition) are chosen from the same simulation. None of these are fatal on their own, but combined with the in-sample issue they mean the quantitative claims are not yet established.\n\nBottom line: this is a paper for the IceCube EHE group and for people working on ML in neutrino telescopes. The conceptual contribution is worth engaging with, but the performance numbers need a proper train/test split and ideally a data/MC comparison before they can be used to guide a future selection. I would send it to peer review and ask for those additions.","headline":"A useful methods paper whose performance numbers are not yet trustworthy: the evaluation appears to be on the training sample, so the claimed gains could be largely overfitting.","tokens_in":11663,"tokens_out":3890,"would_cite":false,"duration_ms":42299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining a graph neural network for muon-bundle rejection with a multitask transformer for direction reconstruction can roughly quintuple the number of IceCube EHE neutrino candidates reconstructed within 5 degrees…","keywords":["IceCube","extremely high energy neutrinos","graph neural network","transformer","atmospheric muon background","directional reconstruction","lateral energy spread","multitask learning"],"falsifier":"Run the trained GNN and MTNN on real IceCube events with independently known directions (for example, muons whose direction is fixed by IceTop or the Moon shadow) and compare the classifier score distribution and the opening-angle residuals to the simulation-based predictions; a significant mismatch in either would directly invalidate the claimed background rejection and reconstruction gains.","tokens_in":10740,"feed_emoji":"🧊","tokens_out":9960,"duration_ms":102642,"temperature":0.7,"pith_summary":"This paper claims that two neural networks can substantially improve IceCube's search for neutrinos above 10 PeV (the extremely-high-energy range), where a recent detection by another telescope has renewed interest. A graph neural network separates neutrino-induced tracks from cosmic-ray muon bundles by using the lateral spread of light deposits, without any prior on the event direction. A multitask transformer then reconstructs each event's direction without assuming it is a single straight track, which is the assumption baked into the current SplineMPE likelihood method. On a simulated 15-year exposure, the combination yields about five times more neutrino candidates reconstructed within 5 degrees of the true direction at the same atmospheric-muon background rate. If the performance carries over to real data, the selection would improve diffuse flux measurements above 10 PeV and increase the chance of catching an IceCube event comparable to the recent very-high-energy detection.","feed_headline":"Neural nets boost sharp EHE neutrino events 5-fold","feed_subtitle":"A transformer and a graph network yield ~89 well-tracked neutrino candidates per 15 years, versus ~16 with current reconstruction.","key_machinery":"The load-bearing machinery has two parts. First, a graph neural network (GNN) with one node per DOM, using position, first-hit time, and total charge as features, trained with a focal cross-entropy loss to separate neutrino events from cosmic-ray muon bundles; its discriminative power tracks the charge-weighted RMS of energy-deposit distances from the leading trajectory, i.e., the lateral spread. Second, a multitask transformer-based neural network (MTNN) that summarizes pulses at the DOM level (total charge, charge-weighted mean time, timing and charge percentiles, and closest times at fixed charge fractions), then uses a classifier token and supporting tasks (visible length, trackness, position, calorimetric energy) before final tasks (direction and energy) to achieve inductive transfer. Both networks process events above a 27,500-photoelectron charge threshold; the comparison baseline is SplineMPE, a likelihood reconstruction that assumes an infinite single muon track.","core_discovery":"The central claim is that the dominant background in IceCube's extremely-high-energy neutrino search—dense muon bundles from cosmic-ray showers—can be rejected using the lateral spread of their light deposits, and that a topology-agnostic neural network can reconstruct neutrino directions better than the standard likelihood method for most events. The graph network classifies each event using each DOM's position, first-hit time, and total charge; its score correlates strongly with the charge-weighted RMS of deposit distances from the event's leading trajectory, so thin single-muon-like events score as neutrino-like and laterally spread bundles are rejected. The multitask transformer reconstructs direction through supporting tasks (trackness, interaction position, visible length, calorimetric energy) that feed final tasks (direction and energy) and make no prior assumption about event topology, unlike SplineMPE's single-infinite-track model. On a simulated 15-year exposure with a bundle-rejection cut of 0.9, the transformer gives 20.8 neutrino events within 1 degree and 88.6 within 5 degrees of the true direction, compared with 12.5 and 15.7 for SplineMPE, at the same atmospheric-muon rate of roughly 85 events; for neutrinos above 10 PeV the 5-degree yield rises from 2.62 to 5.39.","pith_inferences":["Because the GNN score correlates with lateral RMS, a simple analytic lateral-spread cut could capture much of the classifier's power and would be far easier to validate on real data; the network likely learns this physical observable rather than an uninterpretable pattern.","The same topology-agnostic multitask transformer design could transfer to other neutrino telescopes or to IceCube-Gen2, where higher pulse densities make DOM-level summarization and prior-free reconstruction more valuable.","The reported yields rest entirely on simulation; a calibration with real through-going muons of known direction would both validate the claimed gains and, if discrepancies appear, provide the data-driven corrections needed to retain them."],"forward_implications":["A future EHE selection built on these networks would recover roughly 5–6 times more neutrino candidates within 5 degrees of the true direction than the SplineMPE-based selection at the same muon-bundle background rate.","The 15-year sample of about 89 well-reconstructed candidates and about 85 atmospheric muons at a score cut of 0.9 is well matched to a diffuse flux measurement above 10 PeV.","Because the transformer is trained on all flavors and topologies, cascade and starting-track events—where SplineMPE's single-track prior fails—gain the most reconstruction accuracy.","The lateral-spread classifier can be combined with existing overburden and stochastic energy-loss cuts to suppress downgoing-region background, where atmospheric muons are hardest to remove."],"supporting_citations":[{"why":"This paper extends this prior EHE selection, which used stochastic energy-loss profiles.","marker":"[8]"},{"why":"This work defines the focal-loss objective used to train the classifier.","marker":"[9]"},{"why":"This work supplies the graph neural network architecture used for the muon-bundle classification.","marker":"[12]"},{"why":"This work provides the software implementation used to build the neural network models.","marker":"[13]"},{"why":"This work generates the CORSIKA air-shower simulation for the cosmic-ray muon bundle background.","marker":"[14]"},{"why":"This work supplies the SIBYLL 2.3d hadronic interaction model used in the cosmic-ray simulation.","marker":"[15]"},{"why":"This work defines the SplineMPE likelihood reconstruction that serves as the comparison baseline.","marker":"[16]"},{"why":"This work provides the transformer architecture components that the multitask network adapts.","marker":"[18]"},{"why":"This work supplies the per-DOM pulse summarization that makes high-energy events tractable for training.","marker":"[19]"}],"fun_headline_variants":["GNN and transformer improve IceCube's EHE neutrino selection","AI nets enhance IceCube's extreme-energy neutrino detection","Graph and transformer nets upgrade IceCube's EHE neutrino analysis","Machine learning sharpens IceCube's extreme neutrino reconstruction","Neural networks boost IceCube's highest-energy neutrino yield"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The networks are trained and tested only on simulated events, so the claimed gains hinge on the simulation matching how real cosmic-ray muon bundles deposit and spread light in the detector.","fun_headline_variants_meta":{"raw":{"variants":["GNN and transformer improve IceCube's EHE neutrino selection","AI nets enhance IceCube's extreme-energy neutrino detection","Graph and transformer nets upgrade IceCube's EHE neutrino analysis","Machine learning sharpens IceCube's extreme neutrino reconstruction","Neural networks boost IceCube's highest-energy neutrino yield"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1571,"prompt_tokens":1017,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":633,"tokens_out":554,"duration_ms":7177,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:01:10.421836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained GNN and MTNN on real IceCube events with independently known directions (for example, muons whose direction is fixed by IceTop or the Moon shadow) and compare the classifier score distribution and the opening-angle residuals to the simulation-based predictions; a significant mismatch in either would directly invalidate the claimed background rejection and reconstruction gains.","supporting_citations":[{"cited_title":"Focal loss for dense object detection,","cited_arxiv_id":null,"evidence_quote":"This work defines the focal-loss objective used to train the classifier."},{"cited_title":"Abbasiet al.,Journal of Instrumentation17no","cited_arxiv_id":null,"evidence_quote":"This work supplies the graph neural network architecture used for the muon-bundle classification."},{"cited_title":"Søgaardet al.,Journal of Open Source Software8 no","cited_arxiv_id":null,"evidence_quote":"This work provides the software implementation used to build the neural network models."},{"cited_title":"CORSIKA: A Monte Carlo code to simulate extensive air showers,","cited_arxiv_id":null,"evidence_quote":"This work generates the CORSIKA air-shower simulation for the cosmic-ray muon bundle background."},{"cited_title":"Riehn, R","cited_arxiv_id":null,"evidence_quote":"This work supplies the SIBYLL 2.3d hadronic interaction model used in the cosmic-ray simulation."},{"cited_title":"Bradascio and T","cited_arxiv_id":null,"evidence_quote":"This work defines the SplineMPE likelihood reconstruction that serves as the comparison baseline."},{"cited_title":"Bukhariet al.,Eur","cited_arxiv_id":null,"evidence_quote":"This work provides the transformer architecture components that the multitask network adapts."},{"cited_title":"Abbasiet al.,Journal of Instrumentation16no","cited_arxiv_id":null,"evidence_quote":"This work supplies the per-DOM pulse summarization that makes high-energy events tractable for training."}],"review_version":1}