{"id":"159db238-3c17-46c4-9298-9f31d160db00","arxiv_id":"2509.01397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Double descent, a test-error peak followed by recovery at high capacity, appears in jet regression and event classification on ATLAS open data; with early stopping, overparameterized models can beat classical ones.","lead":"This paper reports the first sightings of 'double descent' in particle physics data: models large enough to memorize their training set can get better test performance again after an initial worsening. It maps when overparameterized models beat smaller classical ones, which matters as the field moves toward billion-parameter foundation models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accepting the §4.2 overparameterization benefit requires replacing test-based oracle checkpoint selection with validation-based selection; otherwise the 1B-model gain may be a test-leakage artifact.","rationale":"The paper's central scientific value is twofold: a first observation of double descent in HEP data, and the practical claim that overparameterized models can outperform classical models. The first is supported by the final-error curves in Figs. 1 and 3, subject to the seed-variance caveat. The second is the claim highlighted in the abstract and Sec. 4.2, and it is computed by choosing the training epoch that minimizes test loss. This is an oracle procedure: test labels leak into model selection. The reader's weakest-assumption list already notices this alongside the missing seeds; I single it out because it is a correctness issue that persists even after adding error bars. Averaging over seeds does not cure a selection rule that uses the evaluation set. The proposed validation-based rerun is decisive and cheap: if the 1B advantage vanishes, the headline should be weakened to state that the gain is available only to an oracle that knows the test labels; if it survives, the benefit claim is robust. Since the current manuscript transparently describes the procedure but does not provide the validation-based comparison, the appropriate disposition is the same CONDITIONAL verdict the reader gave, pending the rerun.","tokens_in":5760,"tokens_out":10038,"duration_ms":120876,"concrete_test":"Repeat the Sec. 4.2 model-wise scans (all widths up to the 16-layer, 1.1B model) on the same dataset with the same hyperparameters, but partition the events into training, validation, and held-out test sets. For each width, train for the prescribed number of epochs, evaluate validation loss at each checkpoint, pick the epoch with the lowest validation loss, and evaluate that single checkpoint on the held-out test set. Then plot held-out test error vs. model size alongside the oracle best-test curve from the same runs. If the largest model does not beat the best underparameterized model on held-out test error, the claimed 1B benefit is a test-leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 and Fig. 3 support the paper's strongest practical claim ('a 1B parameter model outperforms classical models') using what the authors call 'early stopping in the sense of taking the epoch checkpoint with the best test-set performance.' This is an oracle selection rule: the test labels are used to choose the training epoch, so the reported early-stopping curve is the running minimum of the test-loss trajectory, not the outcome of any causal model-selection procedure. Larger models can reach a lower oracle minimum even when their final test error is worse; the apparent monotonic benefit for the 1B model may therefore reflect trajectory flexibility rather than better generalization. This does not challenge the final-error double-descent curves in Sec. 3 or Fig. 3, but it directly undermines the abstract's 'performance gain' claim, which is the headline benefit of overparameterization. A fair comparison must select the checkpoint on a validation split and then evaluate on a held-out test split.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports empirical observations of double-descent behavior in particle physics datasets. Using the ATLAS JetSet jet-pT regression task and the ATLAS SUSY Wh1Lbb event-classification dataset, the authors train transformers and MLPs of varying widths and show model-wise and epoch-wise non-monotonic test-loss curves. They claim the first demonstration of double descent in particle physics data, and further claim that in some settings overparameterized models outperform classical models when the best test-loss epoch checkpoint is selected. The paper also compares balanced and unbalanced classification datasets to study the dependence of epoch-wise double descent on class imbalance.","tokens_in":5949,"tokens_out":3635,"duration_ms":42357,"significance":"If the empirical claims are substantiated, the paper would provide useful domain-specific evidence on generalization in HEP machine learning, with implications for the current trend toward large foundation models in particle physics. The paper has strengths: it uses public ATLAS datasets, a transparent training setup, and clear visualizations, and the authors state that code is released. However, the headline claims rest on three load-bearing methodological choices that are not yet adequately supported: the use of oracle test-based early stopping, the absence of uncertainty quantification, and the confounding of class balance with dataset size. These issues can be addressed with additional experiments, so the contribution is potentially valid but requires major revision.","major_comments":[{"comment":"The early-stopping curves are obtained by 'taking the epoch checkpoint with the best test-set performance'. This is an oracle selection rule: the test labels are used to choose the training epoch, so the reported early-stopping value is the running minimum of the test-loss trajectory. A larger model can reach a lower oracle minimum even when its final test error is worse; the apparent benefit of the 1B-parameter model may reflect trajectory flexibility rather than better generalization. The 'performance gain' claim in the abstract is therefore not supported as stated. Please redo the model-selection comparison using a validation split for checkpoint selection and report held-out test error, alongside the final-error curves.","section":"Sec. 4.2, Fig. 3"},{"comment":"All reported curves appear to be single training runs with no seeds, error bars, or confidence intervals. The double-descent signature is a non-monotonicity in test loss; without an estimate of run-to-run variance, the visual features in Figs. 1-3 could be attributable to optimization noise, especially near the interpolation threshold where training dynamics are sensitive. Please run several seeds per configuration and report the spread, or otherwise justify that the phenomena are reproducible across random initializations and data subsamples. This is central to the claim of observing double descent for the first time in particle physics data.","section":"Sec. 2, Figs. 1-3"},{"comment":"The comparison used to support the 'prior dependence' claim contrasts a balanced 800k-event dataset with an unbalanced 7M-event dataset. These two settings differ both in class balance and in total dataset size, so the observed difference in epoch-wise double descent cannot be attributed to class imbalance alone. The hypothesis that class imbalance affects feature-learning speeds is plausible but untested as presented. A controlled comparison should vary class proportions at fixed dataset size, or vary dataset size at fixed class proportions, before drawing this conclusion.","section":"Sec. 4.1, Fig. 2"}],"minor_comments":[{"comment":"Typo: 'straining steps' should be 'training steps'.","section":"Sec. 2"},{"comment":"Typo: 'is a interesting future research direction' should be 'is an interesting future research direction'.","section":"Sec. 3"},{"comment":"The paper states 'we release code' but provides no repository URL or DOI. Please include the actual location of the code.","section":"General"},{"comment":"The caption says 'overparameterised models outperform underparameterised ones', but this is only true under the oracle early-stopping rule. Please qualify this statement or revise it to match the final-error results.","section":"Fig. 3 caption"},{"comment":"Please state explicitly what the 'Early Stopping' curve represents (best test loss over training?) and how it is computed, since this terminology is nonstandard and affects interpretation.","section":"Fig. 1d"},{"comment":"The preprocessing of JetSet (track selection, normalization, tokenization) and the exact train/test split are not described. Please provide enough detail to make the experiments reproducible.","section":"Sec. 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a compact empirical study of double descent in two ATLAS open datasets. The genuinely new thing is the demonstration of model-wise and epoch-wise double descent in jet pT regression and in SUSY event classification, which I don't think has been shown in particle physics data before. That matters: HEP-ML is moving toward large foundation models, and knowing whether the bias-variance curve is monotonic in this domain is useful. The authors deserve credit for using public data, holding the optimizer fixed across model sizes, and being explicit that the behavior is dataset-dependent.\n\nThe soft spots are real, and one is load-bearing. In Section 4.2 and Figure 3, the 'early stopping' claim uses the epoch checkpoint with the best test-set loss. That is an oracle selection rule: it uses test labels to pick the training epoch, so the reported benefit for the 1B-parameter model could just be trajectory flexibility, not a genuine generalization advantage. The authors even write 'early stopping in the sense of taking the epoch checkpoint with the best test-set performance,' so there is no ambiguity. This undermines the abstract's 'performance gain' claim, though it does not necessarily invalidate the existence of double descent in the final-error curves.\n\nA second issue is the lack of error bars: all figures show a single run with no seeds. At the interpolation peak, the differences across widths are large enough to be probably real, but without variance we don't know. A third issue: the balanced vs unbalanced comparison in Section 4.1 changes both class balance and dataset size (800k vs 7M), so the claim that imbalance causes epoch-wise double descent is not actually tested. The 'first in particle physics' claim also reads as an assertion rather than a literature search, though it's plausible.\n\nThe underlying experimental idea is sound and the paper is clearly written. I think it deserves peer review, but a serious referee should require multi-seed runs, a validation-based early stopping procedure, and a controlled imbalance experiment. The code release is claimed but not visible in the arXiv version, so that also needs checking. For a reader working on HEP-ML scaling, this is worth a look and worth citing with caution.\n\nI'd send it to review, expecting major revision.","headline":"A credible first look at double descent in HEP data, but the practical benefit claim rests on test-label oracle checkpoint selection and single runs.","tokens_in":6464,"tokens_out":2807,"would_cite":true,"duration_ms":28911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that double descent—where test error first worsens then improves as model capacity or training time grows—appears in particle physics data, shown for jet pT regression and event classification, and that overparameterized mo","keywords":["double descent","overparameterization","particle physics","jet pT regression","event classification","epoch-wise double descent","model-wise double descent","early stopping"],"falsifier":"Retrain the same model families with several random seeds and a small grid of learning rates, then measure test loss at each width and epoch; the central claim weakens if the double-descent peaks and the overparameterization benefit do not reproduce. Alternatively, rerun the early-stopping comparison using a held-out validation set instead of the test set; if the 1B-model advantage disappears, the benefit claim depends on leaking test information into checkpoint selection.","tokens_in":5599,"feed_emoji":"📉","tokens_out":4770,"duration_ms":56204,"temperature":0.7,"pith_summary":"The paper aims to establish that double descent, a non-monotonic generalization curve previously seen in images, quantum systems, and protein folding, also occurs in particle physics data. It reports the first explicit instances of both model-wise double descent (across model width) and epoch-wise double descent (across training time) in a jet pT regression task, and model-wise double descent in a SUSY-versus-Standard-Model event classification task. It further claims that when the best test-loss checkpoint is selected, a network with roughly one billion parameters can beat classical models even on a modest 150k-event dataset. The broader point is that generalization behavior is task- and dataset-dependent, so scaling decisions in particle physics should be informed by where overparameterization actually helps.","feed_headline":"A 1B-parameter model beats classical models on modest data","feed_subtitle":"First evidence of epoch- and model-wise double descent in particle physics data","key_machinery":"The central object is the double-descent curve: test error plotted against model width (model-wise) or training epoch (epoch-wise), anchored by the interpolation threshold, the capacity at which training loss first vanishes. In the jet regression task, the scaling parameter is the transformer width d_model; in classification, it is the hidden-layer width of a three-layer or sixteen-layer MLP. The experimental protocol holds optimizer and batch size fixed across all model sizes—Adam with learning rate 1e-4, batch size 128, and a 5% warmup for transformers—so that differences in test loss are attributed to model size and training time rather than to optimizer choices.","core_discovery":"The central discovery is empirical: neural networks trained on particle physics data reproduce the non-monotonic generalization curves associated with double descent. In jet transverse-momentum regression with a two-layer transformer, test loss as a function of model width peaks near the interpolation threshold—where training loss first reaches zero—and then decreases again, which is model-wise double descent. Along training time, test loss also decreases, increases, and then decreases again, which is epoch-wise double descent. In a 15-way event classification task, model-wise double descent appears when models train far past the interpolation threshold; if instead the checkpoint with the be","pith_inferences":["A testable extension the paper does not run: reweight the imbalanced 7M-event dataset to equal class frequencies while keeping its size, to confirm that class imbalance, rather than dataset size or process composition, triggers epoch-wise double descent.","The best-test-checkpoint comparison leaks test information into model selection; a fairer practical rule would use a held-out validation set, and the claim that 1B-parameter models beat classical models should be rechecked under that rule.","If the observed curves are stable across random seeds and optimizer settings, particle physics practitioners could adopt a simple recipe: train wide models, evaluate checkpoints on a validation set, and expect the interpolation peak to be avoidable.","The results imply that foundation-model scaling studies in high-energy physics should include per-task scans over model width and training time, because aggregate scaling laws may hide non-monotonic generalization behavior."],"forward_implications":["If correct, test error in particle physics tasks is not a monotone function of model size or training time, so model selection must look beyond the interpolation peak and beyond the first test-loss minimum.","Overparameterized networks can, under best-test-checkpoint selection, outperform smaller classical models even with modest training data, supporting continued scaling of high-capacity models in the field.","Double descent is not universal across particle physics tasks: jet pT regression shows both model-wise and epoch-wise double descent, while event classification shows model-wise but not always epoch-wise double descent.","Class imbalance is a plausible driver of epoch-wise double descent in classification, since balanced and imbalanced datasets of the same task behave differently.","The benefit of overparameterization in the small-data classification case appears only when the best test-loss checkpoint is chosen, not when comparing final checkpoints."],"supporting_citations":[{"why":"Defines the deep double-descent generalization curve that the paper claims to reproduce in particle physics data.","marker":"[13]"},{"why":"Supplies the implicit-bias mechanism invoked to explain why large interpolating models can generalize well.","marker":"[14]"},{"why":"Provides the account of epoch-wise double descent used to frame the epoch-direction test-loss trajectories.","marker":"[15]"},{"why":"Provides the JetSet dataset used for the jet pT regression experiments.","marker":"[19]"},{"why":"Provides the SUSY Wh1Lbb dataset used for the event classification experiments.","marker":"[22]"},{"why":"Specifies the Adam optimizer with fixed hyperparameters across all model sizes, the key isolation assumption of the study.","marker":"[23]"}],"fun_headline_variants":["Double descent found in particle physics models","First evidence of double descent in particle physics","Overparameterized nets beat classical models in particle physics","Particle physics models show double descent benefit"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The generalization curves rest on single runs with fixed Adam hyperparameters, so differences across widths and epochs could in principle be artifacts of training dynamics or randomness rather than genuine double descent; the overparameterization benefit additionally depends on selecting checkpoints by test loss.","fun_headline_variants_meta":{"raw":{"variants":["Double descent found in particle physics models","First evidence of double descent in particle physics","Overparameterized nets beat classical models in particle physics","Particle physics models show double descent benefit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":1752,"prompt_tokens":559,"completion_tokens":1193,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":303,"completion_tokens_details":{"reasoning_tokens":1137}},"tokens_in":303,"tokens_out":1193,"duration_ms":11560,"temperature":1.0,"reasoning_tokens":1137,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:34:33.957590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same model families with several random seeds and a small grid of learning rates, then measure test loss at each width and epoch; the central claim weakens if the double-descent peaks and the overparameterization benefit do not reproduce. Alternatively, rerun the early-stopping comparison using a held-out validation set instead of the test set; if the 1B-model advantage disappears, the benefit claim depends on leaking test information into checkpoint selection.","supporting_citations":[{"cited_title":"The implicit bias of gradient descent on separable data","cited_arxiv_id":null,"evidence_quote":"Supplies the implicit-bias mechanism invoked to explain why large interpolating models can generalize well."},{"cited_title":"ATLAS ttˆ Simulation for ML-based Jet Flavour Tagging (JetSet)","cited_arxiv_id":null,"evidence_quote":"Provides the JetSet dataset used for the jet pT regression experiments."},{"cited_title":"ATLAS SUSY Searches in Wh1Lbb Channel Open Data Set","cited_arxiv_id":null,"evidence_quote":"Provides the SUSY Wh1Lbb dataset used for the event classification experiments."}],"review_version":1}