{"id":"6e5489bb-b681-45a0-8449-7cddcf63da81","arxiv_id":"2510.10541","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training RL models on benchmark test sets yields nearly the same scores as training on training sets, and stress tests show the resulting skills are brittle—evidence that current RL benchmarks fail to measure generalization.","lead":"This paper argues that popular RL reasoning benchmarks no longer measure generalization, because training on the test set performs about the same as training on the training set. It introduces a diagnostic metric (OPG) and stress tests showing brittle, recitation-like behavior that current scores hide.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under-trained RL oracle may explain OPG≈0; no convergence evidence reported.","rationale":"The reader flagged the OPG interpretive criterion (a useful benchmark must produce OPG ≫ 0) as the weakest assumption. I agree that this criterion is asserted rather than justified, but I see an even more immediate empirical threat to the OPG measurement itself: the RL oracle may be undertrained, making OPG ≈ 0 an artifact. This is more load-bearing because it does not depend on whether the OPG criterion is accepted; if the oracle can be trained to much higher test accuracy, the reported near-zero OPG would be wrong, not merely misinterpreted. The paper provides no evidence that the oracle has converged. The low absolute accuracy of the oracle relative to what memorization would imply (and the fact that SFT oracles achieve large OPG while RL oracles do not) strongly suggests hyperparameter or training-budget issues. A simple check—train the RL oracle longer and with weaker KL regularization—would adjudicate this. My recommended verdict is UNCHANGED: the paper should still be conditional pending this test, but the condition should be expanded to explicitly include oracle convergence. I agree only partially with the reader because the interpretive concern is valid but secondary to the empirical validity of the OPG estimate.","tokens_in":14302,"tokens_out":7533,"duration_ms":65466,"concrete_test":"Train the MATH 3B RL oracle for 20 epochs (instead of 5) and with KL coefficient lowered to 1e-3; report its accuracy on the test prompts used for training. If the oracle's test-set accuracy rises substantially above the 100%-train-trained model (e.g., from 64.4% to >80%), then the OPG in Table 1 is an artifact of under-training and the paper's central claim fails. If the oracle's accuracy remains ≈64% even with the extended budget and weaker KL, the vanishing gap is robust to training budget.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central finding is that RL models trained on a benchmark's train split perform about as well as models trained directly on the test split (OPG ≈ 0, Tables 1–2), which is interpreted as test-set unseen-ness being no longer a valid measure of generalization. This interpretation rests on the RL oracle (M_RL,test) serving as a faithful upper bound. Section 2.1.2 defines this oracle as trained with the same GRPO setup as the train model; Appendix A.1 specifies KL regularization with coefficient 1e-2 and only 5 epochs. No learning curves, convergence checks, or the oracle's training-set accuracy are reported. The oracle's absolute test accuracy is surprisingly low (e.g., 64.4% on MATH 3B vs. 64.2% for train-trained), far below what memorization of the test prompts would imply. This raises the possibility that near-zero OPG reflects an undertrained or over-regularized oracle rather than a property of the benchmark. If KL regularization prevents the policy from moving far from the base model, or 5 epochs is insufficient, the oracle cannot exploit the test split even in principle, making OPG artificially small. The paper's subset experiment (10–100% of train data) varies only train-side data quantity and does not test the oracle's capacity. Thus the conclusion that 'unseen-ness is no longer a valid measure' depends on an unverified condition: that the oracle is a genuine upper bound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Oracle Performance Gap (OPG), defined in Eq. (1) as the normalized difference between a model trained on the test split and a model trained on the train split. Across MATH, GSM8K, HeadQA, and DeepScaler with Qwen2.5-3B/7B models, the authors report that OPG is near zero for GRPO-trained RL models but large for SFT models, concluding that unseen-ness is no longer a valid measure of RL generalization. The paper then introduces three stress tests (difficulty stratification, distributional shift, and counterfactual reasoning) to show that RL-learned skills are brittle, and it proposes three benchmark design principles: difficulty stratification, distributional robustness, and counterfactual reasoning.","tokens_in":14681,"tokens_out":5587,"duration_ms":49161,"significance":"If the central claim were well supported, the paper would make an important contribution to RL evaluation for LLMs: it offers a concrete metric (OPG), a multi-benchmark study, and a set of actionable design principles. The difficulty-stratified analysis is a useful idea, and the distribution/counterfactual tests are creative diagnostics. However, the empirical support is currently insufficient. The OPG interpretation depends on an unverified oracle, the headline tables report single runs without variance, the distribution test has severe sampling noise, the counterfactual test lacks a baseline, and the central interpretive standard (Eq. 2) is asserted rather than validated. These issues are load-bearing for the paper's main conclusion, so the paper cannot be accepted in its current form. With additional experiments and careful statistical treatment, the core ideas could become a valuable contribution.","major_comments":[{"comment":"The RL oracle M_RL,test is defined as a GRPO model trained on the test split with the same setup, but no convergence evidence is provided. With only 5 epochs and a KL coefficient of 1e-2, the oracle may be under-trained or over-regularized, unable to exploit the test set even in principle. The paper never reports the oracle's training-set accuracy or learning curves. This is critical because OPG≈0 is interpreted as 'unseen-ness is insufficient'; if the oracle is not a genuine upper bound, the OPG is artificially small. Please report learning curves, training-set accuracy, and verify that longer training or a lower KL coefficient does not substantially improve test accuracy.","section":"§2.1.2 / Appendix A.1"},{"comment":"The headline OPG tables report single runs without variance. Many OPG values are sub-1pp (e.g., 0.31, -0.54, 0.73, 0.89) and are within binomial sampling error for the reported accuracies. More importantly, Table 1 contains a DeepScaler 7B OPG of -5.07%, meaning the train-trained model beats the oracle by 5% relative; this directly contradicts the claim that OPG is 'consistently low' across benchmarks. Please report multiple seeds, confidence intervals, and explicitly discuss the DeepScaler 7B outlier.","section":"§2.2 / Tables 1–2"},{"comment":"The distribution test uses only 80 samples per distance bin. The performance gains at d2–d5 are +0.75, 0.00, -1.25, -2.50 percentage points. For n=80 and accuracies around 45%, the standard error of the difference is roughly 7–8 percentage points, so none of these values is statistically distinguishable from zero. The claimed 'performance inversion' in Finding 4 is not supported by the data as reported. Please provide confidence intervals, increase sample size, or run multiple seeds and report the distribution of gains.","section":"§3.2.1 / Table 3"},{"comment":"The counterfactual test lacks a baseline. The paper reports only RL-trained models on D_bal and D_cf. Without measuring the base model or an SFT model on the same D_cf, one cannot attribute the drop from 64.2/74.8 to 36.0/41.2 to RL training, to the counterfactual transformation, or to a general inability to follow the novel premise. Please add such baselines and also report the size of D_cf.","section":"§3.2.2 / Table 4"},{"comment":"The central interpretive claim is that OPG≫0 defines a benchmark that effectively measures generalization. This threshold is asserted, not derived or empirically validated. A benchmark could have OPG≈0 and still be highly discriminative among RL policies if training on the train split already captures the test distribution; conversely, a large OPG could reflect oracle overfitting rather than a good benchmark. To make the paper's conclusion stand, the authors should demonstrate that OPG correlates with an independent measure of benchmark quality, such as the ability to rank known strong versus weak policies. Without this, the paper's main conclusion is an unvalidated interpretive standard applied to its own measurements.","section":"§2.1.1 / Eqs. (2)–(3)"}],"minor_comments":[{"comment":"'Generalizatgion' is a typo in the section heading.","section":"§2 title"},{"comment":"The worked counterfactual example contains contradictory numbers: the correct answer is given as both 3 and 9, and the model's answer as both 1 and 14/3. These inconsistencies undermine the qualitative evidence.","section":"Appendix E.2"},{"comment":"The axis labels contain 'uni00A0' artifacts; the Unicode issues should be fixed.","section":"Figure 2"},{"comment":"Core training set selection uses t-SNE coordinates, which are stochastic. The t-SNE random seed and exact parameters must be reported for reproducibility; otherwise the protocol is not reproducible.","section":"Appendix D"},{"comment":"The difficulty annotation is performed by a single LLM (Gemini 2.5 Pro) with no validation. Report inter-annotator agreement or a human-reviewed sample to support the rubric's reliability.","section":"Appendix B.1"},{"comment":"The 'oracle gap' in Figure 2(a) appears to be a different quantity from the OPG in Eq. (1). Define it explicitly to avoid confusion.","section":"§3.1.1 / Figure 2(a)"},{"comment":"The sample sizes for D_bal and D_cf are not reported; please include them.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the diagnostic ideas are creative, but the empirical evidence for the central claim is currently insufficient. I would ask the authors to add convergence checks for the oracle, multi-seed variance reporting, baselines for the counterfactual test, and a validation of the OPG criterion. If these are provided, the paper could be a solid contribution to RL evaluation; as it stands, the conclusions outrun the data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result here is real and worth knowing: RL models trained on the train split of MATH, GSM8K, HeadQA, and DeepScaler match the performance of models trained directly on the test split. That is a surprising measurement, and I don't know of it being reported before. If it holds up, it puts real pressure on the standard train/test protocol for RL post-training. The difficulty-stratified analysis is also a nice addition—showing that aggregate scores hide a large asymmetry in transfer from hard to easy versus easy to hard. Those two pieces are the paper's genuine contribution.\n\nThat said, the paper's own framing overshoots. The OPG is defined as a benchmark-failure criterion (Eq. 2: OPG ≫ 0 is required), but that criterion is asserted, not argued for. A benchmark could still separate good from bad RL policies even if train-trained and test-trained models happen to score alike; the paper never addresses that. The stress-test note about the oracle is also on point: no learning curves, no convergence checks, and the oracle is trained with KL regularization at 1e-2 for only 5 epochs. Its absolute test accuracy is suspiciously close to the train-trained model and far below what test-set memorization would imply, so the near-zero OPG could partly reflect an oracle that never had the capacity to exploit the test set in the first place. The 10–100% train-subset experiment doesn't resolve this because it only varies train-side data.\n\nSmaller issues: the headline OPG tables report single runs with no variance, and differences below 1 percentage point are interpreted as meaningful. The distribution test's key inversion (Table 3) rests on 80-problem bins, where a 2.5-point drop is within binomial noise. The counterfactual test has no baseline—showing that an RL model collapses on invented rules is interesting, but without a base model or SFT comparison we don't know if this is an RL-specific failure or a general LLM limitation. These are fixable with more runs and a few extra baselines.\n\nWho is this for? Anyone working on RL evaluation or benchmark design for reasoning models will find the OPG result and the difficulty-stratified diagnostics useful. It deserves a serious referee, but it needs revision before the strong conclusion is accepted: report variance, release the oracle training curves, add baselines to the stress tests, and soften the claim from \"benchmarks are insufficient\" to \"aggregate unseen-ness is no longer a sufficient criterion for these settings.\"","headline":"The OPG measurement is a genuinely new and useful diagnostic, but the strong conclusion that benchmarks are broken rests on an asserted criterion and an oracle that may be undertrained.","tokens_in":15129,"tokens_out":1208,"would_cite":true,"duration_ms":12717,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that RL-trained reasoning models can match test-set-trained 'oracle' models while trained only on the training split, so held-out accuracy no longer measures RL generalization, and stress tests show the learned skills are","keywords":["reinforcement learning","LLM reasoning","benchmark evaluation","generalization gap","Oracle Performance Gap","distribution shift","counterfactual reasoning","difficulty stratification"],"falsifier":"Run the same train-versus-test OPG comparison on a test set that is semantically disjoint from training (for example, generated with a held-out problem-generation routine). A large positive OPG on that version would show the vanishing gap is a property of the current benchmark's homogeneity rather than of RL in general.","tokens_in":14249,"feed_emoji":"🧪","tokens_out":5771,"duration_ms":52019,"temperature":0.7,"pith_summary":"The paper tries to establish that current benchmarks for evaluating RL-trained LLM reasoning are structurally incapable of revealing how well these models generalize. The key evidence is that an RL model trained on a benchmark's training split scores about the same as an RL model trained directly on the test split, so the test set's 'unseen-ness' no longer acts as a meaningful obstacle. The paper then shows that high benchmark scores coexist with brittle skills: specialists trained on easy problems fail on hard ones, narrow-distribution fine-tuning can make the model worse than baseline on far-out-of-distribution data, and switching a problem to a counterfactual rule causes a large accuracy drop. It concludes with three principles for faithful benchmarks: difficulty stratification, distributional robustness, and counterfactual reasoning.","feed_headline":"Train-set RL equals test-set RL: benchmarks miss generalization","feed_subtitle":"Near-zero oracle gap: train-split and test-split RL score alike; new tests should stress robustness and counterfactuals","key_machinery":"The central object is the Oracle Performance Gap (OPG), the normalized difference in pass@1 accuracy between an 'oracle' model fine-tuned directly on the benchmark test set and a standard model fine-tuned only on its train set. A vanishing OPG is taken as the diagnostic signature of a benchmark that no longer tests generalization. Supporting tools are the difficulty-stratified cross-generalization analysis, a semantically clustered distribution test, and a counterfactual test that rewrites a core rule to force a choice between recitation and on-the-fly reasoning.","core_discovery":"The paper's central discovery is that the conventional train/test split no longer separates RL-trained reasoning models: the Oracle Performance Gap—the normalized accuracy difference between a model fine-tuned on the test split and one fine-tuned on the training split—is essentially zero (roughly -5% to +1.5% across the benchmarks), while SFT shows a substantial gap (up to 36%). The paper interprets this as 'unseen-ness' no longer being a diagnostic signal for RL generalization, and reinforces it with three stress tests: performance collapses for easy-trained specialists on harder problems; semantically narrow fine-tuning can invert to a penalty on far out-of-distribution data; and counterfa","pith_inferences":["My inference: a low OPG could also be produced by the base model having encountered near-duplicate problems during pretraining, so a near-zero gap alone may indicate contamination rather than a genuine absence of generalization.","My inference: the difficulty-stratified result suggests a concrete training recipe—curricula should front-load hard problems—though the paper itself stops short of advocating this.","My inference: the three principles could be turned into a composite benchmark-validity metric that combines OPG, difficulty-decay slope, OOD inversion margin, and counterfactual drop, allowing the community to compare benchmarks rather than only model scores.","My inference: a direct follow-up would test whether process-based rewards or explicit rule-following training shrink the counterfactual drop; the paper does not experiment with those alternatives."],"forward_implications":["Benchmark scores reported with a single train/test split can no longer be read as evidence of RL generalization; a model can match near-oracle scores without having seen the test questions.","Future benchmarks should report difficulty-stratified results, because aggregate accuracy masks large asymmetries in which specialist models trained on hard problems transfer while easy-trained models do not.","Benchmarks should include distribution-shift sets; the observed performance inversion shows fine-tuning can actively harm performance on sufficiently distant data.","Benchmarks should include counterfactual rewrites; the accuracy drop shows models default to recitation rather than following novel premises.","Training data difficulty matters: models trained on harder problems have higher cross-difficulty transfer, so benchmark training sets should be sufficiently challenging."],"fun_headline_variants":["Zero oracle gap: RL benchmarks can't separate train from test","RL train=test on benchmarks, exposing generalization blind spot","Benchmarks show RL overfits: train=test, stress tests fail","Near-zero gap signals RL benchmarks measure nothing new","RL generalization failures hidden by zero oracle gap on benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a useful benchmark must show a clearly positive OPG; if a benchmark could separate strong from weak RL policies while train-trained and test-trained models scored alike, the paper's central conclusion would not follow.","fun_headline_variants_meta":{"raw":{"variants":["Zero oracle gap: RL benchmarks can't separate train from test","RL train=test on benchmarks, exposing generalization blind spot","Benchmarks show RL overfits: train=test, stress tests fail","Near-zero gap signals RL benchmarks measure nothing new","RL generalization failures hidden by zero oracle gap on benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1478,"prompt_tokens":682,"completion_tokens":796,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":723}},"tokens_in":426,"tokens_out":796,"duration_ms":7119,"temperature":1.0,"reasoning_tokens":723,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:16:20.245671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same train-versus-test OPG comparison on a test set that is semantically disjoint from training (for example, generated with a held-out problem-generation routine). A large positive OPG on that version would show the vanishing gap is a property of the current benchmark's homogeneity rather than of RL in general.","supporting_citations":[],"review_version":1}