{"id":"45971ac5-4323-44fe-8497-29d6b4628475","arxiv_id":"2603.16140","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Contrary to prior 'noise-robust' RLVR findings, truly incorrect annotations (after re-verification) degrade math reasoning by 8–10% and cannot be mitigated by existing RLVR algorithms.","lead":"This paper argues that previous claims that reinforcement learning with verifiable rewards (RLVR) tolerates fully incorrect training labels are wrong, because those 'noisy' datasets secretly contained many correct answers. After cleaning the data with a re-verification pipeline, the authors show that fully incorrect labels actually cause 8–10% accuracy drops in math reasoning, and that existing algorithm tweaks like DAPO, SAPO, and TIS do not recover the loss.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 8–10% degradation claim rests on single-run training curves with no seed variance; statistical support is missing.","rationale":"The paper is well-designed and the contamination explanation for prior noise-robustness results is plausible. The multi-stage re-verification pipeline is a thoughtful attempt, and the 3% upper bound on judge error is a good-faith statistical check. However, the empirical foundation of the headline result is incomplete: every key comparison appears to be a single run with no reported seed variance. This is a standard requirement for RL comparisons, and without it the reported 8–10% gaps and the 'comparable to format reward' claim are not statistically grounded. The reader's weakest assumption about residual contamination is reasonable, but it is less decisive: leftover correct labels would make the noisy condition perform better, so the reported degradation is if anything conservative. The more load-bearing issue is the absence of uncertainty quantification. I therefore see no reason to change the reader's CONDITIONAL verdict, but the condition should explicitly require seed-level replication (and ideally release of the re-verified dataset for independent audit) before the result is treated as established.","tokens_in":18990,"tokens_out":9538,"duration_ms":106236,"concrete_test":"Rerun the GRPO clean and GRPO 100%-noise conditions on Qwen2.5-Math-7B with at least 5 random seeds (varying data order and rollout sampling), evaluate at the same 600-step checkpoint on MATH-500, AIME, and AMC, and report mean±std with a difference test (e.g., bootstrap CI for clean-minus-noisy). If the 95% CI excludes 0 on all three benchmarks, the central claim is supported; if it includes 0 on any benchmark, the quantitative conclusion is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim — that 100% truly incorrect annotations degrade MATH-500 by 9% and other benchmarks by 8–10% (Fig. 3) — is supported only by single-run training curves. No seed count is reported anywhere, and Figures 3, 7, 9, and 11 show point trajectories without error bars. The only mention of confidence intervals (Appendix A.3) lacks methodology, so it cannot be verified. In RLVR, run-to-run variance from data ordering, rollout sampling, and initialization can be several accuracy points; a 9% gap could fall within that noise. This is directly load-bearing: if the clean-minus-noisy difference is not statistically significant, the refutation of the noise-robustness hypothesis collapses. The contamination concern identified by the reader is real but secondary: residual hidden correct labels would bias the noisy curve upward, making the reported gap a conservative lower bound on destruction rather than an inflated one. The missing variance estimate is the more serious threat to the paper's conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits the claim that reinforcement learning with verifiable rewards (RLVR) is robust to incorrect annotations. The authors argue that previously reported noisy training sets were contaminated with correct labels, and they construct a 'truly noisy' math dataset by re-verifying 16.4% of the prior noisy data as correct via a pipeline based on GPT-5 Pro, Math-Verify, an LLM judge, and manual inspection. Using Qwen2.5-Math-7B and GRPO, they report that training on 100% truly incorrect annotations degrades MATH-500, AIME, and AMC accuracy by 8.5–10.0% relative to clean data, roughly matching a format-only reward baseline and sometimes falling below the base model. They further report that DAPO, SAPO, Dr. GRPO, TIS, and PGFC under 50% noise do not outperform GRPO on the same noisy data and remain 3.1–7.3% below clean GRPO. Finally, in a Text2SQL case study on BIRD, training on the original noisy dataset is reported to be 5.7–12.1% worse than training on the authors' corrected subset across five base models, with PGFC failing to recover the gap. The paper concludes that current RLVR algorithms cannot compensate for poor data quality.","tokens_in":19129,"tokens_out":6564,"duration_ms":64131,"significance":"The strength of the paper is its evidence that prior '100% noisy' datasets may have contained a substantial fraction of correct answers (16.4% removed by the pipeline; 40% in a 20-sample pilot). The contamination finding, if confirmed, undermines the empirical basis of the noise-robustness hypothesis. The authors also make a good-faith effort to bound the judge error rate with a Clopper-Pearson interval and to cover multiple algorithms and benchmarks. If the reported 8–10% clean-vs-noisy gap is statistically robust, the paper is a significant and practically important negative result: data quality remains central to RLVR, and algorithmic fixes do not make noisy annotations harmless. However, the manuscript currently lacks the statistical evidence needed to establish the magnitude of the gap, and the re-verification pipeline's completeness is not fully validated. These are fixable but necessary before the central claim can be accepted.","major_comments":[{"comment":"The central quantitative claim — that 100% truly incorrect annotations degrade MATH-500 by 9% and other benchmarks by 8–10%, and that algorithms fail to recover — is supported only by single-run training trajectories. No seed count is reported anywhere. Figures 3, 7, 9, 11, and 13 show point trajectories without error bars; Appendix A.3 mentions overlapping confidence intervals but provides no methodology for how they were computed or how many runs they cover. RLVR training variance from data ordering, rollout sampling, and initialization can be several accuracy points, so a 9% gap could fall within run-to-run noise. This is load-bearing: without a variance estimate or significance test, the refutation of the noise-robustness hypothesis is not established. Please run at least 3–5 seeds for the core Qwen2.5-Math-7B comparisons and report confidence intervals or pairwise significance tests","section":"§4.2, Fig. 3; §5.3, Fig. 7; Appendix A.3"},{"comment":"The re-verification pipeline's ability to remove all correct annotations depends on GPT-5 Pro correctly enumerating every valid answer for each problem. The manual audit (100 samples, 0% judge errors) and the Clopper-Pearson bound (3% at 95% confidence) validate the LLM-as-judge step only against the set of answers produced by GPT-5 Pro; they do not bound the failure rate of the answer-enumeration step. If GPT-5 Pro systematically misses a class of valid answers, some correct annotations will remain in the 'truly noisy' set. The direction of this bias is favorable to the paper's conclusion — residual correct labels would inflate the noisy training curve, making the reported clean-vs-noise gap an underestimate — but the paper should state this direction explicitly and provide additional evidence, e.g., independent human re-annotation of a random sample of the retained 'incorrect' set. The","section":"§3.3, Fig. 2"},{"comment":"The real-world Text2SQL conclusion that original BIRD data is 5.7–12.1% worse than corrected data is based on five base models, but each condition appears to be a single run without variance reporting. For large models multiple seeds may be expensive, but the paper should at least report the number of runs, the variance if available, and a statistical test or explicit caveat. Without this, the 'up to 12%' claim is not easily distinguishable from training noise, especially for LoRA-based training via an external API. Also, the corrected BIRD-600 dataset was curated by the authors themselves; while the two-person verification process is described, a small independent audit or release of the corrections would strengthen the claim.","section":"§6.2, Fig. 9"}],"minor_comments":[{"comment":"'Clopper-Person exact method' should be 'Clopper-Pearson exact method'.","section":"§3.3"},{"comment":"The labels such as '-10.0-10.4' and '-5.7-5.1' are ambiguous; clarify whether they denote ranges, paired bars, or multiple measurements.","section":"Fig. 9"},{"comment":"The paper promises a reliable dataset for the community but provides no code or data release URL. An availability statement is needed.","section":"Contribution 1"},{"comment":"The random-annotation setting uses early stopping at two epochs based on pilot runs, while other settings use three epochs. Please state the stopping rule more precisely and justify that the comparison is fair given different training durations.","section":"§4.1, Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The contamination finding is likely the most solid and novel contribution; the main gap is the absence of seed-level variance for nearly every quantitative claim. If the authors can add multiple seeds for the core experiments, report confidence intervals, and make the corrected datasets available, I would support publication. The current version does not yet provide the statistical support needed for the headline 8–10% degradation claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main event here is the contamination finding, and it looks real. The authors took the supposedly '100% incorrect' dataset from Shao et al. (2025), re-verified it with a pipeline combining GPT-5 Pro annotation, symbolic checking with math-verify, an LLM judge, and manual inspection, and found at least 16.4% of the 'incorrect' labels were actually correct. That directly explains why prior work saw apparent noise robustness: the noise wasn't noise. This is a genuinely new and useful result, and the dataset of 12,769 truly incorrect annotations plus the real-world Text2SQL correction of 600 BIRD instances are practical contributions. I believe the authors here—they engaged honestly with the prior work and the direction of the bias is if anything conservative: residual hidden correct labels would make noisy training look less destructive, not more.\n\nThe experimental logic is sound: clean vs. noised data across multiple benchmarks and algorithms, plus a real-world domain. The finding that none of the SOTA RLVR variants (DAPO, SAPO, Dr. GRPO, TIS, PGFC) beat vanilla GRPO under 50% noise is consistent across the board. That part I buy.\n\nNow the soft spots. The paper's central quantitative claim—that 100% truly incorrect annotations degrade MATH-500 by 9% and other benchmarks by 8–10%—rests on single training runs. No seed count is reported anywhere. Figures 3, 7, 9, and 11 show point trajectories without error bars. Appendix A.3 mentions 'overlapping confidence intervals' but gives no methodology, so it can't be verified. In RLVR, run-to-run variance from data ordering, rollout sampling, and initialization can be several accuracy points; a 9% gap could shrink or vanish with variance accounted for. This matters, because if the gap is not statistically significant, the refutation of the noise-robustness hypothesis is weakened—though the consistent direction across three benchmarks and the real-world results give me confidence the qualitative conclusion holds.\n\nThe re-verification pipeline is also not fully transparent: no artifacts are shipped, the BIRD-600 correction process lacks enough detail to reproduce, and the manual inspection is by the authors themselves. The 3% upper bound on judge error is a decent statistical gesture, but it doesn't cover systematic failure modes. That said, this is secondary to the missing variance.\n\nWho should read this: anyone working on RLVR post-training, especially teams deciding whether to invest in data cleaning vs. algorithm tweaks. The paper deserves a serious referee. I'd send it out, but I'd ask for seeds, error bars, and better documentation of the verification pipeline before treating the exact numbers as established. The core message—high-quality data remains essential—is likely correct, but the magnitude needs a firmer foundation.","headline":"This paper likely overturns a prominent 'RLVR tolerates 100% noise' claim by showing prior noisy datasets were contaminated with correct labels, but the headline 8–10% degradation number needs seed variance before it's fully established.","tokens_in":19674,"tokens_out":1377,"would_cite":true,"duration_ms":16759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning with verifiable rewards is not robust to wrong answers: once 'noisy' training sets are stripped of accidentally correct labels, training on truly incorrect annotations loses 8–10% accuracy and matches a format-only re","keywords":["RLVR","verifiable rewards","noisy annotations","data contamination","GRPO","reward noise","Text2SQL","mathematical reasoning"],"falsifier":"Take the paper's constructed truly-noisy math set, draw a random sample of several hundred 'incorrect' labels, and have independent human experts or a different automated verifier check each one for mathematical equivalence with all valid answers; if the true proportion of correct labels exceeds the claimed roughly 3% bound, the 8–10% performance gap is partly an artifact of residual contamination.","tokens_in":18789,"feed_emoji":"📉","tokens_out":6042,"duration_ms":63414,"temperature":0.7,"pith_summary":"The paper tries to establish that the apparent noise tolerance of reinforcement learning with verifiable rewards (RLVR) is an artifact of contaminated data. Prior work claimed that models trained on 100% incorrect annotations perform almost as well as models trained on clean data; this paper shows that at least 16% of those supposedly wrong labels were actually correct, and after removing them, 100% truly incorrect annotations cut MATH-500 accuracy by about 9% and other math benchmarks by 8–10%, matching a baseline that only rewards writing an answer in a box. It also shows that five state-of-the-art RLVR algorithm improvements—covering bias correction, adaptive clipping, and dynamic sampling—fail to close this gap under 50% noise, and that real-world annotation errors in a text-to-SQL task cause 5–12% degradation. The paper's contribution is a re-verification pipeline plus systematic experiments locating the bottleneck in data quality rather than in the RLVR algorithm.","feed_headline":"Wrong labels cut RL math accuracy by 9%","feed_subtitle":"Prior claims that noisy data is harmless came from contaminated sets; truly wrong labels match format-only rewards.","key_machinery":"The load-bearing tool is the multi-stage re-verification pipeline used to convert a supposedly noisy set into a truly noisy one. It first prompts an LLM to list all valid answers to each question, then checks each candidate wrong answer with a symbolic equivalence checker, then applies an LLM judge with a refined prompt, and finally uses manual inspection of 100 samples to bound judge error at no more than 3% with 95% confidence. On the training side, the mechanism that transmits noise is the group-relative advantage estimator: it normalizes rewards within each group of sampled outputs, so when the annotation is wrong, correct reasoning receives low relative reward while short, format-compli","core_discovery":"The central claim is that the noise-robustness hypothesis for RLVR is invalid. The paper argues that prior '100% noisy' training sets were contaminated with correct answers because ground-truth solutions were incomplete and equivalence checking was too weak. After re-verifying every retained answer through an LLM annotator, symbolic equivalence checks, iterative LLM judging, and manual inspection, 16.4% of supposedly wrong labels were removed. On the cleaned noisy set, a model trained with 100% genuinely wrong labels scores 9% below the same model trained on clean data on MATH-500, and 8–10% worse on AIME and AMC benchmarks, essentially matching a reward that only checks for the presence of","pith_inferences":["If contamination is widespread in published 'noisy' RLVR datasets, earlier claims about noise tolerance should not be treated as evidence that wrong labels are harmless; re-running those experiments with independently verified wrong labels is the natural next check.","The monotonic drop in response length as noise increases suggests that noisy rewards actively suppress long reasoning chains; a testable extension is to separate 'format' rewards from 'content' rewards to see whether exploration can be preserved under noise.","The 3% upper bound on judge error comes from only 100 manually inspected samples; auditing a larger random sample, or using an independent automated equivalence prover, could either tighten the bound or reveal systematic misses.","In the text-to-SQL case, 62% of a randomly sampled benchmark subset needed correction, implying that raw real-world datasets carry substantial hidden label error; this would bias any train/eval split built from them and should be accounted for in benchmark construction."],"forward_implications":["On math reasoning, training with 100% genuinely wrong annotations performs about as well as rewarding only the presence of a boxed answer, meaning wrong labels teach format adherence without improving reasoning.","Even at 50% wrong labels, five published algorithm improvements do not beat plain group-relative policy optimization and remain 3.1–7.3% below clean-data training.","There is no safe noise threshold: accuracy falls monotonically from about 1.5% degradation at 10% noise to about 9.8% at 100% noise, while response length shrinks by up to about 24%, indicating weaker reasoning chains.","Real-world annotation errors in a text-to-SQL benchmark cost 5.7–12.1% accuracy across five base models, and a noise-correction algorithm fails to recover the loss.","The practical consequence is that RLVR pipelines should invest in annotation verification and data cleaning before tuning algorithms, because current algorithm changes cannot compensate for poor data quality."],"fun_headline_variants":["Noisy data shaves 9% off RL math accuracy","RLVR noise myth: prior 100% noise sets were contaminated","Real noisy labels drop RLVR by up to 12% on SQL","Clean data essential: RLVR noise hurts by 8-10%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the paper's re-verification pipeline correctly labels every retained answer as truly wrong; if the LLM judge or symbolic checker systematically misses answers that are mathematically equivalent, the measured 8–10% degradation gap could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Noisy data shaves 9% off RL math accuracy","RLVR noise myth: prior 100% noise sets were contaminated","Real noisy labels drop RLVR by up to 12% on SQL","Clean data essential: RLVR noise hurts by 8-10%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2806,"prompt_tokens":725,"completion_tokens":2081,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":2005}},"tokens_in":469,"tokens_out":2081,"duration_ms":14723,"temperature":1.0,"reasoning_tokens":2005,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:01:34.255884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's constructed truly-noisy math set, draw a random sample of several hundred 'incorrect' labels, and have independent human experts or a different automated verifier check each one for mathematical equivalence with all valid answers; if the true proportion of correct labels exceeds the claimed roughly 3% bound, the 8–10% performance gap is partly an artifact of residual contamination.","supporting_citations":[],"review_version":1}