Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Reinforcement learning with verifiable rewards is not robust to wrong answers: once 'noisy' training sets are stripped of accidentally correct labels, training on truly incorrect annotations loses 8–10% accuracy and matches a format-only re

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:01 UTC pith:GQHJ7K7W

load-bearing objection This paper likely overturns a prominent 'RLVR tolerates 100% noise' claim by showing prior noisy datasets were contaminated with correct labels, but the headline 8–10% degradation number needs seed variance before it's fully established. the 3 major comments →

arxiv 2603.16140 v2 pith:GQHJ7K7W submitted 2026-03-17 cs.LG

Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards

Yuxuan Zhu , Daniel Kang This is my paper
classification cs.LG
keywords RLVRverifiable rewardsnoisy annotationsdata contaminationGRPOreward noiseText2SQLmathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the apparent noise tolerance of reinforcement learning with verifiable rewards (RLVR) is an artifact of contaminated data. Prior work claimed that models trained on 100% incorrect annotations perform almost as well as models trained on clean data; this paper shows that at least 16% of those supposedly wrong labels were actually correct, and after removing them, 100% truly incorrect annotations cut MATH-500 accuracy by about 9% and other math benchmarks by 8–10%, matching a baseline that only rewards writing an answer in a box. It also shows that five state-of-the-art RLVR algorithm improvements—covering bias correction, adaptive clipping, and dynamic sampling—fail to close this gap under 50% noise, and that real-world annotation errors in a text-to-SQL task cause 5–12% degradation. The paper's contribution is a re-verification pipeline plus systematic experiments locating the bottleneck in data quality rather than in the RLVR algorithm.

Core claim

The central claim is that the noise-robustness hypothesis for RLVR is invalid. The paper argues that prior '100% noisy' training sets were contaminated with correct answers because ground-truth solutions were incomplete and equivalence checking was too weak. After re-verifying every retained answer through an LLM annotator, symbolic equivalence checks, iterative LLM judging, and manual inspection, 16.4% of supposedly wrong labels were removed. On the cleaned noisy set, a model trained with 100% genuinely wrong labels scores 9% below the same model trained on clean data on MATH-500, and 8–10% worse on AIME and AMC benchmarks, essentially matching a reward that only checks for the presence of

What carries the argument

The load-bearing tool is the multi-stage re-verification pipeline used to convert a supposedly noisy set into a truly noisy one. It first prompts an LLM to list all valid answers to each question, then checks each candidate wrong answer with a symbolic equivalence checker, then applies an LLM judge with a refined prompt, and finally uses manual inspection of 100 samples to bound judge error at no more than 3% with 95% confidence. On the training side, the mechanism that transmits noise is the group-relative advantage estimator: it normalizes rewards within each group of sampled outputs, so when the annotation is wrong, correct reasoning receives low relative reward while short, format-compli

Load-bearing premise

The load-bearing premise is that the paper's re-verification pipeline correctly labels every retained answer as truly wrong; if the LLM judge or symbolic checker systematically misses answers that are mathematically equivalent, the measured 8–10% degradation gap could be inflated.

What would settle it

Take the paper's constructed truly-noisy math set, draw a random sample of several hundred 'incorrect' labels, and have independent human experts or a different automated verifier check each one for mathematical equivalence with all valid answers; if the true proportion of correct labels exceeds the claimed roughly 3% bound, the 8–10% performance gap is partly an artifact of residual contamination.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On math reasoning, training with 100% genuinely wrong annotations performs about as well as rewarding only the presence of a boxed answer, meaning wrong labels teach format adherence without improving reasoning.
  • Even at 50% wrong labels, five published algorithm improvements do not beat plain group-relative policy optimization and remain 3.1–7.3% below clean-data training.
  • There is no safe noise threshold: accuracy falls monotonically from about 1.5% degradation at 10% noise to about 9.8% at 100% noise, while response length shrinks by up to about 24%, indicating weaker reasoning chains.
  • Real-world annotation errors in a text-to-SQL benchmark cost 5.7–12.1% accuracy across five base models, and a noise-correction algorithm fails to recover the loss.
  • The practical consequence is that RLVR pipelines should invest in annotation verification and data cleaning before tuning algorithms, because current algorithm changes cannot compensate for poor data quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If contamination is widespread in published 'noisy' RLVR datasets, earlier claims about noise tolerance should not be treated as evidence that wrong labels are harmless; re-running those experiments with independently verified wrong labels is the natural next check.
  • The monotonic drop in response length as noise increases suggests that noisy rewards actively suppress long reasoning chains; a testable extension is to separate 'format' rewards from 'content' rewards to see whether exploration can be preserved under noise.
  • The 3% upper bound on judge error comes from only 100 manually inspected samples; auditing a larger random sample, or using an independent automated equivalence prover, could either tighten the bound or reveal systematic misses.
  • In the text-to-SQL case, 62% of a randomly sampled benchmark subset needed correction, implying that raw real-world datasets carry substantial hidden label error; this would bias any train/eval split built from them and should be accounted for in benchmark construction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper revisits the claim that reinforcement learning with verifiable rewards (RLVR) is robust to incorrect annotations. The authors argue that previously reported noisy training sets were contaminated with correct labels, and they construct a 'truly noisy' math dataset by re-verifying 16.4% of the prior noisy data as correct via a pipeline based on GPT-5 Pro, Math-Verify, an LLM judge, and manual inspection. Using Qwen2.5-Math-7B and GRPO, they report that training on 100% truly incorrect annotations degrades MATH-500, AIME, and AMC accuracy by 8.5–10.0% relative to clean data, roughly matching a format-only reward baseline and sometimes falling below the base model. They further report that DAPO, SAPO, Dr. GRPO, TIS, and PGFC under 50% noise do not outperform GRPO on the same noisy data and remain 3.1–7.3% below clean GRPO. Finally, in a Text2SQL case study on BIRD, training on the original noisy dataset is reported to be 5.7–12.1% worse than training on the authors' corrected subset across five base models, with PGFC failing to recover the gap. The paper concludes that current RLVR algorithms cannot compensate for poor data quality.

Significance. The strength of the paper is its evidence that prior '100% noisy' datasets may have contained a substantial fraction of correct answers (16.4% removed by the pipeline; 40% in a 20-sample pilot). The contamination finding, if confirmed, undermines the empirical basis of the noise-robustness hypothesis. The authors also make a good-faith effort to bound the judge error rate with a Clopper-Pearson interval and to cover multiple algorithms and benchmarks. If the reported 8–10% clean-vs-noisy gap is statistically robust, the paper is a significant and practically important negative result: data quality remains central to RLVR, and algorithmic fixes do not make noisy annotations harmless. However, the manuscript currently lacks the statistical evidence needed to establish the magnitude of the gap, and the re-verification pipeline's completeness is not fully validated. These are fixable but necessary before the central claim can be accepted.

major comments (3)
  1. [§4.2, Fig. 3; §5.3, Fig. 7; Appendix A.3] The central quantitative claim — that 100% truly incorrect annotations degrade MATH-500 by 9% and other benchmarks by 8–10%, and that algorithms fail to recover — is supported only by single-run training trajectories. No seed count is reported anywhere. Figures 3, 7, 9, 11, and 13 show point trajectories without error bars; Appendix A.3 mentions overlapping confidence intervals but provides no methodology for how they were computed or how many runs they cover. RLVR training variance from data ordering, rollout sampling, and initialization can be several accuracy points, so a 9% gap could fall within run-to-run noise. This is load-bearing: without a variance estimate or significance test, the refutation of the noise-robustness hypothesis is not established. Please run at least 3–5 seeds for the core Qwen2.5-Math-7B comparisons and report confidence intervals or pairwise significance tests
  2. [§3.3, Fig. 2] The re-verification pipeline's ability to remove all correct annotations depends on GPT-5 Pro correctly enumerating every valid answer for each problem. The manual audit (100 samples, 0% judge errors) and the Clopper-Pearson bound (3% at 95% confidence) validate the LLM-as-judge step only against the set of answers produced by GPT-5 Pro; they do not bound the failure rate of the answer-enumeration step. If GPT-5 Pro systematically misses a class of valid answers, some correct annotations will remain in the 'truly noisy' set. The direction of this bias is favorable to the paper's conclusion — residual correct labels would inflate the noisy training curve, making the reported clean-vs-noise gap an underestimate — but the paper should state this direction explicitly and provide additional evidence, e.g., independent human re-annotation of a random sample of the retained 'incorrect' set. The
  3. [§6.2, Fig. 9] The real-world Text2SQL conclusion that original BIRD data is 5.7–12.1% worse than corrected data is based on five base models, but each condition appears to be a single run without variance reporting. For large models multiple seeds may be expensive, but the paper should at least report the number of runs, the variance if available, and a statistical test or explicit caveat. Without this, the 'up to 12%' claim is not easily distinguishable from training noise, especially for LoRA-based training via an external API. Also, the corrected BIRD-600 dataset was curated by the authors themselves; while the two-person verification process is described, a small independent audit or release of the corrections would strengthen the claim.
minor comments (4)
  1. [§3.3] 'Clopper-Person exact method' should be 'Clopper-Pearson exact method'.
  2. [Fig. 9] The labels such as '-10.0-10.4' and '-5.7-5.1' are ambiguous; clarify whether they denote ranges, paired bars, or multiple measurements.
  3. [Contribution 1] The paper promises a reliable dataset for the community but provides no code or data release URL. An availability statement is needed.
  4. [§4.1, Appendix C] The random-annotation setting uses early stopping at two epochs based on pilot runs, while other settings use three epochs. Please state the stopping rule more precisely and justify that the comparison is fair given different training durations.

Circularity Check

0 steps flagged

No significant circularity; central empirical comparison is self-contained, with only a non-load-bearing self-citation.

full rationale

Walking the claimed derivation chain: the paper's core claims are empirical comparisons between RLVR trained on clean labels, re-verified 'truly incorrect' labels, random labels, and format-only rewards. The 8–10% degradation on MATH-500, AIME, and AMC is a measured difference between training runs on the same question set with different labels, not a quantity fitted to a pre-specified target. The 'truly noisy' dataset is constructed by a re-verification pipeline (GPT-5 Pro annotation, math-verify, LLM-as-a-judge, manual inspection) that excludes labels judged correct; this is a data-curation definition, not an equation whose output is the paper's conclusion. The algorithm-failure result is also measured against GRPO, and for PGFC the paper even supplies oracle noise rates ('we used the ground-truth noise rates as the correction factors, measuring the method's upper-bound performance'), which is generous rather than circular. For the Text2SQL experiment, the paper relies on the externally provided Arcwise Mini-Dev corrections plus its own manual correction of 600 BIRD instances, so the central real-world comparison does not reduce to a self-citation. The only self-citation of note is Jin et al. 2026 (co-authored by the paper's authors), cited for the claim that BIRD contains annotation errors; this is corroborated by independent prior work (Wretblad et al. 2024; Pourreza & Rafiei 2023), by Arcwise, and by the authors' own correction process finding 62% noisy instances, so it is not load-bearing. Concerns about missing seed variance or the reliability of the LLM judge are statistical-validity or correctness risks, not circularity: they do not make any prediction equivalent to an input by construction. Therefore the central derivation is self-contained, with at most a minor, non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No new physical entities or mathematical objects are introduced. The paper relies on a multi-stage verification pipeline with LLMs as the primary tool for judging correctness. The main assumptions are about the reliability of LLM-based verification and the representativeness of small manual checks. Free parameters are minimal — mostly the amount of data removed, which is a design choice rather than a physical constant.

free parameters (1)
  • percentage of removed data (16.4%) = 16.4% removed from original noisy dataset
    The authors exclude 16.4% of the original noisy data after finding 9.0% had insufficient ground truth annotations. The cutoff is a practical choice they made to form a 'truly noisy' dataset, not a natural constant.
axioms (4)
  • domain assumption GPT-5 Pro's annotations of all valid answers for each math problem are complete (the set of 'correct' answers is fully enumerable).
    Section 3.3 Pipeline Step 1: if GPT-5 Pro misses some valid answers, the pipeline could mislabel correct answers as incorrect.
  • domain assumption Math-Verify and the LLM judge (GPT-5 Pro) can correctly detect all semantically equivalent answers.
    Pipeline Steps 2 and 3: if the judge fails to detect equivalence, correct answers remain and contaminate the noisy set (Section 3.3).
  • domain assumption The 100-sample manual inspection (with 0% observed judge error) is representative of the entire 12,769-sample dataset.
    The authors state a Clopper-Pearson upper bound of 3% with 95% confidence, but this bound is on the judge's binary 'correct/incorrect' classification, not on the joint human+LLM pipeline's accuracy on all samples.
  • domain assumption The reward formulation for Text2SQL (execution-match: +1, missing solution tags: -1, otherwise 0) correctly captures 'correct' SQL behavior.
    Equation in Section 6.1: if execution-match is too coarse, it could mask how incorrect SQL answers are actually rewarded.

pith-pipeline@v1.3.0-alltime-deepseek · 18604 in / 7654 out tokens · 58395 ms · 2026-08-02T18:01:34.255884+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.

Figures

Figures reproduced from arXiv: 2603.16140 by Daniel Kang, Yuxuan Zhu.

Figure 1
Figure 1. Figure 1: Noisy data significantly degrades RLVR performance, and existing algorithmic improvements fail to mitigate this impact. Using Qwen2.5-Math-7B as a base model, we show that training on 100% incorrect annotations sampled from the base model (red line) leads to performance similar to training with format-only rewards (purple line) and underperforms training on clean data (blue line) by 9%. Even with the best-… view at source ↗
Figure 2
Figure 2. Figure 2: Data re-verification pipeline for synthesizing truly noisy dataset. As shown in the two examples, we identify the issues of insufficient ground-truth annotations and inadequate equivalence checking, leading to correct annotations in the prior noisy dataset. We address them via a rigorous pipeline combining LLM annotation and human verification. Thus, our study focuses on real-world annotation errors. 3.3. … view at source ↗
Figure 3
Figure 3. Figure 3: Both incorrect and random annotations significantly decrease the performance of RLVR: RLVR with incorrect annotations achieves comparable or lower accuracy (by 1.2–5.6%) than RLVR with format rewards, and lower accuracy (by 8.5–10.0%) than RLVR with clean data; RLVR with random annotations achieves significantly lower accuracy (by 6.7–27.8%) than the base model. with ground-truth answers, and a random-anno… view at source ↗
Figure 6
Figure 6. Figure 6: Training on noisy data results in shorter responses (5.2–23.9%), indicating that noise induces weaker reasoning. Evaluation. We performed all evaluations using greedy decoding (temperature=0). We include five benchmarks for evaluation: MATH-500 (Lightman et al., 2023), AIME 2024 and 2025 (Community, 2025a), AMC 2023 and 2024 (Community, 2025b). Among them, the AIME 2025 and AMC 2024 datasets were released … view at source ↗
Figure 7
Figure 7. Figure 7: Existing algorithmic improvements fail to mitigate the impact of noisy data. Under 50% noise, none of the improved algorithms achieved a > 2% accuracy improvement on all benchmarks simultaneously compared to GRPO with 50% noise, while all algorithms underperform the vanilla GRPO with clean data by 3.1–7.3%. 1 2 4 8 16 32 64 128 Number of Samples k 40 50 60 70 80 Average Pass@k (%) Algorithms (50% noise) GR… view at source ↗
Figure 8
Figure 8. Figure 8: None of the evaluated algorithmic improvements achieve a higher pass@k score than GRPO under 50% noise. 3. Clipping mechanisms: Prior research suggests that asymmetric clipping can improve learning from noisy data by regulating entropy (Lv et al., 2025). Further￾more, adaptive clipping ratios based on group advan￾tage can stabilize gradient updates when a batch of roll￾outs exhibit a variance of rewards. T… view at source ↗
Figure 9
Figure 9. Figure 9: Real-world noise significantly degrades RLVR perfor￾mance, while algorithmic intervention fails to recover it. Models trained on the original, noisy BIRD dataset (BIRD-Original-600) via GRPO suffers a 5.7–12.1% accuracy drop compared to BIRD￾Corrected-600. The PGFC algorithm, specifically designed to handle noisy data, fails to consistently improve over GRPO and remains 5.1–16.3% behind the clean data base… view at source ↗
Figure 10
Figure 10. Figure 10: Noise leads to lower pass@k than the base model when k > 1, showing that noise does not improve capability boundary. 0% 10% 20% 30% 40% 50% 100% Proportion of Noise in Data 0 20 40 60 80 Average Pass@1 (%) 80.8% -0.5% -1.9% -1.4% -3.0% -3.8% -8.6% (a) MATH-500. 0% 10% 20% 30% 40% 50% 100% Proportion of Noise in Data 0 5 10 15 20 Average Pass@1 (%) 20.2% -1.2% 0.0% -2.3% -4.5% -6.8% -11.4% (b) AIME 2024 an… view at source ↗
Figure 11
Figure 11. Figure 11: Training on noisy data results in increasing performance degradation (by up to 11.4%) as the noise proportion increases. 0% 10% 20% 30% 40% 50% 100% Proportion of Noise in Data 300 400 500 600 Avg. Response Length 577 -24 -40 -45 -13 -21 -18 (a) MATH-500. 0% 10% 20% 30% 40% 50% 100% Proportion of Noise in Data 600 800 1000 1200 1400 Avg. Response Length 1343 -76 -144 -196 -221 -285 -411 (b) AIME 2024 and … view at source ↗
Figure 12
Figure 12. Figure 12: Training on noisy data results in shorter responses, indicating that noise induces weaker reasoning. A. Detailed Experimental Results A.1. Pass@k Results of Models Trained on Noisy Data We show the pass@k performance of models trained on noisy data in [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Existing algorithm improvements fail to mitigate the impact of noisy data. Under 50% noise, none of the improved algorithm achieved > 2% accuracy improvement on all benchmarks simultaneously compared to GRPO with 50% noise, while all algorithms underperform the vanilla GRPO with clean data by 3.1-7.3% compared to the clean baseline. Similarly, MATH-500 and AMC suffer significant drops of 8.6% and 8.9%, re… view at source ↗
Figure 14
Figure 14. Figure 14: None of the evaluated algorithmic improvements achieve higher pass@k score than GRPO under 50% noise. GRPO (correct) SAPO (50% noise) DAPO (50% noise) TIS (50% noise) DR.GRPO (50% noise) PGFC (50% noise) GRPO (50% noise) 300 400 500 600 Avg. Response Length 577 -37 -34 -93 -57 -16 -21 (a) MATH-500. GRPO (correct) SAPO (50% noise) DAPO (50% noise) TIS (50% noise) DR.GRPO (50% noise) PGFC (50% noise) GRPO (… view at source ↗
Figure 15
Figure 15. Figure 15: Training on noisy data with improved algorithms still results in shorter responses. toward shorter and likely incorrect solutions. A.3. Results of Different RLVR Algorithms In [PITH_FULL_IMAGE:figures/full_fig_p013_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: After three epochs (600 steps), no model shows improvements of average pass@1 of all benchmarks in the subsequent 50 steps. C. Determination of Training Duration To establish a rigorous stopping criterion for math reasoning training, we monitored the average performance across all benchmarks over nearly 600 training steps. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p015_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Starting the tenth epoch, no model shows improvements in terms of reward. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

    cs.LG 2026-04 unverdicted novelty 6.0

    Systematic false positives in verifiers can cause RLVR training to reach suboptimal plateaus or collapse, with outcomes driven by error patterns rather than overall error rate.

  2. GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling

    cs.LG 2026-06 unverdicted novelty 5.0

    GeoMin uses geometric distribution modeling on labeled data to assess self-reward reliability, enabling better performance in semi-supervised RLVR with only 10% of typical annotations.

Reference graph

Works this paper leans on

7 extracted references · cited by 2 Pith papers

  1. [1]

    (k in 12, -12)

    If the question has multiple distinct but correct answers, the student only needs to provide one of them to be considered correct. For example, if the ground truth answer is "(k in 12, -12)" and the student’s answer is "11", the student’s answer should be considered correct

  2. [2]

    (11) 6" and the student’s answer is

    Ignore any format mistakes and only grade the mathematical meaning of the answer. For example, if the ground truth answer is "(11) 6" and the student’s answer is "11", the student’s answer should be considered correct. Another example is if the ground truth answer is latex formatted and the student’s answer is not latex formatted but is mathematically equ...

  3. [3]

    If the student’s answer is mathematically equivalent to the ground truth answer, it should be considered correct

  4. [4]

    4\sqrt{2}

    If the student’s answer uses fractions, decimals, or different representations that are mathematically equivalent to the ground truth answer, it should be considered correct. For example, if the ground truth answer is "4\sqrt{2}" and the student’s answer is "5.656854249492381", the student’s answer should be considered correct

  5. [5]

    Yes" or anything equivalent to

    If the question is an Yes/No question and the ground truth answer is an exact solution, the student’s answer is correct as long as they answers "Yes" or anything equivalent to "Yes". For example, for the question "Does there exist a fraction equivalent to $\frac{7}{13}$ such that the difference between the denominator and the numerator is 24?", the ground...

  6. [6]

    2 + 2" and the student’s answer is

    If the student’s answer is a simplification of the ground truth answer, it should be considered correct. For example, if the ground truth answer is "2 + 2" and the student’s answer is "4", the student’s answer should be considered correct. Vice versa is also true

  7. [7]

    5" and the student’s answer is

    If the student’s answer is mathematically inequivalent to the ground truth answer, it should be considered incorrect. For example, if the ground truth answer is "5" and the student’s answer is "-5", the student’s answer should be considered incorrect. Question:{question} Ground Truth:{ground truth} Student Answer:{student answer} Is the student’s answer c...