Pith.

Machine Learning · arXiv:2609.13866 v1 · submitted 12 Sep 2026 · 17 references

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards

Juntao Yu

The paper says

“Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards.”

more of the abstract

When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in 0.1, 0.3, 0.5, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. This is a clean, focused paper that identifies a dangerous configuration mistake in group-relative RL that can cause full policy collapse. This line is wrong

6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 17 references say.

6 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 6

The abstract reports 'EM 0.080 +/- 0.112' and '0.040 +/- 0.008' for the no-filter and shaped-score arms; these standard deviations show high variance in the no-filter arm. Consider noting that this is sample SD over four runs, not a confidence interval, to avoid misinterpretation.

The paper says, at Section 4, paragraph 2

Pith says: minor objection 2 of 6

The description of the gate arms is clear, but the phrase 'zeroes (advantage→0)' could be clarified to state that advantage-zeroing is applied to groups determined by the filter metric, rather than to all groups.

show 4 more

The paper says, at Figure 1

Pith says: minor objection 3 of 6

The left panel's caption says 'Same model, data and hyperparameters; only the group-gating rule differs.' The right panel shows length collapse; consider adding a label for the y-axis (tokens) and noting that the shaded regions are standard deviations across runs.

The paper says, at Section 7, Table 3

Pith says: minor objection 4 of 6

The 7B·GSM8K row says 'no dose—see below'. This is a bit informal; consider stating the measured all-fail rate (0/32 per batch) explicitly in the table or its caption.

The paper says, at Section 9, Dr. GRPO discussion

Pith says: minor objection 5 of 6

The phrase 'The avoidance recurs at 7B on MATH' is supported by single n=1 cells. This is acknowledged in the caveat, but consider adding a footnote in the table itself noting n=1.

The paper says, at Section 10, Limitations

Pith says: minor objection 6 of 6

The A/B test on re-zeroing binary dead groups uses two seeds and shows no consistent effect; this is a useful negative result. The text says 'single-seed claim that “gating hurts by −0.15” disappears on the second seed.' The notation '−0.15' is ambiguous; clarify that this is the change in held-out mean reward.

What would settle it

Run the core GSM8K 1.5B three-arm comparison with 10 independent seeds instead of four, and extend training to 200 steps per arm. Verify that the binary-outcome arm maintains high accuracy (≥0.74) and that the shaped-score and no-filter arms collapse (final EM ≤0.10) in every seed. This would confirm that the observed collapse is not a finite-horizon artifact or a rare-seed fluke.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Sep 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 34 s · 9,433 tokens read, 3,888 written
  3. Editor wrote the verdict above 11 s · 10,126 tokens read, 1,005 written
  4. Referee accept: 6 minor 23 s · 9,711 tokens read, 2,588 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper isolates a real, dangerous configuration pitfall in group-relative RL: when the group filter metric is set to a shaped reward score instead of the binary task outcome, all-fail groups survive filtering, and std normalization rescales tiny differences among failures into full-size phantom advantages. The result is catastrophic policy collapse to very short incorrect outputs. The core three-arm contrast at 1.5B GSM8K is well-replicated (four runs per arm) and the native-trainer telemetry (0/40 vs 29/40 refill steps) is a hard process measure that cleanly shows the control flow being suppressed. The Dr. GRPO intervention—removing the std divisor—avoids collapse at both 1.5B and 7B, providing mechanism evidence. Limitations are honestly stated: off-anchor cells are n=1, only one model family, two tasks, 40-step horizon. These are proportional to the claim: this is a specific empirical demonstration, not a universal theory. The paper's contribution is narrower than 'shaping is bad'—it pins down a specific implementation-level semantic mismatch and shows how to detect it (all-fail group fraction, response length). The field should treat this as a safety warning for practitioners who configure filter metrics under composite rewards. I would cite this if I were working with shaped rewards in GRPO-like algorithms. It deserves a serious referee.

Recommendation: Accept with minor revisions—the n=1 cells are fine as boundary probes if flagged, which they are. No single off-anchor claim needs to carry weight; the paper's argument rests on the replicated anchor and the mechanistic intervention. I would bring this to a reading group focused on RL for LLMs.

The strongest objection

The paper's central claim is well-supported by the controlled three-arm comparison with replication at the core anchor (four runs for each arm), the native-trainer refill telemetry (0/40 vs 29/40 steps, a process measure not subject to sampling noise), and the mechanism intervention (Dr. GRPO) which removes the std divisor and avoids collapse at both 1.5B and 7B. The limitations are honestly stated (small n for off-anchor cells, 40-step horizon, single model family) and do not undermine the core demonstration. I find no internal inconsistency, hidden assumption, or methodological flaw that would threaten the primary conclusion: that a metric–predicate mismatch under composite shaped rewards can cause phantom advantages and policy collapse, and that using an outcome-semantic filter metric avoids the failure in the tested settings.

The reader identified generalization beyond the tested settings as the weakest assumption, and the paper itself acknowledges this limitation. My read confirms that the core claim is empirically solid within its stated scope. The controlled design, mechanism intervention, and native-trainer telemetry provide converging evidence. I see no reason to adjust the verdict. The proposed concrete test would further strengthen confidence in the generality of the phenomenon.

The referee's report

The paper addresses a timely and practical safety hazard in widely used open-source RL frameworks (verl/DAPO). Its central contribution is isolating the metric–predicate semantic mismatch as a distinct failure mode separate from previously known shaping-induced amplification and correct-only filtering. The evidence for the core claim is strong: a four-run anchor on GSM8K, native-trainer control-flow telemetry showing the filter metric changes whether the physical refill loop is exercised, and a causal intervention (Dr. GRPO) that prevents collapse. The paper's limitations are transparently stated, and the practical rules (use an outcome-semantic filter metric, monitor all-fail fraction and length) are concrete and actionable. The work is likely to influence how practitioners configure group-relative RL systems and raise awareness of implementation-level hazards in modular reward pipelines. The exploratory probes across coefficients, shaping directions, scales, and estimators add breadth. Overall, this is a well-executed empirical study that merits publication.

Pith's reading of the paper (a machine's summary, not the authors')

Group-relative policy optimization (GRPO) algorithms can discard rollout groups with no contrast. When the filter metric is set to a shaped training score instead of the binary task outcome, groups where all rollouts are wrong still retain nonzero spread in the shaped score. This spread is amplified by standard-deviation normalization into full-size advantages, creating 'phantom advantages' that encourage the policy to produce very short incorrect responses. The paper demonstrates this collapse across controlled experiments and shows that using a task-outcome signal avoids it.

What carries the argument. The central mechanism is the combination of (1) shaped-score filter allowing all-fail groups with nonzero spread, and (2) standard-deviation normalization (Ai = (ri - mean)/std) which rescales that spread to full advantage magnitude, overriding the absence of genuine contrast between successes and failures.)

Core claim. The paper identifies a metric-predicate semantic mismatch: when the group-filter metric in DAPO-style dynamic sampling is configured to track a composite shaped reward rather than the binary task outcome, all-fail groups (where all rollouts are incorrect) retain nonzero within-group spread and pass the filter predicate. Standard-deviation normalization then promotes shaping differences among failures into full-size phantom advantages, driving the policy to output very short incorrect responses. The discovery is that the choice of filter metric is safety-critical under shaped rewards, not just a performance detail.

The premise it rests on. The causal dependence of collapse on the metric-predicate mismatch and std normalization is supported only by a limited number of runs (four for the anchor, single runs for most other cells), and generalization to all tasks,models, and training horizons is unproven, as the paper itself notes.

If the paper is right (Pith's inferences, not the authors')

Practitioners using GRPO with shaped rewards should set the group filter metric to a binary task-outcome signal, not the shaped score.

All-fail group fraction and mean response length should be monitored as leading indicators of collapse.

Removing standard-deviation normalization (as in Dr. GRPO) can prevent the collapse in tested settings.

The paper's findings generalize to GSPO and other group-relative estimators using shaped rewards.

Before adding shaping terms, measure baseline all-fail frequency to assess exposure risk.

The same phantom-advantage mechanism may affect other RL algorithms with group normalization and shaped rewards, beyond the tested variants.

The finding suggests that any reward shaping that creates within-group variance among failures can be dangerous when combined with group-relative normalization, not just length-based shaping.

The paper's refill telemetry provides a practical diagnostic: if the refill path is rarely exercised, the metric may be letting degenerate groups through.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in 0.1, 0.3, 0.5, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.98

“Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.”

How the main claim is stated: asserted 0.96

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.31 · human participants 0.02 · controlled comparison 0.97 · randomized assignment 0.07 · states quantitative result 0.98 · compares with prior results 0.96 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. background 1.00 0.10 0.03 asserted 0.94
We identify and quantify a metric-predicate mismatch under composite shaped rewards. contribution 0.98 0.93 0.09 asserted 1.00
When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. interpretation 0.62 0.81 0.27 asserted 1.00
In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). result 1.00 0.95 0.12 asserted 1.00
On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. result 0.99 0.95 0.08 asserted 1.00
Both use the same custom shaped-reward hook and unmodified trainer/filter code. method 0.97 0.86 0.02 asserted 1.00
Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. contribution 0.99 0.86 0.09 asserted 1.00
Across tested positive coefficients lambda in {0.1, 0.3, 0.5}, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. result 0.99 0.89 0.10 asserted 0.96
Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping. interpretation 0.90 0.29 0.89 asserted 0.96

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.95

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,319 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Same model, data and hyperparameters; only the group-gating rule differs. Right: the
Figure 2 of the paper
Figure 2. Endpoint EM versus phantom strength λ. Error bars on the λ=0.30 anchor are sample standard deviations across four runs per arm. phantom what the shortcut does outcome length, short-favoring truncates the answer away very short outputs (as low as 1 token) length, long-favoring pads an intact answer no collapse observed format produces well-formed wrong answers no collapse observed These probes support a finite-horizon hypothesis: shaping is most dangerous when its shortcut directly erodes task completion, whereas a completion-preserving shortcut can be less destructive. The…
Figure 3 of the paper
Figure 3. The same three arms across model scale and task difficulty. On MATH, unsafe arms at
Figure 4 of the paper
Figure 4. Confirmation on verl’s official recipe/dapo trainer with physical group deletion and batch refilling. Right: one generation batch always suffices for the shaped-score metric, whereas the accuracy metric exercises the refill path in 29 of 40 steps. val EM @ 40, GSM8K-1.5B, λ=0.30 GRPO (anchor, n=4) GSPO Dr. GRPO no filtering 0.080 ± 0.112 0.030 0.718 filter on shaped score 0.040 ± 0.008 0.000 0.747 filter on binary outcome 0.754 ± 0.005 0.734 0.738 final length (tokens) 3–18 (unsafe) 1–2 (unsafe) 151–202 (all arms) GSPO: the failure is not tied to the importance-ratio granularity. The
Figure 5 of the paper
Figure 5. De-noised held-out accuracy under a faulty verifier.

Figures are the authors', reproduced from the paper.

References (17)
  1. [4] URLhttps://arxiv.org/abs/2501.12948. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.arXiv preprint arXiv:2210.10760, on PithPith's reviewarXiv
  2. [6] URLhttps://arxiv.org/abs/2103.03874. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, on PithPith's reviewarXiv
  3. [7] URLhttps://arxiv.org/abs/2106.09685. Jungseob Lee, Seungyoon Lee, Seongtae Hong, Minhyuk Kim, Chanjun Park, and Heuiseok Lim. Beyond penalizing mistakes: Stabilizing efficiency training in large reasoning models via adaptive correct-only rewards.arXiv preprint arXiv:2606.22716, on PithPith's reviewarXiv
  4. [9] URL https://arxiv.org/abs/2603.10535. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, on PitharXiv
  5. [11] URLhttps://arxiv.org/abs/2503.20783. 10 Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. DeepSeek- Math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, on PithPith's reviewarXiv
  6. [13] URLhttps://arxiv.org/abs/2409.19256. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking.arXiv preprint arXiv:2209.13085, on PithPith's reviewarXiv
  7. [14] URLhttps://arxiv. org/abs/2209.13085. verl community. verl documentation: DAPO recipe.https://github.com/verl-project/verl/ blob/ad2e3c272ee95fc5627c5007af59b5d25100be1a/docs/algo/dapo.md, 2026a. verl community. verl-recipe: Community recipes for verl.https://github.com/verl-p... on PithPith's reviewarXiv
  8. [15] URLhttps://arxiv.org/abs/ 2508.20751. Yu Wang. The dark room in the reward channel: Dense prediction rewards collapse GRPO- trained LLM agents—and the channel, not the content, decides what works.arXiv preprint arXiv:2607.21273, on PithPith's reviewarXiv
  9. [16] URLhttps://arxiv.org/abs/2607.21273. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, on PithPith's reviewarXiv
  10. [18] URL https://arxiv.org/abs/2503.14476. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, on PithPith's reviewarXiv
  11. [19] URLhttps://arxiv. org/abs/2507.18071. 11 on PithPith's reviewarXiv
  12. [2021] URLhttps://arxiv.org/abs/2110.14168. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, on PithPith's reviewarXiv
  13. [2022] URLhttps://arxiv.org/abs/2210.10760. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, on PithPith's reviewarXiv
  14. [2023] URLhttps://arxiv.org/abs/2305.20050. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, on PithPith's reviewarXiv
  15. [2024] URLhttps://arxiv.org/abs/2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, on PithPith's reviewarXiv
  16. [2025] URLhttps://arxiv.org/abs/2506.07982. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXi... on PithPith's reviewarXiv
  17. [2026] URLhttps://arxiv.org/abs/ 2606.22716. Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, and Xing Yu. Tackling length inflation without trade-offs: Group relative reward rescaling for reinforcement learning.ar... on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment