Pith. sign in

REVIEW 4 major objections 7 minor 27 references

Critic-free RL for language models gives the same positive credit to bad low-probability tokens as to good ones; TACO recalibrates that credit so flawed local steps stop getting reinforced.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:24 UTC pith:O4IXK4NV

load-bearing objection Clean diagnosis of a real GRPO credit bug plus a cheap, usable fix that actually moves numbers and long-horizon stability. the 4 major comments →

arxiv 2607.07976 v1 pith:O4IXK4NV submitted 2026-07-08 cs.CL cs.AIcs.LG

When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

Xiuyi Lou , Zicheng Xu , Yu-Neng Chuang , Hoang Anh Duy Le , Zhaozhuo Xu , Guanchu Wang , Vladimir Braverman This is my paper
classification cs.CL cs.AIcs.LG
keywords reinforcement learninglarge language modelscredit assignmentGRPOtail tokensreasoningpolicy optimizationPositive-Credit Contamination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When language models are trained with group-relative, critic-free reinforcement learning, a single success signal for a whole answer is copied onto every token in that answer. The paper argues that this broadcast rule creates Positive-Credit Contamination: locally wrong or nonsensical low-probability tokens that happen to sit inside a rewarded answer receive the same positive update as the tokens that actually carried the reasoning. Over training those bad local habits get strengthened. The authors propose TACO, which scores each token’s risk of belonging to the unreliable tail of the next-token distribution using its probability and the local entropy of the context, then softly reduces only positive credit for high-risk tokens. Across three base models and eight reasoning benchmarks, this calibration beats standard GRPO-style training and related stabilizers, and it keeps improving under longer training where the baseline plateaus or declines. A reader who cares about reliable chain-of-thought training should care because the method attacks a concrete failure mode of the dominant cheap RL recipe without adding a critic or heavy external feedback.

Core claim

The central discovery is that Positive-Credit Contamination is a real failure mode of GRPO-style reinforcement learning with verifiable rewards: implausible low-probability tail tokens inside otherwise correct trajectories receive identical positive sequence-level advantages, and the accumulation of those updates biases the policy toward flawed local continuations. TACO mitigates this by estimating a context-aware tail-risk score and using it to soft-suppress only positive token-level credit, preserving useful rare exploration while dampening incidental noise, with consistent gains on in-domain math and out-of-distribution science benchmarks and more stable long-horizon training.

What carries the argument

Tail-Aware Credit Calibration (TACO): a tail-risk score r_tail = −log p − H + log α that compares a token’s surprisal to the local entropy of the generation context, converted into a weight that multiplies only positive sequence advantages so high-risk tokens are softly down-weighted rather than zeroed out.

Load-bearing premise

The method assumes that a simple score built from a token’s probability and the local entropy of its context can tell useful rare exploration apart from contextually bad tail tokens well enough that soft positive-only down-weighting improves the policy rather than just swapping one bias for another.

What would settle it

Train the same model with TACO and with GRPO, then have independent process-level or human labels mark which tokens in rewarded trajectories are locally erroneous; if TACO’s high-risk tokens do not disproportionately match those erroneous tokens, or if forcing the same down-weighting on randomly chosen rare tokens produces equal gains, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper identifies Positive-Credit Contamination as a failure mode of GRPO-style critic-free RLVR: sequence-level positive advantages are broadcast uniformly, so locally implausible low-probability tokens inside rewarded completions receive the same positive credit as reliable tokens. It proposes TACO, which scores each token with a tail-risk measure r_tail = -log p - H + log α (Eq. 2) and softly down-weights only positive advantages for high-risk tokens (Eqs. 3–4), leaving negative advantages unchanged. Empirically, TACO is evaluated on three base models (Qwen3-1.7B/4B-Base, Qwen2.5-Math-7B) and eight benchmarks (six math, two OOD science), with comparisons to GRPO (clip-higher), advantage reweighting, and STAPO. Reported gains are consistent on average accuracy, with additional evidence from training dynamics, a 600-step long-horizon run where GRPO plateaus while TACO continues to improve, qualitative token-suppression cases, a controlled synthetic MDP isolating broadcast vs. oracle credit, and released code.

Significance. If the diagnosed failure mode and the proposed calibration are correct, the work is a useful, low-overhead contribution to reasoning-oriented RLVR: it names a concrete credit-assignment pathology of broadcast advantages, gives a simple generation-time fix that does not require critics or external process labels, and shows multi-scale, multi-benchmark gains plus improved long-horizon stability. Strengths that should be credited include the synthetic MDP that isolates broadcast contamination under controlled optimal/tail actions (Fig. 3), the multi-model evaluation with OOD science sets (Table 1), the extended-training comparison (Fig. 5), qualitative suppression cases (Figs. 6, 10, 11), training diagnostics showing sparse intervention (~1.9% of positive-advantage tokens; Fig. 9), and public source code. The practical significance is real even if the method is viewed as a carefully designed positive-only low-probability dampener rather than a fully validated semantic reliability filter.

major comments (4)
  1. The central mechanistic claim is that Eq. (2) separates contextually implausible tail tokens from useful rare exploration via the entropy adjustment, so that Eqs. (3)–(4) specifically mitigate Positive-Credit Contamination rather than generically damping low-p tokens under positive advantages. The manuscript motivates this with nucleus-sampling literature, the synthetic MDP (Fig. 3), and qualitative cases (Figs. 6, 10, 11), but does not ablate the entropy term: a pure-rarity baseline (e.g., r = -log p + c, or positive-only reweighting of low-p tokens without H) is missing. Without that ablation, gains over GRPO / Adv. Reweighting / STAPO (Table 1) cannot be confidently attributed to the claimed context-aware separation. Please add this ablation on at least one model and report both accuracy and which tokens are suppressed.
  2. Relatedly, there is no independent token-level check that high r_tail tokens inside rewarded traces are disproportionately the contextually erroneous ones. Fig. 9 shows only ~1.9% of positive-advantage tokens are downweighted on average, so mislabeling of useful rare patterns could still matter. Human or process-level labels on a sample of rewarded traces (or a proxy such as step-level verifier disagreement / self-consistency under local edits) would substantially strengthen the causal link from the diagnosed failure mode to TACO. If such labels are infeasible, at least quantify false-positive suppression rates on known useful rare tokens (e.g., uncommon but correct mathematical symbols) versus clear noise fragments.
  3. Section 5.2 and Table 1 frame TACO as improving by calibrating implausible positive credit 'beyond merely stabilizing' low-p updates as in STAPO and Adv. Reweighting. Those baselines address gradient dominance / spurious large updates, not the same positive-only contamination objective. To make the comparison load-bearing for the paper's claim, either (i) include a positive-only low-p dampener matched to TACO's intervention rate, or (ii) analyze whether TACO's suppressed tokens differ systematically from those affected by STAPO/Adv. Reweighting (overlap, entropy regime, position in the trace). Currently the superiority claim is empirical but the mechanistic contrast is under-specified.
  4. The synthetic MDP (Sec. 3.3, Fig. 3, App. B) usefully shows that broadcast advantages underperform oracle step credit as H grows, n_opt shrinks, and G shrinks. However, the MDP's 'tail' actions are oracle-labeled suboptimal actions with reward 0, whereas LLM implausible tokens are defined via policy probability and local entropy without outcome labels. Please clarify what this experiment does and does not establish: it supports that broadcast contamination can hurt, but it does not validate that r_tail recovers the oracle unreliable set under LLM-like distributions. A short bridging experiment (e.g., applying r_tail-style scoring inside the MDP and measuring recovery of suboptimal actions) would tighten this link.
minor comments (7)
  1. Eq. (2): the role of log α is described as making identification more aggressive for larger α, but the sign convention (adding log α so that r_tail > 0 marks risk) would be clearer with a short derivation or a one-line interpretation of the threshold p < α e^{-H}.
  2. Table 2 reports sensitivity for α ∈ {0.01, 0.005} and λ ∈ {0.6, 0.9}; the text mentions clear early degradation at α = 0.1 but does not tabulate it. Including that failed setting would help readers see the failure mode of over-aggressive tail identification.
  3. Figure 1 caption and main text refer to 'five representative benchmarks' while the abstract and Sec. 5 claim eight; align the wording so the figure is clearly a subset of Table 1.
  4. Notation: ℒ_{i,t} is introduced as the PPO-style clipped surrogate in Sec. 3.2 with a footnote to DeepSeek-R1; a self-contained formula (or explicit pointer to the exact clipped objective used in verl) would improve reproducibility for readers not carrying that report.
  5. In Fig. 6 and related cases, red/blue highlighting is informative; stating the exact mapping from w_{i,t} to color intensity (and whether weights are computed under π_θ_old or π_θ) would make the qualitative evidence easier to interpret.
  6. Minor polish: 'i.os ales' and mixed-language fragments in Fig. 6 are intentional examples of noise, but a few prose typos elsewhere (e.g., spacing around TACO in early sections, 'V olume' in bibliography) should be cleaned in revision.
  7. App. A: evaluation uses top-p=0.7, temperature 0.9 while training uses temperature 1.0, top-p=1.0. A brief note that this is intentional (and shared across methods) would preempt fairness concerns.

Circularity Check

0 steps flagged

No significant circularity: TACO is an empirical credit-shaping heuristic with free hyperparameters, evaluated on held-out benchmarks rather than a fitted quantity renamed as prediction.

full rationale

The paper’s load-bearing chain is (i) diagnose Positive-Credit Contamination under GRPO’s broadcast advantage via literature, qualitative traces, and a synthetic MDP with oracle step rewards (Fig. 3), (ii) propose a generation-time tail-risk score r_tail = −log p − H + log α and positive-only soft weights (Eqs. 2–4), and (iii) measure accuracy and training dynamics against GRPO, advantage reweighting, and STAPO on external math/science benchmarks. None of these steps reduce by construction to their inputs: α and λ are ordinary free hyperparameters selected by validation, not parameters fitted to a target quantity that is then reported as a prediction; the synthetic MDP compares broadcast vs. oracle credit under an independently defined reward, so the performance gap is not tautological; and self-citations (e.g., evaluation-prompt construction) are not used to force uniqueness or smuggle the central ansatz. The method is self-contained as an empirical RLVR intervention; whether r_tail truly separates implausible from useful rare tokens is a correctness/causal-attribution question, not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard RLVR machinery plus two free hyperparameters and the modeling assumption that surprisal excess over local entropy is a valid proxy for contextual implausibility. No new physical entities are postulated; the “invented” items are named phenomena and a scoring formula whose only external handle is downstream task accuracy.

free parameters (2)
  • α (tail-risk strictness) = 0.01 (default)
    Additive constant in r_tail = −log p − H + log α that sets how aggressively low-probability tokens are flagged; default 0.01 chosen by validation; sensitivity table shows performance depends on it.
  • λ (maximum suppression strength) = 0.9 (default)
    Caps the credit weight floor at 1−λ for high-risk tokens; default 0.9; jointly tuned with α.
axioms (4)
  • domain assumption Outcome-level verified reward plus group-normalized sequence advantage is a valid training signal for reasoning LLMs (GRPO/DAPO-style RLVR).
    Taken as the starting point of the paper; all experiments build on this stack without re-deriving its validity.
  • ad hoc to paper Tokens whose surprisal substantially exceeds local entropy are more likely to be contextually unreliable (“implausible tail”) than tokens that are merely rare under high entropy.
    Core modeling choice of Eq. 2; motivated by nucleus-sampling literature but not proved; load-bearing for the credit gate.
  • ad hoc to paper Soft positive-only reweighting preserves useful rare patterns while damping incidental noise; negative advantages should remain unmodulated.
    Design of Eqs. 3–4; alternative hard masks or two-sided reweighting are not exhaustively compared.
  • domain assumption Standard PPO-style clipped surrogate with asymmetric clip-higher remains the correct outer optimizer once advantages are replaced by TACO advantages.
    Algorithm 1 leaves the GRPO surrogate unchanged except for the advantage tensor.
invented entities (2)
  • Positive-Credit Contamination no independent evidence
    purpose: Name the failure mode in which implausible tail tokens inside rewarded trajectories receive the same positive advantage as reliable tokens.
    Conceptual label introduced by the paper; independent evidence is the synthetic MDP gap and qualitative cases, not an external measurement.
  • Tail-risk score r_tail and TACO credit weight w no independent evidence
    purpose: Operational proxy that converts local probability and entropy into a soft positive-credit multiplier.
    New scoring formula; falsifiable only via downstream RL performance, not via an external physical prediction.

pith-pipeline@v1.1.0-grok45 · 20811 in / 3459 out tokens · 43379 ms · 2026-07-10T14:24:03.421134+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: https://github.com/xiuyilou/TACO.

Figures

Figures reproduced from arXiv: 2607.07976 by Guanchu Wang, Hoang Anh Duy Le, Vladimir Braverman, Xiuyi Lou, Yu-Neng Chuang, Zhaozhuo Xu, Zicheng Xu.

Figure 1
Figure 1. Figure 1: TACO consistently improves over GRPO across representative benchmarks and models. irrelevant or erroneous; we refer to such locally unreliable tokens as implausible tail tokens. Yet, these tokens’ local effects may be bypassed or overlooked by the overall correct reasoning process and final answer [22, 10, 6]. These tokens then receive the same positive credit as reliable ones, resulting in indiscriminate … view at source ↗
Figure 2
Figure 2. Figure 2: Examples of unreliable tokens from various sources in real correct reasoning traces. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Demonstration of the effect of Positive-Credit Contamination on a synthetic sequential MDP. Experiments are conducted to analyze how (a) trace length H, (b) optimal-action count nopt, and (c) group size G affect the contamination behavior. ∆ denotes the final-step reward gap between oracle and broadcast; the largest gap in each panel is bolded. The phenomenon that LLMs can produce locally unreliable tokens… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of TACO and baseline methods on Qwen3-1.7B-Base. Results on other models are provided in Appendix C. 0 100 200 300 400 500 600 Training Step 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Accuracy (%) Ours GRPO (a) AIME24 test accuracyprogre 0 100 200 300 400 500 600 Training Step 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Entropy (b) Policy entropy [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of TACO and GRPO on Qwen3-1.7B-Base with extended training. Learning Effectiveness. As shown in Figure 4a, TACO achieves consistently higher training accu￾racy than baselines. This improvement stems from the selective nature of TACO’s credit calibration: by suppressing positive credit for locally implausible tokens, each policy update concentrates reinforce￾ment on contextually more reliable rea… view at source ↗
Figure 6
Figure 6. Figure 6: A case from Qwen3-4B-Base. TACO selectively downweights tail tokens (shown in red; darker red indicates stronger suppression), while preserving coherent reasoning steps (shown in blue) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics of TACO and baseline methods on Qwen3-4B-Base. 0 25 50 75 100 125 150 175 200 Training Step −0.6 −0.4 −0.2 0.0 Reward Ours STAPO GRPO Adv. Reweighting (a) Training reward 0 25 50 75 100 125 150 175 200 Training Step 0.1 0.2 0.3 0.4 0.5 Entropy (b) Policy entropy 0 25 50 75 100 125 150 175 200 Training Step 900 950 1000 1050 1100 Response Length (c) Response length [PITH_FULL_IMAGE:figure… view at source ↗
Figure 8
Figure 8. Figure 8: Training dynamics of TACO and baseline methods on Qwen2.5-Math-7B. Figures 10 and 11 show two additional rewarded traces from different models. The Qwen3-1.7B-Base case contains tail tokens related to instruction or search leakage, such as off-context assistant/search text and unrelated help messages. The Qwen2.5-Math-7B case contains malformed template frag￾ments and serialized control artifacts, such as … view at source ↗
Figure 9
Figure 9. Figure 9: Training diagnostics over positive-advantage response tokens. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A case from Qwen3-1.7B-Base. Prompt. An airline serves a dinner to all the passengers on an airplane. They get their choice of steak or fish. Three steak meals and three fish meals are set aside for the six-member crew. If the meals are distributed to the crew members randomly, what is the probability that both pilots get the fish? The answer is in the form m n , where gcd(m, n) = 1. Please provide the va… view at source ↗
Figure 11
Figure 11. Figure 11: A case from Qwen2.5-Math-7B. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 8 internal anchors

  1. [2]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    work page 2025

  2. [3]

    Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceed...

    work page doi:10.18653/v1/2024.acl-long.211 2024

  3. [4]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview. net/forum?id=rygGQyrFvH

    work page 2020

  4. [6]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    work page 2024

  5. [7]

    URLhttps://openreview.net/forum?id=IkmD3fKBPQ

    OpenReview.net, 2024. URLhttps://openreview.net/forum?id=IkmD3fKBPQ

    work page 2024

  6. [8]

    Execution-grounded credit assignment for grpo in code generation.arXiv preprint arXiv:2603.16158, 2026

    Abhijit Kumar, Natalya Kumar, and Shikhar Gupta. Execution-grounded credit assignment for grpo in code generation.arXiv preprint arXiv:2603.16158, 2026

    work page arXiv 2026

  7. [9]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V . Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. O...

    work page 2022

  8. [11]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [13]

    Heterogeneous adaptive policy optimization: Tailoring optimization to every token’s nature,

    Zheng Liu, Mengjie Liu, Siwei Wen, Mengzhang Cai, Bin Cui, Conghui He, and Wentao Zhang. Heterogeneous adaptive policy optimization: Tailoring optimization to every token’s nature,

    work page

  10. [14]

    URLhttps://arxiv.org/abs/2509.16591. 10

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [15]

    AMC23: American mathematics competitions 2023 test set

    math-ai Team. AMC23: American mathematics competitions 2023 test set. https: //huggingface.co/datasets/math-ai/amc23, 2024. Hugging Face dataset. Accessed: 2026-05-06

    work page 2023

  12. [18]

    Grpo- λ: Credit assignment improves llm reasoning, 2025

    Prasanna Parthasarathi, Mathieu Reymond, Boxing Chen, Yufei Cui, and Sarath Chandar. Grpo- λ: Credit assignment improves llm reasoning, 2025. URL https://arxiv.org/abs/2510. 00194

    work page 2025

  13. [19]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [20]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. URL http://arxiv.org/ abs/1707.06347

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  15. [22]

    Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy.arXiv preprint arXiv:2508.04349, 2025

    Hongze Tan, Zihan Wang, Jianfei Pan, Jinghao Lin, Hao Wang, Yifan Wu, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy.arXiv preprint arXiv:2508.04349, 2025

    work page arXiv 2025

  16. [25]

    verl: V olcano engine reinforcement learning for llms

    verl-project. verl: V olcano engine reinforcement learning for llms. https://github.com/ verl-project/verl, 2024. GitHub repository. Accessed: 2026-05-06

    work page 2024

  17. [26]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela ...

    work page 2024

  18. [27]

    Neural text generation with unlikelihood training

    Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URLhttps://openreview.net/forum?id=SJeYe0NtvH. 11

    work page 2020

  19. [30]

    Self-ensemble: Mitigating confidence distortion for large language models

    Zicheng Xu, Guanchu Wang, Guangyao Zheng, Yu-Neng Chuang, Alex Szalay, Xia Hu, and Vladimir Braverman. Self-ensemble: Mitigating confidence distortion for large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, No...

    work page 2025

  20. [31]

    URLhttps://aclanthology.org/2025.findings-emnlp.902/

    work page 2025

  21. [32]

    Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning

    Zicheng Xu, Ruixuan Zhang, Yu-Neng Chuang, Xiuyi Lou, Hoang Anh Duy Le, Oren Gal, Alexander S Szalay, Zhaozhuo Xu, Guanchu Wang, and Vladimir Braverman. Learning at the right pace: Adaptive data scheduling improves llm reinforcement learning.arXiv preprint arXiv:2606.22305, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [34]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [35]

    Matthew Y . R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, and Aviral Kumar. Int: Self-proposed interventions enable credit assignment in LLM reasoning.CoRR, abs/2601.14209,

    work page arXiv

  24. [37]

    Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

    Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yun- jian Xu. Do not let low-probability tokens over-dominate in rl for llms.arXiv preprint arXiv:2505.12929, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [40]

    American invitational mathematics examination (AIME) 2024

    Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2024. https://huggingface.co/datasets/math-ai/aime24, 2024. Hugging Face dataset. Ac- cessed: 2026-05-06. 12

    work page 2024

  26. [41]

    American invitational mathematics examination (AIME) 2025

    Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2025. https://huggingface.co/datasets/math-ai/aime25, 2025. Hugging Face dataset. Ac- cessed: 2026-05-06

    work page 2025

  27. [42]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization.CoRR, abs/2507.18071, 2025. doi: 10.48550/ARXIV .2507.18071. URL https: //doi.org/10.48550/arXiv.2507.18071. A Implementation Details Training Setup.We train all methods with ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 2025