Pith. sign in

REVIEW 4 major objections 4 minor 57 references

When an LLM is trained by reinforcement learning on a new objective, the KL term meant to stop forgetting should constrain only the distribution over verified-correct responses, not the whole response distribution.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CoKL regularizes only the correctness-conditioned response distribution, decoupling total correctness from mode preservation and improving retention-adaptation balance in LLM RL.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection CoKL is a genuinely cleaner correctness-conditioned KL regularizer with solid theory and honest experiments, though its preservation guarantee leans on the verifier being right and the main LLM results are single-run. the 4 major comments →

arxiv 2608.01743 v1 pith:ZHCSITUN submitted 2026-08-03 cs.LG cs.CL

Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning

Li Wang , Xiaodong Lu , Xiaohan Wang , Jiajun Chai , Wei Lin , Tianhao Peng , Guojun Yin This is my paper
classification cs.LG cs.CL
keywords correctness-conditioned KL regularizationcapability retentioncatastrophic forgettingLLM post-trainingreinforcement learning with verifiable rewardsGRPOKL divergence decompositioncontinual post-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RL-based post-training of LLMs routinely degrades capabilities the base model already has, and the standard remedy — KL regularization toward a frozen reference — constrains the entire response distribution, including outputs the task reward wants to suppress. This paper proposes Correctness-Conditioned KL Regularization (CoKL), which narrows the preservation constraint to the conditional distribution over verifier-correct responses. A decomposition of full-policy forward KL into a correctness-probability term, a correctness-conditioned term, and an incorrect-response term shows that standard regularization anchors all three; CoKL keeps only the middle term, so the policy is free to raise the total probability of correct answers while the relative allocation among correct modes stays anchored to the reference. The paper proves that full-policy forward and reverse KL objectives place their optimal correctness probability strictly between the reference's value and 1 when the reference is imperfect, whereas CoKL's optimum is total correctness 1 with the reference's correct-response shape. Experiments in a controlled multi-solution environment and in continual math-to-chat post-training across three model scales show CoKL retaining more prior capability while matching or exceeding new-task adaptation.

Core claim

CoKL's core claim rests on a decomposition identity plus a consequence. For any policy, full forward KL against the reference decomposes as D_KL(Bern(Z_ref) || Bern(Z_theta)) + Z_ref·D_KL(pi+_ref || pi+_theta) + (1 − Z_ref)·D_KL(pi−_ref || pi−_theta), where Z is the total probability of a verifier-correct response and pi+/pi− are the policy conditioned on producing a correct or incorrect response. Standard regularization keeps all three terms, thereby anchoring the policy's overall correctness level and its incorrect modes to the reference. CoKL keeps only the middle term — the forward KL between correctness-conditioned distributions — so total correctness mass Z_theta is decoupled from the

What carries the argument

The load-bearing object is the correctness decomposition of forward KL and the conditional objective built from it: CoKL = D_KL(pi+_ref || pi+_theta), the forward KL between reference and current policies each conditioned on producing a verifier-correct response. This conditional KL acts as a shape constraint on the correct-response distribution; its gradient, −E_{pi+_ref}[∇log pi_theta] + E_{pi+_theta}[∇log pi_theta], is implemented in the finite-group surrogate by a reference-correct anchor term with stop-gradient self-normalized weights and a current-policy normalization correction, with zero-correct groups falling back to a one-sided recovery pressure. The reference-correct buffer — prom

Load-bearing premise

The verifier's binary correctness label reliably identifies the behaviors that constitute capabilities worth preserving; if the verifier is wrong, CoKL deliberately leaves those behaviors unanchored and may weaken or erase them.

What would settle it

In the paper's controlled bandit with reference correctness probability q around 0.10 and a moderate beta, the theorem predicts CoKL's final correctness approaches 1 while full forward KL approaches (1 − beta + sqrt((beta − 1)^2 + 4 beta q))/2; if CoKL does not exceed full-policy KL's correctness under matched training, the decoupling claim collapses. Alternatively, flip the verifier labels on a subset of correct actions: CoKL's coverage of those actions should drop noticeably relative to full-policy KL, confirming that the preservation guarantee is conditional on verifier correctness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If CoKL is right, KL regularization in LLM RL can be scoped to correctness-conditioned distributions, so raising the probability of correct answers is never penalized by the regularizer — only the relative shape among correct modes is.
  • Full-policy forward and reverse KL are provably biased against maximizing correctness when the reference policy is imperfect; CoKL removes that bias, letting regularized RL reach higher correctness without abandoning reference-supported response modes.
  • The method drops into GRPO as an auxiliary loss with a cached reference-correct buffer and clipped off-policy importance weights, so existing verifiable-reward RL pipelines can adopt it without new value models or reward-model changes.
  • In continual post-training (math then chat), CoKL attains the highest aggregate retention–adaptation score at 0.6B, 1.7B, and 4B scale in the paper's setups, preserving math capability while recovering essentially all chat adaptation of unregularized RL.
  • Both CoKL terms matter: dropping the reference anchor causes severe forgetting, and dropping the current-policy normalization correction (which decouples correctness mass from shape) reduces new-task adaptation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The conditioning event need not be correctness: any verifiable binary property — safety compliance, factual grounding, style adherence — defines a CoKL-style regularizer that preserves the reference's allocation among acceptable responses without taxing the rate of acceptability; this extension is implicit but not tested in the paper.
  • The closed-form optimum formulas for forward and reverse KL are directly measurable in the bandit environment; fitting them empirically would quantify exactly how much correctness headroom full-policy KL sacrifices relative to CoKL.
  • Because CoKL anchors only verifier-labeled correct responses, label noise or verifier bias becomes a first-order risk: systematically mislabeling a valid behavior should produce measurable capability loss for CoKL but not for full-policy KL, a testable stress test.
  • The reference-correct buffer excludes prompts where the reference never produces a correct answer, so capabilities that are rare or absent under the reference receive no preservation anchor; seeding the buffer with a more exploratory reference or a diverse sampler could extend CoKL's reach beyond the reference's support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CoKL, a regularizer for RL-based LLM post-training that constrains only the correctness-conditioned response distribution rather than the full policy. Using a binary verifier, the authors decompose full-policy forward KL into a Bernoulli term over correctness, a correct-conditional KL, and an incorrect-conditional KL (Eq. 7), and retain only D_KL(π+_ref || π+_θ). The population-level objective J = Zπ − β D_KL(π+_ref || π+) is shown to be increasing in Zπ for fixed π+, so global optima have Zπ = 1 and π+ = π+_ref (Theorem 1); full-policy forward and reverse KL induce correctness gaps q < Z* < 1 for an imperfect reference (Theorem 2). A finite-group surrogate (Eq. 17) and a reference-correct buffer are integrated with GRPO. Experiments in a controlled bandit and continual math→chat post-training on Qwen3-0.6B/1.7B/4B compare CoKL with full-policy and correct-only KL baselines.

Significance. The conceptual proposal is well-motivated: restricting preservation to correct-conditioned modes could reduce interference while retaining capabilities. The derivations are clean, self-contained, and not fitted to data; the closed-form correctness-gap analysis for full-policy KL (Theorem 2) is a substantive contribution. The controlled environment is evaluated over five seeds, and code is released. However, the headline empirical claim rests on single-run LLM experiments under a custom dual-anchor normalization, the implemented surrogate differs materially from the population objective analyzed, and the approach is untested under imperfect verifiers. The theoretical result is attractive, but the practical evidence is not yet at the level needed to establish the method's advantage.

major comments (4)
  1. [Table 1 and Appendix E.5] The headline claim of 'highest aggregate performance across all model scales' is based on dual-anchor normalization, and the raw scores show much smaller differences. For example, Table 6 raw Overall for CoKL vs Correct-only Forward-KL is 55.43 vs 53.08 (0.6B), 45.32 vs 44.44 (1.7B), and 56.74 vs 56.62 (4B). The normalized transform (Eqs. 74-77) is sensitive to the two anchors (Base and GRPO w/o KL), and no variance, confidence intervals, or significance tests are reported for any LLM run. A 0.12-point raw difference at 4B becomes a 1.40-point normalized difference. This is not sufficient evidence for the claimed advantage.
  2. [Eq. (17), Algorithm 1, and Appendix B] Theorems 1 and 2 analyze exact population-level objectives over unrestricted distributions, but the trained system minimizes a clipped off-policy surrogate with self-normalized importance weights and a zero-correct-group fallback. Appendix B explicitly states that the clipped off-policy surrogate 'does not admit the same exact recovery-term characterization.' The finite-group analysis in Eqs. (50)-(63) applies to the expected on-policy surrogate, not to the implemented loss with importance clipping and off-policy rollouts. Thus the decoupling guarantee is not directly inherited by the practical objective. The paper should either provide an analogous analysis for the actual surrogate or state this limitation clearly and support the practical claim empirically.
  3. [Eqs. (5)-(6), Method: Correctness Decomposition] The method's preservation guarantee is contingent on the verifier r(x,y) correctly identifying which responses constitute capabilities worth preserving. A false negative places a valid response in π−, which CoKL does not anchor; the RL objective can then suppress it. Full-policy KL anchors both classes and therefore retains some protection under label noise. The paper provides no robustness analysis, and the experiments use exact-match math verifiers, which are comparatively reliable. Since CoKL is proposed as a general post-training technique, the absence of experiments or analysis with noisy verifiers is a load-bearing limitation of the central retention claim.
  4. [Theorems 1 and 2 vs Appendix E.3] The theoretical decoupling results are proved under a shared prompt distribution between the reward objective and the regularizer. In the continual post-training experiments, the RL objective is on a chat task while regularization is on math prompts from a buffer, so Zπ and π+ refer to different prompt distributions. The paper acknowledges this in the transition sentence, but the main empirical design does not directly test the theorem's prediction, and the claimed 'decoupling' is not what drives the reported chat-task result. This gap should be addressed or explicitly qualified as a scope limitation of the theory.
minor comments (4)
  1. [Theoretical Analysis and Appendices C-D] After Theorem 1, the text says 'Theorem 3 shows...' but the intended reference is Theorem 1; Appendix C labels the result 'Theorem 3' and Appendix D labels it 'Theorem 4.' The numbering and cross-references should be harmonized.
  2. [Appendix E.5] The caveat that CoKL's advantage 'naturally reduces toward correct-only replay when forgetting is mild' is an important qualification but appears only in the appendix. Given the abstract's general claim, this limitation should be stated in the main text or conclusion.
  3. [Tables 1 and 6] The caption of Table 1 says 'Best and second-best post-training results are bolded and underlined,' but for Qwen3-4B some rows have negative normalized values and 'second-best' is not always visually clear. Consider reporting raw scores with confidence intervals in the main table.
  4. [Throughout] There are minor grammatical issues, e.g., 'The full derivation are provided in Appendix B' should be 'derivations are.' The paper would also benefit from a sentence in the main text explaining that Appendix E.3 reports the hyperparameter search for baselines while CoKL's β=1 was chosen from Figure 4.

Circularity Check

0 steps flagged

No significant circularity: CoKL's core theoretical claims follow from its stated definition and are independently benchmarked.

full rationale

The paper's central derivation chain is not circular. The full-policy KL decomposition (Eq. 7 / Appendix A) is standard conditional-probability algebra; it does not assume the CoKL conclusion. The CoKL objective is defined as D_KL(pi+_ref || pi+_theta) (Eq. 8), and Theorem 1's decoupling is a direct algebraic consequence of that definition (Eqs. 9-10), not a fitted quantity presented as a prediction. Theorem 2's strict correctness gap under full-policy KL is derived analytically from the Bernoulli decomposition and closed-form optimization (Appendix D), independent of CoKL's definition. The finite-group surrogate properties in Appendix B are derived from self-normalized importance weights and the zero-correct fallback convention; they characterize the estimator rather than assume its effect. The empirical results compare CoKL against external baselines (GRPO, full forward/reverse KL, correct-only forward/reverse KL) on controlled and continual post-training settings, so the retention-adaptation claims are externally falsifiable. The related-work self-citations (e.g., ResRL, Contextual Rollout Bandits, Implicit Hierarchical GRPO) are contextual and not load-bearing for the paper's theorems or experiments. The verifier-correctness limitation noted in the reader's take is an assumption about input labels, not a circular step: CoKL's derivation explicitly conditions on r(x,y), and imperfect labels would degrade the method empirically but do not make the derivation self-referential. The decoupling result is close to the definition of CoKL, which is why the score is 1 rather than 0, but no prediction is forced by construction and no load-bearing self-citation exists.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central theoretical claim rests only on standard probability and KL identities plus a domain assumption about the correctness signal. The practical method introduces no new entities. The main idealizations are the unrestricted policy simplex and the population-level reward equal to Z, which the paper explicitly flags.

free parameters (3)
  • beta (CoKL regularization coefficient) = 1.0
    Chosen via sensitivity sweep (0.1, 1, 10) on Qwen3-1.7B. It is a standard regularization weight balancing retention and adaptation, not fitted to the theoretical claims.
  • tau (correctness threshold for continuous reward scores) = task-dependent, unspecified
    Defines the correctness partition when rewards are continuous. The paper states this extension but does not specify a value or use it in the main experiments.
  • G_ref (reference samples per prompt) and KL batch size = G_ref=8, batch size=8
    Design choices for the reference-correct buffer. Larger batches improve retention but increase cost. Not fitted to the theory.
axioms (4)
  • domain assumption For each prompt, responses are assigned a binary correctness label r(x,y) that partitions the response space.
    Section 'Method: Correctness Decomposition'. The entire CoKL method and its theory presume this correctness signal is meaningful and stable.
  • ad hoc to paper Policies are arbitrary distributions over a finite response set, so (Z, pi+, pi-) can be varied independently.
    Used in the proofs of Theorems 1 and 2 (Appendices C and D). Real neural policies share parameters, so this unrestricted-simplex assumption does not hold in practice.
  • ad hoc to paper The population-level RL objective equals the expected correctness probability Z_pi.
    Theorems 1 and 2 analyze J = Z - beta*KL, but GRPO uses clipped group-relative advantages. The paper acknowledges the gap between this idealization and the practical objective.
  • standard math 0 < Z_ref < 1 (imperfect reference policy) and Z_theta in (0,1) for the decomposition.
    Assumed in Eq. 7 and Theorem 2. Boundary cases are handled by extended-value conventions for KL.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZHCSITUN

@misc{pith2026260801743,
  author       = {Pith},
  title        = {Pith review of: Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHCSITUN}},
  note         = {Machine review of arXiv:2608.01743}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) has become a central paradigm for large language model (LLM) post-training, but optimization toward new objectives can degrade capabilities already present in the base model. KL regularization is widely used to mitigate such forgetting by constraining policy drift toward a reference model. However, standard full-policy KL regularization constrains the entire response distribution and may unnecessarily restrict exploration and target-task learning. This raises a natural question: can a more precise constraint preserve existing capabilities while minimizing interference with learning new tasks? To this end, we propose \underline{Co}rrectness-Conditioned \underline{KL} Regularization (CoKL), a conditional regularization framework that narrows the preservation constraint from the full output distribution to correctness-conditioned response distributions. We instantiate CoKL with forward KL divergence and derive a practical finite-group training objective for RL-based LLM post-training. At the population level, CoKL decouples the total probability assigned to correct responses from their correctness-conditioned distribution, thereby regularizing the relative probability allocation among reference-supported correct responses without directly anchoring incorrect outputs or total correctness mass. We further show that full-policy forward and reverse KL regularization induce a strict optimal correctness gap when the reference policy is imperfect, whereas CoKL avoids this limitation. Experiments in controlled multi-solution environments and continual post-training settings across multiple model scales demonstrate that CoKL achieves a more favorable balance between target-task improvement and prior-capability retention than existing regularization methods. Our code is available at https://github.com/Lumina04/CoKL.

Figures

Figures reproduced from arXiv: 2608.01743 by Guojun Yin, Jiajun Chai, Li Wang, Tianhao Peng, Wei Lin, Xiaodong Lu, Xiaohan Wang.

Figure 1
Figure 1. Figure 1: Schematic comparison of KL constraint scopes. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Trade-offs between correctness and coverage@64 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (Left) Dynamics of correct-mode coverage@64 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effects of KL batch size (Left) and loss coefficient [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity of the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of different methods on Qwen3- [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt templates used for the math and chat tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Representative output of the CoKL-trained Qwen3-0.6B model on the math task. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Representative output of the CoKL-trained Qwen3-0.6B model on the general chat task. The reported score is provided [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 20 canonical work pages · 5 internal anchors

  1. [4]

    Chen, Z.; Lu, R.; Zhao, A.; Wang, Z.; Yue, Y.; Song, S.; and Huang, G. 2026. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Advances in Neural Information Processing Systems, 38: 57654--57689

    work page 2026

  2. [10]

    He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3828--3850

    work page 2024

  3. [12]

    Kotha, S.; Springer, J.; and Raghunathan, A. 2024. Understanding catastrophic forgetting in language models via implicit inference. In International Conference on Learning Representations, volume 2024, 24110--24139

    work page 2024

  4. [14]

    H.; Gonzalez, J.; Zhang, H.; and Stoica, I

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; and Stoica, I. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, 611--626

    work page 2023

  5. [20]

    Lin, Y.; Lin, H.; Xiong, W.; Diao, S.; Liu, J.; Zhang, J.; Pan, R.; Wang, H.; Hu, W.; Zhang, H.; et al. 2024. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 580--606

    work page 2024

  6. [27]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730--27744

    2022

  7. [29]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2024. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  8. [30]

    Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33: 3008--3021

    work page 2020

  9. [31]

    Vassoyan, J.; Beau, N.; and Plaud, R. 2025. Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning. In Findings of the Association for Computational Linguistics: NAACL 2025, 6108--6118

    work page 2025

  10. [38]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 113222--113244

    work page 2026

  11. [41]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  12. [42]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  13. [43]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  14. [44]

    arXiv preprint arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

    Pith/arXiv arXiv 1909

  15. [45]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  16. [46]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  17. [47]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  18. [48]

    arXiv preprint arXiv:2509.07430 , year=

    The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward , author=. arXiv preprint arXiv:2509.07430 , year=

    arXiv

  19. [49]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  20. [50]

    Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

    Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning , author=. arXiv preprint arXiv:2605.18500 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [51]

    arXiv preprint arXiv:2510.03865 , year=

    Unlocking reasoning capabilities in llms via reinforcement learning exploration , author=. arXiv preprint arXiv:2510.03865 , year=

    work page arXiv

  22. [52]

    SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs

    SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs , author=. arXiv preprint arXiv:2605.18864 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [53]

    arXiv preprint arXiv:2510.20817 , year=

    KL-Regularized Reinforcement Learning is Designed to Mode Collapse , author=. arXiv preprint arXiv:2510.20817 , year=

    arXiv

  24. [54]

    expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling

    expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling , author=. arXiv preprint arXiv:2605.09923 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [55]

    arXiv preprint arXiv:2507.15778 , year=

    Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr , author=. arXiv preprint arXiv:2507.15778 , year=

    Pith/arXiv arXiv

  26. [56]

    arXiv preprint arXiv:2505.22617 , year=

    The entropy mechanism of reinforcement learning for reasoning language models , author=. arXiv preprint arXiv:2505.22617 , year=

    Pith/arXiv arXiv

  27. [57]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    work page 2025

  28. [58]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  29. [59]

    arXiv preprint arXiv:2507.18071 , year=

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    Pith/arXiv arXiv

  30. [60]

    arXiv preprint arXiv:2509.02479 , year=

    Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning , author=. arXiv preprint arXiv:2509.02479 , year=

    Pith/arXiv arXiv

  31. [61]

    arXiv preprint arXiv:2509.21826 , year=

    ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models , author=. arXiv preprint arXiv:2509.21826 , year=

    arXiv

  32. [62]

    ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

    Resrl: Boosting llm reasoning via negative sample projection residual reinforcement learning , author=. arXiv preprint arXiv:2605.00380 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [63]

    arXiv preprint arXiv:2602.08499 , year=

    Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards , author=. arXiv preprint arXiv:2602.08499 , year=

    Pith/arXiv arXiv

  34. [64]

    Advances in Neural Information Processing Systems , volume=

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? , author=. Advances in Neural Information Processing Systems , volume=

    work page

  35. [65]

    arXiv preprint arXiv:2507.14783 , year=

    Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling , author=. arXiv preprint arXiv:2507.14783 , year=

    arXiv

  36. [66]

    arXiv preprint arXiv:2507.17512 , year=

    Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning , author=. arXiv preprint arXiv:2507.17512 , year=

    Pith/arXiv arXiv

  37. [67]

    arXiv preprint arXiv:2602.12566 , year=

    To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models , author=. arXiv preprint arXiv:2602.12566 , year=

    arXiv

  38. [68]

    arXiv preprint arXiv:2602.02301 , year=

    Advancing General-Purpose Reasoning Models with Modular Gradient Surgery , author=. arXiv preprint arXiv:2602.02301 , year=

    arXiv

  39. [69]

    arXiv preprint arXiv:2505.17508 , year=

    On the design of kl-regularized policy gradient algorithms for llm reasoning , author=. arXiv preprint arXiv:2505.17508 , year=

    arXiv

  40. [70]

    Advances in Neural Information Processing Systems , volume=

    Afterburner: Reinforcement learning facilitates self-improving code efficiency optimization , author=. Advances in Neural Information Processing Systems , volume=

    work page

  41. [71]

    arXiv preprint arXiv:2507.02259 , year=

    Memagent: Reshaping long-context llm with multi-conv rl-based memory agent , author=. arXiv preprint arXiv:2507.02259 , year=

    Pith/arXiv arXiv

  42. [72]

    Advances in Neural Information Processing Systems , volume=

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  43. [73]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  44. [74]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  45. [75]

    WildChat: 1M Chat

    Wenting Zhao and Xiang Ren and Jack Hessel and Claire Cardie and Yejin Choi and Yuntian Deng , booktitle=. WildChat: 1M Chat. 2024 , url=

    2024

  46. [76]

    Advances in Neural Information Processing Systems , volume=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=

  47. [77]

    arXiv preprint arXiv:2512.15489 , year =

    Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision , author =. arXiv preprint arXiv:2512.15489 , year =

    arXiv

  48. [78]

    arXiv preprint arXiv:2509.20357 , year=

    Language models that think, chat better , author=. arXiv preprint arXiv:2509.20357 , year=

    arXiv

  49. [79]

    arXiv preprint arXiv:2410.18451 , year=

    Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs , author=. arXiv preprint arXiv:2410.18451 , year=

    Pith/arXiv arXiv

  50. [80]

    arXiv preprint arXiv:2512.05962 , year=

    Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity , author=. arXiv preprint arXiv:2512.05962 , year=

    work page arXiv

  51. [81]

    Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

    Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity , author=. arXiv preprint arXiv:2605.00365 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  52. [82]

    arXiv preprint arXiv:2602.19895 , year=

    Dsdr: Dual-scale diversity regularization for exploration in llm reasoning , author=. arXiv preprint arXiv:2602.19895 , year=

    arXiv

  53. [83]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Mitigating the alignment tax of rlhf , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  54. [84]

    International Conference on Learning Representations , volume=

    Understanding catastrophic forgetting in language models via implicit inference , author=. International Conference on Learning Representations , volume=

    work page

  55. [85]

    arXiv preprint arXiv:2204.05862 , year=

    Training a helpful and harmless assistant with reinforcement learning from human feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    Pith/arXiv arXiv

  56. [86]

    arXiv preprint arXiv:2507.14843 , year=

    The invisible leash: Why rlvr may or may not escape its origin , author=. arXiv preprint arXiv:2507.14843 , year=

    arXiv

  57. [87]

    arXiv preprint arXiv:2503.20783 , year=

    Understanding r1-zero-like training: A critical perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.