Pith. sign in

REVIEW 3 major objections 5 minor 55 references

GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rollout RL instability in LLMs is linked to transient overlap with pretrained principal subspaces, and a hard bilateral-orthogonality constraint (GCPO) outperforms GRPO and recent variants on math, code, and tool use.

desk verdict A genuinely useful diagnostic and a plausible geometric constraint for rollout RL, but the empirical case for GCPO is tied to a single k=8 peak selected on the same benchmarks, so 'consistent outperformance' is not yet established. read the letter →

arxiv 2608.11674 v1 pith:BNKJZS3C submitted 2026-08-12 cs.LG cs.AI

Kai Yang , Jingwei Xu , Wanyu Wang , Kai-Yuan Guo , Zhenbo Yu , Yi Wang , Yu Qiao This is my paper · ORCID
classification cs.LGcs.AI
keywords rolloutreinforcementlearningGRPOLLMpost-trainingsingularsubspacegeometrypolicyoptimizationresponse-lengthinflationlow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rollout-based reinforcement learning for large language models frequently destabilizes training, degrades off-task capabilities, and inflates response length. This paper argues that the stepwise parameter updates generated by methods like GRPO sometimes enter the dominant singular subspaces of pretrained weights, and that transient excess principal-subspace overlap is a warning signal that precedes performance drops. To test the causal reading, the authors run a controlled intervention in which the overlapping component of an update is rescaled while preserving update norm, and find a dose-dependent accuracy loss. They then propose GCPO, which constrains every effective layer update to lie in the bilateral orthogonal complement of those principal subspaces, and report consistent gains over GRPO, DAPO, GSPO, GMPO, and matched LoRA baselines on math, code, and tool-use tasks across two model families.

What carries the argument

The central objects are the principal singular subspaces of each pretrained weight matrix, obtained from its SVD $W_{\mathrm{ref}} = \Phi\Sigma\Psi^{\top}$; the top-$k$ columns of $\Phi$ and $\Psi$ span the principal output and input subspaces. The diagnostic is the dimension-corrected excess overlap $O^{\mathrm{excess}}_t = O_t - O_{\mathrm{null}}$, where $O_t$ is the fraction of update energy in the three blocks with at least one-sided principal overlap and $O_{\mathrm{null}}$ is the expected value for an isotropic update. The intervention is GCPO, which reparameterizes each layer update as $\delta W^{(\ell)} = \alpha \Pi^\perp_\Phi L^{(\ell)} R^{(\ell)} \Pi^\perp_\Psi$, so both left and right projections are hard constraints by construction rather than penalties. This machinery turns the observed off-principal tendency into an exact feasible set and yields a layer-level guarantee: inputs in the principal input subspace see an unchanged layer response.

What would settle it

Train GRPO and GCPO on a wider set of backbones and track the raw excess-overlap time series: if validation accuracy declines occur with no preceding overlap spike, or if spikes occur without any later decline, the warning-signal claim fails. Separately, if GCPO's advantage over GRPO-LoRA (same rank, no projection) disappears or reverses on those settings, then low-rank parameterization rather than bilateral orthogonality would be the cause of the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that the geometry of individual rollout updates, not just their aggregate statistics, is diagnostic of, and partly responsible for, instability in rollout RL for LLMs. Concretely, it introduces excess principal-subspace overlap, the fraction of update energy landing on the top-k left and right singular subspaces of a pretrained weight matrix minus the value expected under an isotropic update of the same dimensions. Although average overlap is small, episodic spikes precede validation declines, and amplifying overlap under norm-matched layer-wise scaling causes accuracy to fall in a dose-dependent way. The proposed method, GCPO, enforces hard bilateral orthogonality by parameterizing each update as projected low-rank factors, which guarantees zero overlap at every layer while leaving a large complementary space for task adaptation. The empirical consequence is that GCPO beats the strongest baseline by 1.02–2.37 points across six model–task settings, attains the best worst-case cross-task retention, and avoids response-length inflation.

Load-bearing premise

The method assumes the top-$k$ singular subspaces of each pretrained weight matrix are the right structure to protect, and that a single rank $k=8$ works across all layers, models, and tasks; if the protected subspace or rank is ill-chosen, the constraint could block useful adaptation or protect the wrong directions.

Editorial extensions

If this is right

  • If GCPO's constraint is the cause of its gains, then any rollout-based policy optimizer can be stabilized by adding hard bilateral orthogonality to the top-$k$ singular subspaces of adapted layers, with the same LoRA-level memory cost.
  • Constraining updates away from the pretrained principal subspaces should transfer to other on-policy objectives beyond GRPO, since the constraint is on the effective update, not on the loss.
  • The excess-overlap diagnostic can act as a cheap monitoring signal during RL post-training: sustained spikes indicate a risky phase and flag checkpoints at risk.
  • GCPO's retention results imply that RL on one domain (e.g., math) does not have to degrade other capabilities (e.g., tool use) if update directions are restricted appropriately.
  • Response-length inflation is not inevitable for rollout RL; a geometric constraint on parameter updates can prevent it without an explicit length penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed protected rank $k=8$ across all layers and tasks is a pragmatic choice; an adaptive, layer-wise selection of $k$ (e.g., based on the spectrum of each weight matrix) would likely improve both the diagnostic and the constraint further.
  • The controlled-intervention result suggests a testable early-stopping rule: pause or shrink updates when excess overlap spikes, before validation accuracy drops.
  • The same bilateral-orthogonality idea could be tested in non-rollout post-training paradigms such as DPO, KTO, or OPD, which the paper explicitly leaves open.
  • If the dose-dependent harm of principal-subspace injection is a general property, then the top-$k$ singular subspaces act as a protective prior inherited from pretraining; one could search for other structural references (e.g., token-embedding subspaces) that play a similar role.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies the geometry of per-step policy updates in rollout-based RL for LLMs. It introduces Principal-Subspace Overlap (PSO), a dimension-corrected statistic measuring the energy of a realized update that lies in the top-k left/right singular subspaces of the pretrained weights. Observational and single-update intervention evidence suggest that transient excess overlap is associated with validation degradation. The authors then propose GCPO, which parameterizes each layer update as a low-rank product sandwiched between orthogonal-complement projectors, enforcing zero overlap with the selected principal subspaces by construction. On Qwen3-8B and GLM4-9B across MATH500, HumanEval+, and ToolAlpaca, GCPO is reported to outperform GRPO, GSPO, DAPO, GMPO, and GRPO-LoRA by 1.02-2.37 points over the strongest baseline, with lower variance, improved cross-task retention, and reduced response-length inflation. The paper includes proofs of the linear-algebra properties, ablations, and code release.

Significance. The conceptual contribution, moving from aggregate geometry to stepwise overlap and making the off-principal preference an exact feasibility constraint, is interesting, and the linear-algebra development in Appendix B is correct and clearly presented. The method is simple, computationally cheap (memory comparable to LoRA), and complementary to KL regularization rather than a replacement for it. If the empirical advantage survives proper hyperparameter validation, this is a useful design principle for stabilizing rollout RL. The paper is honest about the correlational character of the diagnostic and about the limitations of the intervention evidence. The main significance is conditional on whether the k=8 choice is robust or can be selected without touching the reported test benchmarks.

major comments (3)
  1. [Section 5.4, Figure 7] The headline result depends on k=8, but Figure 7 shows GCPO is best only for k=8: on MATH500 with k in {4,16,32,64} the accuracy is 74.75, 75.63, 74.38, and 74.00, all below the strongest baseline DAPO (78.33), and on ToolAlpaca the corresponding values are 63.13, 62.76, 64.73, and 64.78, all below GSPO (66.18). Since k was selected on the same two benchmarks and the same test partitions used to produce Table 1, the reported 1.02-2.37 point margins may reflect test-set selection rather than a general geometric benefit. Please provide a k-selection protocol that does not use the reported test benchmarks (e.g., a dedicated validation subset or a principled default based on layer width), or report the main results for a range of k showing that GCPO stays competitive with or above the strongest baseline.
  2. [Table 1 and Section 5.4] Half of the headline results are for GLM4-9B, yet the k-sensitivity analysis is only reported for Qwen3-8B. The suitability of k=8 for GLM4-9B is assumed, not demonstrated. Please add k-sensitivity or transfer evidence for GLM4-9B, or at minimum justify why the optimal k should be identical across backbones of different widths and training dynamics.
  3. [Section 3.2 and Appendix E] The controlled intervention modifies a single update step and evaluates the immediate effect; it does not demonstrate that avoiding overlap is what produces the long-term benefits claimed for GCPO (stability, length control, entropy decay). The causal link between overlap and training dynamics remains a plausible mechanism rather than a tested one. This does not invalidate the method, but the diagnostic and the method should be presented as linked by hypothesis rather than by demonstrated causation.
minor comments (5)
  1. [Figure 5] The y-axis label is missing; it should indicate that the values are mean response lengths in tokens, and ideally include error bars or per-seed variation.
  2. [Abstract] The phrase 'improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively' is grammatically ambiguous; consider recasting it as two separate margins.
  3. [Section 4.2 and Table 9] Equation (8) uses alpha as the scaling constant, while Algorithm 1 defines s = alpha/r; please clarify the relationship between alpha and s and state which quantity is reported as alpha = 16 in Table 9.
  4. [Related Work (Section 2)] The sentence on aggregate off-principal geometry cites Cai et al. (2026); since one of the paper's own authors appears in that reference, please verify that the characterization is attributed to prior work rather than to the present paper's own claims.
  5. [Table 3] The Random subspace ablation yields 66.47, which is below the unconstrained baseline (72.34); a sentence explaining why a random protected subspace is actively harmful would help readers interpret the ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the excess-overlap diagnostic and GCPO's hard orthogonality constraint are implemented independently, and the headline empirical claims rest on external benchmark measurements rather than on self-citations or fitted inputs.

full rationale

The paper's derivation chain is self-contained. The diagnostic defines O_excess as a dimension-corrected alignment statistic (Eqs. 2-4) and supports it with observational trajectories and controlled interventions; the link to validation degradation is explicitly correlational and empirically demonstrated, not derived from the definition. GCPO imposes bilateral orthogonality by construction through the projected low-rank parameterization (Eq. 8), so the subspace-preservation guarantee (Prop. B.3, Cor. B.4) is an algebraic property of the constraint, not a prediction of accuracy. Performance is then measured on held-out subsets of MATH500, HumanEval+, and ToolAlpaca against external baselines. The protected rank k=8 is selected from a sensitivity scan on two of the same benchmarks (Fig. 7); this is a hyperparameter-selection/correctness risk, but it is not a circular reduction because no equation maps the constraint to the reported accuracy, and neighboring k values are openly shown to underperform. The only self-citation (Cai et al. 2026, with one overlapping author) is background for aggregate off-principal geometry and is independently corroborated by the paper's own Figure 1 numbers, so it is not load-bearing. No uniqueness theorem, ansatz-by-citation, or renamed known result carries the argument.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on a small number of tunable choices: the protected rank k (selected by benchmark performance), and LoRA rank/scale. The mathematical proofs are standard linear algebra. No new entities are introduced.

free parameters (3)
  • Protected rank k = 8
    Chosen via ablation on MATH500 and ToolAlpaca test splits (Section 5.4, Figure 7) where accuracy peaks at k=8; used for all layers and all main experiments. This is a hand-selected value fitted to the evaluation benchmarks.
  • Adaptation rank r = 32
    LoRA rank used for both GRPO-LoRA and GCPO; a standard hyperparameter chosen by hand, not tuned per task.
  • Scaling alpha = 16
    Standard LoRA scaling coefficient; chosen by hand.
assumptions (4)
  • domain assumption The top-k left and right singular subspaces of pretrained weights are a functionally meaningful structure whose preservation avoids harmful updates.
    Stated in Section 3.1 as a structural proxy, not a literal partition of semantic knowledge; the method's value depends on this structural choice.
  • domain assumption A random-direction null is the appropriate baseline for defining excess overlap.
    Proposition B.2 defines Onull using a uniformly random direction; if real updates have a different null distribution, the excess-overlap statistic could misstate alignment.
  • domain assumption Optimizing the rollout objective within the projected feasible subspace does not change the optimization landscape in a way that harms convergence.
    Assumed implicitly in Section 4.2; the experiments test it but there is no theoretical guarantee.
  • standard math Standard linear algebra facts: SVD, Eckart-Young, orthogonal projectors, trace cyclic invariance.
    Used in Lemmas B.1, Propositions B.2-B.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs." pith.science (2026). https://pith.science/paper/BNKJZS3C

@misc{pith2026260811674,
  author       = {Pith},
  title        = {Pith review of: GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNKJZS3C}},
  note         = {Machine review of arXiv:2608.11674}
}
read the original abstract

On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.

Figures

Figures reproduced from arXiv: 2608.11674 by the authors.

Figure 1
Figure 1. Stepwise update overlap and validation performance. The red curve is the 5-step moving average of excess principal [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Controlled intervention on the step-150 GRPO [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training stability and accuracy trajectory on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Policy entropy dynamics on MATH500 with GLM4-9B. GCPO maintains a smooth and gradual entropy decay, avoiding both the severe oscillations of GRPO and the premature entropy collapse in other baselines. while most baselines exhibit rapid entropy decay, suggest￾ing premat…
Figure 5
Figure 5. Figure 5: Response length on MATH500 with Qwen3-8B. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Peak per-GPU training memory on MATH500 with [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Stepwise update overlap and validation performance. The red curve is the 5-step moving average of excess principal [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Additional controlled intervention on the step-125 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 12 canonical work pages

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

Show all 55 references
  1. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  2. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  3. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  4. [12]

    2017 , eprint=

    Proximal Policy Optimization Algorithms , author=. 2017 , eprint=

    2017

  5. [13]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  6. [14]

    2026 , eprint=

    On the Geometry of On-Policy Distillation , author=. 2026 , eprint=

    2026

  7. [15]

    2026 , eprint=

    Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation , author=. 2026 , eprint=

    2026

  8. [16]

    M i L o RA : Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

    Wang, Hanqing and Li, Yixia and Wang, Shuo and Chen, Guanhua and Chen, Yun. M i L o RA : Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational ...

    2025 doi

  9. [17]

    2026 , eprint=

    TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning , author=. 2026 , eprint=

    2026

  10. [18]

    arXiv preprint arXiv:2509.24610 , year=

    OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment , author=. arXiv preprint arXiv:2509.24610 , year=

  11. [19]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  12. [20]

    2026 , eprint=

    LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection , author=. 2026 , eprint=

    2026

  13. [21]

    2026 , eprint=

    Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs , author=. 2026 , eprint=

    2026

  14. [22]

    arXiv preprint arXiv:2502.21321 , year=

    Llm post-training: A deep dive into reasoning large language models , author=. arXiv preprint arXiv:2502.21321 , year=

    arXiv

  15. [23]

    arXiv preprint arXiv:2604.00626 , year=

    A survey of on-policy distillation for large language models , author=. arXiv preprint arXiv:2604.00626 , year=

    arXiv

  16. [24]

    International Conference on Machine Learning , pages=

    Scaling laws for reward model overoptimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  17. [25]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  18. [26]

    arXiv preprint arXiv:2310.03716 , year=

    A long way to go: Investigating length correlations in rlhf , author=. arXiv preprint arXiv:2310.03716 , year=

    arXiv

  19. [27]

    arXiv preprint arXiv:2307.04964 , year=

    Secrets of rlhf in large language models part i: Ppo , author=. arXiv preprint arXiv:2307.04964 , year=

    arXiv

  20. [28]

    International Conference on Learning Representations , volume=

    Confronting reward model overoptimization with constrained RLHF , author=. International Conference on Learning Representations , volume=

  21. [29]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine-tuning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  22. [30]

    G eo RA : Geometry-Aware Low-Rank Adaptation for RLVR

    Zhang, Jiaying and Shi, Lei and Li, Jiguo and Xu, Jun and Gao, Jiuchong and Hao, Jinghua and He, Renqing. G eo RA : Geometry-Aware Low-Rank Adaptation for RLVR. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). ...

    2026 doi

  23. [31]

    arXiv preprint arXiv:2507.20673 , year=

    Geometric-mean policy optimization , author=. arXiv preprint arXiv:2507.20673 , year=

  24. [32]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  25. [33]

    arXiv preprint arXiv:2507.18071 , year=

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    arXiv

  26. [34]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  27. [35]

    2024 , eprint=

    ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools , author=. 2024 , eprint=

    2024

  28. [36]

    arXiv preprint arXiv:2306.05301 , year=

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases , author=. arXiv preprint arXiv:2306.05301 , year=

    arXiv

  29. [37]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  30. [38]

    International Conference on Learning Representations , volume=

    Let's verify step by step , author=. International Conference on Learning Representations , volume=

  31. [39]

    Encyclopedia of measurement and statistics , volume=

    Singular value decomposition (SVD) and generalized singular value decomposition , author=. Encyclopedia of measurement and statistics , volume=. 2007 , publisher=

    2007

  32. [40]

    2024 , eprint=

    GeoLoRA: Geometric integration for parameter efficient fine-tuning , author=. 2024 , eprint=

    2024

  33. [41]

    2026 , eprint=

    Reinforcement Learning via Self-Distillation , author=. 2026 , eprint=

    2026

  34. [42]

    2026 , eprint=

    Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation , author=. 2026 , eprint=

    2026

  35. [43]

    Is Your Code Generated by Chat

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by Chat. 2023 , url =

    2023

  36. [44]

    2024 , eprint=

    Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study , author=. 2024 , eprint=

    2024

  37. [45]

    arXiv preprint arXiv:2402.01306 , year=

    Kto: Model alignment as prospect theoretic optimization , author=. arXiv preprint arXiv:2402.01306 , year=

    arXiv

  38. [46]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Deep reinforcement learning: A survey , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2022 , publisher=

    2022

  39. [47]

    Rethinking Entropy Interventions in RLVR : An Entropy Change Perspective

    Hao, Zhezheng and Wang, Hong and Liu, Haoyang and Luo, Jian and Yu, Jiarui and Dong, Hande and Lin, Qiang and Wang, Can and Chen, Jiawei. Rethinking Entropy Interventions in RLVR : An Entropy Change Perspective. Proceedings of the 64th Annual Meeting of the A ssociation for C ...

    2026 doi

  40. [48]

    arXiv preprint arXiv:2505.22617 , year=

    The entropy mechanism of reinforcement learning for reasoning language models , author=. arXiv preprint arXiv:2505.22617 , year=

    arXiv

  41. [49]

    Linear Algebra and its applications , volume=

    A generalization of the Eckart-Young-Mirsky matrix approximation theorem , author=. Linear Algebra and its applications , volume=. 1987 , publisher=

    1987

  42. [50]

    arXiv preprint arXiv:2603.10535 , year=

    Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning , author=. arXiv preprint arXiv:2603.10535 , year=

  43. [51]

    arXiv preprint arXiv:2510.17776 , year=

    Mapping Post-Training Forgetting in Language Models at Scale , author=. arXiv preprint arXiv:2510.17776 , year=

  44. [52]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Orthogonal subspace learning for language model continual learning , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  45. [53]

    International conference on machine learning , pages=

    Constrained policy optimization , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  46. [54]

    arXiv preprint arXiv:2508.10480 , year=

    Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers , author=. arXiv preprint arXiv:2508.10480 , year=

  47. [55]

    2024 , eprint=

    Identifying Policy Gradient Subspaces , author=. 2024 , eprint=

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.