pith. sign in

arxiv: 2605.21851 · v2 · pith:OFRQ6TRFnew · submitted 2026-05-21 · 💻 cs.LG · cs.AI

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning

Yu Li , Rui Miao , Tian Lan , Zhengling Qi This is my paper

Pith reviewed 2026-05-25 05:45 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords token-level credit assignmentBayesian value recursionoracle signalLLM reasoningpolicy optimizationreinforcement learningGRPO alternatives
0
0 comments X

The pith

OPPO accumulates oracle signals along trajectories to produce closed-form token-level advantages and success probabilities with one extra forward pass.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard RL methods for LLM reasoning assign one advantage to an entire trajectory, which blurs credit at important steps. OPPO starts from the observation that the oracle signal already used in distillation methods is the Bayesian update to the model's belief about eventual success. Accumulating this signal along the generated sequence produces, in closed form, a running estimate of success probability at each position. The resulting advantage is the per-token discrimination signal scaled by a state weight that concentrates credit on pivotal tokens and carries a directional variance-reduction guarantee. The approach requires no learned critic and recovers on-policy distillation as a special case when the student itself scores the evidence.

Core claim

The oracle signal used by prior distillation-style methods for local discrimination is also the natural Bayesian update of the model's belief about eventual success. Accumulating the signal along a trajectory yields, in closed form and at the cost of one extra forward pass, a running estimate of the success probability at every position, together with a token-level advantage that requires no learned value network and no additional rollouts.

What carries the argument

Bayesian value recursion that accumulates the oracle signal to compute position-wise success probability estimates and a modulated per-token advantage.

If this is right

  • The advantage factorizes into the per-token discrimination signal modulated by a state weight that concentrates credit on pivotal tokens.
  • The framework admits a self-oracle estimator that recovers the on-policy distillation reward as a strict special case.
  • A teacher-oracle variant delegates scoring to a stronger frozen model and yields further gains.
  • Empirical gains over GRPO, DAPO, and SDPO reach +6.0 on AMC'23 and +5.2 on AIME'24 and widen with response length.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be combined with process-level supervision to further sharpen credit at intermediate reasoning milestones.
  • Because the recursion is closed-form, it may scale more gracefully than learned value heads when response lengths increase.
  • Replacing the oracle with a learned but cheap proxy signal might preserve most of the benefit while removing dependence on ground-truth verification.

Load-bearing premise

The oracle signal supplied by prior distillation methods is precisely the Bayesian update to the model's belief about eventual success.

What would settle it

Run the same trajectories with and without the accumulation step; if the token-level advantages from accumulation produce no measurable improvement in policy gradient updates on long responses, the central claim is falsified.

Figures

Figures reproduced from arXiv: 2605.21851 by Rui Miao, Tian Lan, Yu Li, Zhengling Qi.

Figure 1
Figure 1. Figure 1: Empirical analysis on 200 MATH-500 trajectories from Qwen3-4B-Instruct. (a) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Without anchoring, reward collapses after step [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
read the original abstract

Reinforcement learning with verifiable rewards has become the standard recipe for improving LLM reasoning, but the dominant algorithm GRPO assigns a single trajectory-level advantage to every token, diluting the signal at pivotal reasoning steps and injecting noise at uninformative ones. Critic-free alternatives derived from on-policy distillation supply per-token signals through oracle-conditioned likelihood ratios, yet apply each signal in isolation from the trajectory-level evidence accumulated up to that position. We propose Oracle-Prompted Policy Optimization (OPPO), which rests on a single observation: the oracle signal used by prior distillation-style methods for local discrimination is also the natural Bayesian update of the model's belief about eventual success. Accumulating the signal along a trajectory yields, in closed form and at the cost of one extra forward pass, a running estimate of the success probability at every position, together with a token-level advantage that requires no learned value network and no additional rollouts. A first-order analysis factorizes the advantage into the per-token discrimination signal used by distillation methods modulated by a state weight that concentrates credit on genuinely pivotal tokens, with a directional variance-reduction guarantee. The framework admits two estimators differing only in which model scores the evidence: a \textit{self-oracle} that reuses the student and recovers the on-policy distillation reward as a strict special case, and a \textit{teacher-oracle} that delegates scoring to a stronger frozen model. On two base LLMs across seven mathematics, science, and code reasoning benchmarks, OPPO improves over GRPO, DAPO, and SDPO by up to $+6.0$ points on AMC'23 and $+5.2$ points on AIME'24, with gains that widen monotonically with response length.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes Oracle-Prompted Policy Optimization (OPPO) for token-level credit assignment in LLM reasoning. It rests on the observation that oracle-conditioned likelihood ratios from prior distillation methods are the natural Bayesian update of the model's belief about eventual success; accumulating these signals along a trajectory yields, in closed form and with one extra forward pass, a running estimate of per-position success probability and a corresponding token-level advantage. The method requires no learned value network and no additional rollouts. It admits self-oracle (recovering on-policy distillation as a special case) and teacher-oracle estimators, provides a first-order factorization of the advantage with a directional variance-reduction guarantee, and reports empirical gains of up to +6.0 on AMC'23 and +5.2 on AIME'24 over GRPO, DAPO, and SDPO across seven benchmarks.

Significance. If the Bayesian equivalence holds rigorously, OPPO supplies a parameter-free, low-overhead route to token-level advantages that concentrates credit on pivotal tokens while recovering known methods as special cases. The closed-form accumulation and first-order analysis constitute a genuine theoretical contribution; the reported benchmark improvements, which widen with response length, indicate practical utility for long-horizon reasoning tasks.

major comments (2)
  1. [§3] §3 (Bayesian-update observation): the claim that the oracle signal is exactly the natural Bayesian update multiplier requires an explicit likelihood model p(oracle_signal_t | success) and p(oracle_signal_t | failure). The manuscript presents the equivalence as a direct observation rather than a derived result; without these likelihoods the subsequent accumulation formula does not necessarily recover the true posterior p(success | history) and the first-order factorization loses its value-estimate interpretation.
  2. [Results section] Experimental protocol (results section): the abstract states benchmark gains but the manuscript provides neither error bars across seeds nor a precise description of the data-generation and evaluation protocol (e.g., number of trajectories per prompt, temperature, length filtering). These omissions make it impossible to assess whether the reported +6.0 / +5.2 point margins are statistically reliable or sensitive to implementation details.
minor comments (2)
  1. [Method] Notation: the distinction between the self-oracle and teacher-oracle estimators should be stated with explicit scoring equations rather than prose descriptions.
  2. [Figure 1] Figure clarity: the trajectory diagram illustrating signal accumulation would benefit from an accompanying equation that shows the recursive update step.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the presentation of the Bayesian foundation and improve the experimental reporting. We address each point below.

read point-by-point responses

  1. Referee: [§3] §3 (Bayesian-update observation): the claim that the oracle signal is exactly the natural Bayesian update multiplier requires an explicit likelihood model p(oracle_signal_t | success) and p(oracle_signal_t | failure). The manuscript presents the equivalence as a direct observation rather than a derived result; without these likelihoods the subsequent accumulation formula does not necessarily recover the true posterior p(success | history) and the first-order factorization loses its value-estimate interpretation.

    Authors: We agree that an explicit likelihood model would make the derivation more rigorous. In the revised manuscript we will expand §3 to include the likelihood model p(oracle_signal_t | success) = 1 and p(oracle_signal_t | failure) = 0 (corresponding to a perfect oracle) together with the resulting closed-form posterior recursion; this recovers the accumulation formula as the exact Bayesian update and preserves the first-order factorization interpretation. revision: yes

  2. Referee: [Results section] Experimental protocol (results section): the abstract states benchmark gains but the manuscript provides neither error bars across seeds nor a precise description of the data-generation and evaluation protocol (e.g., number of trajectories per prompt, temperature, length filtering). These omissions make it impossible to assess whether the reported +6.0 / +5.2 point margins are statistically reliable or sensitive to implementation details.

    Authors: We agree that the current experimental section lacks sufficient detail for assessing statistical reliability. In the revision we will add (i) error bars computed over at least three independent random seeds for all reported metrics and (ii) a precise protocol subsection describing the number of trajectories per prompt, sampling temperature, length filtering criteria, and evaluation procedure. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation rests on explicit Bayesian observation with independent content in teacher-oracle case

full rationale

The paper states its core premise as a single observation equating the distillation oracle signal to a Bayesian update of success probability, then derives the running estimate and advantage factorization from that premise in closed form. The self-oracle estimator is explicitly noted to recover the known on-policy distillation reward as a special case, while the teacher-oracle estimator delegates scoring to a separate frozen model and supplies the reported empirical gains. No equations reduce a claimed prediction to a fitted parameter by construction, no self-citation chain is load-bearing for the uniqueness of the Bayesian equivalence, and the accumulation step is presented as a direct consequence of the stated observation rather than a renaming or self-definition. The derivation is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on one domain assumption about the oracle signal; no free parameters or invented entities are mentioned in the abstract.

axioms (1)
  • domain assumption The oracle signal used by prior distillation-style methods is the natural Bayesian update of the model's belief about eventual success.
    Explicitly identified as the single observation on which the entire method rests.

pith-pipeline@v0.9.0 · 5847 in / 1289 out tokens · 21188 ms · 2026-05-25T05:45:58.239120+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 20 internal anchors

  1. [1]

    Phi-4-reasoning Technical Report

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4- reasoning technical report.arXiv preprint arXiv:2504.21318, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InThe twelfth international conference on learning representations, 2024

    work page 2024

  3. [3]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  4. [5]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [6]

    Process Reinforcement through Implicit Rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [7]

    Bayesian reinforce- ment learning: A survey.Foundations and Trends® in Machine Learning, 8(5-6):359–483, 2015

    Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforce- ment learning: A survey.Foundations and Trends® in Machine Learning, 8(5-6):359–483, 2015

    work page 2015

  7. [8]

    MiniLLM: On-Policy Distillation of Large Language Models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models.arXiv preprint arXiv:2306.08543, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [9]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [10]

    Segment policy optimization: Ef- fective segment-level credit assignment in rl for large language models.arXiv preprint arXiv:2505.23564, 2025

    Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Ef- fective segment-level credit assignment in rl for large language models.arXiv preprint arXiv:2505.23564, 2025

    work page arXiv 2025

  10. [11]

    Treerl: Llm reinforce- ment learning with on-policy tree search

    Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Llm reinforce- ment learning with on-policy tree search. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12355–12369, 2025

    work page 2025

  11. [12]

    Reinforcement Learning via Self-Distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [13]

    OpenAI o1 System Card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [14]

    Learning to reason with mixture of tokens.arXiv preprint arXiv:2509.21482, 2025

    Adit Jain and Brendan Rappazzo. Learning to reason with mixture of tokens.arXiv preprint arXiv:2509.21482, 2025

    work page arXiv 2025

  14. [15]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [16]

    Towards understanding the optimization landscape of grpo and its variants

    Samyak Jain, Ayush Agrawal, and Navin Goyal. Towards understanding the optimization landscape of grpo and its variants. InFirst Workshop on Foundations of Reasoning in Language Models, 2025. 10

    work page 2025

  16. [17]

    An in- troduction to variational methods for graphical models.Machine learning, 37(2):183–233, 1999

    Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An in- troduction to variational methods for graphical models.Machine learning, 37(2):183–233, 1999

    work page 1999

  17. [18]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960

    work page 1960

  18. [19]

    Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. 2024

    work page 2024

  19. [20]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694, 2024

    work page arXiv 2024

  20. [21]

    Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization

    Yu Li, Sizhe Tang, and Tian Lan. Reason in chains, learn in trees: Self-rectification and grafting for multi-turn agent policy optimization.arXiv preprint arXiv:2604.07165, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [22]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [23]

    Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature

    Zheng Liu, Mengjie Liu, Siwei Wen, Mengzhang Cai, Bin Cui, Conghui He, and Wentao Zhang. From uniform to heterogeneous: Tailoring policy optimization to every token’s nature.arXiv preprint arXiv:2509.16591, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [24]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [25]

    On-policy distillation.Thinking Machines Lab: Con- nectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Con- nectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy- distillation

    work page doi:10.64434/tml.20251026 2025

  25. [26]

    Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

    work page 2025

  26. [27]

    Advancing reasoning in large language models: Promising methods and approaches.arXiv preprint arXiv:2502.03671, 2025

    Avinash Patil and Aryan Jadon. Advancing reasoning in large language models: Promising methods and approaches.arXiv preprint arXiv:2502.03671, 2025

    work page arXiv 2025

  27. [28]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst conference on language modeling, 2024

    work page 2024

  28. [29]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [30]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025

    work page 2025

  30. [31]

    A Survey of On-Policy Distillation for Large Language Models

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  31. [32]

    A survey of reasoning with foundation models: Concepts, methodologies, and outlook.ACM Computing Surveys, 57(11):1–43, 2025

    Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models: Concepts, methodologies, and outlook.ACM Computing Surveys, 57(11):1–43, 2025

    work page 2025

  32. [33]

    Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    Richard S Sutton. Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    work page 1988

  33. [34]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [35]

    Self-Distilled RLVR

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  35. [36]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [37]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [38]

    Geometric-mean policy optimization.arXiv preprint arXiv:2507.20673, 2025

    Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, et al. Geometric-mean policy optimization.arXiv preprint arXiv:2507.20673, 2025

    work page arXiv 2025

  38. [39]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025. 12 A Extended Related Work This appendix expands the discussion in Section 1 along three axes that frame OPPO within the broader landscape of credit assignment ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025