Pith. sign in

REVIEW 3 major objections 6 minor 58 references

RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RRPO replaces task-provided verifiers with learned contrastive comparisons against weak positive and negative anchors, preserving group-relative policy optimization for tasks where correctness is not a single yes/no label.

desk verdict RRPO's core idea is worth exploring, but the GSM8K results are confounded by gold-reasoning leakage and the empirical reporting is too thin to support the current claims. read the letter →

arxiv 2607.18470 v1 pith:E35PDHT4 submitted 2026-07-20 cs.LG cs.AI

Yuxin Xiong , Xunyi Jiang , Rohan Surana , Xintong Li , Sheldon Yu , Nikki Lijing Kuang , Ryan A. Rossi , Jingbo Shang
show 3 more authors
Tong Yu Julian McAuley Junda Wu
This is my paper
classification cs.LGcs.AI
keywords Reference-RelativePolicyOptimizationgroup-relativeadvantagescontrastivemetriclearningweaksupervisionRLVRopen-endedgenerationLLMpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that group-relative policy optimization—the recipe behind GRPO-style RL from verifiable rewards—can work without any task ground-truth verifier. Its proposal, RRPO, builds per-input pools of desirable and undesirable reference outputs using weak text prompts, learns an offline metric that scores any new rollout by how much closer it is to the desirable pool than to the undesirable one, and then feeds those scores through the standard within-group standardization. On GSM8K the method matches or beats a verifier-based baseline on most model scales even though it never sees the exact answer during training; on open-ended ELI5 and SAMSum it improves on the base model more reliably than pseudo-target fine-tuning; and it adds a small consistent gain after supervised fine-tuning. A sympathetic reader would care because it points to a cheap, label-free way to reuse the group-relative RL machinery.

What carries the argument

Stratified conditional rollouts: for each input x, a frozen anchor policy samples K rollouts conditioned on a hand-written positive condition c+(x) and K on a negative condition c−(x), forming anchor sets A+(x) and A−(x). A frozen MLP projection head g_ψ maps prompt-completion pairs into a normalized embedding space; a set-contrastive loss L_sc = -log σ(S+ - S-) trains this head offline so that unconditional query rollouts align more with positive than with negative anchors. The reference-relative advantage R_ψ(y;x) = S+ - S- is then standardized within each group of G sampled rollouts to produce the advantages used in the Dr.GRPO-style clipped policy objective.

What would settle it

Run RRPO twice on the same task, once with the intended conditions and once with c+ and c− swapped; the swapped version should clearly underperform Base (or at least fall well below the intended run). If performance stays flat, the reported gains are not caused by the reference-relative comparison.

Watch

Extended reading notes

Core claim

The central claim is that a contrastive score R_psi(y;x) = logsumexp of cosine similarities between a rollout and K positive anchors minus the same against K negative anchors, learned offline and frozen, produces advantages that carry enough signal to drive group-relative policy updates. The paper shows this signal alone, with no task verifier, keeps RRPO competitive with verifier-based Dr.GRPO on GSM8K, outperforms pseudo-target fine-tuning on open-ended tasks, and improves SFT-initialized policies.

Load-bearing premise

The paper's entire advantage signal depends on hand-written weak conditions c+(x) and c−(x) actually separating desirable from undesirable outputs for each task; if those prompts are poorly chosen, the contrastive metric has no signal and RRPO degenerates to noise.

Editorial extensions

If this is right

  • RRPO offers a verifier-free alternative to RLVR: on GSM8K it improves over base on all three backbones and matches or beats Dr.GRPO on several Pass@k metrics without using exact answers as rewards.
  • On open-ended tasks, group-relative contrastive advantages are a more stable weak-supervision signal than single-target pseudo-SFT, which degrades the smallest model on SAMSum.
  • RRPO can be stacked on top of SFT: SFT+RRPO improves or matches SFT on all reported open-ended metrics, with small consistent gains.
  • The advantage signal is reusable across pipeline stages: anchors are cached once and the metric head is frozen, so the policy update loop uses the same relative comparison indefinitely.
  • The optimization structure of RRPO is the same as GRPO; only the reward source changes, meaning existing GRPO infrastructure can adopt it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the anchor-condition quality is the real lever, then RRPO should be sensitive to the wording of c+ and c−; a natural test is swapping or weakening these prompts and measuring the drop, which the paper did not run.
  • The same pipeline may transfer to other generation domains (code explanation, medical Q&A) as long as one can phrase directional conditions; nothing in the method is language-specific beyond prompt writing.
  • Because the metric head is trained on the initial policy's rollouts, RRPO's signal may degrade if the policy drifts far during optimization; KL regularization mitigates this, but an online-updated head could be a stronger extension.
  • The contrastive scoring could double as a lightweight reward model for rejection sampling or best-of-n, without human labels, although the paper only uses it for policy gradients.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces RRPO, a two-stage group-relative policy optimization method intended for settings without task-provided verifiers. In Stage 1, RRPO generates positive and negative anchor rollouts from a frozen anchor policy conditioned on weak task-specific conditions, trains an MLP projection head with a set-contrastive logistic loss (Eqs. 5–7), and caches anchor representations. In Stage 2, the metric head is frozen and the difference of log-sum-exp similarities to positive and negative anchors, S+_ψ(y;x) − S−_ψ(y;x), is used as a scalar score that is standardized within each rollout group (Eq. 8) and fed into a Dr.GRPO-style clipped objective (Eq. 2). Experiments cover GSM8K, ELI5, SAMSum, and post-SFT settings, with the central claim that RRPO is competitive with verifier-based Dr.GRPO on GSM8K and improves over weak-supervision baselines on open-ended tasks.

Significance. If the claims held, RRPO would provide a practical route for extending GRPO-style relative policy optimization to tasks without exact verifiers, while preserving the relative-update structure. The method is clearly specified, the two-stage pipeline is simple, and the idea of using stratified conditional rollouts to construct contrastive anchors is a reasonable and falsifiable proposal. However, the empirical support is currently inadequate in two load-bearing respects: the GSM8K positive condition leaks gold reasoning traces, and no uncertainty estimates are reported for any quantitative result. The paper's own Limitations section admits dependence on condition quality and the absence of random-seed variation, but it does not acknowledge the reference-solution leakage in the GSM8K setup. With those issues addressed, the contribution could be a useful addition to the weak-supervision RL literature.

major comments (3)
  1. [§4.2, Table 4, Eqs. (5)–(8)] The GSM8K positive condition c+(x) in Table 4 is 'CoT-style weak condition that encourages step-by-step mathematical reasoning, such as partial or masked reference reasoning.' Because GSM8K training instances contain gold reasoning traces, this condition is constructed from ground-truth solutions, and the positive anchors A+(x) are sampled conditional on it. The metric head (Eq. 6) is then optimized so that unconditional rollouts align with these reference-derived anchors, and Rψ in Eq. (8) scores policy rollouts by proximity to them. This is ground-truth supervision, not merely weak directional guidance, and it contradicts the Abstract's claim of operating 'without relying on task ground-truth verifiers' and §4.2's statement that RRPO 'does not use ground-truth final answers as policy-optimization supervision.' Dr.GRPO receives only sparse 0/1 final-answer correctness. The Table 1 compa
  2. [Tables 1–3; §4.3] All quantitative claims rest on single point estimates with no reported error bars, standard deviations, or number of seeds. The claim that RRPO is 'more stable' than PSEUDO-SFT and 'improves' over baselines is load-bearing for the paper's empirical contribution. Several reported differences are small — e.g., Qwen3-1.7B GSM8K Pass@1 is 0.768 for RRPO versus 0.765 for Dr.GRPO (Table 1), and Qwen3-1.7B ELI5 Mean ROUGE is 0.118 for RRPO versus 0.117 for Base (Table 2) — so without variance estimates these differences are not distinguishable from noise. The Limitations paragraph acknowledges 'random seeds' as a limitation, but this is central to the paper's headline results, not a minor caveat. Please provide at least three independent runs with means and standard deviations, or explicitly restrict the claims to single-run observations.
  3. [§3.2, §4.4, Algorithm 1] The metric head is trained offline on rollouts from the initialization policy πθ0 and anchors from πanc, then frozen during Stage 2. For the post-SFT experiment, the paper states that 'SFT+RRPO initializes RRPO policy optimization from the SFT checkpoint while keeping the RRPO anchor construction and metric-learning procedure unchanged.' If the metric was trained on base-policy rollouts but then applied to an SFT-initialized policy, the scoring function is being used out of distribution. The paper mentions KL regularization as a safeguard, but provides no measurement of policy drift or of the frozen metric's sensitivity to it. This is especially important because the post-SFT gains are one of the three headline claims. Please clarify whether the metric and anchors are retrained for SFT+RRPO, and if not, include a distribution-shift analysis (e.g., metric score stability across policy che
minor comments (6)
  1. [§1] Typo: 'comlimenting' should be 'complementing.'
  2. [Tables 2–3] Abbreviations R-1, R-2, R-L, Mean-R, BERT are used without definition in the captions; define them or refer explicitly to Section B.3.
  3. [§4.3 / §B.2] Capitalization of the baseline is inconsistent: 'PSEUDO-SFT' in the tables and 'Pseudo-SFT' in the text. Use one form throughout.
  4. [Algorithm 1, line 6] The algorithm says 'update ψ using Σ_m L_sc(x,y_m)' while Eq. (7) averages over inputs and sums over m within an input; clarify whether the per-input sum is normalized by G_metric or by dataset size.
  5. [§B.3] The pass@k estimator is stated but the number of samples n per prompt is not reported. Specify n and the number of prompts used for evaluation.
  6. [Figure 1] Figure 1 is difficult to read at the current resolution; the event types in the right panel are hard to distinguish. A larger or vector version would help.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: RRPO's advantage is a learned score fitted to external anchor sets, not a re-labeling of its own output; the GSM8K reference-condition concern is a validity confound, not a circular derivation.

full rationale

Walking the claimed derivation chain: RRPO constructs anchors via Eq. 4 from weak task conditions, trains the metric head with Eq. 6 on those anchors and query rollouts, defines R_psi = S+ - S- in Section 3.3, and then standardizes these scores into group-relative advantages (Eq. 8) used in a Dr.GRPO-style update (Eq. 2). Each step uses quantities defined earlier in the paper, and no equation takes the paper's conclusion (e.g., 'RRPO beats Dr.GRPO on GSM8K') as an input. R_psi is a fitted function, but it is fitted to anchor sets and rollouts, not to the reported benchmark numbers, so the reported results are empirical outcomes of an optimization loop rather than identities. The heavy self-citations (Surana et al. 2026a,b,c; Mundada et al. 2026; Yu et al. 2026b) appear in related-work context and are not used to justify a uniqueness theorem, a forced ansatz, or a derivation step; they are not load-bearing. The more serious issue is a confound: Table 4 says the GSM8K positive condition c+ is 'a CoT-style weak condition that encourages step-by-step mathematical reasoning, such as partial or masked reference reasoning,' so the positive anchors can encode gold reference reasoning. This weakens the 'without relying on task ground-truth verifiers' claim and makes the comparison to Dr.GRPO unfair, since Dr.GRPO only receives sparse 0/1 final-answer correctness. However, this is a data-supervision/leakage concern, not a circular definition: the method does not define its target result in terms of the fitted score by construction, and the same mechanism is evaluated on open-ended tasks where no such reference reasoning is used. The Limitations section also acknowledges dependence on condition quality, consistent with a fragile but non-circular method.

Assumptions & free parameters 6 free parameters · 2 assumptions · 0 invented entities

The central claim rests on hand-crafted condition prompts and hyperparameters rather than a derived invariant. These choices are not independently benchmarked, and the frozen metric head assumption is a fragile point. No new physical or ontological entities are introduced.

free parameters (6)
  • metric temperature tau = 0.1
    Chosen by hand for all tasks; affects sharpness of similarity aggregation in Eq. 5-6.
  • projection dimension = 512
    Chosen by hand; dimension of the metric head output.
  • anchor count K = 4
    Number of positive and negative anchors per stratum; chosen by hand.
  • rollout group size G = 8
    Number of policy rollouts per input during optimization; standard GRPO size chosen by hand.
  • KL coefficient beta and clip epsilon = not specified
    Regularization and clipping coefficients are not reported, but are needed to define the policy objective (Eq. 2).
  • Weak condition prompts c+ and c- = dataset-specific (e.g., 'partial or masked reference reasoning' for GSM8K)
    Hand-designed per dataset; these are the most consequential parameters because they define anchor quality and thus the entire reward signal.
assumptions (2)
  • domain assumption Hand-crafted weak conditions generate anchors that separate desirable from undesirable behavior.
    The entire method depends on this; stated in Sec. 3.1 and Limitations, but no independent evidence that the chosen prompts are generally effective.
  • domain assumption The frozen metric head remains valid as the policy changes during optimization.
    Sec. 3.2 relies on KL regularization to keep the policy close to the metric-training distribution; acknowledged as a limitation in Sec. 7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts." pith.science (2026). https://pith.science/paper/E35PDHT4

@misc{pith2026260718470,
  author       = {Pith},
  title        = {Pith review of: RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E35PDHT4}},
  note         = {Machine review of arXiv:2607.18470}
}
read the original abstract

Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals. However, extending group-relative optimization beyond verifiable settings is challenging because success in many tasks is not captured by a single correctness criterion. We propose \textbf{Reference-Relative Policy Optimization (RRPO)}, which generalizes GRPO by replacing direct correctness-based advantage construction with reference-relative contrastive comparisons. RRPO first uses \emph{stratified conditional rollouts} to construct positive and negative anchor sets, and then trains a metric projection head with a set-contrastive objective to compare candidate rollouts against these anchors. The resulting alignment scores directly define contrastive advantages: during policy optimization, the projection head is frozen, and the scores are centered within each rollout group in a standard group-relative objective. We evaluate RRPO using anchor-based contrastive advantages throughout policy optimization, without relying on task ground-truth verifiers. Across verifiable reasoning, open-ended generation, and post-SFT settings, RRPO remains competitive with verifier-based optimization, improves over weakly supervised baselines, and provides additional gains after supervised fine-tuning.

Figures

Figures reproduced from arXiv: 2607.18470 by the authors.

Figure 1
Figure 1. Stratified conditional rollouts in RRPO. For each input, weak positive and negative conditions generate [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Loss function calculation during training process [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Learning curves during RRPO training. Evalu [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 18 linked inside Pith

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  2. [2]

    Large-scale kernel machines , volume=

    Scaling learning algorithms towards AI , author=. Large-scale kernel machines , volume=

  3. [3]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  4. [4]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  5. [5]

    ICML , year=

    Eligibility traces for off-policy policy evaluation , author=. ICML , year=

  6. [6]

    NeurIPS , year=

    Safe and efficient off-policy reinforcement learning , author=. NeurIPS , year=

  7. [7]

    NeurIPS , year=

    DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections , author=. NeurIPS , year=

  8. [8]

    NeurIPS , year=

    Actor-critic algorithms , author=. NeurIPS , year=

Show all 58 references
  1. [9]

    ICLR , year=

    High-dimensional continuous control using generalized advantage estimation , author=. ICLR , year=

  2. [10]

    ICML , year=

    Policy invariance under reward transformations: Theory and application to reward shaping , author=. ICML , year=

  3. [11]

    Proceedings of the 2nd Workshop on New Frontiers in Summarization , pages=

    SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization , author=. Proceedings of the 2nd Workshop on New Frontiers in Summarization , pages=

  4. [12]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  5. [13]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  6. [14]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  7. [15]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  8. [16]

    arXiv preprint arXiv:2509.23067 , year=

    Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks , author=. arXiv preprint arXiv:2509.23067 , year=

  9. [17]

    arXiv preprint arXiv:2602.11549 , year=

    Native Reasoning Models: Training Language Models to Reason on Unverifiable Data , author=. arXiv preprint arXiv:2602.11549 , year=

  10. [18]

    arXiv preprint arXiv:2602.03876 , year=

    GOPO: Policy Optimization using Ranked Rewards , author=. arXiv preprint arXiv:2602.03876 , year=

  11. [19]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  12. [20]

    Advances in Neural Information Processing Systems , volume=

    Beyond verifiable rewards: Scaling reinforcement learning in language models to unverifiable data , author=. Advances in Neural Information Processing Systems , volume=

  13. [21]

    arXiv preprint arXiv:2601.08427 , year=

    Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering , author=. arXiv preprint arXiv:2601.08427 , year=

  14. [22]

    arXiv preprint arXiv:2411.15124 , year=

    Tulu 3: Pushing frontiers in open language model post-training , author=. arXiv preprint arXiv:2411.15124 , year=

    arXiv

  15. [23]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  16. [24]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  17. [25]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  18. [26]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  19. [27]

    arXiv preprint arXiv:2402.01306 , year=

    Kto: Model alignment as prospect theoretic optimization , author=. arXiv preprint arXiv:2402.01306 , year=

    arXiv

  20. [28]

    arXiv preprint arXiv:2203.11171 , year=

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    arXiv

  21. [29]

    Advances in Neural Information Processing Systems , volume=

    Star: Bootstrapping reasoning with reasoning , author=. Advances in Neural Information Processing Systems , volume=

  22. [30]

    arXiv preprint arXiv:2308.08998 , year=

    Reinforced self-training (rest) for language modeling , author=. arXiv preprint arXiv:2308.08998 , year=

    arXiv

  23. [31]

    5-math technical report: Toward mathematical expert model via self-improvement , author=

    Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement , author=. arXiv preprint arXiv:2409.12122 , year=

    arXiv

  24. [32]

    arXiv preprint arXiv:2407.10759 , year=

    Qwen2-audio technical report , author=. arXiv preprint arXiv:2407.10759 , year=

    arXiv

  25. [33]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  26. [34]

    arXiv preprint arXiv:2401.10020 , year=

    Self-rewarding language models , author=. arXiv preprint arXiv:2401.10020 , year=

    arXiv

  27. [35]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  28. [36]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    ELI5: Long form question answering , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  29. [37]

    arXiv preprint arXiv:1807.03748 , year=

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    arXiv

  30. [38]

    Advances in neural information processing systems , volume=

    Supervised contrastive learning , author=. Advances in neural information processing systems , volume=

  31. [39]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  32. [40]

    Sentence-bert: Sentence embeddings using siamese bert-networks , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  33. [41]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Facenet: A unified embedding for face recognition and clustering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  34. [42]

    Advances in neural information processing systems , volume=

    Hindsight experience replay , author=. Advances in neural information processing systems , volume=

  35. [43]

    IJCAI , volume=

    Learning to achieve goals , author=. IJCAI , volume=

  36. [44]

    arXiv preprint arXiv:2503.20783 , year=

    Understanding r1-zero-like training: A critical perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    arXiv

  37. [45]

    arXiv preprint arXiv:2503.06639 , year=

    Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification , author=. arXiv preprint arXiv:2503.06639 , year=

  38. [46]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  39. [47]

    arXiv preprint arXiv:2509.21154 , year=

    Grpo is secretly a process reward model , author=. arXiv preprint arXiv:2509.21154 , year=

    arXiv

  40. [48]

    Advances in Neural Information Processing Systems , volume=

    Cppo: Accelerating the training of group relative policy optimization-based reasoning models , author=. Advances in Neural Information Processing Systems , volume=

  41. [49]

    2026 , eprint=

    WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning , author=. 2026 , eprint=

    2026

  42. [50]

    2026 , eprint=

    MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization , author=. 2026 , eprint=

    2026

  43. [51]

    arXiv preprint arXiv:2605.12995 , year=

    F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking , author=. arXiv preprint arXiv:2605.12995 , year=

    arXiv

  44. [52]

    arXiv preprint arXiv:2605.11169 , year=

    OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents , author=. arXiv preprint arXiv:2605.11169 , year=

    arXiv

  45. [53]

    Advances in Neural Information Processing Systems , volume=

    Listwise preference diffusion optimization for user behavior trajectories prediction , author=. Advances in Neural Information Processing Systems , volume=

  46. [54]

    arXiv preprint arXiv:2605.02913 , year=

    Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning , author=. arXiv preprint arXiv:2605.02913 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:2601.05600 , year=

    SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes , author=. arXiv preprint arXiv:2601.05600 , year=

  48. [56]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Image Difference Captioning via Adversarial Preference Optimization , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  49. [57]

    The Fourteenth International Conference on Learning Representations , year=

    Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization , author=. The Fourteenth International Conference on Learning Representations , year=

  50. [58]

    In-context Ranking Preference Optimization , author=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.