Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Routing each reasoning trace by correctness — RLSD-style credit refinement on successes, reverse-KL correction on failures — outperforms RLVR, OPSD, and RLSD on three logic-puzzle benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:47 UTC pith:LP2MQEYN

load-bearing objection Clean routing idea and useful ablations, but the failure-branch teacher target is unmeasured and could be the weakest link; single-run evidence also undercuts the 'consistently outperforms' claim. the 3 major comments →

arxiv 2607.18955 v1 pith:LP2MQEYN submitted 2026-07-21 cs.LG cs.CL

H²SD: Hybrid Hindsight Self-Distillation

Qiye Cai , Yichuan Ma , Linyang Li , Peiji Li , Yongkang Chen , Qipeng Guo , Yicheng Zou , Tao Gui
show 2 more authors
Xiaocheng Feng Bing Qin
This is my paper
classification cs.LG cs.CL
keywords H2SDreinforcement learning with verifiable rewardson-policy self-distillationtoken-level credit assignmentreverse KL divergenceprivileged hintslogical reasoning benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that reinforcement learning with verifiable rewards (RLVR) for LLM reasoning is limited by scalar outcome rewards that cannot say which token helped or hurt. It proposes H2SD, a hybrid hindsight self-distillation objective that lets the same model act as its own teacher: for a trajectory verified correct, the teacher receives the student's response plus a rephrasing instruction and only rescales the reward-driven token updates; for a failed trajectory, the teacher receives a natural-language hint with key reasoning steps and a verified answer, and the student minimizes reverse KL divergence to that hint-conditioned distribution. The claim is that this correctness-conditioned split combines precise credit assignment with explicit corrective direction, and experiments across Sudoku, Calcudoku, and Arrow Maze show it consistently beats pure RLVR, on-policy self-distillation, and RLSD-style magnitude modulation while using fewer generated tokens. A sympathetic reader would care because it suggests dense supervision can come from the model itself plus offline hints, removing the need for a strong teacher at inference.

Core claim

The central claim is equation (16): the objective is L_H2SD = E[ m L_RLSD + gamma(1-m) L_RKL ], with m marking trajectory correctness. Correct trajectories use RLSD-style magnitude modulation: the teacher is the same model conditioned on the verified-correct response with a rephrasing instruction, and its token probabilities only rescale reward-determined update magnitudes. Failed trajectories use reverse-KL distillation from a teacher conditioned on a hint containing key reasoning steps and a verified answer, with gradients stopped through the teacher. The argument: success gives a reliable direction needing finer credit; failure needs explicit corrective direction that magnitude modulation

What carries the argument

The mechanism is outcome-conditioned routing over a same-model self-teacher. The router m = 1[R(x,y)=1] sends each sample to one branch. Success branch: the teacher sees the verified-correct response with a rephrasing instruction; token weights w_t = exp(sign(A) * (log P_T(y_t) - log P_S(y_t))), clipped, rescale each token's advantage while preserving the reward direction. Failure branch: reverse KL D_KL(student || stopgrad(teacher_hint)) pulls the student toward the model conditioned on a hint h = H(x) that contains key reasoning steps and a verified answer; the hint comes from a stronger LLM but the teacher is still the current model. Rephrasing on correct trajectories avoids over-constrai

Load-bearing premise

The method rests on the assumption that when a hint contains the right reasoning steps, the model's distribution conditioned on that hint points toward a better answer for failed attempts; if the hint is wrong, the failure-correction branch teaches the model the wrong answer.

What would settle it

Train H2SD on a verifiable task while deliberately corrupting a known fraction of hints (e.g., swapping in wrong answers or omitted steps) and watch the failure branch's reverse-KL loss and pass@1. If the corrupted-hint runs degrade as fast as or faster than a pure RLVR baseline, the failure branch is merely amplifying hint errors rather than correcting the student; conversely, if H2SD resists corrupted hints, the teacher distribution is self-correcting. A simpler check: compute the hint-conditioned teacher's accuracy on failed trajectories before training; if it does not exceed the student's

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, dense token-level supervision for RLVR no longer requires a teacher model at inference; a stronger model is used only once to write hints.
  • The same correctness-conditioned split can be applied to any task with a verifiable reward, provided the failure branch has a hint that is at least as corrective as the student's unconditional distribution.
  • Because OPSD, RLSD+hint, and H2SD all use identical hints, the paper establishes that how privileged context is converted into a learning signal matters more than the context itself.
  • Success-branch rephrasing is beneficial under magnitude modulation but harmful under direct reverse-KL matching, suggesting that correct trajectories should be refined, not re-distilled.
  • The method generalizes to unseen puzzle sizes and maintains higher actor entropy, indicating the gains do not come from mode collapse or longer responses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An obvious extension is to math and code reasoning, where verifiers are cheaper; the main new risk is hint quality, so a natural test is to corrupt hints and measure whether the reverse-KL branch pulls the policy toward wrong answers.
  • Since sibling solutions (verified correct rollouts of the same problem) already give most of the Sudoku gain, the hint generator may be replaceable by self-generated verified rollouts, which would remove the external strong model entirely.
  • The reversed-routing collapse suggests the objective could serve as a diagnostic: if router accuracy is low, the two branches interfere; monitoring actor entropy may provide an early stopping signal before divergence.
  • The reverse-KL failure branch is a form of conservative policy update on bad trajectories; combining it with a trust region or adaptive gamma could prevent overcorrection when hints are sparse.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes H2SD (Hybrid Hindsight Self-Distillation), an objective that routes on-policy trajectories by correctness. Successful trajectories are trained with an RLSD-style magnitude-modulated policy-gradient update, where a teacher prompted to rephrase the correct response provides token-level weights. Failed trajectories are trained by minimizing a reverse-KL divergence from the student to the same model conditioned on an externally generated hint containing key reasoning steps and a verifier-confirmed answer. The combined objective is Eq. (16). Experiments on Sudoku, Calcudoku, and Arrow Maze report large improvements over GRPO, OPSD, RLSD, SDPO, SRPO, and RLSD+hint, with ablations on routing, privileged context, and rephrasing.

Significance. If the empirical results are robust, H2SD offers a simple and plausible resolution of the trade-off between distribution-level correction (OPSD) and stable reward-grounded credit assignment (RLSD). The paper includes useful controlled comparisons: OPSD and H2SD share identical hints, and RLSD+hint isolates the effect of the privileged context. The routing ablation (Table 2) and the privileged-context study (Table 3) are informative. However, the main empirical claim rests on single-run point estimates, and the core mechanism of the failed-trajectory branch is not directly verified. These are fixable with additional analysis and experiments.

major comments (3)
  1. [§4.2, Eq. (15)] The failed-trajectory objective minimizes reverse KL between the student π_θ(·|x, y_<t) and the stop-grad teacher sg[π_θ(·|x, h, y_<t)] on trajectories where y_<t is the student's own erroneous prefix. The paper asserts this provides an 'explicit correction direction,' but it never measures whether the hint-conditioned teacher is actually a better next-token predictor on failed prefixes. In constraint puzzles, a wrong prefix can make the puzzle unsatisfiable, and the teacher's distribution may be arbitrary or may reinforce the error. The claimed advantage over RLSD—which only modulates magnitude—therefore rests on an unverified assumption. Please report (i) teacher vs. student next-token accuracy on failed prefixes, (ii) the probability the teacher assigns to the verifier-correct next token compared with the student's unconditional probability, and (iii) a corrupted-hint control or an an
  2. [Table 1 / §5.3] All reported numbers are single-run point estimates with no error bars or multiple seeds. The abstract and Section 5.3 claim H2SD 'consistently outperforms' the baselines, but Table 1 itself contains a counterexample: on Calcudoku 6×6, H2SD (14.67) is below RLSD (15.33). With no variance estimates, the large Sudoku advantage cannot be distinguished from run-to-run variation. Please provide mean±std over at least three seeds (or confidence intervals) for the main table and the routing ablations, and soften the wording to match the evidence.
  3. [§5.2] The implementation description states that, to reduce memory, exact probabilities are retained only for the student's top-100 tokens and the remaining mass is merged into a tail bucket, for OPSD and related distillation branches. It is not stated explicitly whether the H2SD reverse-KL term in Eq. (15) uses the same truncation, nor is any sensitivity analysis given. Reverse KL is sensitive to the support of the student distribution, especially on failed trajectories where the student may be uncertain; truncating to the top-100 tokens could misestimate the gradient. Please state clearly which objectives use the truncation and report a sensitivity check over the truncation size (e.g., top-50/top-200) or justify why the tail can be neglected.
minor comments (5)
  1. [Eq. (16)] The hyperparameter γ controls the strength of the corrective distillation term, but no sensitivity analysis is reported for γ (or for λ and ε_w from RLSD). A small sweep would help the reader assess robustness.
  2. [§4.1] The hint generator is described as a 'stronger LLM' and some details are given, but the prompt used to elicit the hint and the procedure for verifying the hint's final answer are not specified. Please add enough detail for reproducibility.
  3. [Table 1] The table is visually dense; the column groups (Sudoku, Calcudoku, Arrow Maze) are not labeled in the table body. Adding explicit group headers would improve readability.
  4. [Figures 3–5] Axis labels are incomplete: specify whether the x-axis in Figures 4 and 5 denotes rollout steps or environment steps, and clarify the units in Figure 3. The entropy figures would also benefit from explicit captions describing the y-axis.
  5. [§5.3] The statement 'consistently ranks first across all Arrow Maze sizes' is supported by Table 1, but the abstract's 'consistently outperforms' is not, as noted above. Please adjust the abstract.

Circularity Check

0 steps flagged

No circularity: H2SD's routing is driven by an external verifier and its teacher targets are conditioned on an externally generated, verified hint; no equation reduces to its own input.

full rationale

Walking the derivation chain (Eqs. 1-16), the learning signal is externally grounded at every load-bearing step. GRPO in Eq. (4) uses a verifier-provided scalar reward. The routing variable in Eq. (16), m = 1[R(x,y)=1], is computed from that external reward, not from the student's own distribution or from a parameter fitted to the target result. On successful trajectories, the policy-gradient direction determined by the reward is preserved (Eqs. 3 and 12), and the teacher only modulates token-level magnitudes; the teacher does not supply the optimization direction. On failed trajectories, the target in Eq. (15) is πθ(·|x, h, y<t), where h = H(x) is generated by an external stronger model (Kimi-K2.6) with a verifier-confirmed final answer, as stated in Secs. 4.1 and 5.2. The teacher distribution is therefore not defined in terms of the student's success outcome or in terms of the quantities being predicted. The identical-hint controls (OPSD and RLSD+hint in Table 1) and the routing ablations (Table 2) further show that the claimed gains are not forced by construction. The skeptical concern that the hint-conditioned teacher may be unreliable on invalid student prefixes is a correctness/robustness issue, not circularity: a bad teacher would degrade performance, but it would not make the objective definitionally equivalent to its input. There are no load-bearing self-citations; the cited OPSD/RLSD/SRPO works are external prior papers, and no fitted parameter is renamed as a prediction. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on the verifier, the hint generator, and the assumption that hint-conditioned same-model distributions are useful corrective targets. None of these are fitted to the target result; they are external inputs. The paper does not audit hint quality or the effect of top-100 token truncation, so these remain load-bearing assumptions.

free parameters (4)
  • gamma (corrective distillation weight)
    Eq. 16 weights the reverse-KL term for failed trajectories; value is not reported and must be tuned per benchmark.
  • lambda (RLSD magnitude-modulation strength)
    Eq. 12 controls how strongly teacher probabilities rescale token advantages; inherited from RLSD but not reported for these experiments.
  • clipping thresholds (epsilon, epsilon_w)
    GRPO and RLSD use clipping constants for the surrogate objective and token weights; values are not reported.
  • top-100 token truncation and tail bucket
    Introduced in Sec. 5.2 to reduce memory for OPSD distillation; a modeling choice that may affect baseline comparisons and is not ablated.
axioms (5)
  • domain assumption Verifier rewards are correct and binary; a trajectory reward of 1 genuinely means a valid solution.
    RLVR framework in Sec. 3.1 and Sec. 4.2 treats R=1 as ground truth for routing, so verifier errors propagate into both credit assignment and reverse-KL routing.
  • ad hoc to paper The external hint generator (Kimi-K2.6) produces hints with key reasoning steps and a verified final answer.
    Sec. 4.1 defines h = H(x) but gives no quality control beyond verifier confirmation of the final answer; intermediate steps are unverified and assumed useful.
  • domain assumption A model conditioned on the hint produces a teacher distribution better than the student's unconditional distribution for failed trajectories.
    This is the premise of reverse-KL corrective guidance in Sec. 4.2; if false, failed-trajectory distillation would reinforce wrong reasoning.
  • ad hoc to paper Top-100 token approximation of teacher distributions is sufficient for OPSD-style distillation.
    Introduced in Sec. 5.2 to reduce memory overhead; no sensitivity analysis is provided.
  • domain assumption Sudoku, Calcudoku, and Arrow Maze pass@1 are representative proxies for reasoning ability.
    The empirical claim is scoped to these benchmarks, and generalization to math or code is not established.

pith-pipeline@v1.3.0-alltime-deepseek · 10910 in / 12696 out tokens · 120934 ms · 2026-08-01T13:47:15.473290+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce $\mathrm{H}^{2}\mathrm{SD}$, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H$^2$SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 15 linked inside Pith

  1. [1]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Confer- ence on Learning Representations, volume 2024, pages 21246–21263, 2024. 2

    2024

  2. [2]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pages 32694–32717, 2024. 2

    2024

  3. [3]

    Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 1

    Pith/arXiv arXiv 2025

  4. [4]

    Self-distillation zero: Self-revision turns binary rewards into dense supervision

    Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and San- jeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision. arXiv preprint arXiv:2604.12002, 2026. 2

    Pith/arXiv arXiv 2026

  5. [5]

    Cepo: Rlvr self-distillation using contrastive evi- dence policy optimization.arXiv preprint arXiv:2605.19436, 2026

    Ahmed Heakl, Abdelrahman M Shaker, Youssef Mohamed, Rania Elbadry, Omar Fetouh, Fa- had Shahbaz Khan, and Salman Khan. Cepo: Rlvr self-distillation using contrastive evi- dence policy optimization.arXiv preprint arXiv:2605.19436, 2026. 2

    Pith/arXiv arXiv 2026

  6. [6]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buen- ing, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026. 1, 2

    Pith/arXiv arXiv 2026

  7. [7]

    Respecting self-uncertainty in on-policy self-distillation for efficient llm rea- soning.arXiv preprint arXiv:2605.13255, 2026

    Junlong Ke, Zichen Wen, Weijia Li, Conghui He, and Linfeng Zhang. Respecting self-uncertainty in on-policy self-distillation for efficient llm rea- soning.arXiv preprint arXiv:2605.13255, 2026. 2

    Pith/arXiv arXiv 2026

  8. [8]

    Uni- fying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Uni- fying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026. 1, 2

    arXiv 2026

  9. [9]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025. 1

    Pith/arXiv arXiv 2025

  10. [10]

    On-policy distillation.Think- ing Machines Lab: Connectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Think- ing Machines Lab: Connectionism, 2025. https://thinkingmachines.ai/blog/on-policy- distillation. 1, 2

    2025

  11. [11]

    Toolrl: Reward is all tool learning needs.Advances in Neural Information Processing Systems, 38:105523–105553, 2026

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.Advances in Neural Information Processing Systems, 38:105523–105553, 2026. 1

    2026

  12. [12]

    Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 1, 2

    Pith/arXiv arXiv 2024

  13. [13]

    Self-distillation enables con- tinuallearning.arXiv preprint arXiv:2601.19897,

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables con- tinuallearning.arXiv preprint arXiv:2601.19897,

    Pith/arXiv arXiv

  14. [14]

    Gates: Self-distillation under privileged con- text with consensus gating.arXiv preprint arXiv:2602.20574, 2026

    Alex Stein, Furong Huang, and Tom Goldstein. Gates: Self-distillation under privileged con- text with consensus gating.arXiv preprint arXiv:2602.20574, 2026. 2

    arXiv 2026

  15. [15]

    Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026. 1

    Pith/arXiv arXiv 2026

  16. [16]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 1

    Pith/arXiv arXiv 2025

  17. [17]

    Self- distilled rlvr.arXiv preprint arXiv:2604.03128,

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weip- ing Wang, Jiaqi Wang, and Nan Duan. Self- distilled rlvr.arXiv preprint arXiv:2604.03128,

    Pith/arXiv arXiv

  18. [18]

    On-policy context distilla- tion for language models.arXiv preprint arXiv:2602.12275, 2026

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distilla- tion for language models.arXiv preprint arXiv:2602.12275, 2026. 2

    Pith/arXiv arXiv 2026

  19. [19]

    Dapo: An open-source llm reinforcement learn- ing system at scale.Advances in Neural Infor- mation Processing Systems, 38:113222–113244,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learn- ing system at scale.Advances in Neural Infor- mation Processing Systems, 38:113222–113244,

  20. [20]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. 1

    Pith/arXiv arXiv 2026

  21. [21]

    Self-distilled reasoner: On-policy self- distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self- distillation for large language models.arXiv preprint arXiv:2601.18734, 2026. 1, 2

    Pith/arXiv arXiv 2026

  22. [22]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong- Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025. 2 11

    Pith/arXiv arXiv 2025

  23. [2026]

    2 10 H2SD: Hybrid Hindsight Self-Distillation