Pith. sign in

REVIEW 4 major objections 5 minor 69 references

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BCSD rescales the GRPO advantage using token-level gaps from a Meta-Skill-augmented view and a pruned skill view, and reports the strongest overall success rates on ALFWorld and WebShop across three model scales.

desk verdict A real incremental method for skill-based agent RL with a consistently positive trend, but the direction-preservation claim needs an explicit stop-gradient statement and multi-seed results before I trust it. read the letter →

arxiv 2608.09555 v1 pith:BTKCVRI3 submitted 2026-08-10 cs.AI

Tianjun Pan , Yuan Li , Hongda Wang , Linbo Jin , Mengfei Song , Lei Gao , Qiming Shi , Shaokang Fu
show 3 more authors
Jiarong Zhao Chengyu Wang Chengfu Huo
This is my paper · ORCID
classification cs.AI
keywords skill-basedLLMagentsself-distillationreinforcementlearningGRPOexternalskillutilizationmeta-skilltoken-levelcreditassignmentadvantagerescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets a gap in training LLM agents that receive external natural-language skills: agents often ignore or misuse the guidance. It proposes BCSD, a training method that pairs reinforcement learning with self-distillation from two complementary skill-context views, so the agent learns to apply the provided skills rather than memorize task knowledge. The augmented view adds a policy-dependent Meta-Skill summarizing how the skill should be used, while the pruned view condenses general guidance to focus on the task-specific skill. Token-level gaps between these views and the original context rescale the GRPO advantage without flipping its sign, and the strength of that rescaling decays over training. On ALFWorld and WebShop, BCSD reports the strongest overall success rates across three model scales, and removing the task-specific skill at inference hurts its performance more than it hurts a standard GRPO policy, which the paper reads as evidence of genuine skill utilization.

What carries the argument

The machinery is the bidirectional context pair and the rescaling weight built from it. Given base context $C_0=(Q,G,S_c)$, BCSD constructs $C_1=(Q,G,S_c,M_c)$ by adding a Meta-Skill $M_c$ extracted from successful and failed rollouts, and $C_2=(Q,G_p,S_c)$ by pruning the general skill $G$. For each realized token, it computes the log-probability gaps $\Delta^{\text{aug}}_{i,t}$ and $\Delta^{\text{pru}}_{i,t}$ between these views and $C_0$, combines them as $\Delta_{i,t} = \alpha\,\Delta^{\text{aug}}_{i,t} + (1-\alpha)\,\Delta^{\text{pru}}_{i,t}$, and forms a clipped multiplicative weight $w_{i,t} = \operatorname{clip}\big(\exp(\operatorname{sign}(A^{\text{GRPO}}_i)\Delta_{i,t}),\, 1-c,\, 1+c\big)$. The final token advantage is $A^{\text{BCSD}}_{i,t} = A^{\text{GRPO}}_i\,\big[(1-\lambda_n) + \lambda_n w_{i,t}\big]$, with $\lambda_n$ decaying to zero. This lets the token-level context signals modulate the magnitude of the advantage while never reversing its sign.

What would settle it

Train a control variant of BCSD in which, for each trajectory, the token-level combined gaps $\Delta_{i,t}$ are replaced by randomly permuted gaps sampled from the same set of magnitudes (so only the token-to-token assignment is destroyed), keeping all other settings identical; if this control matches BCSD's success rate, the specific gap content is not what drives the improvement. A more direct check is to substitute an unrelated text for the Meta-Skill in the augmented view: genuine skill-utilization guidance should collapse BCSD's advantage toward the single-view baseline.

Watch

Extended reading notes

Core claim

The central claim is that evaluating each trajectory from two opposite directions of skill-context editing yields more reliable token-level credit for skill utilization than any single privileged context. From a base context of query, general skill, and task-specific skill, the augmented view appends a Meta-Skill distilled from the policy's own successes and failures, and the pruned view replaces the general skill with a condensed version. The same policy re-scores the realized tokens under both views; the two log-probability gaps are combined with a weight alpha and used to rescale the group-relative GRPO advantage token by token, with the coefficient linearly decayed to zero over training. The paper argues this preserves the update direction fixed by trajectory rewards while injecting dense guidance on whether each token follows the skills. Experiments report consistent gains over skill-conditioned RL and single-view self-distillation baselines on ALFWorld and WebShop at three model scales from 1.7B to 7B parameters.

Load-bearing premise

The load-bearing premise is that the token-level log-probability gaps between the augmented, pruned, and base contexts carry genuine information about how well each token follows the provided skills; if those gaps are mostly noise, the rescaling would randomize update magnitudes and the reported gains would not be stable.

Editorial extensions

If this is right

  • Skill-based agents can be trained to use external skills more effectively without internalizing them, keeping skills explicit, editable, and transferable at inference time.
  • The advantage-rescaling scheme injects dense token-level credit while preserving the sign of the trajectory reward, so it can be layered onto any group-relative RL objective.
  • Removing either the Meta-Skill view or the pruned context view degrades performance, and a decaying rescaling coefficient is required to avoid over-relying on privileged guidance late in training.
  • Because the augmented view is refreshed as the policy evolves, the method tracks the agent's changing skill-utilization patterns during reinforcement learning.
  • On the two long-horizon benchmarks tested, BCSD achieves the best overall success rates across three model scales, with the largest gains on WebShop success rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-50% sign agreement between the two token-level gaps suggests the rescaling may act partly as a variance-reducing regularizer rather than a pure credit signal; a direct test would be to compare against GRPO with a token-level entropy or KL bonus of matched strength.
  • The same bidirectional context trick could transfer beyond skills to any externally supplied natural-language constraints, such as safety rules or instruction hierarchies, by comparing full-context and context-reduced views.
  • The Meta-Skill extraction might be simplifiable: if a static summary from a handful of successful and failed trajectories suffices, the periodic refresh loop could be dropped, lowering the method's cost.
  • The larger performance drop when the task-specific skill is removed after training is consistent with the policy learning to attend to the most informative context, but whether that counts as semantic 'use' of the skill would require probing attention or activation patterns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BCSD, a method for training skill-based LLM agents in which the standard GRPO advantage is rescaled token-by-token by a combination of two self-distillation signals: an augmented context that adds a policy-dependent Meta-Skill and a reduced context that prunes the general skill. The rescaling factor is a clipped exponentiated convex combination of the token-level log-probability gaps between each context and the base context. The authors claim that this rescaling changes only the magnitude of the advantage, preserving the update direction determined by the trajectory-level reward. Experiments on ALFWorld and WebShop with Qwen2.5-7B, Qwen2.5-3B, and Qwen3-1.7B report improvements over baselines, and ablations examine the two context views, the coefficient schedule, and the Meta-Skill refresh mechanism.

Significance. If the results hold, BCSD addresses a real gap in skill-based agent training: how to make a policy follow explicit external skills without internalizing them into parameters. The method is clearly formulated, the experimental design covers multiple model scales and two standard benchmarks, and the paper includes useful ablations and a hyperparameter sensitivity analysis. The skill-utilization analysis in Figure 6 is a thoughtful attempt to show that gains come from reliance on the task-specific skill rather than from memorization. However, the central claim of direction preservation depends on an unstated stop-gradient treatment, and the empirical evidence is weakened by the absence of variance information and by hyperparameter selection on the evaluation benchmarks. These issues are load-bearing for the paper's conclusions.

major comments (4)
  1. [Methods, Eqs. (12)-(15)] The claim that BCSD 'changes only the magnitude of the advantage while preserving the update direction' is not guaranteed by the equations as written. The loss in Eq. (15) contains the advantage A_BCSD_{i,t} defined in Eq. (13), which depends on w_{i,t} and hence on the token gaps in Eqs. (9)-(11). Those gaps are functions of the current policy parameters θ. Unless the author explicitly detach A_BCSD from the computation graph (e.g., using A_BCSD = A_GRPO * (1 - λ_n + λ_n * w.detach())), the gradient of L_BCSD w.r.t. θ includes extra terms of the form ∂A_BCSD/∂θ. These terms can change per-token update directions independently of the trajectory reward and can even amplify tokens for which the policy is more likely under C1 than under C0, effectively optimizing the context gap rather than skill utilization. The manuscript does not state whether the rescaling is detached, and Algorithm 1 does not clarify this. The authors must either specify the stop-gradient treatment or provide a full gradient derivation demonstrating that the extra terms do not alter the optimization direction.
  2. [Experiments, Table 1] All reported results come from a single run with no indication of the number of seeds, no error bars, and no significance tests. Given the stochasticity of policy gradient training and the fact that the headline improvements are modest (e.g., 3.1 points on ALFWorld with Qwen2.5-7B, and 0.8 points below the best WebShop score), the aggregate 'strongest overall performance' claim cannot be assessed for reliability. The paper should report mean and variance over at least three seeds, and ideally a paired significance test across tasks.
  3. [Experiments, Table 3 and Figure 3] The hyperparameters α and λ_0 are tuned on the same benchmarks (ALFWorld and WebShop) where the final results are reported. Table 3 and Figure 3 present performance under different settings on these test tasks, and the chosen values (α=0.9, λ_0=0.1) are selected because they give the best reported numbers. This constitutes tuning on the test set and inflates the apparent advantage of BCSD over baselines that may not have received the same per-benchmark tuning. The authors should either use a held-out validation split for hyperparameter selection or acknowledge the selection bias and provide a sensitivity analysis over a range of values on unseen tasks.
  4. [Further Analysis, Figure 5] The near-chance token-level sign agreement ratio (fluctuating between 47% and 53% in Figure 5) is interpreted as evidence that the two context views are complementary. However, it is equally consistent with at least one of the two gaps being dominated by noise. Because the whole method rests on the assumption that the token-level gaps carry reliable, task-relevant credit information, the paper should provide direct evidence that the combined gap is informative: for example, the gap could be compared between successful and failed trajectories, correlated with per-token outcomes, or examined for consistency across checkpoints. Without such analysis, the mechanism underlying the reported gains remains speculative.
minor comments (5)
  1. [Experimental Setup] The implementation details omit the prompts used for the ExtractLLM and PruneLLM calls, the evidence buffer sizes, and the exact definition of a validation window; these details are needed for reproducibility and should be included in an appendix.
  2. [Figure 2] The schematic in Figure 2 is dense and hard to follow; in particular, the box labeled '1) Context Construction' mixes equations with qualitative labels, making it difficult to map the figure to Eqs. (6)-(8). A clearer separation of the context construction and the rescaling pipeline would improve readability.
  3. [Ablation Studies] There is a typographical error in the text 'withaparticularly' (in the paragraph on disabling Meta-Skill updates); it should read 'with a particularly'.
  4. [Figure 5] The y-axis of Figure 5 ranges only from 0.47 to 0.53, which visually amplifies small fluctuations; the paper should either rescale the axis to the full [0,1] interval or explicitly note the truncated range in the caption.
  5. [Algorithm 1] In Algorithm 1, line 8 says 'Re-score the realized tokens under C0, C1, C2 using πθ', but in the main text the gaps are defined using the same policy πθ evaluated at the current parameters; it would be clearer to specify whether this scoring is done under θ_old or θ, since the gradient treatment depends on it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BCSD's claims are tested on external benchmarks and its objective reduces to standard GRPO.

full rationale

The paper's central empirical claims are evaluated on ALFWorld and WebShop, external benchmarks whose success rates are not constructed from BCSD's equations or fitted values. The token-level gaps in Eqs. (9)-(11) are self-referential in the sense that they are computed from the current policy under augmented and pruned contexts, but they enter the loss only as a positive scalar multiplier on the group-relative advantage (Eqs. (12)-(13)), and lambda_n decays to zero, so Eq. (15) reduces to the standard clipped GRPO objective. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The paper's direction-preservation claim is a construction-level property of Eq. (13); whether the implementation must stop gradients through w_i,t is an implementation/correctness detail, not a circularity. The only overlapping-author citation (Zhang et al. 2026b, OPSDL, co-author Tianjun Pan) is used to justify that pruning reduces distracting context; it is not load-bearing for the central derivation. Figure 5's roughly 50 percent cross-view agreement raises a validity concern about signal quality, but that is an empirical robustness issue, not a circularity. Overall, the derivation is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on several unverified modeling choices: the reliability of self-gaps as credit signals, the fidelity of LLM-based meta-skill extraction and pruning, and the combination weights, all tuned on the target benchmarks. The method also introduces an auxiliary construct (Meta-Skill) with no independent evidence of its content or validity.

free parameters (5)
  • alpha (view weight) = 0.9
    Weights the augmented Meta-Skill gap against the pruned-skill gap in Eq. (11). Chosen via the sensitivity sweep in Figure 3 on ALFWorld with Qwen2.5-3B.
  • lambda_0 (initial rescaling coefficient) = 0.1
    Initial strength of the token-level rescaling in Eq. (13). Selected via the sensitivity study in Table 3 on ALFWorld and WebShop.
  • c (clipping bound for w_i,t) = not reported
    Clipping bound in Eq. (12) is never given a value, so the exact nonlinearity is unspecified.
  • lambda_n decay schedule = linear decay from 0.1 to 0 over 150 steps
    The schedule is described verbally but not specified as a formula; the rate of decay is a free choice that affects early versus late training behavior.
  • Meta-Skill refresh interval = every 15 steps (evaluation interval)
    The refresh period in Algorithm 1 is set to the validation interval; no sensitivity to this choice is reported.
assumptions (5)
  • standard math GRPO group-relative advantage (Eq. 3) is a valid optimization target
    Uses DeepSeek-R1's GRPO objective unmodified as the base RL algorithm.
  • ad hoc to paper The policy's own token-level log-prob gaps under C1/C2 versus C0 encode how skill-informed each token is
    This is the core credit-assignment assumption in Eqs. (9) to (13). It is not validated independently, and Figure 5 shows cross-view agreement near 50 percent, suggesting the gaps may be partly noise.
  • ad hoc to paper A Meta-Skill extracted by an external LLM from the policy's own trajectories summarizes effective and ineffective skill usage and stays reliable as the policy changes
    The extraction and refresh procedure in Eqs. (4) and (5) assumes the frozen extractor tracks the policy's evolving utilization patterns without becoming stale or biased.
  • ad hoc to paper Pruning the general skill G to Gp preserves essential procedural guidance and yields a complementary signal
    Eq. (7) relies on a one-time LLM pruning step; the paper gives no check that Gp remains faithful to G or that the resulting gap is complementary rather than redundant.
  • ad hoc to paper Combining the two gaps by a fixed convex weight and rescaling via Eq. (13) does not distort the reward signal
    No proof or stability analysis is given; the success of the combination depends on hyperparameters tuned on the evaluation benchmarks.
invented entities (2)
  • Meta-Skill (Mc)
    purpose: A high-level, policy-dependent textual summary of how to apply a task-specific skill, used only in the augmented teacher context and never at inference.
    This is a new ancillary construct generated by an external LLM from the policy's own rollouts. Its quality is only measured indirectly through downstream benchmark performance; there is no direct falsifiable handle on the Meta-Skill content itself.
  • Pruned skill (Gp)
    purpose: A shortened version of the general skill, intended to reduce distraction and keep the task-specific skill in focus for the reduced teacher context.
    Gp is another derived context without standalone validation. The paper does not show that Gp preserves all essential procedural content or that its construction is stable across tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents." pith.science (2026). https://pith.science/paper/BTKCVRI3

@misc{pith2026260809555,
  author       = {Pith},
  title        = {Pith review of: Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BTKCVRI3}},
  note         = {Machine review of arXiv:2608.09555}
}
read the original abstract

External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

Figures

Figures reproduced from arXiv: 2608.09555 by the authors.

Figure 1
Figure 1. Comparison of three paradigms for skill-based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of BCSD. It combines skill-conditioned rollouts, trajectory-based meta-skill extraction, and bidirectional context self-distillation for token-wise advantage rescaling. a terminal outcome reward R(τ ) after task completion. For the Skill setup, we follow SkillRL (Xia et al. 2026) and adopt a hierarchical SkillBank setup. For each query Q of task type c, the SkillBank provides a general skill G and a retriev… view at source ↗
Figure 4
Figure 4. WebShop Success Rate on different methods with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Effect of different weight settings on ALFWorld [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 6
Figure 6. Figure 6: Success Rate delta under the General-Skill-only [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 23 canonical work pages

  1. [1]

    International Conference on Learning Representations , year =

    Shridhar, Mohit and Yuan, Xingdi and C. International Conference on Learning Representations , year =

  2. [2]

    2022 , url =

    Yao, Shunyu and Chen, Howard and Yang, John and Narasimhan, Karthik , booktitle =. 2022 , url =

    2022

  3. [3]

    2017 , eprint =

    Proximal Policy Optimization Algorithms , author =. 2017 , eprint =

    2017

  4. [4]

    International Conference on Learning Representations , year =

    High-Dimensional Continuous Control Using Generalized Advantage Estimation , author =. International Conference on Learning Representations , year =

  5. [6]

    2025 , url =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and Liu, Juncai and Liu, LingJun and Liu, Xin and Lin, Haibin and Lin, Zhiqi and Ma, Bole and Sheng, Guangming and Tong, Yuxuan and Zhang, Chi and Zhang, Mofan and Zhang, Ru and Zhang, Wang and Zhu, Hang and Zhu, Ji...

    work page 2025

  6. [7]

    2025 , eprint =

    Group Sequence Policy Optimization , author =. 2025 , eprint =

    2025

  7. [8]

    2023 , url =

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =

    2023

  8. [9]

    International Conference on Learning Representations , year =

    Let's Verify Step by Step , author =. International Conference on Learning Representations , year =

Show all 69 references
  1. [10]

    International Conference on Learning Representations (ICLR) , year =

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author =. International Conference on Learning Representations (ICLR) , year =

  2. [11]

    Second Conference on Language Modeling , year =

    Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Yoon, Jinsung and Ar. Second Conference on Language Modeling , year =

  3. [12]

    2026 , eprint =

    Xia, Peng and Chen, Jianwen and Wang, Hanyang and Liu, Jiaqi and Zeng, Kaide and Wang, Yu and Han, Siwei and Zhou, Yiyang and Zhao, Xujiang and Chen, Haifeng and Zheng, Zeyu and Xie, Cihang and Yao, Huaxiu , booktitle =. 2026 , eprint =

    2026

  4. [20]

    2026 , eprint =

    Self-Distilled Agentic Reinforcement Learning , author =. 2026 , eprint =

    2026

  5. [21]

    2605.11853 , archivePrefix =

    Li, Sijia and Huang, Yuchen and Liu, Zifan and Li, Yanping and Fu, Jingjing and Zhao, Li and Bian, Jiang and Zhang, Ling and Zhang, Jun and Wang, Rui , year =. 2605.11853 , archivePrefix =

    arXiv

  6. [25]

    2023 , url =

    Shinn, Noah and Cassano, Federico and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu , booktitle =. 2023 , url =

    2023

  7. [26]

    2026 , month = may, eprint =

    Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward , author =. 2026 , month = may, eprint =

    2026

  8. [30]

    Proceedings of the 43rd International Conference on Machine Learning , volume =

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author =. Proceedings of the 43rd International Conference on Machine Learning , volume =. 2026 , eprint =

    2026

  9. [31]

    Proceedings of the 43rd International Conference on Machine Learning , volume =

    Reinforcement Learning via Self-Distillation , author =. Proceedings of the 43rd International Conference on Machine Learning , volume =. 2026 , eprint =

    2026

  10. [37]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  11. [40]

    Advances in Neural Information Processing Systems , volume=

    Group-in-group policy optimization for llm agent training , author=. Advances in Neural Information Processing Systems , volume=

  12. [41]

    Forty-third International Conference on Machine Learning , year=

    Segment-Aligned Policy Optimization for Multi-Modal Reasoning , author=. Forty-third International Conference on Machine Learning , year=

  13. [42]

    Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms , author=. Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    2026

  14. [47]

    Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    SALT: Step-level advantage assignment for long-horizon agents via trajectory graph , author=. Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    2026

  15. [50]

    An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chengyuan Li and Dayiheng Liu and Fei Huang and Haoran Wei and Huan Lin and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and Jingren Zhou and Junyang Lin and...

  16. [51]

    An Yang and Anfeng Li and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Gao and Chengen Huang and Chenxu Lv and Chujie Zheng and Dayiheng Liu and Fan Zhou and Fei Huang and Feng Hu and Hao Ge and Haoran Wei and Huan Lin and Jialong Tang and...

  17. [53]

    Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Ramos, S.; Geist, M.; and Bachem, O. 2024. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations (ICLR)

    2024

  18. [54]

    a henb \

    Chen, K.; Cusumano-Towner, M.; Huval, B.; Petrenko, A.; Hamburger, J.; Koltun, V.; and Kr \"a henb \"u hl, P. 2025. Reinforcement learning for long-horizon interactive llm agents. arXiv preprint arXiv:2502.01600

    2025 arXiv

  19. [55]

    Feng, L.; Xue, Z.; Liu, T.; and An, B. 2026. Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems, 38: 46375--46408

    2026

  20. [56]

    Gao, H.; Chen, H.; Wang, C.; Guo, S.; Pang, L.; Liu, Z.; Shen, H.; and Cheng, X. 2026 a . SkillAudit : Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing. arXiv:2606.14239

    2026

  21. [57]

    Gao, L.; Li, Z.; Jia, M.; Yuan, J.; Sun, H.; Sun, H.; and Li, X. 2026 b . Segment-Aligned Policy Optimization for Multi-Modal Reasoning. In Forty-third International Conference on Machine Learning

    2026

  22. [58]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  23. [59]

    H.; Liu, X.; Wei, L.; Pan, L.; Zeng, K.; and Cai, X

    He, Z.; Li, Y.; Huang, F.; Chen, T.; Chen, S.; Li, X.; Yu, M. H.; Liu, X.; Wei, L.; Pan, L.; Zeng, K.; and Cai, X. 2026. SIRI : Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training. arXiv:2606.02355

    2026 arXiv

  24. [60]

    u botter, J.; L \

    H \"u botter, J.; L \"u beck, F.; Behric, L. D.; Baumann, A.; Bagatella, M.; Marta, D.; Hakimi, I.; Shenfeld, I.; Kleine Buening, T.; Guestrin, C.; and Krause, A. 2026. Reinforcement Learning via Self-Distillation. In Proceedings of the 43rd International Conference on Machine...

    2026

  25. [61]

    Jiang, Y.; Li, D.; Deng, H.; Ma, B.; Wang, X.; Wang, Q.; and Yu, G. 2026. SoK : Agentic Skills---Beyond Tool Use in LLM Agents. arXiv:2602.20867

    2026 arXiv

  26. [62]

    Lazaridis, A.; Bates, D.; Sharma, A.; King, B.; Lu, V.; and FitzGerald, J. 2026. EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation. arXiv preprint arXiv:2605.23493

    2026 arXiv

  27. [63]

    Li, J.; Wang, Y.; Yan, Q.; Tian, Y.; Xu, Z.; Song, H.; Xu, P.; and Cheong, L. L. 2026 a . SALT: Step-level advantage assignment for long-horizon agents via trajectory graph. In Findings of the Association for Computational Linguistics: EACL 2026, 4709--4725

    2026

  28. [64]

    P.; Li, L.; and Li, Y

    Li, J.; Zhou, P.; Meng, R.; Vadera, M. P.; Li, L.; and Li, Y. 2026 b . Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms. In Findings of the Association for Computational Linguistics: EACL 2026, 6227--6243

    2026

  29. [65]

    W.; Sun, J.; Wang, S.; et al

    Li, X.; Liu, Y.; Chen, W.; You, B.; Di, Z.; He, Y.; Zheng, S.; Choe, K. W.; Sun, J.; Wang, S.; et al. 2026 c . SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670

    2026 arXiv

  30. [66]

    Li, Z.; and Hu, Y. 2026. SkillHone : A Harness for Continual Agent Skill Evolution Through Persistent Decision History. arXiv:2606.08671

    2026 arXiv

  31. [67]

    Liu, H.; Zhang, Y.; Li, X.; Lyu, B.; and Shang, J. 2026 a . HERO : Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation. arXiv:2606.11559

    2026 arXiv

  32. [68]

    Liu, Y.; Ji, J.; An, L.; Jaakkola, T.; Zhang, Y.; and Chang, S. 2026 b . How well do agentic skills work in the wild: Benchmarking llm skill usage in realistic settings. arXiv preprint arXiv:2604.04323

    2026 arXiv

  33. [69]

    Lu, Z.; Yao, Z.; Han, Z.; Wang, Z.-H.; Wu, J.; Gu, Q.; Cai, X.; Lu, W.; Xiao, J.; Zhuang, Y.; and Shen, Y. 2026. Self-Distilled Agentic Reinforcement Learning. arXiv:2605.15155

    2026 arXiv

  34. [70]

    Ma, W.; Wei, J.; Zhao, L.; Zhang, H.; Xiao, B.; Li, L.; Yang, Q.; Gao, B.; Wang, Y.; Li, R.; et al. 2026 a . Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406

    2026 arXiv

  35. [71]

    Ma, Z.; Yang, S.; Ji, Y.; Wang, X.; Wang, Y.; Hu, Y.; Huang, T.; and Chu, X. 2026 b . SkillClaw : Let Skills Evolve Collectively with Agentic Evolver. arXiv:2604.08377

    2026 arXiv

  36. [72]

    Ni, J.; Liu, Y.; Liu, X.; Sun, Y.; Zhou, M.; Cheng, P.; Wang, D.; Zhao, E.; Jiang, X.; and Jiang, G. 2026. Trace2Skill : Distill Trajectory-Local Lessons into Transferable Agent Skills. arXiv:2603.25158

    2026 arXiv

  37. [73]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730--27744

    2022

  38. [74]

    Pan, L.; Tao, S.; Zhai, Y.; Zhang, L.; Liu, Z.; Ding, B.; Liu, A.; and Wen, L. 2026. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. arXiv preprint arXiv:2606.11709

    2026 arXiv

  39. [75]

    Sang, H.; Xu, Y.; Zhou, Z.; He, R.; Wang, Z.; and Sun, J. 2026. CRISP : Compressed Reasoning via Iterative Self-Policy Distillation. arXiv:2603.05433

    2026 arXiv

  40. [76]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  41. [77]

    Shen, G.; Cheng, X.; Zhao, C.; Huang, L.; Li, J.; Zhao, D.; and Yu, X. 2026. Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information. arXiv preprint arXiv:2605.11609

    2026 arXiv

  42. [78]

    Shi, Y.; Chen, Y.; Lu, Z.; Miao, Y.; Liu, S.; Gu, Q.; Cai, X.; Wang, X.; and Zhang, A. 2026. Skill1 : Unified Evolution of Skill-Augmented Agents via Reinforcement Learning. arXiv:2605.06130

    2026 arXiv

  43. [79]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion : Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36, 8634--8652. Curran Associates, Inc

    2023

  44. [80]

    Shridhar, M.; Yuan, X.; C \^o t \'e , M.-A.; Bisk, Y.; Trischler, A.; and Hausknecht, M. 2021. ALFWorld : Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations

    2021

  45. [81]

    Wang, H.; Wang, G.; Xiao, H.; Zhou, Y.; Pan, Y.; Wang, J.; Xu, K.; Wen, Y.; Ruan, X.; Chen, X.; and Qi, H. 2026 a . Skill-SD : Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. arXiv:2604.10674

    2026 arXiv

  46. [82]

    Wang, J.; Zhang, W.; Shi, W.; Li, Y.; and Cheng, J. 2026 b . Tcod: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2604.24005

    2026 arXiv

  47. [83]

    Xia, P.; Chen, J.; Wang, H.; Liu, J.; Zeng, K.; Wang, Y.; Han, S.; Zhou, Y.; Zhao, X.; Chen, H.; Zheng, Z.; Xie, C.; and Yao, H. 2026. SkillRL : Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. In ICLR 2026 Workshop on Memory for LLM-Based Agentic Systems ...

    2026

  48. [84]

    Xu, R.; and Yan, Y. 2026. Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward. Agent Skills '26 Workshop at ACM CAIS 2026. arXiv:2602.12430

    2026 arXiv

  49. [85]

    Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; Zheng, C.; Liu, D.; Zhou, F.; Huang, F.; Hu, F.; Ge, H.; Wei, H.; Lin, H.; Tang, J.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Zhou, J.; Lin, J.; Dang, K.; Bao, K.; ...

    2025 arXiv

  50. [86]

    Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, ...

    2024 arXiv

  51. [87]

    Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N. 2026 a . Self-Distilled RLVR . arXiv:2604.03128

    2026 arXiv

  52. [88]

    Yang, S.; Wu, J.; Lu, Z.; Shen, Y.; Zhang, F.; Feng, L.; Zhang, S.; Luo, H.; Lian, Z.; Wen, Z.; and Tao, J. 2026 b . OPID : On-Policy Skill Distillation for Agentic Reinforcement Learning. arXiv:2606.26790

    2026 arXiv

  53. [89]

    Yang, Y.; Wang, X.; and Zhang, W. 2026. OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning. arXiv preprint arXiv:2605.12400

    2026 arXiv

  54. [90]

    Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. WebShop : Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems, volume 35, 20744--20757. Curran Associates, Inc

    2022

  55. [91]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct : Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations (ICLR)

    2023

  56. [92]

    Zhang, G.; Zhu, E.; Zhou, J.; Jia, C.; and Wang, H. 2026 a . SkillEvolver : Skill Learning as a Meta-Skill. arXiv:2605.10500

    2026 arXiv

  57. [93]

    Zhang, X.; Ding, Z.; Pan, T.; Yang, R.; Kang, C.; Xiong, X.; and Gu, J. 2026 b . OPSDL : On-Policy Self-Distillation for Long-Context Language Models. arXiv:2604.17535

    2026 arXiv

  58. [94]

    Zhang, Y.; Lin, X.; and Wu, C. 2026. StepOPSD : Step-Aware Online Preference Distillation for Agent Reinforcement Learning. arXiv:2605.27140

    2026 arXiv

  59. [95]

    Zhao, S.; Xie, Z.; Liu, M.; Huang, J.; Pang, G.; Chen, F.; and Grover, A. 2026. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306. PMLR

    2026

  60. [96]

    Zheng, B.; Ma, X.; Liang, Y.; Ruan, J.; Fu, X.; Lin, K.; Zhu, B.; Zeng, K.; and Cai, X. 2026. Scope: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688

    2026 arXiv

  61. [97]

    Zhong, Q.; Zheng, M.; Song, M.; Lin, X.; Sun, J.; Jiang, H.; Wang, X.; and Fang, J. 2026. Sod: Step-wise on-policy distillation for small language model agents. arXiv preprint arXiv:2605.07725

    2026 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.