Pith. sign in

REVIEW 2 major objections 7 minor 20 references

Soft Clamp Cuts Tool Over-Calling by 34% in Multi-Teacher Distillation

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 20:56 UTC pith:UOXHR7CD

load-bearing objection Real problem, useful method, but the mechanism story has a gap between diagnosis and intervention the 2 major comments →

arxiv 2607.07050 v1 pith:UOXHR7CD submitted 2026-07-08 cs.CL cs.LG

Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation

Jiabin Shen , Guang Chen , Chengjun Mao This is my paper
classification cs.CL cs.LG
keywords on-policy distillationmulti-teacher distillationtool-usebehavior leveragetoken-level calibrationknowledge distillationagentic language modelsover-calling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-teacher on-policy distillation can shift a model's behavior in ways invisible to aggregate loss monitoring. In a two-teacher tool-use setting, vanilla generalized knowledge distillation (GKD) improves tool-call recall but also increases over-calling—calling tools on examples that should receive direct answers. The authors show this shift cannot be explained by aggregate metrics: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. Instead, they introduce the concept of behavior leverage—the degree to which a token position controls the future generation mode. Mode-entry tokens like the tool-call opening tag and function names have high behavior leverage because changing them redirects the entire continuation into a different trajectory. When one teacher's signals concentrate on high-leverage positions and another teacher's signals spread across lower-leverage content tokens, the student shifts toward the high-leverage behavior even without aggregate signal dominance. The authors propose Soft Clamp, which sets a dynamic threshold at k times the batch-mean per-token divergence, caps the forward value for extreme tokens at that threshold, but scales their gradient by the ratio C/d_i so learning continues. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching decision accuracy, and in a BFCL multi-turn diagnostic it lowers tool-call loops and repeated calls.

Core claim

The paper identifies behavior leverage imbalance as a failure mode in multi-teacher on-policy distillation for tool-use models. When one teacher specializes in tool calls and another in direct responses, the student can develop a bias toward over-calling tools even though aggregate statistics show no dominance by the tool-call teacher in token exposure, divergence, or gradient magnitude. The explanation is that certain token positions—mode-entry tokens like the tool-call tag and function names—have disproportionate control over the generation trajectory: a small local signal at such a position can redirect the entire continuation into a tool-call mode, while response-side signals spreadacros

What carries the argument

The central object is the per-token divergence d_i = JSD(p_teacher || p_student) at each supervised position, and the observation that its behavioral effect depends on where in the generation it lands. Soft Clamp operates on this quantity by setting a batch-dynamic threshold C = k * mean(d_i), then replacing d_i with d_i * C / stopgrad(d_i) for tokens above the threshold—capping the forward contribution while scaling the gradient by C/d_i so it remains nonzero.

Load-bearing premise

The paper assumes that the observed correlation between response-side tool-call probability metrics and final over-calling reflects a causal mechanism where local signals at mode-entry tokens drive the global behavior shift. The authors themselves acknowledge this is diagnostic evidence, not complete causal proof. If the over-calling instead stems from data distribution properties, the supervised format anchor, or base-model-specific interactions, then the proposed mechanism—

What would settle it

If a control experiment showed that compressing extreme divergence at randomly chosen token positions (not mode-entry positions) reduces over-calling equally well, the behavior leverage mechanism would be undermined—Soft Clamp would work for a different reason than the one proposed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-teacher distillation systems should monitor where teacher signals act (which token positions), not just how much total signal each teacher provides, because high-leverage positions can dominate behavior even with small aggregate contributions.
  • The behavior leverage concept extends beyond tool-use: any setting with heterogeneous teachers supervising different generation modes (refusal vs. helpfulness, code vs. natural language, short vs. long reasoning) could exhibit similar imbalance at mode-entry tokens.
  • Dynamic per-token divergence calibration like Soft Clamp could be a general-purpose technique for any distillation setting where certain token positions have disproportionate behavioral control, not limited to tool-use or multi-teacher configurations.
  • The finding that behavior shifts can be invisible from aggregate losses suggests that standard training monitoring (loss curves, total divergence) is insufficient for agentic models, and decision-boundary metrics should become standard diagnostics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If behavior leverage imbalance is the true mechanism, then explicitly identifying and down-weighting mode-entry tokens (rather than blindly compressing all extreme divergences) should outperform Soft Clamp, which does not explicitly identify such tokens.
  • The concept predicts that the severity of over-calling should correlate with the density of high-leverage positions in the tool-call teacher's training data, not with total divergence—testable by varying the proportion of mode-entry tokens while holding total divergence constant.
  • If the mechanism generalizes, single-teacher distillation should also exhibit behavior leverage effects when the teacher's signals happen to concentrate at mode-entry positions, suggesting the phenomenon is not specific to multi-teacher settings.
  • The effectiveness of Soft Clamp with k=3.0 raises the question of whether the optimal clamp multiplier relates to the ratio of high-leverage to low-leverage token positions in the training data, which would make the method tunable rather than ad hoc.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper identifies a failure mode in multi-teacher on-policy distillation (MOPD) for agentic tool-use: vanilla generalized knowledge distillation (GKD) improves tool-call recall but simultaneously increases over-calling (calling tools on examples that should be answered directly). The authors show that aggregate statistics (token exposure, full-sequence per-token JSD) do not explain this shift. They propose the concept of 'behavior leverage imbalance'—the idea that local token-level signals at mode-entry positions (e.g., <tool call>, function names) have disproportionate control over the global generation mode. To mitigate this, they introduce Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level JSD while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching decision accuracy. A BFCL multi-turn diagnostic shows lower tool-call loops and repeated calls.

Significance. The paper addresses a practically important problem: tool overuse in agentic LLMs trained via distillation. The observation that aggregate loss statistics can hide behavior-level shifts is valuable and underexplored. The Soft Clamp method is simple, lightweight, and does not require additional infrastructure (learned reward models, teacher routing policies, or held-out calibration sets). The multi-turn loop diagnostic (Appendix D) is a useful practical stress test. The paper is honest about limitations, explicitly acknowledging that the evidence is diagnostic rather than a complete causal proof. The core empirical finding—that vanilla GKD trades over-calling for call recall—is clearly demonstrated across multiple benchmarks.

major comments (2)
  1. §4.2 and §3.2: The central mechanism claim is that behavior leverage imbalance occurs because local signals at mode-entry and structural positions have disproportionate control over the global generation mode. However, Soft Clamp operates on any token whose divergence exceeds the batch-relative threshold C, regardless of where that token sits in the sequence. The paper itself acknowledges: 'It does not explicitly identify mode-entry tokens; instead, it limits the effect of any extreme local divergence' (§4.2). This creates a gap between the diagnostic story (behavior leverage at structural tokens) and the intervention (compressing any extreme divergence). If the extreme divergence values that Soft Clamp compresses are distributed across ordinary content tokens rather than concentrated at mode-entry positions, then the proposed mechanism is misidentified, and Soft Clamp's effectiveness is
  2. Table 4 (BFCL results): All GKD variants underperform the base and base SFT models on overall BFCL scores (e.g., Vanilla GKD 79.8 vs. Base SFT 82.9). The paper frames Soft Clamp as a GKD-internal calibration method, but this regression is significant. The paper should discuss whether the GKD framework itself is counterproductive on BFCL, or whether the format anchor (sft_alpha=0.3) is insufficient to prevent degradation on this benchmark. This affects the practical applicability of the entire MOPD approach, not just Soft Clamp.
minor comments (7)
  1. §2.2: The tool-call target rendering is described as an implementation-specific serialization detail. However, since the entire mechanism story hinges on mode-entry tokens like <tool call>, the specific rendering format used in training is load-bearing for reproducibility. Appendix A.2 gives one example, but the paper should clarify whether the results depend on this specific format.
  2. Table 5 (When2Call): All GKD variants perform worse than base on MCQ accuracy (e.g., Vanilla GKD 64.9 vs. Base 72.8). The paper acknowledges this but does not analyze why distillation degrades out-of-domain decision-making. A brief discussion of whether this is related to over-calling or a separate issue would strengthen the paper.
  3. Figure 1: The y-axis label 'Tool / response ratio' is clear, but the figure would benefit from explicitly marking which direction (above or below 1.0) indicates tool-call dominance. A reader unfamiliar with the ratio may not quickly parse the result.
  4. §4.1, Eq. (3): The notation d_i * C/stopgrad(d_i) is slightly ambiguous on first read. It would help to explicitly state that the forward value equals C (for d_i > C) in the equation caption).
  5. Appendix C.2: The choice of k=3.0 for the clamp multiplier is stated without justification. A brief sensitivity analysis or explanation of why 3.0 is a reasonable default would improve the method's practical guidance.
  6. Table 7: The 'GKD rollout mixture lambda 0.8' and 'GKD beta 0.5' parameters are listed but not explained in the main text. A brief note on what these control would help reproducibility.
  7. §5.5: The BFCL multi-turn diagnostic uses 800 tasks and 3,136 user turns, but the maximum step limit is not specified in the main text (only mentioned as a metric). Appendix D should state the exact step limit value used.

Circularity Check

0 steps flagged

No circularity: empirical study with fixed parameters evaluated on external benchmarks

full rationale

The paper's derivation chain is: (1) observe that vanilla GKD raises over-calling (Table 3, empirical), (2) rule out aggregate explanations via independent measurements (Table 1, token exposure and JSD ratios computed from training logs, not fitted), (3) observe correlation between response-side P(tool) and final over-calling (Table 2, Figure 2, empirical diagnostic), (4) hypothesize behavior leverage imbalance as mechanism, (5) propose Soft Clamp with fixed k=3.0, (6) evaluate on external benchmarks (APIGen-MT, BFCL, When2Call). No step reduces to its inputs by construction. The diagnostic metrics (P(tool) on response samples, top-1 tool rate) are computed from the student's distribution independently of the over-calling metric they are said to correlate with — they measure teacher-forced first-token probability, while over-calling measures free-generation behavior on should-respond examples. Soft Clamp's parameter is fixed, not fitted to the target. No load-bearing self-citation chain exists; the foundational work (GKD by Agarwal et al., MOPD by Ma et al.) is external. The gap between the mechanism story (mode-entry tokens) and the position-agnostic intervention is a validity concern (the skeptic's attack is about mechanism misidentification), not a circularity concern — the paper does not claim to derive the mechanism from the intervention or vice versa.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The axiom ledger captures the key parameters and assumptions. The free parameters are fixed across experiments, not fitted to the target metric, which supports the non-circularity of the method evaluation. The invented entity 'behavior leverage' is supported by correlational evidence, which the authors acknowledge is not a complete causal proof.

free parameters (4)
  • k (clamp multiplier) = 3.0
    Used in Soft Clamp to set the dynamic threshold C = k * mean(d_i). Stated as fixed in Appendix C.2, not fitted to the over-calling metric.
  • sft_alpha (format anchor weight) = 0.3
    Weight for the supervised loss used as a format anchor. Stated as fixed across all GKD variants in Table 7.
  • c (Hard Clip threshold) = 0.5
    Threshold for the Hard Clip baseline. Stated in Appendix C.3.
  • alpha, z_max, w_min, w_max (Global Reweight params) = 0.3, 3.0, 0.25, 2.0
    Parameters for the Global Reweight baseline. Stated in Appendix C.4.
axioms (3)
  • domain assumption Mode-entry tokens (e.g., <tool call>) have high behavior leverage, meaning changing them redirects the continuation.
    Invoked in §2.3 to define the concept of behavior leverage. It is a reasonable domain assumption for autoregressive generation but is not formally proven.
  • domain assumption The tool-call teacher and response teacher, starting from the same base model and trained on respective target types, serve as valid online supervisors.
    Invoked in §2.2 and §5.1. Assumes that this training procedure produces teachers whose distributions are meaningful targets for the student.
  • domain assumption The BFCL multi-turn loop diagnostic measures usability risk under a simulated environment.
    Invoked in §5.2 and §5.5. Assumes the simulated tool environment is representative of real deployment risks.
invented entities (2)
  • Behavior leverage independent evidence
    purpose: Conceptual framework to explain why local token signals can have disproportionate global effects.
    The paper provides empirical evidence (Table 2, Figure 2) that metrics at these positions correlate with final behavior, serving as a falsifiable handle.
  • Soft Clamp independent evidence
    purpose: Mitigation method for behavior leverage imbalance.
    Evaluated on external benchmarks (APIGen-MT, BFCL) with results that could falsify its utility.

pith-pipeline@v1.1.0-glm · 15171 in / 2431 out tokens · 237245 ms · 2026-07-09T20:56:29.204106+00:00 · methodology

0 comments
read the original abstract

Agentic language models must learn when to call tools, when to consume tool responses, and when to answer directly. This makes multi-teacher on-policy distillation a natural training strategy: one teacher can specialize in tool calls, another in direct responses, and the student can learn from both on its own generated distribution. We show that this strategy can induce a behavior shift that is invisible from aggregate losses alone. In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, where it calls tools on examples that should be answered directly. Aggregate explanations are insufficient: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. We instead analyze behavior leverage imbalance: local token-level signals at mode- entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy. In a BFCL multi-turn diagnostic, it also lowers tool-call loops and repeated calls among GKD variants. These results suggest that multi-teacher OPD should monitor where teacher signals act, not only how large they are in aggregate.

Figures

Figures reproduced from arXiv: 2607.07050 by Chengjun Mao, Guang Chen, Jiabin Shen.

Figure 1
Figure 1. Figure 1: Token exposure and aggregate JSD sanity checks. Tool-call signals do not dominate in aggregate, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Response-side decision pressure tracks final over-calling across GKD variants. Each point is one [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Step-level response-side decision pressure. Teacher-forced process metrics show that the pressure [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Intervention strength and behavior. Stronger compression of extreme token-level signals aligns with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: APIGen-MT behavior trade-off. Soft Clamp reduces over-calling relative to vanilla GKD while [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: BFCL multi-turn loop diagnostic. Over-calling in single-turn evaluation corresponds to more tool [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 14 internal anchors

  1. [1]

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/ abs/2306.13649. GKD: foundation work for on-policy distillation with...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational Conference on Machine Learning (ICML), 2018. Gradient normalization for multi-task learning. Referenced as related approach

    work page 2018

  3. [4]

    Analyzes collapse in Search-R1-style tool-integrated GRPO via Lazy Likelihood Displacement

    URL https://arxiv.org/abs/2512.04220. Analyzes collapse in Search-R1-style tool-integrated GRPO via Lazy Likelihood Displacement. Useful adjacent evidence that scalar training curves can hide behavior-level failures

    work page arXiv

  4. [5]

    Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization

    D. Hao, Zhiwei Jin, Chen Chen, and H. Lu. Stabilizing on-policy distillation for mllm reasoning with global normalization.arXiv preprint arXiv:2606.09091, 2026. URL https://arxiv.org/abs/ 2606.09091. Batch-level global normalization of token distillation rewards as relative advantages. Suppresses gradient spikes. Single-teacher

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [6]

    Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

    Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, and Jun Zhao. Why multi-step tool-use reinforce- ment learning collapses and how supervisory signals fix it.arXiv preprint arXiv:2606.26027, 2026. URL https://arxiv.org/abs/2606.26027. Analyzes catastrophic collapse in multi-step tool-use RL, including failed tool-invocation structures driven by control-toke...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [7]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Wang Dong, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025. URL https://arxiv.org/abs/2503.09516. Tool-integrated RL framework for multi-turn search interaction. Cited as adjacent background for behavior-level ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [8]

    Entropy-Aware On-Policy Distillation of Language Models

    Jongho Jin et al. Entropy-aware on-policy distillation of language models. InInternational Con- ference on Machine Learning (ICML), 2026. URL https://arxiv.org/abs/2603.07079. Single-teacher entropy-based KL switch (forward/reverse). Binary divergence choice, not magnitude calibration

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [9]

    Todi: Token-wise distillation via fine-grained divergence control

    Seongryong Jung et al. Todi: Token-wise distillation via fine-grained divergence control. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. URL https://arxiv.org/abs/2505.16297. Per-token adaptive forward/reverse KL selection via sigmoid weighting. Single-teacher. Outstanding Paper Award Nominee

    work page arXiv 2025

  9. [10]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Roberto Cipolla, and Yarin Gal. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. Homoscedastic uncertainty for multi-task loss weighting. Theoretical basis for uncertainty weighting baseline. 11

    work page 2018

  10. [11]

    DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

    Jongwoo Ko et al. Distillm-2: A contrastive approach boosts the distillation of llms. InInternational Conference on Machine Learning (ICML), 2025. URL https://arxiv.org/abs/2503.07067. Contrastive distillation with different divergences for different data types. Single-teacher

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [12]

    MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

    Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, and Fuli Luo. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026. URL https://arxiv.org/abs/2606.30406. Multi-teacher OPD for capability ...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [13]

    Smart: Self-aware agent for tool overuse mitigation

    Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. InFindings of the Associ- ation for Computational Linguistics: ACL, 2025. URL https://aclanthology.org/2025. findings-acl.239/. Studies unnecessary tool use and calibrates when agents should re...

    work page 2025

  13. [14]

    A Survey of On-Policy Distillation for Large Language Models

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026. URL https://arxiv.org/abs/2604.00626. First OPD survey; unifies f-divergence framework. Must cite

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [15]

    MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

    Zhihui Wang et al. Mad-opd: Breaking the ceiling in on-policy distillation via multi-agent debate.arXiv preprint arXiv:2605.01347, 2026. URL https://arxiv.org/abs/2605.01347. Multi-teacher OPD via debate + confidence-weighted teacher selection. Solves ceiling problem, not bias

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  15. [16]

    Llm-oriented token-adaptive knowledge distillation

    Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. Llm-oriented token-adaptive knowledge distillation. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2026. URL https://ojs.aaai.org/index.php/AAAI/ article/view/40701. Token-adaptive KD based on student needs. Single-teacher + adaptive...

    work page 2026

  16. [17]

    TIP: Token Importance in On-Policy Distillation

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084, 2026. URL https: //arxiv.org/abs/2604.14084. Token importance taxonomy: high student entropy + low entropy with high divergence. Single-teacher descriptive, not calibrative

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [18]

    MiniLLM: On-Policy Distillation of Large Language Models

    Yuxian Xu et al. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2306.08543. Reverse KL + REINFORCE for LLM distillation. Single-teacher

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [19]

    SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

    Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning.arXiv preprint arXiv:2509.02479, 2025. URL https://arxiv.org/abs/2509.02479. Identifies distribu- tional drift from external tool feedback as a source of multi-turn TIR RL instability...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [20]

    The Tool-Overuse Illusion: Why Does LLM Prefer External Tools over Internal Knowledge?

    Yirong Zeng, Shen You, Yufei Liu, Qunyao Du, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Bibo Cai, and Ting Liu. The tool-overuse illusion: Why does llm prefer external tools over internal knowledge?arXiv preprint arXiv:2604.19749, 2026. URL https: //arxiv.org/abs/2604.19749. Studies tool overuse and shows outcome-only RL can incre...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [21]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026. URL https://arxiv.org/abs/2601.18734. Single-teacher hard clip (min(loss, tau)) for token-level imbalance. Gradient zeroed above threshold. 13 A Ad...

    work page internal anchor Pith review Pith/arXiv arXiv 2026