Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation

Chengjun Mao; Guang Chen; Jiabin Shen

Soft Clamp Cuts Tool Over-Calling by 34% in Multi-Teacher Distillation

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 20:56 UTC pith:UOXHR7CD

load-bearing objection Real problem, useful method, but the mechanism story has a gap between diagnosis and intervention the 2 major comments →

arxiv 2607.07050 v2 pith:UOXHR7CD submitted 2026-07-08 cs.CL cs.LG

Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation

Jiabin Shen , Guang Chen , Chengjun Mao This is my paper

classification cs.CL cs.LG

keywords on-policy distillationmulti-teacher distillationtool-usebehavior leveragetoken-level calibrationknowledge distillationagentic language modelsover-calling

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-teacher on-policy distillation can shift a model's behavior in ways invisible to aggregate loss monitoring. In a two-teacher tool-use setting, vanilla generalized knowledge distillation (GKD) improves tool-call recall but also increases over-calling—calling tools on examples that should receive direct answers. The authors show this shift cannot be explained by aggregate metrics: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. Instead, they introduce the concept of behavior leverage—the degree to which a token position controls the future generation mode. Mode-entry tokens like the tool-call opening tag and function names have high behavior leverage because changing them redirects the entire continuation into a different trajectory. When one teacher's signals concentrate on high-leverage positions and another teacher's signals spread across lower-leverage content tokens, the student shifts toward the high-leverage behavior even without aggregate signal dominance. The authors propose Soft Clamp, which sets a dynamic threshold at k times the batch-mean per-token divergence, caps the forward value for extreme tokens at that threshold, but scales their gradient by the ratio C/d_i so learning continues. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching decision accuracy, and in a BFCL multi-turn diagnostic it lowers tool-call loops and repeated calls.

Core claim

The paper identifies behavior leverage imbalance as a failure mode in multi-teacher on-policy distillation for tool-use models. When one teacher specializes in tool calls and another in direct responses, the student can develop a bias toward over-calling tools even though aggregate statistics show no dominance by the tool-call teacher in token exposure, divergence, or gradient magnitude. The explanation is that certain token positions—mode-entry tokens like the tool-call tag and function names—have disproportionate control over the generation trajectory: a small local signal at such a position can redirect the entire continuation into a tool-call mode, while response-side signals spreadacros

What carries the argument

The central object is the per-token divergence d_i = JSD(p_teacher || p_student) at each supervised position, and the observation that its behavioral effect depends on where in the generation it lands. Soft Clamp operates on this quantity by setting a batch-dynamic threshold C = k * mean(d_i), then replacing d_i with d_i * C / stopgrad(d_i) for tokens above the threshold—capping the forward contribution while scaling the gradient by C/d_i so it remains nonzero.

Load-bearing premise

The paper assumes that the observed correlation between response-side tool-call probability metrics and final over-calling reflects a causal mechanism where local signals at mode-entry tokens drive the global behavior shift. The authors themselves acknowledge this is diagnostic evidence, not complete causal proof. If the over-calling instead stems from data distribution properties, the supervised format anchor, or base-model-specific interactions, then the proposed mechanism—

What would settle it

If a control experiment showed that compressing extreme divergence at randomly chosen token positions (not mode-entry positions) reduces over-calling equally well, the behavior leverage mechanism would be undermined—Soft Clamp would work for a different reason than the one proposed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Multi-teacher distillation systems should monitor where teacher signals act (which token positions), not just how much total signal each teacher provides, because high-leverage positions can dominate behavior even with small aggregate contributions.
The behavior leverage concept extends beyond tool-use: any setting with heterogeneous teachers supervising different generation modes (refusal vs. helpfulness, code vs. natural language, short vs. long reasoning) could exhibit similar imbalance at mode-entry tokens.
Dynamic per-token divergence calibration like Soft Clamp could be a general-purpose technique for any distillation setting where certain token positions have disproportionate behavioral control, not limited to tool-use or multi-teacher configurations.
The finding that behavior shifts can be invisible from aggregate losses suggests that standard training monitoring (loss curves, total divergence) is insufficient for agentic models, and decision-boundary metrics should become standard diagnostics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If behavior leverage imbalance is the true mechanism, then explicitly identifying and down-weighting mode-entry tokens (rather than blindly compressing all extreme divergences) should outperform Soft Clamp, which does not explicitly identify such tokens.
The concept predicts that the severity of over-calling should correlate with the density of high-leverage positions in the tool-call teacher's training data, not with total divergence—testable by varying the proportion of mode-entry tokens while holding total divergence constant.
If the mechanism generalizes, single-teacher distillation should also exhibit behavior leverage effects when the teacher's signals happen to concentrate at mode-entry positions, suggesting the phenomenon is not specific to multi-teacher settings.
The effectiveness of Soft Clamp with k=3.0 raises the question of whether the optimal clamp multiplier relates to the ratio of high-leverage to low-leverage token positions in the training data, which would make the method tunable rather than ad hoc.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Real problem, useful method, but the mechanism story has a gap between diagnosis and intervention

read the letter

The paper identifies a genuine problem: in multi-teacher on-policy distillation for tool use, vanilla GKD improves tool-call recall but also increases over-calling — the model calls tools on examples that should get direct answers. This shift is invisible from aggregate losses. That's a useful and practically important observation for anyone doing agentic LLM post-training. The diagnostic work ruling out simple aggregate explanations (Table 1, Figure 1) is solid — they show tool-call samples don't get more token exposure or larger full-sequence divergence. Soft Clamp, their fix, is lightweight and the results on APIGen-MT and the BFCL multi-turn diagnostic are consistent and believable. Credit where due: the problem framing and the negative-result diagnostics are the strongest parts. The method works in practice and is easy to adopt. Now the soft spots. The stress-test concern lands. The paper's mechanism story is that behavior leverage imbalance occurs because local signals at mode-entry tokens (like , function names) have disproportionate control over the generation mode. But Soft Clamp compresses ANY token with extreme divergence, regardless of position. The paper never verifies that the extreme divergence values it compresses actually land on mode-entry positions. The diagnostic evidence in §3.2 shows that response-side P(tool) correlates with over-calling — but that measures the student's output probability at the first token, not where the extreme per-token JSD values sit. The authors half-acknowledge this in §4.2 (‘It does not explicitly identify mode-entry tokens’), but this isn't a minor implementation detail; it's the gap between the diagnostic story and the intervention. Soft Clamp could be working through general gradient regularization or variance reduction, with nothing to do with mode-entry tokens specifically. The paper would be stronger if they logged which positions have extreme divergence and checked whether they concentrate at structural tokens. Other issues are smaller but worth noting: single base model (Qwen3.5-9B), no multi-seed variance, and the When2Call results actually show GKD variants doing worse on request-for-information and cannot-answer cases. The hyperparameters for Soft Clamp (k=3.0) and baselines appear untuned or at least the tuning process isn't described. This paper is for practitioners doing multi-teacher distillation for agentic models, and for researchers working on token-level loss calibration. The diagnostic insight (monitor where signals act, not just aggregate magnitude) is valuable even if the causal chain is incomplete. It deserves a serious referee who should push for the position-level analysis that would close the gap between mechanism and intervention.

Referee Report

2 major / 7 minor

Summary. This paper identifies a failure mode in multi-teacher on-policy distillation (MOPD) for agentic tool-use: vanilla generalized knowledge distillation (GKD) improves tool-call recall but simultaneously increases over-calling (calling tools on examples that should be answered directly). The authors show that aggregate statistics (token exposure, full-sequence per-token JSD) do not explain this shift. They propose the concept of 'behavior leverage imbalance'—the idea that local token-level signals at mode-entry positions (e.g., <tool call>, function names) have disproportionate control over the global generation mode. To mitigate this, they introduce Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level JSD while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching decision accuracy. A BFCL multi-turn diagnostic shows lower tool-call loops and repeated calls.

Significance. The paper addresses a practically important problem: tool overuse in agentic LLMs trained via distillation. The observation that aggregate loss statistics can hide behavior-level shifts is valuable and underexplored. The Soft Clamp method is simple, lightweight, and does not require additional infrastructure (learned reward models, teacher routing policies, or held-out calibration sets). The multi-turn loop diagnostic (Appendix D) is a useful practical stress test. The paper is honest about limitations, explicitly acknowledging that the evidence is diagnostic rather than a complete causal proof. The core empirical finding—that vanilla GKD trades over-calling for call recall—is clearly demonstrated across multiple benchmarks.

major comments (2)

§4.2 and §3.2: The central mechanism claim is that behavior leverage imbalance occurs because local signals at mode-entry and structural positions have disproportionate control over the global generation mode. However, Soft Clamp operates on any token whose divergence exceeds the batch-relative threshold C, regardless of where that token sits in the sequence. The paper itself acknowledges: 'It does not explicitly identify mode-entry tokens; instead, it limits the effect of any extreme local divergence' (§4.2). This creates a gap between the diagnostic story (behavior leverage at structural tokens) and the intervention (compressing any extreme divergence). If the extreme divergence values that Soft Clamp compresses are distributed across ordinary content tokens rather than concentrated at mode-entry positions, then the proposed mechanism is misidentified, and Soft Clamp's effectiveness is
Table 4 (BFCL results): All GKD variants underperform the base and base SFT models on overall BFCL scores (e.g., Vanilla GKD 79.8 vs. Base SFT 82.9). The paper frames Soft Clamp as a GKD-internal calibration method, but this regression is significant. The paper should discuss whether the GKD framework itself is counterproductive on BFCL, or whether the format anchor (sft_alpha=0.3) is insufficient to prevent degradation on this benchmark. This affects the practical applicability of the entire MOPD approach, not just Soft Clamp.

minor comments (7)

§2.2: The tool-call target rendering is described as an implementation-specific serialization detail. However, since the entire mechanism story hinges on mode-entry tokens like <tool call>, the specific rendering format used in training is load-bearing for reproducibility. Appendix A.2 gives one example, but the paper should clarify whether the results depend on this specific format.
Table 5 (When2Call): All GKD variants perform worse than base on MCQ accuracy (e.g., Vanilla GKD 64.9 vs. Base 72.8). The paper acknowledges this but does not analyze why distillation degrades out-of-domain decision-making. A brief discussion of whether this is related to over-calling or a separate issue would strengthen the paper.
Figure 1: The y-axis label 'Tool / response ratio' is clear, but the figure would benefit from explicitly marking which direction (above or below 1.0) indicates tool-call dominance. A reader unfamiliar with the ratio may not quickly parse the result.
§4.1, Eq. (3): The notation d_i * C/stopgrad(d_i) is slightly ambiguous on first read. It would help to explicitly state that the forward value equals C (for d_i > C) in the equation caption).
Appendix C.2: The choice of k=3.0 for the clamp multiplier is stated without justification. A brief sensitivity analysis or explanation of why 3.0 is a reasonable default would improve the method's practical guidance.
Table 7: The 'GKD rollout mixture lambda 0.8' and 'GKD beta 0.5' parameters are listed but not explained in the main text. A brief note on what these control would help reproducibility.
§5.5: The BFCL multi-turn diagnostic uses 800 tasks and 3,136 user turns, but the maximum step limit is not specified in the main text (only mentioned as a metric). Appendix D should state the exact step limit value used.

Circularity Check

0 steps flagged

No circularity: empirical study with fixed parameters evaluated on external benchmarks

full rationale

The paper's derivation chain is: (1) observe that vanilla GKD raises over-calling (Table 3, empirical), (2) rule out aggregate explanations via independent measurements (Table 1, token exposure and JSD ratios computed from training logs, not fitted), (3) observe correlation between response-side P(tool) and final over-calling (Table 2, Figure 2, empirical diagnostic), (4) hypothesize behavior leverage imbalance as mechanism, (5) propose Soft Clamp with fixed k=3.0, (6) evaluate on external benchmarks (APIGen-MT, BFCL, When2Call). No step reduces to its inputs by construction. The diagnostic metrics (P(tool) on response samples, top-1 tool rate) are computed from the student's distribution independently of the over-calling metric they are said to correlate with — they measure teacher-forced first-token probability, while over-calling measures free-generation behavior on should-respond examples. Soft Clamp's parameter is fixed, not fitted to the target. No load-bearing self-citation chain exists; the foundational work (GKD by Agarwal et al., MOPD by Ma et al.) is external. The gap between the mechanism story (mode-entry tokens) and the position-agnostic intervention is a validity concern (the skeptic's attack is about mechanism misidentification), not a circularity concern — the paper does not claim to derive the mechanism from the intervention or vice versa.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The axiom ledger captures the key parameters and assumptions. The free parameters are fixed across experiments, not fitted to the target metric, which supports the non-circularity of the method evaluation. The invented entity 'behavior leverage' is supported by correlational evidence, which the authors acknowledge is not a complete causal proof.

free parameters (4)

k (clamp multiplier) = 3.0
Used in Soft Clamp to set the dynamic threshold C = k * mean(d_i). Stated as fixed in Appendix C.2, not fitted to the over-calling metric.
sft_alpha (format anchor weight) = 0.3
Weight for the supervised loss used as a format anchor. Stated as fixed across all GKD variants in Table 7.
c (Hard Clip threshold) = 0.5
Threshold for the Hard Clip baseline. Stated in Appendix C.3.
alpha, z_max, w_min, w_max (Global Reweight params) = 0.3, 3.0, 0.25, 2.0
Parameters for the Global Reweight baseline. Stated in Appendix C.4.

axioms (3)

domain assumption Mode-entry tokens (e.g., <tool call>) have high behavior leverage, meaning changing them redirects the continuation.
Invoked in §2.3 to define the concept of behavior leverage. It is a reasonable domain assumption for autoregressive generation but is not formally proven.
domain assumption The tool-call teacher and response teacher, starting from the same base model and trained on respective target types, serve as valid online supervisors.
Invoked in §2.2 and §5.1. Assumes that this training procedure produces teachers whose distributions are meaningful targets for the student.
domain assumption The BFCL multi-turn loop diagnostic measures usability risk under a simulated environment.
Invoked in §5.2 and §5.5. Assumes the simulated tool environment is representative of real deployment risks.

invented entities (2)

Behavior leverage independent evidence
purpose: Conceptual framework to explain why local token signals can have disproportionate global effects.
The paper provides empirical evidence (Table 2, Figure 2) that metrics at these positions correlate with final behavior, serving as a falsifiable handle.
Soft Clamp independent evidence
purpose: Mitigation method for behavior leverage imbalance.
Evaluated on external benchmarks (APIGen-MT, BFCL) with results that could falsify its utility.

pith-pipeline@v1.1.0-glm · 15171 in / 2431 out tokens · 237245 ms · 2026-07-09T20:56:29.204106+00:00 · methodology

0 comments

read the original abstract

Agentic language models must learn when to call tools, when to consume tool responses, and when to answer directly. This makes multi-teacher on-policy distillation a natural training strategy: one teacher can specialize in tool calls, another in direct responses, and the student can learn from both on its own generated distribution. We show that this strategy can induce a behavior shift that is invisible from aggregate losses alone. In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, where it calls tools on examples that should be answered directly. Aggregate explanations are insufficient: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. We instead analyze behavior leverage imbalance: local token-level signals at mode- entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy. In a BFCL multi-turn diagnostic, it also lowers tool-call loops and repeated calls among GKD variants. These results suggest that multi-teacher OPD should monitor where teacher signals act, not only how large they are in aggregate.