REVIEW 3 major objections 4 minor 29 references
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read CREST claims that a self-teacher should set update magnitudes while the verifier sets update directions, and that this split unlocks dense token-level credit assignment for multi-turn tool-use agents without losing the verifier-bounded…
desk verdict The verifier-bounded ceiling claim overreaches the token-wise proof, and the single-seed numbers need support, but the sign/magnitude decomposition is a fresh, buildable idea that deserves refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the factored per-token advantage $$A_t = $A^{{\mathrm{turn}}$}_{[t]}\,\phi_t, \qquad \phi_t = 1 + \$lambda^{{\mathrm{eff}}$}_t\,(w_t - 1),$$ where $A^{\mathrm{turn}}_{[t]}$ is a group-relative normalized verified reward computed independently within each turn, and $\phi_t$ is a positive multiplicative factor built from a teacher--student divergence $\Delta_t$, clipped into $w_t$, then gated by a direction gate that suppresses teacher--verifier disagreement and an entropy gate that concentrates amplification on high-uncertainty content tokens. Because $\phi_t > 0$ always, the scalar sign of every token's update is identical to the sign of the turn-segmented verified advantage; the teacher can only enlarge an update along a verifier-approved direction. The direction gate is the mechanism that keeps the method verifier-bounded rather than teacher-bounded, while the entropy gate prevents the gradient-concentration collapse observed in plain self-distillation.
What would settle it
Run CREST on a multi-turn benchmark where only session-level success is observable, or where rollouts within a group have different turn counts and orders; if the method then no longer beats GRPO and MT-GRPO on the held-out splits, the turn-alignment assumption fails. A second check is to match the OPSD setup and measure the top-5% token gradient share: if CREST's share approaches OPSD's roughly 77% instead of the reported middle ground, the entropy gate is not preventing concentration collapse.
Extended reading notes
Core claim
CREST's central claim is that dense token-level supervision from a privileged self-teacher can be added to verifier-based policy optimization by splitting the teacher's role: verified rewards set the sign of every per-token gradient, while the self-teacher only modulates its magnitude. A turn-segmented, group-relative advantage $A^{\mathrm{turn}}_{[t]}$ isolates each turn's own outcome, and an entropy-gated teacher factor $\phi_t$ reweights tokens inside the turn without ever changing the sign of $A^{\mathrm{turn}}_{[t]}$. The paper reports that this combination reaches 52.0% average accuracy on BFCL V3 with Qwen3-4B and 9.38% session accuracy on WildToolBench with Qwen3-8B, outperforming all RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics.
Load-bearing premise
The load-bearing premise is that per-turn verifier rewards exist and are semantically aligned across the rollouts of a training group, meaning all rollouts share the same fixed session and ordered user turns so that turn $k$ denotes the same benchmark request; if rewards are terminal-only or turn structures vary, the turn-segmented advantage no longer isolates one turn's credit and the mechanism collapses.
Editorial extensions
If this is right
- CREST's direction gate guarantees, per token, that the update sign equals the sign of the turn-segmented verified advantage, so a privileged teacher can be used without replacing the verifier as the optimization target.
- The largest measured gains appear on the splits where credit dilution is worst, namely BFCL V3 Long Context and WildToolBench session accuracy, indicating that the method targets the failure regime that limits multi-turn agent training.
- Ablations show that inter-turn and intra-turn credit are complementary: neither turn segmentation alone nor teacher modulation alone reaches the combined result, and removing both gates drops performance back to GRPO-level accuracy.
- With default settings $\lambda=0.3$, $\epsilon=0.28$, the teacher's per-token amplification is bounded by 8.4% of the inter-turn per-token gradient norm, making the teacher's influence a small, controlled perturbation.
- On-policy distillation baselines plateau below RL baselines in these experiments, and CREST surpasses the distillation ceiling by step 20 on the reported training-accuracy curve, consistent with the claim that its direction gate decouples teacher magnitude from verifier direction.
Reading between the lines
- The same magnitude-only principle may extend to other hierarchical generation tasks, such as multi-hop retrieval, collaborative dialogue, or long-horizon planning, whenever a local verifier can score each segment; the paper names these as future directions but does not test them.
- A testable variant is an adaptive or scheduled $\lambda$: the paper fixes $\lambda=0.3$ and reports that $\lambda=0.5$ over-concentrates gradient, so decaying teacher influence across training could improve later-step stability.
- Because the entropy gate uses token surprisal as a proxy for importance, replacing it with a learned token-importance estimator while keeping the sign-preserving gates might sharpen the teacher's focus; this is an open variant, not a claim of the paper.
- The formal sign-preservation guarantee is token-wise rather than aggregate: reweighting changes how token gradients cancel in the sum, so 'verifier-bounded' should be read as a ceiling on each token's scalar direction rather than a guarantee that the final stationary point is verifier-determined.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CREST, a hierarchical credit-assignment method for training multi-turn, multi-step LLM tool-use agents. The method replaces trajectory-level advantages with per-turn group-relative advantages computed from verifier rewards, and multiplies each token's advantage by an entropy-gated, self-teacher-derived magnitude factor. The authors claim that this decomposition keeps gradient directions determined by the verifier while letting a privileged self-teacher modulate only per-token magnitudes, and they report experiments on BFCL V3 and WildToolBench with Qwen3-4B and Qwen3-8B, together with ablations and a formal appendix.
Significance. If the claims were fully supported, the paper would make a useful practical contribution: the turn-segmented advantage is a simple and plausible fix for inter-turn reward dilution, and the entropy-gated modulation is a sensible guard against gradient concentration collapse. The paper is clearly written and the appendix is unusually candid, explicitly stating that the formal guarantee is token-wise scalar-sign preservation and that the perturbation bound is absolute, not relative to the aggregate gradient. The two-level ablation is a helpful design analysis. However, the headline conceptual claim, that the teacher is restricted to magnitude modulation without sacrificing a verifier-bounded ceiling, is not established beyond the token-wise level, and the main empirical tables rest on single-seed runs with no variance estimates. The contribution is therefore currently overstated relative to the evidence, but the underlying approach remains plausible and the issues appear addressable in revision.
major comments (3)
- [§3.4 and Appendix A.2] The main text's statement of property P2 is not supported by the proof. Section 3.4 states that ||Bias|| ≤ λε·||∇J_GRPO|| ≈ 8.4%, which is a relative bound on the aggregate GRPO gradient. Appendix A.2 proves only the absolute bound ||Σ_t (φ_t−1)g_t|| ≤ λε Σ_t ||g_t||. The latter does not imply the former because Σ_t ||g_t|| can be much larger than ||Σ_t g_t|| when token gradients partially cancel. The proof itself is correct, but the main-text formulation and the derived '8.4%' claim must be corrected, and all downstream statements that rely on the relative interpretation should be revised.
- [Abstract, §5, Appendix A.1, A.4] The central claim that CREST 'preserves the verifier-bounded ceiling' overreaches what is proved. Proposition A.1 establishes only that sign(A_t) = sign(A_turn[t]) for each token individually, because φ_t > 0. Appendix A.1 and A.4 explicitly state that reweighting can change how token gradients cancel in the aggregate and that the result does not imply preservation of aggregate gradient directions, stationary points, or a global performance ceiling. The abstract and Section 5 nevertheless assert that the verifier-bounded ceiling is preserved 'by construction' and that the teacher's role is reduced to magnitude modulation. This mismatch between the formal content and the framing is load-bearing because the paper's title and motivation rest on the verifier-bounded-ceiling claim. The authors should either prove an aggregate-level guarantee or reframe the claim as a token-wise anchor plus a heuristic, and adjust the abstract and conclusion accordingly.
- [Table 2 and §4.1] The empirical evidence for the central 'consistently outperforms all baselines' claim is weakened by the single-training-seed protocol. Section 4.1 states that each method uses a single training seed and that the three evaluation decodes are not independent training replicates; no standard errors or multiple-seed results are reported. Several margins in Table 2 are small relative to what would be expected from training noise, for example Qwen3-8B WildToolBench Task Accuracy (CREST 52.34 vs MT-GRPO 49.61) and Qwen3-4B BFCL Base (CREST 67.0 vs MT-GRPO 63.0). Variance estimates or additional seeds are needed to support the strength of the empirical claim. Additionally, the Qwen3-4B CREST row in Table 2 reports an Average of 52.00, but the four split values (67, 48, 38, 60) have arithmetic mean 53.25, and the reported improvement +29.88 does not match the column values; because the abstract quotes 52.0%, this numerical inconsistency must be resolved.
minor comments (4)
- [Title and §1] The title contains a typo, 'Multi-T urn', which should read 'Multi-Turn'; the same issue appears in the first line of the full text.
- [§3.4] The symbol 'Bias' in the P2 statement is used before it is defined; the authors should define it as the teacher-induced perturbation D = Σ_t (φ_t−1)g_t relative to the inter-turn-only gradient.
- [Appendix B / C.1] Algorithm 1 and Section C.1 describe the self-teacher as conditioning on ground-truth tool-call results, but the construction of the privileged context h^T_t is not fully specified; a concrete description of how the teacher context is assembled for each turn would improve reproducibility.
- [Appendix C.2] The sentence 'we adopt the fixed teacher with per-token KL clipping (threshold 0.2) for the OPSD baseline' is clear, but it would help to state explicitly that this fixed-teacher variant is the one plotted as the OPSD curve in Figure 3(a), since the caption alone does not say so.
Circularity Check
No circularity: CREST's formal properties follow by construction from its definitions, and its empirical claims rest on independent benchmark rollouts, not on fitted inputs or load-bearing self-citations.
full rationale
The paper's formal properties are consequences of its own definitions rather than hidden imports. P1 (sign(A_t)=sign(A_turn)) follows because phi_t >= 1 whenever the direction gate is active and phi_t = 1 otherwise (Proposition A.1), and the appendix explicitly limits this to a 'local, token-wise anchoring property,' stating that 'reweighting can still change how token gradients cancel in the aggregate.' P2 is proved as an absolute bound on the sum of per-token gradient norms (Proposition A.2), and P3 is a direct consequence of the gating definitions. None of these steps fits a parameter to a target result or defines an input in terms of an output. The empirical gains in Section 4 come from rollout comparisons on BFCL V3 and WildToolBench against independently specified baselines; lambda is the only tuned hyperparameter and is selected by model selection, not by solving for the reported results. The one self-citation (Lu et al. 2026a, EnvTuning) appears only as a baseline and in a future-work remark, so it is not load-bearing for the central claim. The main text's '8.4%' and 'verifier-bounded ceiling' phrasings overstate what Appendix A proves, but overclaiming a theorem's strength is a correctness and scope issue, not circularity. Accordingly, no circular step can be exhibited, and the derivation is self-contained.
Assumptions & free parameters
free parameters (2)
- lambda (modulation strength) =
0.3
- fixed constants (epsilon, tau, rho) =
epsilon=0.28, tau=2.0, rho=0.5
assumptions (4)
- domain assumption Per-turn verifiable rewards are available and semantically aligned across rollouts, so turn k denotes the same benchmark request for every rollout in the group.
- domain assumption Student surprisal is a sufficient proxy for token importance in tool-use trajectories.
- domain assumption A privileged self-teacher conditioned on ground truth provides useful token-level supervision to the student.
- standard math Standard policy-gradient optimization with group-relative normalization is a valid framework.
Cite this review
Pith. "Pith review of Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents." pith.science (2026). https://pith.science/paper/YWVNOHHJ
@misc{pith2026260813179,
author = {Pith},
title = {Pith review of: Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/YWVNOHHJ}},
note = {Machine review of arXiv:2608.13179}
}
abstract
Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
Are: Scaling up agent environments and evaluations.arXiv preprint arXiv:2509.17158,
Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, et al. Are: Scaling up agent environments and evaluations.arXiv preprint arXiv:2509.17158,
-
[5]
Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,
-
[6]
Gpt-4o system card.arXiv preprint arXiv:2410.21276,
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,
-
[7]
Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pp
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pp. 54107–54157,
2024
-
[8]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,
-
[9]
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472,
-
[10]
Llms get lost in multi-turn conversation
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120,
-
[11]
Api-bank: A comprehensive benchmark for tool-augmented llms
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. InProceedings of the 2023 conference on empirical methods in natural language processing, pp. 3102–3116,
2023
Show all 29 references
-
[13]
Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,
Yaxuan Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,
-
[14]
Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,
-
[15]
Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,
-
[16]
Siyuan Lu, Zechuan Wang, Hongxuan Zhang, Qintong Wu, Leilei Gan, Chenyi Zhuang, Jinjie Gu, and Tao Lin
Blog post. Siyuan Lu, Zechuan Wang, Hongxuan Zhang, Qintong Wu, Leilei Gan, Chenyi Zhuang, Jinjie Gu, and Tao Lin. Don’t just fine-tune the agent, tune the environment. InThe Fourteenth International Conference on Learning Representations, 2026a. URLhttps://openreview.net/foru...
-
[17]
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez
Open-source personal AI assistant, version 2026.3.8, accessed 2026-03-09. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of ...
2026
-
[18]
Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,
Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,
-
[19]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
-
[21]
Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026a
Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026a. Idan Shenfeld et al. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026b. Mohit Shridhar, Xingd...
-
[23]
Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073,
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073,
-
[24]
Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025a
11 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecom...
-
[25]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[26]
Self-distilled RLVR.arXiv preprint arXiv:2604.03128,
Chenxu Yang et al. Self-distilled RLVR.arXiv preprint arXiv:2604.03128,
-
[27]
Benchmarking LLM tool-use in the wild
Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, and feng zhang. Benchmarking LLM tool-use in the wild. InThe Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview.net/forum?id=yz7fL5vfpn. Qiying Yu, Zheng Zhang, Ruo...
-
[28]
Deep- researcher: Scaling deep research via reinforcement learning in real-world environments
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep- researcher: Scaling deep research via reinforcement learning in real-world environments. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processi...
2025
-
[29]
to multi-turn tool-use trajectories results in training collapse within∼20 steps, consistent with findings reported by the original authors. To establish a fair OPSD baseline, we investigated three teacher update strategies (Figure 4(a)): • Online teacher(updated with policy):...
2026
-
[2017]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2021]
Mingyang Song and Mao Zheng
URLhttps://openreview.net/forum?id=0IOX0YcCdTn. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,
-
[2023]
Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383,
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383,
-
[2024]
On SFT, RL, and on-policy distillation
Will Brown. On SFT, RL, and on-policy distillation. https://x.com/willccbb/status/2050038277454143918,
-
[2025]
Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536,
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536,
-
[2026]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al
Blog post. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agen...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.