{"id":"86b63a73-4741-4c6e-9d16-19e088413624","arxiv_id":"2510.18383","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A teacher-guided dense reward for GRPO distillation improves small-model tool-use accuracy and out-of-domain generalization on executable-tool benchmarks.","lead":"This paper teaches small language models to use tools by combining reinforcement learning with a dense reward built from a larger teacher model's reference solution. The method improves out-of-domain tool-use performance over supervised fine-tuning and sparse-reward RL on math, retrieval, and function-calling benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward composition selected on same OOD benchmarks used for evaluation; claimed cross-domain gains may reflect test-set tuning, not true generalization.","rationale":"The reader's CONDITIONAL verdict already flags unreported reward weights and selection on OOD test sets in the rationale, but the stated weakest_assumption focuses on the teacher-alignment reward's strict set-match. I identify the test-set selection issue as the more load-bearing concern because it directly threatens the validity of the evidence for the paper's main contribution: cross-domain generalization. If the reward design was chosen after seeing BFCL-v4 and RAG results, the OOD table cannot support the claim that MENTOR's dense reward is generally superior. The concrete test would settle this by decoupling reward design from evaluation. Since the reader's conclusion (CONDITIONAL) already accommodates this concern, the verdict should not change, but the stress test sharpens the condition: the authors must either pre-register the reward configuration or evaluate on untouched OOD benchmarks.","tokens_in":18597,"tokens_out":5223,"duration_ms":47101,"concrete_test":"Hold out BFCL-v4 and the three RAG datasets from all development. Pre-specify a single reward composition and fixed weights, or select weights on a separate math-only validation split. Then run MENTOR and both baselines on the held-out OOD tasks exactly once. If MENTOR's OOD gains over Sparse/SFT shrink or disappear, the selection-bias concern lands. Report at least three training seeds with error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MENTOR improves out-of-domain tool-use generalization rests on BFCL-v4 and RAG results (Table 2). These same benchmarks are used in the ablation study (Table 3) to select Setting 5 as the final reward design; the paper reports no held-out validation split and never lists the reward weights (wc, wa, wv). Thus the OOD numbers are not unbiased estimates of a pre-specified method—they are the best of several configurations chosen by inspecting performance on the evaluation target itself. Because the paper's main novelty is the reward composition, the evidence that this composition is superior is substantially weaker if it was selected by looking at the test set. This is load-bearing: without a validation holdout, the headline cross-domain improvements over SFT and Sparse cannot be distinguished from tuning to the evaluation benchmarks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MENTOR, a reinforcement-learning (GRPO) framework for distilling tool-use skills from a large teacher model (Qwen3-235B) into small student models. Instead of the sparse final-answer reward used in standard RL or the rigid trajectory imitation of SFT, MENTOR uses a dense composite reward (Eq. 4) combining final-answer correctness (Rc), exact set-match with the teacher's tool calls (Ra, Eq. 6), and tool-execution validity (Rv, Eq. 7). The method is trained on 1.27k filtered AceReason-Math trajectories and evaluated on in-domain math benchmarks and out-of-domain tool-use tasks (BFCL-v4 and retrieval QA), with the claim that MENTOR improves OOD generalization over SFT and sparse-reward RL baselines.","tokens_in":18779,"tokens_out":5840,"duration_ms":52419,"significance":"If the results hold, the paper makes a useful contribution to small-model tool-use distillation: it provides a concrete recipe for combining GRPO with a teacher-reference reward, and the idea of using teacher tool-set alignment as a dense signal is intuitive and relevant to a growing literature. The code release and the inclusion of multiple OOD benchmarks (BFCL-v4, HotpotQA, 2WikiMultiHopQA, Bamboogle) are strengths. However, the significance is tempered by methodological concerns: the reward design appears to be selected on the very benchmarks used to evaluate the OOD claim, the training/evaluation is single-seed with small, noisy evaluation sets, and key hyperparameters (the reward weights) are not reported.","major_comments":[{"comment":"The OOD benchmarks BFCL-v4 and RAG are used both as the evaluation target in Table 2 and as the basis for choosing the reward composition in the ablation study in Table 3. The paper reports no held-out validation split and never states the values of wc, wa, wv in Eq. (4). As a result, the headline OOD improvements may reflect tuning of the reward setting to the evaluation benchmarks rather than a pre-specified method. This is load-bearing: please report a validation split used to select Setting 5 and the weights, or present results for a fixed configuration without reference to test-set performance.","section":""},{"comment":"Training is performed with a single seed, only 1.27k trajectories (Appendix A.1), and only 2 epochs (Table 5). Several reported deltas are small, e.g., Qwen3-8B Math 46.22 vs. 45.68 for Sparse and RAG 34.70 vs. 34.00 for Sparse; AIME24/AIME25/amc23 have only 30-40 examples, so one question changes accuracy by ~2.5-3.3 points. Without multiple seeds, confidence intervals, or significance tests, the claim of 'significantly improves' is not supported for these cases. Please provide repeated runs with error bars, and either temper the claims or identify where the advantage is robust.","section":""},{"comment":"The exact set-match reward Ra penalizes any alternative tool combination, even if valid and equally effective for the student. The justification for strictness rests on the ablation in Table 3 and the AS correlation in Fig. 3, both measured on the same OOD benchmarks used to select the final setting. This creates a circularity risk for the OOD generalization claim. As a concrete test, please evaluate the chosen setting against a pre-specified teacher-independent reward and/or on a validation split not used for ablation selection.","section":""},{"comment":"The reward weights wc, wa, wv are never reported anywhere, and the notation in Table 3 is internally inconsistent: Setting (1) is labeled 'R_a (Sparse)' but described as 'only a reward for final answer correctness', which corresponds to R_c in Eq. (5); Setting (5) is written as 'R_a + R_a + R_v (Ours)' but presumably should be R_c + R_a + R_v. This makes the method impossible to reproduce and the ablation difficult to interpret. Please correct the notation and provide the exact weight values.","section":""}],"minor_comments":[{"comment":"Algorithm 1 line 3 generates the reference trajectory inside the per-question loop, whereas Section 3.1 describes pre-generating reference trajectories. Clarify whether the teacher trajectory is generated online or precomputed.","section":""},{"comment":"The alignment score (AS) is defined in Eq. (8), but Figure 3 labels the x-axis 'Tool Distribution Similarity to Teacher.' Specify whether the same 12-tool distribution from Appendix A.4 is used and how it is computed for each model.","section":""},{"comment":"The term 'flexible' in the title and abstract is not reflected in Eq. (6), which is an exact set-match reward. The flexibility appears only in that the sequence of tool calls need not match. Please define 'flexible' explicitly to avoid overstating the design.","section":""},{"comment":"The masking function M used to exclude tool output tokens is never defined. State which tokens are masked and why.","section":""},{"comment":"The hyperparameter table lists 'Number of Rollout 10'; align this with the group size G used in Eq. (3) and Algorithm 1.","section":""},{"comment":"The 'Overall' columns are computed differently for MATH (macro-average) and BFCL-v4 (official weighted average), and the RAG overall appears to be a macro-average of three datasets. Please state these calculation rules in the table caption or text.","section":""}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible, and the Qwen2.5-7B results are promising. However, the current evaluation protocol—reward-setting selection on the same OOD benchmarks, single-seed results with small evaluation sets, and unreported reward weights—prevents the central OOD generalization claim from being taken at face value. The requested fixes (validation split, multiple seeds, reported weights, corrected notation) are within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MENTOR is a plausible and well-motivated answer to a real problem: SFT distillation of tool-use trajectories doesn't generalize, and sparse RL rewards leave small models flailing. The idea of using a dense teacher-guided reward with three components — final-answer correctness, exact tool-set match, and tool-execution validity — is sensible, and the authors show through ablations that each piece contributes and that strict set-matching beats a flexible F1 version. The evaluation across four SLMs on out-of-domain BFCL and RAG tasks is a plus, and the tool-invocation analyses (Figures 3–5) support the claim that RL-trained students adopt teacher-like calling patterns.\n\nThat said, the central claim of OOD generalization is weaker than the paper makes it look. The ablation in Table 3 is run on the same BFCL-v4 and RAG benchmarks used for the headline results, with no held-out validation split. The reward weights wc, wa, wv are never reported. So Setting 5, the MENTOR reward, was effectively chosen after looking at the OOD test sets. That means the reported gains over SFT and Sparse cannot be cleanly distinguished from tuning the reward to the evaluation benchmarks. This is the load-bearing concern, and it's not addressed.\n\nThere are also smaller but real issues: a single seed, only two epochs on 1.27k trajectories, no confidence intervals, and several deltas under two EM points despite the word 'significant' in the abstract. Reproducibility is shaky: the abstract points to one GitHub repo, the appendix points to an anonymous OpenReview-style URL, and the training set isn't released, so exact reproduction requires rerunning a 235B teacher.\n\nThe binary teacher-alignment reward Ra is rough — it pays 1 only for an exact set match, which can penalize valid alternative tool strategies — but the ablation shows strictness wins empirically in this setting, so I'd call that a design choice rather than a fatal flaw.\n\nWho should read this: people working on tool-use distillation or reward design for SLM RL. It's a solid empirical recipe, not a breakthrough. The paper deserves a serious referee. For that, the authors need to publish the reward weights, hold out a validation split for any ablation- or weight-selection, add repeated-seed variance, and fix the code links. If those are done, this could be a useful and citable contribution; as it stands, it's a conditional accept.\n\nRecommendation: don't desk-reject. Send it to review, but ask for those changes.","headline":"Solid RL-distillation recipe, but the OOD gains are undercut by reward selection on the test set and unreported weights.","tokens_in":19358,"tokens_out":3221,"would_cite":true,"duration_ms":26594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that dense, teacher-guided rewards train small language models to use tools more effectively and generalize further than supervised imitation or sparse-reward RL.","keywords":["tool-use distillation","small language models","reinforcement learning","teacher-guided reward","group relative policy optimization","out-of-domain generalization","reward design","tool invocation"],"falsifier":"Train the same setup with a teacher whose tool selections are deliberately suboptimal (for example, always calling one expensive tool); if MENTOR students then perform no better than sparse-RL or SFT students, the gains come from the teacher's specific choices rather than from the dense reward structure. Alternatively, compute the fraction of correct-answer rollouts that Ra penalizes because the student used a different but valid tool set; a high fraction would show the reward is suppressing valid strategies.","tokens_in":18416,"feed_emoji":"🔧","tokens_out":6975,"duration_ms":56416,"temperature":0.7,"pith_summary":"SFT teaches a small model to imitate a fixed teacher trajectory, which stops working when the task shifts. Sparse-reward RL lets the student explore but gives too little signal for a small model to find good tool-use strategies. MENTOR's claim is that a dense reward built from the teacher's reference - final-answer agreement, exact tool-set match, and error-free execution - gives the student enough guidance to learn a transferable tool-use policy. On math benchmarks and, more important, on out-of-domain tool-calling and retrieval tasks, models trained this way outperform both SFT and sparse-reward RL at the same scale. A distinctive part of the argument is that strict exact-set alignment with the teacher beats a flexible F1-based match, so the paper argues unambiguous signals matter more than permissive ones.","feed_headline":"Teacher-guided rewards lift small-model tool use beyond SFT","feed_subtitle":"A composite reward - correctness, exact tool-set match, validity - generalizes to unfamiliar tools where imitation training stalls.","key_machinery":"The composite teacher-guided reward from Eq. 4: R(O(s), O(t)) = wc*Rc + wa*Ra + wv*Rv, where Rc = 1 if the student's final answer equals the teacher's answer, Ra = 1 only when the student's set of tool calls exactly equals the teacher's set, and Rv = 1 only when every tool call in the student's trajectory executes without error. In GRPO, groups of student rollouts are scored with this dense signal, groupwise advantages are computed, and tool-output tokens are masked out of the loss so the student learns the strategy rather than memorizing the formatting.","core_discovery":"On its own terms, MENTOR's discovery is that a dense composite reward - Rc for matching the teacher's final answer, Ra for calling exactly the same set of tools as the teacher, and Rv for executing every tool call without error - provides the fine-grained guidance that small models need when learning tool use, and that this guidance transfers. Across four student models from 1.5B to 8B parameters, MENTOR outperforms SFT and sparse-reward RL on in-domain math (for example, the 7B student reaches 27.88% overall accuracy vs 22.75% after SFT and 25.35% after sparse RL) and, more strikingly, on out-of-domain retrieval QA (21.23% exact match vs 13.70% and 18.13%) and function-calling benchmarks. A","pith_inferences":["If the teacher's tool choices are mediocre, the strict set-match reward would actively discourage the student from finding better alternatives; a direct test is to train with teachers of varying tool-use skill and check whether student performance tracks teacher skill.","The exact-set reward may succeed mainly because it provides a low-variance, unambiguous training signal rather than because the teacher's particular choice of tools is optimal; ablating with a random but consistent teacher tool policy could separate these explanations.","The same reward structure should extend to other verifiable tool environments, such as web search or API calls; a natural next test is a browser-based agent benchmark where teacher trajectories are noisier and where exact-set alignment may be too brittle."],"forward_implications":["Small models can acquire tool-use policies that transfer to tools unseen in training, without the need to curate teacher trajectories for every possible scenario.","A strict teacher-alignment reward is more effective than a flexible F1-based one in a verifiable tool environment.","Adding a tool-validation reward drives invalid tool-call rates to near zero during training, while a teacher-alignment reward prevents the student from drifting into a tool-avoidant policy.","RL-based distillation is a viable alternative to SFT for transferring strategic tool use to small models, especially where the deployment domain is not covered by training trajectories."],"fun_headline_variants":["Flexible teacher rewards beat SFT for tool-use distillation","Teacher rewards, not imitation, boost small-model tool use","MENTOR: flexible rewards for small-model tool mastery","Teacher-optimized rewards help small models use new tools","Flexible teacher rewards beat strict imitation for tool learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the teacher's exact set of tool calls is a reliable guide: the reward pays in full only when the student's tool set exactly equals the teacher's, so any valid alternative strategy is penalized.","fun_headline_variants_meta":{"raw":{"variants":["Flexible teacher rewards beat SFT for tool-use distillation","Teacher rewards, not imitation, boost small-model tool use","MENTOR: flexible rewards for small-model tool mastery","Teacher-optimized rewards help small models use new tools","Flexible teacher rewards beat strict imitation for tool learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000756,"raw_usage":{"total_tokens":3200,"prompt_tokens":748,"completion_tokens":2452,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":2373}},"tokens_in":492,"tokens_out":2452,"duration_ms":13702,"temperature":1.0,"reasoning_tokens":2373,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:50:27.128385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same setup with a teacher whose tool selections are deliberately suboptimal (for example, always calling one expensive tool); if MENTOR students then perform no better than sparse-RL or SFT students, the gains come from the teacher's specific choices rather than from the dense reward structure. Alternatively, compute the fraction of correct-answer rollouts that Ra penalizes because the student used a different but valid tool set; a high fraction would show the reward is suppressing valid strategies.","supporting_citations":[],"review_version":1}