Pith. sign in

REVIEW 4 major objections 6 minor 64 references

This paper shows that dense, teacher-guided rewards train small language models to use tools more effectively and generalize further than supervised imitation or sparse-reward RL.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:50 UTC pith:2N3GJTI6

load-bearing objection Solid RL-distillation recipe, but the OOD gains are undercut by reward selection on the test set and unreported weights. the 4 major comments →

arxiv 2510.18383 v3 pith:2N3GJTI6 submitted 2025-10-21 cs.CL cs.AI

MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation

ChangSu Choi , Hoyun Song , Dongyeon Kim , WooHyeon Jung , Minkyung Cho , Sunjin Park , NohHyeob Bae , Seona Yu
show 1 more author
KyungTae Lim
This is my paper
classification cs.CL cs.AI
keywords tool-use distillationsmall language modelsreinforcement learningteacher-guided rewardgroup relative policy optimizationout-of-domain generalizationreward designtool invocation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SFT teaches a small model to imitate a fixed teacher trajectory, which stops working when the task shifts. Sparse-reward RL lets the student explore but gives too little signal for a small model to find good tool-use strategies. MENTOR's claim is that a dense reward built from the teacher's reference - final-answer agreement, exact tool-set match, and error-free execution - gives the student enough guidance to learn a transferable tool-use policy. On math benchmarks and, more important, on out-of-domain tool-calling and retrieval tasks, models trained this way outperform both SFT and sparse-reward RL at the same scale. A distinctive part of the argument is that strict exact-set alignment with the teacher beats a flexible F1-based match, so the paper argues unambiguous signals matter more than permissive ones.

Core claim

On its own terms, MENTOR's discovery is that a dense composite reward - Rc for matching the teacher's final answer, Ra for calling exactly the same set of tools as the teacher, and Rv for executing every tool call without error - provides the fine-grained guidance that small models need when learning tool use, and that this guidance transfers. Across four student models from 1.5B to 8B parameters, MENTOR outperforms SFT and sparse-reward RL on in-domain math (for example, the 7B student reaches 27.88% overall accuracy vs 22.75% after SFT and 25.35% after sparse RL) and, more strikingly, on out-of-domain retrieval QA (21.23% exact match vs 13.70% and 18.13%) and function-calling benchmarks. A

What carries the argument

The composite teacher-guided reward from Eq. 4: R(O(s), O(t)) = wc*Rc + wa*Ra + wv*Rv, where Rc = 1 if the student's final answer equals the teacher's answer, Ra = 1 only when the student's set of tool calls exactly equals the teacher's set, and Rv = 1 only when every tool call in the student's trajectory executes without error. In GRPO, groups of student rollouts are scored with this dense signal, groupwise advantages are computed, and tool-output tokens are masked out of the loss so the student learns the strategy rather than memorizing the formatting.

Load-bearing premise

The load-bearing premise is that the teacher's exact set of tool calls is a reliable guide: the reward pays in full only when the student's tool set exactly equals the teacher's, so any valid alternative strategy is penalized.

What would settle it

Train the same setup with a teacher whose tool selections are deliberately suboptimal (for example, always calling one expensive tool); if MENTOR students then perform no better than sparse-RL or SFT students, the gains come from the teacher's specific choices rather than from the dense reward structure. Alternatively, compute the fraction of correct-answer rollouts that Ra penalizes because the student used a different but valid tool set; a high fraction would show the reward is suppressing valid strategies.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Small models can acquire tool-use policies that transfer to tools unseen in training, without the need to curate teacher trajectories for every possible scenario.
  • A strict teacher-alignment reward is more effective than a flexible F1-based one in a verifiable tool environment.
  • Adding a tool-validation reward drives invalid tool-call rates to near zero during training, while a teacher-alignment reward prevents the student from drifting into a tool-avoidant policy.
  • RL-based distillation is a viable alternative to SFT for transferring strategic tool use to small models, especially where the deployment domain is not covered by training trajectories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the teacher's tool choices are mediocre, the strict set-match reward would actively discourage the student from finding better alternatives; a direct test is to train with teachers of varying tool-use skill and check whether student performance tracks teacher skill.
  • The exact-set reward may succeed mainly because it provides a low-variance, unambiguous training signal rather than because the teacher's particular choice of tools is optimal; ablating with a random but consistent teacher tool policy could separate these explanations.
  • The same reward structure should extend to other verifiable tool environments, such as web search or API calls; a natural next test is a browser-based agent benchmark where teacher trajectories are noisier and where exact-set alignment may be too brittle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MENTOR, a reinforcement-learning (GRPO) framework for distilling tool-use skills from a large teacher model (Qwen3-235B) into small student models. Instead of the sparse final-answer reward used in standard RL or the rigid trajectory imitation of SFT, MENTOR uses a dense composite reward (Eq. 4) combining final-answer correctness (Rc), exact set-match with the teacher's tool calls (Ra, Eq. 6), and tool-execution validity (Rv, Eq. 7). The method is trained on 1.27k filtered AceReason-Math trajectories and evaluated on in-domain math benchmarks and out-of-domain tool-use tasks (BFCL-v4 and retrieval QA), with the claim that MENTOR improves OOD generalization over SFT and sparse-reward RL baselines.

Significance. If the results hold, the paper makes a useful contribution to small-model tool-use distillation: it provides a concrete recipe for combining GRPO with a teacher-reference reward, and the idea of using teacher tool-set alignment as a dense signal is intuitive and relevant to a growing literature. The code release and the inclusion of multiple OOD benchmarks (BFCL-v4, HotpotQA, 2WikiMultiHopQA, Bamboogle) are strengths. However, the significance is tempered by methodological concerns: the reward design appears to be selected on the very benchmarks used to evaluate the OOD claim, the training/evaluation is single-seed with small, noisy evaluation sets, and key hyperparameters (the reward weights) are not reported.

major comments (4)
  1. The OOD benchmarks BFCL-v4 and RAG are used both as the evaluation target in Table 2 and as the basis for choosing the reward composition in the ablation study in Table 3. The paper reports no held-out validation split and never states the values of wc, wa, wv in Eq. (4). As a result, the headline OOD improvements may reflect tuning of the reward setting to the evaluation benchmarks rather than a pre-specified method. This is load-bearing: please report a validation split used to select Setting 5 and the weights, or present results for a fixed configuration without reference to test-set performance.
  2. Training is performed with a single seed, only 1.27k trajectories (Appendix A.1), and only 2 epochs (Table 5). Several reported deltas are small, e.g., Qwen3-8B Math 46.22 vs. 45.68 for Sparse and RAG 34.70 vs. 34.00 for Sparse; AIME24/AIME25/amc23 have only 30-40 examples, so one question changes accuracy by ~2.5-3.3 points. Without multiple seeds, confidence intervals, or significance tests, the claim of 'significantly improves' is not supported for these cases. Please provide repeated runs with error bars, and either temper the claims or identify where the advantage is robust.
  3. The exact set-match reward Ra penalizes any alternative tool combination, even if valid and equally effective for the student. The justification for strictness rests on the ablation in Table 3 and the AS correlation in Fig. 3, both measured on the same OOD benchmarks used to select the final setting. This creates a circularity risk for the OOD generalization claim. As a concrete test, please evaluate the chosen setting against a pre-specified teacher-independent reward and/or on a validation split not used for ablation selection.
  4. The reward weights wc, wa, wv are never reported anywhere, and the notation in Table 3 is internally inconsistent: Setting (1) is labeled 'R_a (Sparse)' but described as 'only a reward for final answer correctness', which corresponds to R_c in Eq. (5); Setting (5) is written as 'R_a + R_a + R_v (Ours)' but presumably should be R_c + R_a + R_v. This makes the method impossible to reproduce and the ablation difficult to interpret. Please correct the notation and provide the exact weight values.
minor comments (6)
  1. Algorithm 1 line 3 generates the reference trajectory inside the per-question loop, whereas Section 3.1 describes pre-generating reference trajectories. Clarify whether the teacher trajectory is generated online or precomputed.
  2. The alignment score (AS) is defined in Eq. (8), but Figure 3 labels the x-axis 'Tool Distribution Similarity to Teacher.' Specify whether the same 12-tool distribution from Appendix A.4 is used and how it is computed for each model.
  3. The term 'flexible' in the title and abstract is not reflected in Eq. (6), which is an exact set-match reward. The flexibility appears only in that the sequence of tool calls need not match. Please define 'flexible' explicitly to avoid overstating the design.
  4. The masking function M used to exclude tool output tokens is never defined. State which tokens are masked and why.
  5. The hyperparameter table lists 'Number of Rollout 10'; align this with the group size G used in Eq. (3) and Algorithm 1.
  6. The 'Overall' columns are computed differently for MATH (macro-average) and BFCL-v4 (official weighted average), and the RAG overall appears to be a macro-average of three datasets. Please state these calculation rules in the table caption or text.

Circularity Check

1 steps flagged

Minor self-referentiality in the alignment metric; the central training/evaluation chain is otherwise externally anchored and not circular.

specific steps
  1. self definitional [Section 3.3 (Eq. 6); Section 4.1 and Appendix A.4 (Eq. 8)]
    "The reward is assigned only if the set of tool calls made in the student’s trajectory, τ(s), is identical to the set of tool calls in the teacher’s trajectory, τ(t). ... We also quantify policy alignment using an alignment score (AS) based on the Jensen-Shannon divergence. This score measures the divergence between a model’s tool usage distribution and the teacher’s reference distribution from Figure 1."

    AS is a distributional relaxation of the exact tool-set match that the teacher-alignment reward Ra (Eq. 6) explicitly optimizes. The paper then presents high AS (Figures 3-4) as evidence that MENTOR 'internalizes the teacher's strategy' and transfers it OOD. Since both the reward and the evaluation metric share the same reference — the teacher's tool-use distribution — the observed alignment is in part an expected byproduct of the training objective, not an independent confirmation of strategic superiority.

full rationale

The core reward (Eq. 4-7) is anchored to an external teacher (Qwen3-235B) and to ground-truth answers, not to the student's own outputs; the training set is AceReason-Math, and the main math benchmarks are held out. No load-bearing self-citation or imported uniqueness theorem appears. The only circular element is the AS-based analysis, which is a mild self-referentiality rather than a central prediction. A separate validation concern — the ablation in Table 3 uses the same BFCL/RAG benchmarks later reported as the main OOD evidence, and no reward weights are given — is a test-set selection / reproducibility issue, but it does not make the reward construction itself definitional or equivalent to its inputs; hence the low circularity score.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method introduces no new entities (no new forces, particles, or mechanistic objects). Free parameters are mostly hyperparameters that are not reported with sensitivity analysis; the principal assumption is that teacher tool-set choice is a good training target. The few fitted values (weights, temperature, group size, filter size) are standard RL knobs rather than exotic inventions.

free parameters (6)
  • weight wc for correctness reward = implied equal weighting; exact values not reported in main text
    Eq. 4 combines Rc, Ra, Rv with weights wc, wa, wv; the paper never reports the weight values, and ablations use unweighted (implied) settings. The central result depends on this balance, so it is a free parameter.
  • weight wa for teacher-alignment reward = not reported
    Eq. 4; not specified in main text or hyperparameter tables.
  • weight wv for tool-validation reward = not reported
    Eq. 4; not specified.
  • rollout temperature (Qwen3 0.6 / Qwen2.5 0.7) = 0.6 / 0.7
    Table 5: sampling temperature during rollouts affects exploration and hence which trajectories get rewarded; chosen by hand per model family.
  • group size G=10 rollouts = 10
    Table 5: number of rollouts per question for GRPO advantage computation; hand-chosen hyperparameter.
  • 1.27k trajectory training-set filter = 1.27k successful trajectories
    Appendix A.1: the training set is the subset of AceReason-Math where Qwen3-235B's answer matched the ground truth; the size of this filtered set is a data-selection choice that shapes the reward signal.
axioms (5)
  • domain assumption The teacher's reference trajectory is a reliable exemplar of an effective tool-use strategy, so rewarding exact tool-set match transfers genuine capability.
    Central to Eq. 6 (teacher-alignment reward); the paper validates it only via correlation (Fig. 3) and the ablation's strict-vs-flexible comparison on a single benchmark battery.
  • domain assumption Ground-truth answers in AceReason-Math and the evaluation benchmarks are correct and the teacher's matching-answer trajectories are therefore 'successful'.
    Appendix A.1 filters on answer match to ground truth; benchmark correctness is assumed.
  • standard math GRPO with the given loss, KL coefficient (0.001), clip ratio (0.2), and LoRA-based SFT for baselines is an appropriate optimization setup.
    Uses the standard GRPO objective (Eq. 3, from Shao et al. 2024) with standard hyperparameters; not an ad-hoc invention.
  • domain assumption Exact-match (EM) accuracy with answer normalization is a meaningful measure of tool-use capability on these benchmarks.
    Section 4.1 defines accuracy as EM after normalization; this is a standard but assumption-laden metric for multi-hop QA and math.
  • domain assumption The retrieval environment (FlashRAG + E5-base-v2 + Dec 2018 Wikipedia snapshot, top-5) provides a faithful proxy for a real search tool.
    Section 4.1 Implementation Details; OOD generalization claims for 'search' rest on this proxy.

pith-pipeline@v1.3.0-alltime-deepseek · 18153 in / 9615 out tokens · 62304 ms · 2026-08-04T08:50:27.128385+00:00 · methodology

0 comments
read the original abstract

Distilling the tool-use capabilities of large language models (LLMs) into small language models (SLMs) is essential for their practical application. The predominant approach, supervised fine-tuning (SFT), suffers from poor out-of-domain (OOD) generalization due to its rigid alignment with static teacher trajectories. While reinforcement learning (RL) offers an alternative, the capacity limitations of SLMs pose a severe dilemma: sparse outcome rewards provide insufficient guidance, whereas strict trajectory matching imposes overly restrictive constraints. To bridge this capacity-driven gap, we propose MENTOR, which introduces a flexible yet process-aware reward structure. Instead of enforcing rigid replication, MENTOR uses the teacher's reference to guide tool-use behavior, balancing behavioral alignment with downstream performance. Extensive experiments on controlled executable-tool benchmarks demonstrate that MENTOR improves OOD tool-use performance compared to SFT and strict RL baselines. Our findings suggest that within verifiable tool-use environments, flexible tool-use alignment offers a more effective approach than strict trajectory replication for developing adaptable small models.

Figures

Figures reproduced from arXiv: 2510.18383 by Changsu Choi, Dongyeon Kim, Hoyun Song, KyungTae Lim, MinKyung Cho, NohHyeob Bae, Seona Yu, Sunjin Park, WooHyeon Jung.

Figure 1
Figure 1. Figure 1: Comparative analysis of the tool effect on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MENTOR training framework. A problem-solving trajectory (τ ) consists of a sequence of Reasoning (R), Tool-Use (T), and Observation (O), and a final Answer (A). Each student rollout is evaluated by a set of reward models, which generate a reward signal by aligning the student’s actions against the teacher’s reference trajectory. imitation-based learning, as its exploratory nature can, in pr… view at source ↗
Figure 3
Figure 3. Figure 3: Correlation between task performance (Math) and alignment score (AS). 30 20 10 0 10 20 30 Difference (%) add subtract multiply divide sum_numbers floor ceil round_number power sqrt abs_value modulo Vanilla 30 20 10 0 10 20 30 Difference (%) SFT 30 20 10 0 10 20 30 Difference (%) add subtract multiply divide sum_numbers floor ceil round_number power sqrt abs_value modulo Sparse 30 20 10 0 10 20 30 Differenc… view at source ↗
Figure 5
Figure 5. Figure 5: Tool-use efficiency on in-domain and out-of￾domain tasks, measured by the distribution of tool calls per sample. in-domain and out-of-domain tasks. While the Teacher model is highly efficient, the Vanilla and SFT baselines tend to use tools inefficiently, evi￾denced by their wide distributions and numerous outliers. In contrast, our RL-based approach ef￾fectively transfers the teacher’s efficient strategy,… view at source ↗
Figure 7
Figure 7. Figure 7: Tool usage rate over training steps their error rates, which remain high and unstable throughout training. In contrast, settings that in￾clude the tool validation reward (Settings 3 and 5) learn to avoid invalid calls, with their error rates dropping rapidly to near-zero. This suggests that directly penalizing invalid calls with Rv is a highly effective strategy for training a more reliable agent. Alignmen… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 3 canonical work pages

  1. [1]

    AI-MO. 2024. Aime. https://huggingface.co/datasets/AI-MO/aimo-validation-aime

    2024

  2. [2]

    Mathematical Association of America AMC. 2023. 2023 AMC 12a and 12b: American mathematics competitions. https://www.maa.org/math-competitions/amc-12. Official competition information available at the MAA website. Problem statements referenced via the Art of Problem Solving archive: https://artofproblemsolving.com/wiki/index.php/2023_AMC_12A_Problems and h...

    2023

  3. [3]

    Anthropic. 2024. Introducing the model context protocol . https://www.anthropic.com/news/model-context-protocol/

    2024

  4. [4]

    Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, and 1 others. 2025 a . Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470

    Pith/arXiv arXiv 2025

  5. [5]

    Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025 b . Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400

    Pith/arXiv arXiv 2025

  6. [6]

    Li Chenglin, Qianglong Chen, Liangyue Li, Caiyu Wang, Feng Tao, Yicheng Li, Zulong Chen, and Yin Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.91 Mixed distillation helps smaller language models reason better . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1673--1690, Miami, Florida, USA. Association for Co...

    work page doi:10.18653/v1/2024.findings-emnlp.91 2024

  7. [7]

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024 a . Beyond imitation: Learning key reasoning steps from dual chain-of-thoughts in reasoning distillation. arXiv preprint arXiv:2405.19737

    Pith/arXiv arXiv 2024

  8. [8]

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024 b . Improve student's reasoning generalizability through cascading decomposed cots distillation. arXiv preprint arXiv:2405.19842

    Pith/arXiv arXiv 2024

  9. [9]

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2025. https://aclanthology.org/2025.acl-long.21/ Capture the key in reasoning to enhance C o T distillation generalization . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 441--465. Association for Computational Linguistics

    2025

  10. [10]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536

    Pith/arXiv arXiv 2025

  11. [11]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985

    Pith/arXiv arXiv 2024

  12. [12]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR

    2023

  13. [13]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. Tora: A tool-integrated reasoning agent for mathematical problem solving. The Twelfth International Conference on Learning Representations

    2024

  14. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem solving with the MATH dataset . In Proceedings of the Neural Information Processin...

    2021

  15. [15]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. https://aclanthology.org/2023.acl-long.830/ Large language models are reasoning teachers . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852--14882

    2023

  16. [16]

    Xanh Ho, Anh - Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020 , pages...

    doi:10.18653/v1/2020.coling-main.580 2020

  17. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    Pith/arXiv arXiv 2021

  18. [18]

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. 2025. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004

    Pith/arXiv arXiv 2025

  19. [19]

    Tenghao Huang, Dongwon Jung, Vaibhav Kumar, Mohammad Kachuee, Xiang Li, Puyang Xu, and Muhao Chen. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.61 Planning and editing what you retrieve for enhanced tool learning . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 975--988, Mexico City, Mexico. Association for Comput...

    work page doi:10.18653/v1/2024.findings-naacl.61 2024

  20. [20]

    Tatsuro Inaba, Hirokazu Kiyomaru, Fei Cheng, and Sadao Kurohashi. 2023. https://doi.org/10.18653/v1/2023.acl-short.130 M ulti T ool- C o T : GPT -3 can use multiple external tools with chain of thought prompting . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1522--1532, Toronto,...

    doi:10.18653/v1/2023.acl-short.130 2023

  21. [21]

    Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji - Rong Wen. 2025. https://doi.org/10.1145/3701716.3715313 Flashrag: A modular toolkit for efficient retrieval-augmented generation research . In Companion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 202...

    arXiv 2025

  22. [22]

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237--285

    1996

  23. [23]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International conference on machine learning, pages 15696--15707. PMLR

    2023

  24. [24]

    Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, and Sung Ju Hwang. 2025. Distilling llm agent into small models with retrieval and code tools. arXiv preprint arXiv:2505.17612

    arXiv 2025

  25. [25]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769--6781

    2020

  26. [26]

    Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, and 1 others. 2023. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world systems. arXiv preprint arXiv:2311.11315

    Pith/arXiv arXiv 2023

  27. [27]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. https://arxiv.org/abs/2206.14858 Solving quantitative reasoning problems with language models . Preprint, arXiv:2206.14858

    Pith/arXiv arXiv 2022

  28. [28]

    Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, and 1 others. 2025. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374

    Pith/arXiv arXiv 2025

  29. [29]

    Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, and Jun Zhao. 2025. Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24567--24575

    2025

  30. [30]

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, and 1 others. 2024. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning

    2024

  31. [31]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2025 a . Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. The Twelfth International Conference on Learning Representations

    2025

  32. [32]

    Ne Luo, Aryo Pradipta Gema, Xuanli He, Emile Van Krieken, Pietro Lesci, and Pasquale Minervini. 2025 b . Self-training large language models for tool-use without demonstrations. arXiv preprint arXiv:2502.05867

    Pith/arXiv arXiv 2025

  33. [33]

    Yuanjie Lyu, Chengyu Wang, Jun Huang, and Tong Xu. 2025. From correction to mastery: Reinforced distillation of large language model agents. arXiv preprint arXiv:2509.14257

    Pith/arXiv arXiv 2025

  34. [34]

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014

    Pith/arXiv arXiv 2023

  35. [35]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning

    2025

  36. [36]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Measuring and narrowing the compositionality gap in language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687--5711, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.378 2023

  37. [37]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958

    Pith/arXiv arXiv 2025

  38. [38]

    Cheng Qian, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.48 Toolink: Linking toolkit creation and using through chain-of-solving on open-source model . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...

    work page doi:10.18653/v1/2024.naacl-long.48 2024

  39. [39]

    Jiahao Qiu, Xinzhe Juan, Yimin Wang, Ling Yang, Xuan Qi, Tongcheng Zhang, Jiacheng Guo, Yifu Lu, Zixin Yao, Hongru Wang, and 1 others. 2025. Agentdistill: Training-free agent distillation with generalizable mcp boxes. arXiv preprint arXiv:2506.14728

    Pith/arXiv arXiv 2025

  40. [40]

    Qwen3. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  41. [41]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539--68551

    2023

  42. [42]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  43. [43]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  44. [44]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  45. [45]

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.01441

    Pith/arXiv arXiv 2025

  46. [46]

    Hoyun Song, Huije Lee, Jisu Shin, Sukmin Cho, Changgeon Ko, and Jong C. Park. 2025. https://doi.org/10.18653/v1/2025.findings-acl.1119 Does rationale quality matter? enhancing mental disorder detection via selective reasoning distillation . In Findings of the Association for Computational Linguistics: ACL 2025, pages 21738--21756, Vienna, Austria. Associa...

    doi:10.18653/v1/2025.findings-acl.1119 2025

  47. [47]

    Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. 2025. Climbing the ladder of reasoning: What llms can-and still can't-solve after sft? arXiv preprint arXiv:2504.11741

    arXiv 2025

  48. [48]

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7601--7614

    2024

  49. [49]

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2024 a . Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations

    2024

  50. [50]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533

    Pith/arXiv arXiv 2022

  51. [51]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024 b . Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning

    2024

  52. [52]

    Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. 2025. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512

    Pith/arXiv arXiv 2025

  53. [53]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, and 1 others. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    Pith/arXiv arXiv 2022

  54. [54]

    Haoze Wu, Yunzhi Yao, Wenhao Yu, Huajun Chen, and Ningyu Zhang. 2025 a . Recode: Updating code api knowledge with reinforcement learning. arXiv preprint arXiv:2506.20495

    arXiv 2025

  55. [55]

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. 2025 b . On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629

    arXiv 2025

  56. [56]

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch every step! llm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406.11176

    Pith/arXiv arXiv 2024

  57. [57]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. https://arxiv.org/abs/2409.12122 Qwen2.5-math technical report: Toward mathematical expert model via self-improvement . ArXiv

    Pith/arXiv arXiv 2024

  58. [58]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369--2380, Brussels...

    doi:10.18653/v1/d18-1259 2018

  59. [59]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  60. [60]

    Maxwell J Yin, Dingyi Jiang, Yongbing Chen, Boyu Wang, and Charles Ling. 2025. Enhancing generalization in chain of thought reasoning for smaller models. arXiv preprint arXiv:2501.09804

    Pith/arXiv arXiv 2025

  61. [61]

    Yuanqing Yu, Zhefan Wang, Weizhi Ma, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. 2024. Steptool: Enhancing multi-step tool usage in llms through step-grained reinforcement learning. arXiv preprint arXiv:2410.07745

    Pith/arXiv arXiv 2024

  62. [62]

    Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and 1 others. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921

    Pith/arXiv arXiv 2023

  63. [63]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  64. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...