Pith. sign in

REVIEW 3 major objections 5 minor 12 cited by

This paper claims that SCoRe, a student-centered distillation method, closes the agentic performance gap between small and large LLMs by having the student explore and the teacher correct only the earliest error, with a 7B student matching

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:51 UTC pith:2WDWFUCL

load-bearing objection A genuinely new distillation scheme with consistent gains, but the headline 'gap closing' claim rests on train/test overlap that the paper never rules out. the 3 major comments →

arxiv 2509.14257 v3 pith:2WDWFUCL submitted 2025-09-12 cs.CL cs.AI

Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs

Yuanjie Lyu , Chengyu Wang , Jun Huang , Tong Xu This is my paper
classification cs.CL cs.AI
keywords SCoReagent distillationreinforcement learningbehavior cloningLLM agentscompounding errorsshort-horizon RLtool-use agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that a small language model can match a much larger one on agentic tasks if distillation reverses the usual direction: instead of the student cloning the teacher's full solution, the student attempts the task and the teacher fixes only the first wrong step. The method, SCoRe, first fine-tunes the student on these corrected, mostly student-generated trajectories, then runs short-horizon reinforcement learning starting from the verified prefix before the corrected step, with rewards at that key step. On 12 benchmarks, a 7B-parameter student comes within 0.9 points of a 72B teacher on math and factual reasoning, and exceeds it on deep-search tasks. The paper also argues that this 'student-explores, teacher-corrects' scheme reduces worst-case compounding errors from quadratic to linear in the number of steps. If true, it would mean smaller, cheaper models can be trained to near-expert agent performance without needing enormous backbones.

Core claim

The central claim is that agent distillation works better when the student, not the teacher, generates the trajectory. In SCoRe, a student attempts a task, and the teacher inspects the result, locates the earliest deviation from the correct path, and replaces only that single step with a corrected one. The student then continues from the corrected prefix, producing a mostly student-authored solution that is used for supervised fine-tuning. A subsequent reinforcement learning phase starts rollouts from the verified prefix and assigns extra rewards for reproducing the teacher's fix or avoiding the original mistake. The paper reports that this method lets a 7B-parameter student achieve agentic

What carries the argument

The key mechanism is Mentored Problem-Solving (MPS): the student rolls out a full trajectory, the teacher identifies and corrects the earliest wrong step, and the student continues from that corrected prefix. This produces capability-matched, deficiency-localized training data, and it is paired with a short-horizon RL phase that starts rollouts from the verified prefix and uses a key-step reward for reproducing the teacher's correction or avoiding the original error. The theoretical backbone is Theorem 3.2, which bounds the student's expected cost under its own state distribution, replacing the teacher-distribution assumption of behavior cloning.

Load-bearing premise

The load-bearing assumption is that the training questions taken from HotpotQA, 2WikiMultiHopQA, and WebWalker are disjoint from the test questions of those same benchmarks; the paper does not report running an exclusion step.

What would settle it

Compute the overlap between the 35k training question–answer pairs and the official test splits of HotpotQA, 2WikiMultiHopQA, and WebWalker. If even a small percentage of test questions appear in training, SCoRe's parity numbers would not transfer to truly unseen data; if zero overlap is confirmed, the central claim stands on its own.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A 7B-parameter student trained with SCoRe scores 50.8 on average across math and factual reasoning benchmarks, within 0.9 points of a 72B teacher (51.7), and it surpasses the 72B teacher on deep-search tasks (30.5 vs 27.3).
  • Correction-based SFT alone outperforms behavior cloning at the same data budget, and adding short-horizon RL with key-step rewards produces the largest gains.
  • The worst-case expected cost of compounding errors is bounded by O(H) instead of O(H^2), where H is the number of reasoning-action steps.
  • The hard-to-teach subset—tasks the teacher could not fix even after multiple attempts—goes from 0% accuracy to 24.3% after SCoRe-RL.
  • SCoRe-RL outperforms GRPO, DPO, and ARPO baselines on most of the 12 evaluated benchmarks, matching or exceeding teacher-level performance on several deep-search tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'correct the earliest error' principle is a natural minimal-intervention curriculum: it keeps the student on a learnable path while exposing the exact step that causes failure, and this could generalize to sequential decision problems beyond LLM agents, such as program repair, web navigation, or embodied control.
  • A direct test of the claim would be to run SCoRe on a dataset family entirely disjoint from the training questions—for example, questions written after training time or drawn from a different benchmark—and see whether the documented gap reopens.
  • The paper does not report excluding HotpotQA, 2WikiMultiHopQA, or WebWalker training questions from the test splits of those same benchmarks; if overlap exists, the reported parity numbers would be inflated, and the true effect of SCoRe would be smaller.
  • SCoRe's short-horizon RL idea—starting rollouts from a verified prefix—could be combined with other verifier-based training signals, such as process supervision or learned reward models, potentially reducing variance in long-horizon agentic tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SCoRe, a two-phase distillation framework for LLM agents. In the first phase, a student generates full trajectories; a teacher locates the earliest erroneous step and provides a one-step correction, producing student-centric corrected trajectories for SFT. In the second phase, short-horizon GRPO rollouts start from the verified prefix preceding the original error, with additional key-step rewards for reproducing the teacher's correction or avoiding the original error. The paper claims that a 7B student trained with SCoRe closes the agentic performance gap with a 72B teacher on 12 benchmarks, and that the method reduces the worst-case compounding-error growth from O(H^2) to O(H).

Significance. If the empirical claims hold, SCoRe would be a practically useful contribution to agent distillation: it is simple, requires only a teacher that can localize errors, and the public code and detailed implementation (VERL, LLaMAFactory, etc.) support reproducibility. The theoretical derivations, while not all novel, are a useful formalization of the intuition that student-led trajectory generation reduces distribution shift. However, the significance is conditional because the evaluation currently has a data-hygiene risk that directly affects the headline result, and the main theoretical comparison is not as clean as stated.

major comments (3)
  1. [§4.1 Implementation; Tables 1–2] Training seed Q–A pairs are collected 'mainly from the ToolStar dataset... including HotpotQA, 2WikiMultiHopQA, and WebWalker for factual QA' (35k pairs). Evaluation uses the same benchmark families: HotpotQA and 2Wiki in Table 1, WebWalker in Table 2. The paper never states that training queries were excluded from the official test splits of these benchmarks. With test sizes of 200, 200, and 100, a small overlap can inflate SCoRe-RL scores by several points. Please report the exact exclusion/overlap procedure; if overlap exists, re-run using disjoint training/test splits.
  2. [Tables 1–2, §4.1] No error bars, confidence intervals, or repeated-seed results are reported. The GRPO and ARPO rows are 'mostly taken from Dong et al. (2025b)' (Table 1 caption), i.e., not run under this paper's tool/environment; comparisons to SCoRe-RL may be mismatched. Given that the reasoning-table headline is a 0.9-point gap to the 72B teacher (50.8 vs. 51.7), run-to-run noise could change the conclusion. Report variance and rerun baselines under identical conditions, or clearly mark imported results and accompany them with uncertainty estimates.
  3. [Theorem 3.2 and Appendix A.2] The bound c(π̂) ≤ c(πE) + Hε is not specific to SCoRe: it assumes a per-step error bound under d^{π̂}_t, whereas the BC O(H²) bound in Theorem 3.1 uses error under d^{πE}_t. Comparing bounds under different state distributions does not establish that SCoRe reduces compounding errors from O(H²) to O(H). The proof also uses E[c_t(s)|G_t] ≤ E_{s∼d^{πE}_t}[c_t(s)], which is unjustified because the same action can have different cost in an out-of-distribution state. Restate as a proposition under an explicit on-policy assumption, or provide a proof of the actual training-to-test transfer.
minor comments (5)
  1. [§3.3, Eq. (8)] The notation G_t is used both for the return in Eq. (8) and for the event in Theorem 3.2; please disambiguate.
  2. [§4.1, Table 4] Test sizes are small (e.g., HotpotQA 200, WebWalker 100). Report bootstrap CIs or per-item variance estimates, especially for the deep-search results.
  3. [§4.1] The paper states no web browser was used. For WebWalker, a web-traversal benchmark, this deviates from the benchmark's intended setting; please describe how the search-only interface maps to WebWalker queries and whether this affects comparability with reported baselines.
  4. [Appendix C, Figure 4] Figure 4 lacks error bars and the axis labels are inconsistent with the text (SFT-10k vs. 10K). Clarify whether the plot shows a single run.
  5. [Abstract vs. Table 1] The abstract says the student 'matches' the 72B teacher, but Table 1 shows 50.8 vs. 51.7 on the reasoning average. Use consistent wording such as 'closes the gap' or 'nears'.

Circularity Check

0 steps flagged

No significant circularity: the central empirical claim is evaluated on external benchmark test splits, not on the method's own reward or correction labels. The main validity concern is a possible train/test overlap, which is a data-hygiene issue rather than a circular derivation.

full rationale

SCoRe's headline claim is measured by task success on external benchmark test splits (Tables 1 and 2), using token-level F1 or LLM-as-a-judge correctness against ground truth. The key-step reward is a training-time credit-assignment signal defined from the teacher's correction, but the final evaluation is not that reward, so the result is not the training objective by construction. The O(H) vs O(H^2) theoretical comparison is a conditional inequality: Theorem 3.2 assumes per-step misalignment under the student's own distribution is at most ε and concludes expected cost is at most c(π_E)+Hε. The proof in Appendix A simply sums the assumed per-step bound and does not actually invoke the first-error-correction mechanism; this makes the theorem weak or definitional, but it is a stated assumption-conclusion pair rather than a fitted quantity renamed as a prediction. There are no load-bearing self-citations or imported uniqueness theorems; the baselines and prior results cited (Ross et al., Kang et al., Dong et al.) are external work. The most serious validity threat is not circularity: Section 4.1 states that the 35k seed question–answer pairs are collected mainly from ToolStar, 'including HotpotQA, 2WikiMultiHopQA, and WebWalker for factual QA,' and the same three datasets appear in the evaluation tables, with no reported exclusion from the official test splits. If test questions overlap with training data, the reported gains would be inflated, but this is a data-contamination/missing-support concern, not a derivation that equals its input by construction. No circular step is established.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central results depend on hand-chosen reward weights and cutoffs, on the teacher's localization ability, on the judge's reliability, and on an unstated train/test disjointness assumption. The last is the most consequential and is flagged as a red flag.

free parameters (3)
  • Key-step reward weights = R_final=1.0, R_key=0.5, R_avoid=0.1 (from Figure 2)
    Hand-chosen rewards in the RL objective; the paper does not report a sweep or sensitivity analysis.
  • Maximum correction attempts = 5
    Tasks unsolved after 5 one-step corrections are labeled hard-to-teach; this cutoff affects what data enters SFT/RL.
  • Maximum rollout steps = 8
    Rollouts exceeding 8 reasoning-action cycles are treated as incomplete during RL and inference.
axioms (4)
  • domain assumption The teacher can reliably identify the first deviation step and produce a correct, minimal correction.
    Invoked throughout MPS (Section 3.2); if the teacher mislocates or mis-corrects, training data is corrupted.
  • domain assumption The LLM judge (Qwen2.5-72B-Instruct) gives reliable correctness judgments on math and deep search tasks.
    Used for all evaluation labels in Tables 1 and 2 (Section 4.1 Evaluation).
  • domain assumption The student's per-step error rate under its own rollouts is bounded by epsilon, as assumed in Theorem 3.2.
    The O(H) bound depends on this hypothesis; the paper does not provide empirical estimates of epsilon under d^pi_hat_t.
  • ad hoc to paper Training data topics from HotpotQA, 2WikiMultiHopQA, and WebWalker are disjoint from the test splits of those benchmarks.
    Implicitly assumed in Section 4.1; not stated or verified.

pith-pipeline@v1.3.0-alltime-deepseek · 21306 in / 12864 out tokens · 132325 ms · 2026-08-04T17:51:19.132703+00:00 · methodology

0 comments
read the original abstract

Large Language Model agents achieve strong performance on multi-step reasoning and tool-use tasks, but their impressive capabilities typically rely on extremely large backbones. Existing distillation approaches train smaller students to imitate full teacher trajectories, yet reasoning and knowledge gaps between the teacher and student can cause compounding errors. We propose SCoRe, a student-centered framework in which the student generates training trajectories and the teacher corrects only the earliest error, producing training data matched to the student's abilities and exposing specific weaknesses. The student is first fine-tuned on corrected trajectories. Subsequently, short-horizon reinforcement learning starts from the verified prefix preceding the earliest error, with target rewards assigned at that step. This design enables the student to solve problems through unconstrained RL exploration rather than teacher imitation, while the short-horizon setup improves training stability. On 12 challenging benchmarks, a 7B-parameter student distilled with SCoRe closes the agentic performance gap with a 72B-parameter teacher.

Figures

Figures reproduced from arXiv: 2509.14257 by Chengyu Wang, Jun Huang, Tong Xu, Yuanjie Lyu.

Figure 1
Figure 1. Figure 1: Comparison between imitation-based distillation and our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The SCoRe framework. (a) A student agent attempts a task, and the teacher provides a single-step correction at the first error, creating student-centric training data. (b) The student is initially trained to imitate full solution trajectories via supervised fine-tuning. (c) The student is further improved through reinforcement learning, using shortened rollouts starting from the prefix preceding the teache… view at source ↗
Figure 3
Figure 3. Figure 3: Teacher intervention frequency and performance on “hard data” after training. Cat￾egories: 0 = solved by the student solely, 1 = one teacher correction, ≥2 = two or more than two corrections. Hard data = unsolved samples even with the teacher’s help. SCoRe matches teacher performance on rea￾soning and deep search with smaller back￾bones. On smaller backbones, our method lever￾ages MPS trajectories generate… view at source ↗
Figure 4
Figure 4. Figure 4: Performance of models SFT on MPS-generated data (data scales: 10K, 5K, 2K), compared [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of SCoRe-SFT, SCoRe-RL, and a DPO baseline. While DPO [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt for the teacher to find the wrong step [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template for the teacher to correct the wrong step. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Rollout On-Policy Distillation via Peer Successes and Failures

    cs.LG 2026-05 unverdicted novelty 7.0

    MOPD improves on-policy distillation for LLMs by using peer successes for positive patterns and failures for negative examples to create more informative teacher signals.

  2. ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

    cs.LG 2026-06 unverdicted novelty 6.0

    ReNIO reweights negative student-generated trajectories in LLM on-policy distillation using probability ratios, reporting relative gains up to 10% on reasoning benchmarks.

  3. Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents

    cs.LG 2026-05 unverdicted novelty 6.0

    LearnWeak specializes small CUAs via weakness detection by a reference agent, targeted task synthesis, and error-aware training, delivering 11+ point gains on OSWorld.

  4. Multi-Rollout On-Policy Distillation via Peer Successes and Failures

    cs.LG 2026-05 unverdicted novelty 6.0

    MOPD improves on-policy distillation by using peer successes and failures from multiple rollouts to construct more informative teacher signals, yielding consistent gains over baselines on reasoning benchmarks.

  5. It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

    cs.AI 2026-05 conditional novelty 6.0

    W2SPO trains a reasoning LLM by inserting short 8-token branches proposed by a weaker model into its own rollouts and updating only those branch tokens, improving Pass@1 over vanilla GRPO (64.2% vs 62.3%) with a 3.55x...

  6. AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use

    cs.CL 2026-04 unverdicted novelty 6.0

    AgenticQwen small models trained via reasoning and agentic RL with dual data flywheels achieve strong benchmark performance and close the gap to larger models on industrial search and data analysis tasks.

  7. MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation

    cs.CL 2025-10 conditional novelty 6.0

    A teacher-guided dense reward for GRPO distillation improves small-model tool-use accuracy and out-of-domain generalization on executable-tool benchmarks.

  8. Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces

    cs.CL 2026-06 unverdicted novelty 5.0

    Reasoning in large output spaces proceeds via shortlisting then fine-grained reasoning; this characterization enables a mechanistic distillation strategy that outperforms standard distillation.

  9. Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

    cs.AI 2026-05 unverdicted novelty 5.0

    Reshaping outcome rewards, process signals, and rollout comparability in GRPO raises strict compile-and-semantic accuracy in agentic code repair from 0.385 to 0.535 under weak feedback.

  10. A Survey of On-Policy Distillation for Large Language Models

    cs.LG 2026-04 unverdicted novelty 4.0

    On-Policy Distillation is formalized as f-divergence minimization over student-sampled trajectories and organized along three design axes for LLM capability transfer.

  11. A Survey of On-Policy Distillation for Large Language Models

    cs.LG 2026-04 unverdicted novelty 3.0

    A survey that formalizes on-policy distillation as f-divergence minimization over student-sampled trajectories and organizes the literature along three design axes while linking it to KL-constrained RL.

  12. A Survey of On-Policy Distillation for Large Language Models

    cs.LG 2026-04 unverdicted novelty 2.0

    On-policy distillation reframes LLM knowledge transfer as iterative correction on student trajectories rather than single-pass imitation, with the survey organizing the field along divergence design, feedback sources,...

Reference graph

Works this paper leans on

56 extracted references · 33 linked inside Pith · cited by 9 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Never give up: Learning directed exploration strategies

    Adri \`a Puigdom \`e nech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Mart \' n Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020

    Pith/arXiv arXiv 2002

  4. [4]

    xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations

    Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, et al. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations. arXiv preprint arXiv:2506.13651, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning

    Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. arXiv preprint arXiv:2505.16410, 2025 a

    Pith/arXiv arXiv 2025

  7. [7]

    Agentic reinforced policy optimization

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025 b

    Pith/arXiv arXiv 2025

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024

    2024

  9. [9]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp.\ 10764--10799. PMLR, 2023

    2023

  11. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020

    Pith/arXiv arXiv 2011

  13. [13]

    Distilling llm agent into small models with retrieval and code tools

    Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, and Sung Ju Hwang. Distilling llm agent into small models with retrieval and code tools. arXiv preprint arXiv:2505.17612, 2025

    arXiv 2025

  14. [14]

    Hg-dagger: Interactive imitation learning with human experts

    Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp.\ 8077--8083. IEEE, 2019

    2019

  15. [15]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13 0 (9): 0 9, 2024

    2024

  16. [16]

    Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl

    Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, et al. Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl. arXiv preprint arXiv:2508.13167, 2025 a

    Pith/arXiv arXiv 2025

  17. [17]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025 b

    Pith/arXiv arXiv 2025

  18. [18]

    Webthinker: Empowering large reasoning models with deep research capability

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025 c

    Pith/arXiv arXiv 2025

  19. [19]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  20. [20]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Teaching small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022

    Pith/arXiv arXiv 2022

  22. [22]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  23. [23]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  24. [24]

    Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification

    Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.06639, 2025

    arXiv 2025

  25. [25]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    Pith/arXiv arXiv 1910

  26. [26]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    Toolrl: Reward is all tool learning needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023

    2023

  30. [30]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.\ 3505--3506, 2020

    2020

  31. [31]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St \'e phane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.\ 627--635. JMLR Workshop and Conference Proceedings, 2011

    2011

  32. [32]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 0 68539--68551, 2023

    2023

  33. [33]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C \^o t \'e , Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  37. [37]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017

    Pith/arXiv arXiv 2017

  38. [38]

    Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models

    Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv preprint arXiv:2503.21380, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Behavioral cloning from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018

    Pith/arXiv arXiv 2018

  42. [42]

    Musique: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10: 0 539--554, 2022

    2022

  43. [43]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024

    2024

  44. [44]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    Pith/arXiv arXiv 2022

  45. [45]

    Webwalker: Benchmarking llms in web traversal

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68 0 (2): 0 121101, 2025

    2025

  47. [47]

    A comprehensive survey of deep research: Systems, methodologies, and applications

    Renjun Xu and Jingwen Peng. A comprehensive survey of deep research: Systems, methodologies, and applications. arXiv preprint arXiv:2506.12594, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    Pith/arXiv arXiv 2018

  50. [50]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  51. [51]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  52. [52]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  54. [54]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  55. [55]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  56. [56]

    _7 tlPzKTþr ;]샦3Ӯ쀥ԤN #,¤HE ! , ۹ ]XO< iVzK L m;YcT !Z :4Uy

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    1970