Pith. sign in

REVIEW 4 major objections 83 references

Post-training alone can lift a 3B-activated MoE model to match or beat 100B-class models on reasoning, agents, and safety.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 03:28 UTC pith:YWPBAXJK

load-bearing objection Solid industrial post-training systems report: MOPD fusion and HMPO are useful recipes, and the efficiency claim is real on public benches even if the 100B-class headline is partly propped by in-house agent/safety suites. the 4 major comments →

arxiv 2607.09375 v1 pith:YWPBAXJK submitted 2026-07-10 cs.LG cs.CL

Mach-Mind-4-Flash Technical Report

Foundation Model Team (Li Auto Inc) This is my paper
classification cs.LG cs.CL
keywords Mixture-of-Expertspost-trainingreinforcement learningon-policy distillationagentic modelstoken efficiencymulti-teacher fusiontool use
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This report claims that a compact 35B Mixture-of-Experts model with only 3B parameters active at inference can reach the performance tier of models with ten to thirty times more activated parameters, without scaling pre-training compute. The authors do this by specializing then integrating: they train many domain-specific reinforcement-learning experts in parallel (reasoning, general skills, and agentic tool use), fuse them with a routed reverse-KL distillation on the student’s own rollouts, and finally compress reasoning length with a single-stage median-budget reward. A unified training stack with multi-teacher scheduling and operator-level speedups makes the pipeline practical. The resulting model leads or matches much larger systems on competition math, instruction following, behavioral safety, tool calling, web browsing, and autonomous agent tasks while keeping inference cheap. A sympathetic reader cares because this is a concrete recipe for closing the capability gap with post-training rather than ever-larger base models.

Core claim

The central claim is that aggressive post-training—parallel specialist RL experts, Multi-Teacher On-Policy Distillation that routes each sample to a frozen domain teacher and supervises the student via token-level reverse KL on its own rollouts, plus Hybrid Median-length Policy Optimization for length compression—can elevate a 35B MoE with 3B activated parameters to parity or better with 100B-plus models across reasoning, instruction following, safety, tool use, deep search, and agent benchmarks, all without scaling pre-training.

What carries the argument

Multi-Teacher On-Policy Distillation (MOPD): each training sample is routed to its domain specialist teacher; the student is trained on its own rollouts with a token-level reverse-KL objective (implemented via a clipped policy-gradient surrogate), replacing mixed-reward RL so specialists fuse without see-saw degradation. Hybrid Median-length Policy Optimization (HMPO) then sets a group-adaptive length budget from the median of correct rollouts and multiplies accuracy by a cosine length reward so short wrong answers get zero reward.

Load-bearing premise

That specialists trained on verifiable or rubric rewards, then fused by routing reverse-KL distillation onto the student’s own rollouts, keep their strengths in one generalist without serious loss on the long-horizon agent tasks that matter.

What would settle it

Retrain the same base under identical MOPD fusion but drop the reasoning or agent teachers, then re-evaluate on AIME, LiveCodeBench, SWE-bench Verified, and ClawBench: if the fused model still matches the full-teacher scores and the expert checkpoints on those axes, the claim that routed multi-teacher fusion preserves specialist capability holds; if large drops appear, it fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Compact MoE models can be pushed into the 100B-class performance band by post-training alone, cutting inference cost by large factors.
  • Training specialists in parallel then fusing with routed reverse-KL removes the need for mixed multi-objective rewards and the associated capability see-saw.
  • Length control trained only on math generalizes to code, science, and instruction following, cutting tokens 19–46% with at most 0.7 points accuracy loss.
  • Scalable agent environments (file-system, programmatic, model-simulated) plus trajectory-level rewards become a primary lever for real-world tool-use and autonomous-agent gains.
  • Operator-level MoE acceleration and dynamic multi-teacher scheduling make multi-expert post-training pipelines practically deployable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If MOPD’s incomplete retention on repository-level software engineering is systematic, long-horizon scaffold-specific habits may need turn-aware or trajectory-structure-preserving distillation rather than pure token reverse-KL.
  • Strong results on in-house behavioral-safety and claw-agent benches invite independent re-implementation of those suites; external validity would strengthen or weaken the agent-safety lead.
  • HMPO’s math-only training transferring to other domains suggests a general “answer as shortly as correctness allows” policy that could be tested as a plug-in stage on non-MoE reasoning models.
  • The EnvScaling idea—scale environments rather than static traces—could become a standard data recipe for any multi-turn tool-use RL setup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The manuscript presents Mach-Mind-4-Flash, a 35B MoE model (3B activated) trained from Qwen3.5-35B-A3B via post-training only. The pipeline has three stages: (1) a unified RL/OPD infrastructure with dynamic multi-teacher scheduling and SonicMoE operator optimizations (claimed 17% end-to-end speedup); (2) parallel domain RL experts (Reasoning, General, Agent) fused by Multi-Teacher On-Policy Distillation (MOPD), a routed token-level reverse-KL objective on student rollouts intended to avoid mixed-reward see-saw; (3) Hybrid Median-length Policy Optimization (HMPO), which sets a group-adaptive length budget from the median of correct rollouts and applies a multiplicative correctness-first reward. Reported scores include AIME'26 92.70, IFBench 82.82, Behavioral-SafetyBench 80.74, BFCL-v4 75.80, BrowseComp-zh 72.31, and ClawBench 84.20, with the claim that the model matches or exceeds models with 10–30× activated parameters at lower inference cost.

Significance. If the results hold under consistent public evaluation, this is a strong systems contribution: it shows that specialization-then-integration post-training, rather than pre-training scale, can push a compact MoE into the performance band of much larger models. MOPD is a concrete, implementable alternative to mixed-reward RL with useful ablations (Table 3; Appendix C code-teacher transfer). EnvScaling for tool-use RL, the Claw sandbox loop, and HMPO’s single-stage median-budget design are practically relevant. The infra work (unified RL/OPD loss, multi-teacher Ray scheduling, SonicMoE + shared-expert fusion) is documented with end-to-end step-time numbers (Appendix A). These are real engineering assets even if some absolute ranking claims need qualification. The paper does not ship code or multi-seed public replications, so significance currently rests on the reported pipeline and harness results rather than fully independent verification.

major comments (4)
  1. Table 2 and §5.1: The central claim that the model is “on par with or surpassing” 100B-class systems mixes official report numbers (marked *) with single-harness reproductions under “identical settings,” without multi-seed variance, confidence intervals, or a fully public-only comparison. Several axes that drive the “leading” narrative (Behavioral-SafetyBench 80.74, ClawBench 84.20, LexInstructEval 74.63) are in-house or internal-augmented. Please (i) report a public-only subset with the same decoding protocol for all models, (ii) state seeds/avg@N variance for AIME, LiveCodeBench, BFCL, and BrowseComp, and (iii) either open the in-house suites or demote them from headline ranking claims so the 100B-class comparison is externally checkable.
  2. Table 3 and §5.2 / §6: MOPD is claimed to eliminate see-saw degradation and retain expert performance, yet SWE-bench Verified falls from expert 73.80 to MOPD 71.10 (and PinchBench 77.10→75.90), while ClawBench/ClawEval improve. §6 already concedes residual long-horizon gaps. For the agentic half of the abstract claim, incomplete retention on repository-level SWE is load-bearing. Please quantify which agent behaviors are lost (scaffold-specific planning, recovery, tool schemas), add ablations that isolate routing vs. reverse-KL vs. response-length cap (Table 1: max_response_length 8K), and state clearly which capabilities are fully retained versus partially smoothed.
  3. §4.9 and §5.3 / Figure 14: HMPO is trained only on ~6.5K math problems (G=10, λ=0.8) yet is claimed to compress chains by 19–46% with ≤0.7pp accuracy loss and to generalize to code, science, and instruction following. Figure 14 shows AIME’26 token–accuracy only. Please report per-domain length and accuracy deltas (code, STEM, IF, multi-turn agent) for the HMPO checkpoint versus the MOPD checkpoint, and test whether the median-of-correct budget remains well-defined when correctness is sparse or multi-turn (as §6 notes for agent trajectories).
  4. Eq. (3) and Appendix B: The production objective is a clipped off-policy surrogate L_distill on a k1 reverse-KL estimator with asynchronous vLLM rollouts (θ vs θ_old), not pure on-policy reverse KL. The abstract’s “routed reverse-KL” wording overstates the implemented estimator. Please state the clip ε, the measured off-policy drift, and whether pure on-policy (synchronized) distillation matches the reported fusion quality; otherwise the causal attribution of “eliminates see-saw” to reverse KL alone is not fully secured.

Circularity Check

1 steps flagged

Empirical post-training report: headline scores are measured outcomes, not forced by construction; only minor non-load-bearing self-citation of methods papers.

specific steps
  1. self citation load bearing [§4.9 Hybrid Median-length Policy Optimization (HMPO); ref [66]]
    "To this end we apply HMPO (Hybrid Median-length Policy Optimization) [66], a cost-effective single-stage RL framework that, unlike prior length-control methods relying on rigid manual budgets or expensive multi-stage pruning, dynamically derives a group-adaptive budget b from the median length of correct rollouts and applies a length-aware token reward."

    HMPO is introduced via a concurrent/prior arXiv by overlapping authors (Zheng et al., arXiv:2606.01934). This is ordinary method self-citation, not a uniqueness theorem that forces the paper’s accuracy or compression numbers: those remain measured after training. Flagged only as minor non-load-bearing self-citation; it does not make the central ‘par with 100B-class’ claim circular.

full rationale

Mach-Mind-4-Flash is a systems/training technical report, not a first-principles derivation. The load-bearing claim—that post-training alone lifts a 3B-activated 35B MoE to ~100B-class performance—is supported by measured benchmark tables (Table 2, Table 3, Figure 14), not by an equation that rewrites its own inputs. MOPD (Eq. 3 / Appendix B) is a routed reverse-KL on-policy distillation objective with a clipped off-policy surrogate; student performance after fusion is an empirical outcome (including incomplete retention on SWE-bench Verified: expert 73.80 → MOPD 71.10), not a quantity fixed by the loss definition. HMPO redesigns the GRPO reward with a group-median budget and multiplicative correctness-first composition; the reported 19–46% length cuts and ≤0.7pp accuracy loss are post-hoc measurements on AIME and transfer domains, not fitted parameters renamed as predictions. Public axes (AIME, LiveCodeBench, BFCL-v4, BrowseComp-zh, SWE-bench) provide independent external grounding. Self-citations of HMPO (arXiv:2606.01934), LexInstructEval, and RubricHub are ordinary method references by overlapping authors; they do not supply a uniqueness theorem that forbids alternatives or force the headline scores. In-house suites (Behavioral-SafetyBench, ClawBench, LexInstructEval) raise external-validity concerns for correctness review, but defining an evaluation set is not circular derivation. No self-definitional loop, no fitted-input-called-prediction, and no ansatz smuggled as a uniqueness result. Score 1 only for the mild, non-load-bearing methods self-citations.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 5 invented entities

Load-bearing content is empirical systems work: a strong open MoE base, GRPO-style RL with domain rewards, multi-teacher reverse-KL distillation, and hand-chosen RL/length hyperparameters. Free parameters are training knobs that shape the reported Pareto; axioms are standard RL/LM assumptions plus domain assumptions that synthetic verifiable environments and internal benches track real capability; invented entities are the named methods and in-house evaluation suites introduced to operationalize the pipeline.

free parameters (6)
  • MOPD learning_rate / batch / response cap
    Production MOPD uses lr=1e-6, batch 64, max_response_length 8192 (Table 1); these are chosen settings that control fusion quality and cost.
  • HMPO group size G and reward offset λ
    G=10, λ=0.8 on ~6.5K math problems; they define the adaptive budget and cosine token reward scale.
  • Tool-use GRPO clip range and turn limit
    ε_low=0.20, ε_high=0.28, T_max=40, K=8 trajectories; asymmetric clip and horizon are design choices affecting agent RL.
  • SFT lr, batch, epochs, context
    Global batch 32, lr 1e-5 cosine, 2 epochs, 131072 context; establish the shared initialization for all experts.
  • Difficulty pruning pass@8 thresholds
    Samples with 0/8 or 8/8 (or agent pass@8 outside 0.1–0.9) are dropped; this filters the RL data distribution that produces reported gains.
  • Unified loss weights α, β for RL/OPD
    Mode switching and joint training depend on hand-set mixture weights between distillation and RL losses.
axioms (6)
  • domain assumption Group-relative policy optimization with outcome or process rewards improves specialist policies on verifiable tasks.
    Sections 4.2–4.7 treat GRPO + verifiable/rubric rewards as the working optimization engine without new convergence proofs.
  • domain assumption Token-level reverse KL from a frozen specialist on student on-policy (or off-policy-corrected) rollouts transfers that specialist’s behavior into a shared student.
    Eq. (3) and Appendix B; central to the claim that MOPD avoids mixed-reward see-saw.
  • domain assumption Qwen3.5-35B-A3B already contains sufficient latent capacity that post-training can reach ~100B-class scores.
    Pipeline starts from that base (Section 4); the ‘no pretraining scale’ claim is relative to this strong initialization.
  • domain assumption Programmatic validators, execution tests, and LLM judges used as rewards are faithful enough proxies for task success and safety.
    Reward design across IF, writing, tool-use, SWE, Claw, and safety tracks depends on this.
  • standard math Standard probability, KL, and clipped importance-sampling identities used in the k1 estimator and PPO-style surrogate.
    Appendix B Eqs. (5)–(9).
  • ad hoc to paper Median length of correct rollouts is a suitable adaptive efficiency budget that generalizes beyond math.
    HMPO Section 4.9 trains only on math yet claims 19–46% compression across domains with ≤0.7pp loss.
invented entities (5)
  • Multi-Teacher On-Policy Distillation (MOPD) no independent evidence
    purpose: Fuse many frozen RL specialists into one student via routed reverse-KL on student rollouts.
    Named method and objective Eq. (3); independent evidence is empirical ablations, not external theory.
  • Hybrid Median-length Policy Optimization (HMPO) no independent evidence
    purpose: Single-stage length compression with median-of-correct budget and multiplicative correctness-first reward.
    Final pipeline stage; companion arXiv cited but results here are self-reported.
  • EnvScaling environment pool no independent evidence
    purpose: Scale tool-use RL via file-system, programmatic, and model-simulated multi-turn environments.
    Section 4.4 data strategy; not released as a public benchmark suite.
  • Content-SafetyBench / Behavioral-SafetyBench no independent evidence
    purpose: Measure content and agentic behavioral safety where the model claims large leads.
    In-house suites (Section 5.1); external falsifiability limited without public release.
  • Claw Agent / ClawBench-style sandbox agent loop no independent evidence
    purpose: Train and score long-horizon environment-interactive agents beyond API tool calling.
    Section 4.7 and OpenClaw results; partly tied to internal harnesses.

pith-pipeline@v1.1.0-grok45 · 33809 in / 4366 out tokens · 49477 ms · 2026-07-13T03:28:16.356040+00:00 · methodology

0 comments
read the original abstract

We present Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model with 3B activated parameters. Through post-training optimization alone without scaling pre-training compute, the model achieves performance on par with or surpassing that of 100B-parameter-class models. By introducing scalable agentic interaction environments for large-scale reinforcement learning, the model attains significant performance gains on real-world application tasks. Our pipeline comprises three stages: (1) a unified RL/OPD training infrastructure with dynamic multi-teacher scheduling and operator-level acceleration, delivering 17\% end-to-end training speedup; (2) multiple domain-specific RL experts trained in parallel across Reasoning, General, and Agent tracks, then fused into a single generalist via Multi-Teacher On-Policy Distillation (MOPD) -- a routed reverse-KL objective that eliminates the see-saw degradation of mixed-reward RL; (3) Hybrid Median-length Policy Optimization (HMPO), a single-stage token-efficiency method that compresses reasoning chains by 19--46\% with $\le$0.7 percentage-point accuracy loss. Mach-Mind-4-Flash scores 92.70 on AIME'26, 82.82 on IFBench, 80.74 on Behavioral-SafetyBench, 75.80 on BFCL-v4, 72.31 on BrowseComp-zh, and 84.20 on ClawBench -- leading or matching models with 10--30$\times$ its activated size at a fraction of the inference cost.

Figures

Figures reproduced from arXiv: 2607.09375 by Foundation Model Team (Li Auto Inc).

Figure 1
Figure 1. Figure 1: Mach-Mind-4-Flash matches or exceeds much larger models across diverse capabil￾ity axes. With only 3B activated parameters, Mach-Mind-4-Flash leads on IFBench, Behavioral￾SafetyBench, and BrowseComp-zh, while remaining competitive on reasoning, tool use, and agentic coding against models with 3–30× its activated size. arXiv:2607.09375v1 [cs.LG] 10 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Unified training framework for RL and OPD. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Underlying operator framework diagram. In practical large-scale post-training scenarios, a single teacher model often struggles to compre￾hensively cover all task domains. Conversely, building independent training pipelines for different tasks easily leads to resource waste and engineering fragmentation. Therefore, we design a dynamic multi-teacher scalable architecture that supports the flexible, on-deman… view at source ↗
Figure 4
Figure 4. Figure 4: The post-training pipeline of Mach-Mind-4-Flash. The process starts from a base model, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of our SFT corpus across domains. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of reward strategies on a structured reasoning task (Table QA). [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Overview of EnvScaling for Tool-Use RL. We scale three complementary environment [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Overview of the DeepSearch High-Difficulty QA Data Synthesis Framework. We design a [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Overview of the Code Agent Data Curation Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Claw Agent training framework with sandboxed tool execution. A multi-harness training [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Pilot study on the tool-agent do￾main: the teacher–student top-K token overlap rate (K=16) climbs monotonically from 0.73 to 0.84 over training. 0 20 40 60 80 Training step 0.04 0.05 0.06 0.07 0.08 0.09 abs raw EMA smoothed (a) Monitoring quantity Labs. 0 20 40 60 80 Training step 0.00 0.01 0.02 0.03 distill raw EMA smoothed (b) Back-propagated quantity Ldistill [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Distillation training dynamics on the MOPD production run. Light traces are per-step [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Overview of HMPO. Left: For each query, the policy samples a group of rollouts (G). Right: Instead of relying on a static threshold, HMPO dynamically derives an adaptive budget b from the median length of only the correct rollouts to construct a smooth cosine-decay token reward. Bottom: The final reward is combined multiplicatively to enforce a strict “correctness-first, length-second” objective, mathemat… view at source ↗
Figure 14
Figure 14. Figure 14: Token efficiency on AIME’26. Each point represents a model’s accuracy vs. average [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: End-to-end acceleration effect diagram. B Derivation of the MOPD Objective Section 4.8 (Eq. 3) writes the MOPD loss as a routed mixture of token-level reverse KL divergences under the student’s on-policy distribution. This appendix records the three quantities we defer for brevity: the single-sample estimator we use in place of the exact reverse KL, the magnitude￾preserving diagnostic we log for monitorin… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 54 linked inside Pith

  1. [1]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  2. [2]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  3. [3]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis.arXiv preprint arXiv:2305.15334, 2023. URL https: //arxiv.org/abs/2305.15334

    Pith/arXiv arXiv 2023

  4. [4]

    Toolbench: Towards better instruction tuning for tool learning.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin et al. Toolbench: Towards better instruction tuning for tool learning.arXiv preprint arXiv:2307.16789, 2023. URLhttps://arxiv.org/abs/2307.16789

    Pith/arXiv arXiv 2023

  5. [5]

    Api-bank: A comprehensive benchmark for tool-augmented llms.arXiv preprint arXiv:2304.08244, 2023

    Minghao Li et al. Api-bank: A comprehensive benchmark for tool-augmented llms.arXiv preprint arXiv:2304.08244, 2023. URLhttps://arxiv.org/abs/2304.08244

    Pith/arXiv arXiv 2023

  6. [6]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025. URL https://arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2025

  7. [7]

    Glm-5: from vibe coding to agentic engineering, 2026

    GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunx- iang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Z...

    Pith/arXiv arXiv 2026

  8. [8]

    Step 3.5 flash: Open frontier-level intelligence with 11b active parameters

    Ailin Huang et al. Step 3.5 flash: Open frontier-level intelligence with 11b active parameters. arXiv preprint arXiv:2602.10604, 2026. URLhttps://arxiv.org/abs/2602.10604

    arXiv 2026

  9. [11]

    URLhttps://arxiv.org/abs/2603.00729. 24

    Pith/arXiv arXiv

  10. [12]

    Patil et al

    Shishir G. Patil et al. Berkeley function calling leaderboard (bfcl). https://gorilla.cs. berkeley.edu/leaderboard.html, 2024. URL https://gorilla.cs.berkeley.edu/ leaderboard.html

    2024

  11. [13]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, et al. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024. URL https://arxiv.org/ abs/2406.12045

    Pith/arXiv arXiv 2024

  12. [14]

    Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use

    Jiarui Lu et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use. InFindings of the Association for Computational Linguistics: NAACL, 2025. URL https://aclanthology.org/2025.findings-naacl.65/

    2025

  13. [15]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023. URLhttps://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2023

  14. [16]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022

    2022

  15. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  16. [18]

    Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025. URLhttps://arxiv.org/abs/2501.12599

    Pith/arXiv arXiv 2025

  17. [19]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    Pith/arXiv arXiv 2025

  18. [21]

    URLhttps://arxiv.org/abs/2505.22312

    Pith/arXiv arXiv

  19. [22]

    Multi-turn reinforcement learning for tool-calling agents with iterative reward calibration.arXiv preprint arXiv:2604.02869, 2026

    Wachiravit Modecrua et al. Multi-turn reinforcement learning for tool-calling agents with iterative reward calibration.arXiv preprint arXiv:2604.02869, 2026. URL https://arxiv. org/abs/2604.02869

    Pith/arXiv arXiv 2026

  20. [23]

    Training long-context, multi-turn software engineering agents with reinforcement learning.arXiv preprint arXiv:2508.03501, 2025

    Alexander Golubev et al. Training long-context, multi-turn software engineering agents with reinforcement learning.arXiv preprint arXiv:2508.03501, 2025. URL https://arxiv.org/ abs/2508.03501

    arXiv 2025

  21. [24]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. URLhttps://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2023

  22. [25]

    Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024. URL https://arxiv.org/abs/ 2403.07691

    Pith/arXiv arXiv 2024

  23. [26]

    Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024

    Yu Meng et al. Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024. URLhttps://arxiv.org/abs/2405.14734

    Pith/arXiv arXiv 2024

  24. [27]

    Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization

    Zhanhui Zhou et al. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. InFindings of the Association for Computational Linguistics: ACL, 2024. URL https://aclanthology.org/2024.findings-acl.630/. 25

    2024

  25. [28]

    Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023

    Huiqiang Jiang et al. Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023. URL https://arxiv.org/abs/ 2310.05736

    Pith/arXiv arXiv 2023

  26. [29]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.arXiv preprint arXiv:2310.06839, 2023

    Huiqiang Jiang et al. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.arXiv preprint arXiv:2310.06839, 2023. URL https://arxiv.org/ abs/2310.06839

    Pith/arXiv arXiv 2023

  27. [30]

    Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024

    Zhuoshi Pan et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024. URL https://arxiv.org/abs/2403. 12968

    Pith/arXiv arXiv 2024

  28. [31]

    Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026. URLhttps://arxiv.org/html/2606.19348

    arXiv 2026

  29. [32]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    Aili Chen et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. URLhttps://arxiv.org/abs/2506.13585

    Pith/arXiv arXiv 2025

  30. [33]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon et al. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM Symposium on Operating Systems Principles, 2023. URLhttps://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  31. [34]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving.arXiv preprint arXiv:2401.09670, 2024

    Yinmin Zhong et al. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving.arXiv preprint arXiv:2401.09670, 2024. URL https://arxiv. org/abs/2401.09670

    Pith/arXiv arXiv 2024

  32. [35]

    Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    Fabian Gloeckle et al. Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024. URLhttps://arxiv.org/abs/2404.19737

    Pith/arXiv arXiv 2024

  33. [36]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  34. [38]

    URLhttps://arxiv.org/abs/2411.01111

    Pith/arXiv arXiv

  35. [39]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. URLhttps://arxiv.org/abs/2312.06674

    Pith/arXiv arXiv 2023

  36. [40]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

    2026

  37. [41]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    Pith/arXiv arXiv 2024

  38. [42]

    Lexinstructeval: Lexical instruction following evaluation for large language models

    Huimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun, Hengtong Lu, Kaike Zhang, and Chen Wei. Lexinstructeval: Lexical instruction following evaluation for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 25117–25123, 2026

    2026

  39. [43]

    Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation

    Sunzhu Li, Jiale Zhao, Huimin Ren, Zhenlin Wei, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Chen Wei. Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31320–31344, 2026

    2026

  40. [44]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  41. [45]

    Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359, 2021

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359, 2021. 26

    Pith/arXiv arXiv 2021

  42. [46]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021

    2021

  43. [47]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, 2020

    2020

  44. [48]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Pith/arXiv arXiv 2021

  45. [49]

    Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  46. [50]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    Pith/arXiv arXiv 1909

  47. [51]

    A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  48. [52]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI soft...

    2025

  49. [53]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URLhttps://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  50. [54]

    Immersion in the github universe: Scaling coding agents to mastery, 2026

    Jiale Zhao, Guoxin Chen, Fanzhe Meng, Minghao Li, Jie Chen, Hui Xu, Yongshuai Sun, Xin Zhao, Ruihua Song, Yuan Zhang, Peng Wang, Cheng Chen, Jirong Wen, and Kai Jia. Immersion in the github universe: Scaling coding agents to mastery, 2026. URL https: //arxiv.org/abs/2602.09892

    arXiv 2026

  51. [55]

    R2e- gym: Procedural environment generation and hybrid verifiers for scaling open-weights swe agents, 2025

    Jain Naman, Singh Jaskirat, Shetty Manish, Zheng Liang, Sen Koushik, and Stoica Ion. R2e- gym: Procedural environment generation and hybrid verifiers for scaling open-weights swe agents, 2025. URLhttps://arxiv.org/abs/xxx.xxxx

    2025

  52. [56]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025. URLhttps://arxiv.org/abs/2504.21798

    Pith/arXiv arXiv 2025

  53. [57]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025

    GLM Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bo...

    Pith/arXiv arXiv 2025

  54. [58]

    Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving, 2026

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, and Haoli Bai. Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving, 2026. URL https://arxiv.org/abs/2601.01426

    arXiv 2026

  55. [59]

    slime: An llm post-training framework for rl scaling

    Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv

    2025

  56. [60]

    Your efficient rl framework secretly brings you off-policy rl training, August 2025

    Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. URL https://fengyao.notion.site/off-policy-rl

    2025

  57. [61]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL https: //arxiv.org/abs/2503.20783

    Pith/arXiv arXiv 2025

  58. [62]

    Dhillon, David Brandfonbrener, and Rishabh Agarwal

    Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling re- inforcement learning compute for llms, 2025. URL https://arxiv.org/abs/2510.13786

    Pith/arXiv arXiv 2025

  59. [63]

    Less is more: Early stopping rollout for on-policy distillation.arXiv preprint arXiv:2605.27028, 2026

    Zhou Ziheng, Jiaqi Li, Huacong Tang, Ying Nian Wu, and Demetri Terzopoulos. Less is more: Early stopping rollout for on-policy distillation.arXiv preprint arXiv:2605.27028, 2026

    Pith/arXiv arXiv 2026

  60. [64]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Pith/arXiv arXiv 2026

  61. [65]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  62. [66]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  63. [67]

    Do not think that much for 2+ 3=? on the overthinking of long reasoning models

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of long reasoning models. InForty-second International Conference on Machine Learning, 2025

    2025

  64. [68]

    Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025

    Pith/arXiv arXiv 2025

  65. [69]

    Hmpo: Hybrid median-length policy optimization for chain-of-thought compression.arXiv preprint arXiv:2606.01934, 2026

    Minghui Zheng, Hongxu Chen, Huimin Ren, Hongsheng Xin, Xiaoyang Qu, Ze Wang, Shul- ing Yang, Ziyu Peng, Kaike Zhang, Pan Zhou, et al. Hmpo: Hybrid median-length policy optimization for chain-of-thought compression.arXiv preprint arXiv:2606.01934, 2026. 28

    Pith/arXiv arXiv 2026

  66. [70]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  67. [71]

    Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Pith/arXiv arXiv 2026

  68. [72]

    Nemotron 3 super: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.arXiv preprint arXiv:2604.12374, 2026

    Aakshita Chandiramani, Aaron Blakeman, Abdullahi Olaoye, Abhibha Gupta, Abhilash So- masamudramath, Abhinav Khattar, Adeola Adesoba, Adi Renduchintala, Adil Asif, Aditya Agrawal, et al. Nemotron 3 super: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.arXiv preprint arXiv:2604.12374, 2026

    Pith/arXiv arXiv 2026

  69. [73]

    American invitational mathematics examination (aime) 2025, 2025

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025

    2025

  70. [74]

    American invitational mathematics examination (aime) 2026, 2026

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2026, 2026

    2026

  71. [75]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Repre- sentations, volume 2025, pages 58791–58831, 2025

    2025

  72. [76]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  73. [77]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  74. [78]

    Generalizing verifiable instruction following

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generalizing verifiable instruction following. Advances in Neural Information Processing Systems, 38, 2026

    2026

  75. [79]

    Writingbench: A comprehensive benchmark for generative writing.Advances in Neural Information Processing Systems, 38, 2026

    Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, et al. Writingbench: A comprehensive benchmark for generative writing.Advances in Neural Information Processing Systems, 38, 2026

    2026

  76. [80]

    Agent-safetybench: Evaluating the safety of llm agents.arXiv preprint arXiv:2412.14470, 2024

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-safetybench: Evaluating the safety of llm agents.arXiv preprint arXiv:2412.14470, 2024

    Pith/arXiv arXiv 2024

  77. [81]

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InForty-second International Conference on Machine Learning, 2025

    2025

  78. [82]

    τ 2-bench: Eval- uating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2-bench: Eval- uating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    Pith/arXiv arXiv 2025

  79. [83]

    Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei et al. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025. URLhttps://arxiv.org/abs/2504.12516

    Pith/arXiv arXiv 2025

  80. [84]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.arXiv preprint arXiv:2504.19314, 2025

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.arXiv preprint arXiv:2504.19314, 2025

    Pith/arXiv arXiv 2025

Showing first 80 references.