Pith. sign in

REVIEW 5 major objections 5 minor 6 cited by

Efficient Agents: Building Effective Agents While Reducing Cost

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By varying one component at a time on GAIA and ranking by cost-of-pass, this paper assembles an agent that keeps 96.7% of a leading baseline's accuracy while cutting the expected cost of a correct answer by 28.4%.

desk verdict Useful component-level efficiency audit, but the headline numbers don't add up and the OWL baseline is ambiguous; the qualitative takeaway is plausible, the specifics are not. read the letter →

arxiv 2508.02694 v1 pith:CTUKKGOY submitted 2025-07-24 cs.AI cs.CLcs.MA

Ningning Wang , Xavier Hu , Pai Liu , He Zhu , Yue Hou , Heyuan Huang , Shengyu Zhang , Jian Yang
show 6 more authors
Jiaheng Liu Ge Zhang Changwang Zhang Jun Wang Yuchen Eleanor Jiang Wangchunshu Zhou
This is my paper · ORCID
classification cs.AIcs.CLcs.MA
keywords LLMagentscost-of-passefficiency-effectivenesstrade-offGAIAbenchmarktest-timescalingplanningtoolusememory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that much of the cost of modern LLM-based agents is optional: on the GAIA benchmark, the most expensive design choices—long planning horizons, best-of-N sampling, and elaborate memory modules—buy little accuracy while driving up the expected cost per correct answer. It measures every component with the cost-of-pass metric, $v(m,p)=C(m,p)/R(m,p)$, the dollar cost of one attempt divided by the success rate, so that accuracy and cost are compared on one scale. From one-at-a-time ablations of the backbone, step limit, replan interval, search sources, query-expansion count, best-of-N, and memory type, the paper assembles a configuration it calls Efficient Agents (GPT-4.1, at most 8 ReAct steps, replanning every step, five search sources and query expansions, no best-of-N, simple memory). It reports that this configuration scores 51.52% on GAIA dev with cost-of-pass 0.55, versus 53.33% and 0.75 for a leading open-source agent, retaining 96.7% of the baseline's accuracy while improving cost-of-pass by 28.4%. The point is that such gains come from measured component selection, not from a new architecture.

What carries the argument

The load-bearing object is the cost-of-pass metric, $v(m,p)=C(m,p)/R(m,p)$, where $C(m,p)$ is the dollar cost of one inference attempt by model $m$ on problem $p$ and $R(m,p)$ is the success rate; it compresses accuracy and cost into one number, so every design choice can be ranked on the same scale. The argument-carrying method is single-factor ablation: fix a default agent configuration, change one component at a time, and read off accuracy, cost, and cost-of-pass. The final configuration is then chosen by the paper's selection rule: for each component, take the variant with the lowest cost-of-pass among those that do not substantially cut accuracy. The metric is what lets the paper convert component rankings into a whole-system design without adding new machinery.

What would settle it

Reproduce the final configuration on GAIA dev and then run a small factorial sweep around it (for example, step limits 4/8/12, search counts 3/5/10, memory simple/none). If any combination reaches cost-of-pass below 0.55 with accuracy at or above 51.52%, or if switching one chosen setting back (say the step limit to 12) improves cost-of-pass, the component-wise selection claim does not hold as stated.

Watch

Extended reading notes

Core claim

The central discovery is an efficiency-effectiveness map of agent components. On GAIA, the backbone model sets the accuracy ceiling: GPT-4.1 reaches 53.33% at cost-of-pass 0.98, while a stronger but pricier model reaches 61.82% at cost-of-pass 3.54, and cheap sparse models stay below 30% accuracy. Around that ceiling, the framework's cost is controlled by a few levers: cutting the step limit from 12 to 8 loses only 0.6 accuracy points and improves cost-of-pass from 0.98 to 0.70; going from one search source to five improves accuracy by six points and improves cost-of-pass from 1.32 to 0.81; simple memory outperforms no-extra, summarized, fixed, and hybrid memories on both accuracy and cost; and best-of-N inflates cost-of-pass without meaningful accuracy gain. The paper composes the individually best settings into one configuration and reports 51.52% accuracy at cost-of-pass 0.55, which it frames as 96.7% of a leading open-source agent's accuracy (53.33%) with a 28.4% better cost-of-pass (0.75 to 0.55).

Load-bearing premise

The load-bearing premise is that the cost and accuracy effects found when each component is varied one at a time add up when the chosen settings are combined, so the reported 0.55 cost-of-pass and 51.52% accuracy are inferred from separate tests rather than measured on the final joint configuration.

Editorial extensions

If this is right

  • Shorter planning horizons are the cheapest accuracy-preserving lever: capping an agent at 8 ReAct steps instead of 12 keeps accuracy nearly flat and cuts cost-of-pass by about 29%.
  • Best-of-N test-time scaling is not worth its price on GAIA: raising N from 1 to 4 adds under a point of accuracy but raises cost-of-pass from 0.98 to 1.28.
  • Elaborate memory modules can be pure overhead: simple memory beats no-extra-memory and summarized or hybrid memories on both accuracy and cost-of-pass in this benchmark.
  • Tool-use configuration matters more than memory sophistication: adding search sources and query expansions improves both accuracy and cost-of-pass at the same time.
  • A competitive efficient agent can be assembled from component-level evidence alone, reaching 96.7% of a top open-source agent's accuracy with a 28.4% better cost-of-pass.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper's selection rule assumes component effects add; if step limit, search count, and memory interact, the combined 0.55 cost-of-pass is not guaranteed for every task mix, and a task-adaptive controller could do better or worse.
  • Inference: the same 'lowest cost-of-pass per component' recipe could be run on other benchmarks or with other backbone families, but the specific winning settings (8 steps, 5 queries) are tied to GAIA and to May-2025 API prices.
  • Inference: because cost-of-pass divides by success rate, it rewards cheap models that occasionally fail and can be retried; a budget-constrained deployment with a hard cap on attempts per task might rank models differently.
  • Inference: the steep rise in cost-of-pass from Level 1 to Level 3 suggests that routing tasks by predicted difficulty to different step limits or backbone models could further reduce cost without the accuracy loss seen at fixed low step limits.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents an empirical study of efficiency-effectiveness trade-offs in LLM-driven agents on the GAIA benchmark. Using the cost-of-pass metric, it ablates five design dimensions (backbone LLM, test-time scaling, planning, tool use, and memory) and then proposes Efficient Agents, a configuration in which each component is chosen from the component-level results. The paper claims that Efficient Agents retains 96.7% of the accuracy of the OWL agent while improving cost-of-pass by 28.4%, and reports comparisons against OWL and Smolagents. The contribution is framed as the first systematic efficiency analysis of general agent frameworks and a demonstration of an efficient agent configuration.

Significance. If the reported comparisons were valid, the paper would be a useful empirical contribution to cost-aware agent design: it provides a systematic ablation over several components and uses a meaningful economic metric rather than token count alone. The release of code and prompts is a strength. However, the central quantitative claims are not currently supported. The headline cost improvement is computed with the wrong metric, the OWL baseline is ambiguous and inconsistent with the paper's own description, the configuration is selected and evaluated on the same development set, and cost numbers differ across tables for nominally identical configurations. These issues affect the main claim, so the paper needs substantial revision, but the underlying study is potentially salvageable and can be strengthened with corrected numbers, a clearly specified baseline, and a held-out evaluation.

major comments (5)
  1. [Abstract / §1 / Table 7] The headline cost claim is internally inconsistent. The abstract and §1 state that costs fall from $0.398 to $0.228 and that cost-of-pass improves by 28.4%. Table 7 reports overall cost per attempt of $0.285 for Efficient Agents, not $0.228; $0.228 is the Level-1 figure. Furthermore, the cost-of-pass values in Table 7 are 0.75 for OWL and 0.55 for Efficient Agents, which is an improvement of (0.75−0.55)/0.75 ≈ 26.7%, not 28.4%. The 28.4% figure is instead the reduction in cost per attempt, (0.398−0.285)/0.398 ≈ 28.4%. The abstract and introduction therefore misstate the paper's central quantitative result.
  2. [§5.1 / Table 7] The OWL baseline used in Table 7 is not reconciled with the paper's own description of OWL. §5.1 states that OWL 'scored an impressive average of 69.7% on the GAIA benchmark,' but Table 7 reports OWL accuracy as 53.33%, which is exactly the GPT-4.1 default accuracy from Table 1. If the correct OWL score is 69.7%, then Efficient Agents retains 51.52/69.7 ≈ 73.9% of OWL's accuracy, not 96.7%. The manuscript must specify the model, settings, and evaluation protocol used for the OWL row in Table 7, and the headline retention claim must be recomputed against a consistent baseline.
  3. [§4 / Tables 1–7] The Efficient Agents configuration is selected and evaluated on the same GAIA development set. Section 4 says each component is chosen based on the preceding empirical studies (Tables 1–5), and Table 7 then reports the resulting system on the same benchmark. This makes the headline result an in-sample selection outcome rather than an independent evaluation. The paper should either evaluate the selected configuration on a held-out split or clearly state that the reported numbers are tuning-set results, not predictions of out-of-sample performance.
  4. [Table 2 vs. Table 1 / §3.2] The same default configuration appears with inconsistent cost numbers across tables. Table 1 reports GPT-4.1 with 243K tokens and cost $0.705; Table 2, N=1 row, reports the same 243K tokens but cost $0.521, and Table 5's 'w/o extra' row repeats $0.521. Since the token counts are identical, the cost difference must be explained (e.g., different input/output token mix or pricing date), and the cost-of-pass values derived from these costs are therefore not comparable across tables.
  5. [§4 / Table 6 / Table 4] The selected configuration in Table 6 combines 'Multi' search source with search number 5, but Table 4 does not test this combination: Multi is evaluated only with search number 10, and search number variations are evaluated only with the Simple source. The paper assumes that the component-wise improvements add, but no interaction experiment supports this. The reported cost-of-pass of 0.55 for the combined system is therefore an extrapolation, not a measured property of the configuration described in Table 6.
minor comments (5)
  1. [§3.3] The text states that when the maximum steps increase from 4 to 8, accuracy rises from 58.49% to 69.81%, but these are the Level-1 accuracy values; the overall accuracy in Table 3 goes from 41.82% to 52.73%. The text should be corrected to avoid confusing level-specific and aggregate numbers.
  2. [§6] The conclusion contains the placeholder 'reducing the operational cost by xx times'; this should be replaced with the concrete figure or removed.
  3. [References] References [20] and [27] are the same arXiv paper (cost-of-pass) and should be merged into a single citation.
  4. [Table 5 / Appendix A] The term 'Simple' is used both for the default memory setting in Appendix A and as a distinct experimental row in Table 5; the relationship between these two uses should be clarified in the text.
  5. [§2.2] The paper should state how the aggregate cost-of-pass in the tables is computed (mean of per-problem ratios versus ratio of aggregate cost to aggregate accuracy) and should report variance or confidence intervals, since the cost-of-pass metric is central to all claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the final configuration is an in-sample empirical result, not a definitionally forced prediction.

full rationale

The paper's derivation chain is empirical rather than deductive. Section 3 measures cost-of-pass and accuracy for backbone, test-time scaling, planning, tool use, and memory on GAIA dev; Section 4 then selects the combination with the lowest cost-of-pass that does not substantially degrade accuracy; Table 7 reports a run of that combination. No equation defines the final cost-of-pass in terms of the selection rule, and the 0.55 value is not identical to any single row in Tables 1-5, so interactions could in principle have changed it. The in-sample selection means the headline improvement is an optimization outcome on the evaluation set rather than an independent prediction, but that is a selection-bias and validation concern, not circularity in the sense of a result reducing to its inputs by construction. The OWL discrepancy (53.33 in Table 7 vs 69.7 in Section 5.1) is a serious baseline-consistency problem for the 96.7% retention claim, but it is not a circularity. Self-citations (e.g., refs [2, 3, 6]) are contextual and not load-bearing for the central claims.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of configuration choices selected from the GAIA dev set, on single-run pass@1 estimates, and on the assumption that component effects are independent. The paper does not introduce new theoretical entities. The main burden is the benchmark-specific fitting of every component choice.

free parameters (7)
  • Backbone LLM = GPT-4.1
    Chosen because it had a low cost-of-pass among high-accuracy backbones in Table 1; selection made on GAIA dev results.
  • Maximum steps = 8
    Chosen from {4, 8, 12} because Table 3 shows cost-of-pass 0.70 at 8 versus 0.98 at 12, with little accuracy loss; selection made on GAIA dev results.
  • Plan interval = 1
    Chosen from {1, 2, 4} based on Table 3; interval 1 had the lowest cost-of-pass, though interval 2 had higher accuracy; selection made on GAIA dev results.
  • Search source = Multi
    Chosen because multi crawler lowered cost-of-pass from 1.32 to 0.81 in Table 4; selection made on GAIA dev results.
  • Search number = 5
    Table 4 shows search number 10 beats 5 on cost-of-pass for the simple crawler, yet the final configuration uses 5; the selection criterion is unclear.
  • Best-of-N = 1
    Chosen because Table 2 shows Best-of-N increases cost-of-pass without meaningful accuracy gain; selection made on GAIA dev results.
  • Memory design = Simple
    Chosen because Table 5 shows Simple Memory has the lowest cost-of-pass and highest accuracy among the six designs; selection made on GAIA dev results.
assumptions (5)
  • domain assumption The GAIA development set is a representative proxy for agentic task complexity and cost behavior.
    All findings and the final configuration are derived from GAIA dev; no other benchmark or real-task validation is provided.
  • domain assumption Single-run pass@1 on GAIA estimates the success rate R_m(p) used in cost-of-pass.
    The paper reports one pass per problem and no variance; cost-of-pass divides cost by this single-run accuracy.
  • domain assumption Official API prices as of May 2025 are fixed and accurate for all models.
    Costs depend on per-token prices that are not listed in the paper and may change.
  • ad hoc to paper Component effects are independent, so individually selected settings combine into the claimed Efficient Agents performance.
    Each component is chosen from single-factor tables, then the combined configuration is reported without interaction checks.
  • domain assumption LLM outputs are stable enough that accuracy differences of one or two points are meaningful.
    Tables treat 53.33 vs 53.94 and similar small gaps as trends, with no repeated runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Agents: Building Effective Agents While Reducing Cost." pith.science (2026). https://pith.science/paper/CTUKKGOY

@misc{pith2026250802694,
  author       = {Pith},
  title        = {Pith review of: Efficient Agents: Building Effective Agents While Reducing Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CTUKKGOY}},
  note         = {Machine review of arXiv:2508.02694}
}
abstract

The remarkable capabilities of Large Language Model (LLM)-driven agents have enabled sophisticated systems to tackle complex, multi-step tasks, but their escalating costs threaten scalability and accessibility. This work presents the first systematic study of the efficiency-effectiveness trade-off in modern agent systems, addressing the critical need for cost-effective designs without sacrificing performance. We investigate three key questions: (1) How much complexity do agentic tasks inherently require? (2) When do additional modules yield diminishing returns? (3) How much efficiency can be gained through the design of efficient agent frameworks? Through an empirical analysis on the GAIA benchmark, we evaluate the impact of LLM backbone selection, agent framework designs, and test-time scaling strategies. Using the cost-of-pass metric, we quantify the efficiency-performance trade-off across these dimensions. Our findings inform the development of Efficient Agents , a novel agent framework that has an optimal complexity to task requirements. Efficient Agents retains 96.7% of the performance of OWL, one leading open-source agent framework, while reducing operational costs from $0.398 to $0.228, resulting in a 28.4% improvement in cost-of-pass. Our work provides actionable insights for designing efficient, high-performing agent systems, advancing the accessibility and sustainability of AI-driven solutions.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

    cs.AI 2026-07 accept novelty 7.0 of 10

    Storage footprint is a measurable, independent resource axis for LLM agents: equally accurate configurations differ 15.7× in retained bytes and most of the spread is removable duplication.

  2. OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review

    cs.SE 2026-08 conditional novelty 6.0 of 10

    OpenCodeReview reports up to 2.17x higher SEM-F1 than Claude Code and Codex on AACR-Bench, using 5-15x fewer tokens, by constraining agent review with rules, bounded tools, and diff-only reflection.

  3. CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems

    cs.SE 2026-02 conditional novelty 6.0 of 10

    CAM ranks how much each intermediate output of a multi-agent code-generation system causally matters for final correctness, then uses that ranking to repair failures and prune features.

  4. Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models

    cs.IR 2026-01 reject novelty 5.0 of 10

    A two-framework testbed comparison claims mem0 is Pareto-optimal over Graphiti for distributed LLM agents because its lower cost is paired with accuracy that is not significantly different.

  5. A Completely Blind Channel Estimation Technique for OFDM Using Constellation Splitting

    eess.SP 2025-07 unverdicted novelty 5.0 of 10

    A blind OFDM channel estimation scheme uses linear precoding and constellation splitting on alternate subcarriers to eliminate the complex-scalar ambiguity normally resolved with pilots.

  6. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

Reference graph

Works this paper leans on

52 extracted references · 15 canonical work pages · cited by 6 Pith papers

  1. [1]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. URLhttps://arxiv.org/abs/2308.08155

    arXiv 2023

  2. [2]

    Agents: An open-source framework for autonomous language agents

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents

  3. [3]

    Symbolic learning enables self-evolving agents

    Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang. Symbolic learning enables self-evolving agents. 2024. URL https://arxiv.org/abs/2406.18532

    arXiv 2024

  4. [4]

    Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023. URL https: //arxiv.org/abs/2305.13304

    arXiv 2023

  5. [5]

    Autogpt, 2025

    Significant Gravitas. Autogpt, 2025. URL https://github.com/Significant-Gravitas/AutoGPT. Accessed: 2025-05-13

    work page 2025

  6. [6]

    Oagents: An empirical study of building effective agents, 2025

    He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang, Changwang Zhang, Xiangru Tang, Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Jiaheng Liu, and Wangchunshu Zhou. Oagents: An empirical study of building effective agen...

    arXiv 2025

  7. [7]

    Scaling test-time compute for llm agents, 2025

    King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou. Scaling test-time compute for llm agents, 2025. URLhttps://arxiv.org/abs/2506.12928

    arXiv 2025

  8. [8]

    Infiagent-dabench: Evaluating agents on data analysis tasks, 2024

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. Infiagent-dabench: Evaluating agents on data analysis tasks, 2024. URLhttps://arxiv.org/abs/2401.05507

    arXiv 2024

Show all 52 references
  1. [9]

    Os agents: A survey on mllm-based agents for computer, phone and browser use, 2024

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for computer, phone and browser use, 2024

    2024

  2. [10]

    Taskcraft: Automated generation of agentic tasks, 2025

    Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Taskcraft: Automated generation of agentic ...

    2025 arXiv

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URLhttps://arxiv.org/abs/1810.04805

    2019 arXiv

  4. [12]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  5. [13]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URLhttps: //arxiv.org/abs/2001.08361

    2020 arXiv

  6. [14]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  7. [15]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URLhttps://arxiv.org/abs/1910.01108

    2020 arXiv

  8. [16]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung 11 Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa,...

    2024 arXiv

  9. [17]

    Deep research, 2025

    OpenAI. Deep research, 2025. URL https://openai.com/index/deep-research-system-card/. Accessed: 2025-05-13

    2025

  10. [18]

    manus, 2025

    Monica.ai. manus, 2025. URLhttps://manus.im/app. Accessed: 2025-05-13

    2025

  11. [19]

    Gaia: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023. URLhttps://arxiv.org/abs/2311.12983

    2023 arXiv

  12. [21]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation

    CAMEL-AI.org. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. https://github.com/camel-ai/owl, 2025. Accessed: 2025-03-07

    2025

  13. [22]

    Reasoning in token economies: Budget-aware evaluation of llm reasoning strategies

    Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkun. Reasoning in token economies: Budget-aware evaluation of llm reasoning strategies. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1991...

    2024

  14. [23]

    Understanding the planning of llm agents: A survey, 2024

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey, 2024. URLhttps://arxiv.org/abs/ 2402.02716

    2024 arXiv

  15. [24]

    Tool learning with foundation models, 2024

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, ...

    2024 arXiv

  16. [25]

    A survey on the memory mechanism of large language model based agents, 2024

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Rong Wen. A survey on the memory mechanism of large language model based agents, 2024. URLhttps: //arxiv.org/abs/2404.13501

    2024 arXiv

  17. [26]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URLhttps://arxiv.org/abs/2408.03314

    2024 arXiv

  18. [27]

    Cost-of-pass: An economic framework for evaluating language models.arXiv preprint arXiv:2504.13359, 2025

    Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, and James Zou. Cost-of-pass: An economic framework for evaluating language models.arXiv preprint arXiv:2504.13359, 2025

    2025

  19. [28]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  20. [29]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  21. [30]

    Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    2022

  22. [31]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025. URLhttps://arxiv.org/abs/2412.21187

    2025 arXiv

  23. [32]

    GPT-4.1: An Advanced Multimodal AI Model, 2025

    OpenAI. GPT-4.1: An Advanced Multimodal AI Model, 2025. Developed by OpenAI, available at https: //openai.com

    2025

  24. [33]

    Claude 3.7 sonnet.https://www.anthropic.com/news/claude-3-7-sonnet, February 2025

    Anthropic. Claude 3.7 sonnet.https://www.anthropic.com/news/claude-3-7-sonnet, February 2025. Accessed: 2025-05-11

    2025

  25. [34]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 12

    2025 arXiv

  26. [35]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URLhttps://qwenlm. github.io/blog/qwq-32b/

    2025

  27. [36]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URLhttps: //arxiv.org/abs/2407.21787

    2024 arXiv

  28. [37]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URLhttps://arxiv.org/abs/2210.03629

    2023 arXiv

  29. [38]

    Webgpt: Browser-assisted question-answering with human feedback, 2022

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webg...

    2022 arXiv

  30. [39]

    Memory and new controls for ChatGPT, February 2025

    OpenAI. Memory and new controls for ChatGPT, February 2025. URL https://openai.com/index/ memory-and-new-controls-for-chatgpt/ . Accessed: 2025-05-14

    2025

  31. [40]

    ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    2025

  32. [41]

    Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URLhttps://arxiv.org/abs/2504.12516

    2025 arXiv

  33. [42]

    BERT-of-theseus: Compressing BERT by progressive module replacing

    Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of-theseus: Compressing BERT by progressive module replacing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859–7869, Online, November 2020. Associatio...

    2020

  34. [43]

    Bert loses patience: Fast and robust inference with early exit

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330–18341. Curran Associates, Inc., 2020. URLhttps://proceedings.n...

    2020

  35. [44]

    Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression

    Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, and Furu Wei. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference ...

    2021 doi

  36. [45]

    BERT learns to teach: Knowledge distillation with meta learning

    Wangchunshu Zhou, Canwen Xu, and Julian McAuley. BERT learns to teach: Knowledge distillation with meta learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volum...

    2022 doi

  37. [46]

    Modular transformers: Compressing transformers into modularized layers for flexible efficient inference

    Wangchunshu Zhou, Ronan Le Bras, and Yejin Choi. Modular transformers: Compressing transformers into modularized layers for flexible efficient inference. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics:...

    2023 doi

  38. [47]

    Efficient prompting via dynamic in-context learning, 2023

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Ryan Cotterell, and Mrinmaya Sachan. Efficient prompting via dynamic in-context learning, 2023. URLhttps://arxiv.org/abs/2305.11170

    2023 arXiv

  39. [48]

    EfficientVLM: Fast and accurate vision- language models via knowledge distillation and modal-adaptive pruning

    Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. EfficientVLM: Fast and accurate vision- language models via knowledge distillation and modal-adaptive pruning. In Anna Rogers, Jordan Boyd-Graber, and NaoakiOkazaki, editors,FindingsoftheAssociation forComputationalL...

    2023 doi

  40. [49]

    Uora: Uniform orthogonal reinitialization adaptation in parameter-efficient fine-tuning of large models

    Xueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong, Shuhao Guan, Linbo Cao, and Yining Wang. Uora: Uniform orthogonal reinitialization adaptation in parameter-efficient fine-tuning of large models. 2025. URL https://arxiv.org/abs/2505.20154

    2025 arXiv

  41. [50]

    Token-budget-aware llm reasoning

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547, 2024

    2024 arXiv

  42. [51]

    Cut the crap: An economical communication pipeline for llm-based multi-agent systems

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024

    arXiv

  43. [52]

    analysis

    Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Budgetmlagent: A cost-effective llm multi-agent system for automating machine learning tasks, 2025. URLhttps://arxiv.org/abs/2411.07464. 14 Appendix A Default Setup Table 8 details the default setup for conduc...

    2025 arXiv

  44. [2023]

    URL https://arxiv.org/abs/2309.07870

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.