Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Adapted harnesses let small models match frontier agents on routine business tasks at roughly 4% of the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 05:38 UTC pith:3KEHE2B4

load-bearing objection Solid systems paper: automated harness search can make SLMs recover most frontier accuracy on repetitive business agents at ~4% cost, with honest limits on diversity and base capability. the 3 major comments →

arxiv 2607.08938 v1 pith:3KEHE2B4 submitted 2026-07-09 cs.SE

Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation

Chenyang Yang , Xinran Zhao , Tongshuang Wu , Christian K\"astner This is my paper
classification cs.SE
keywords agentssmall language modelsharness adaptationcost-efficient AI deploymentmeta-agent optimizationtool-usebusiness workflows
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Frontier language-model agents can automate routine business work, but their inference bills make large-scale use expensive. Smaller open models are far cheaper, yet they usually fail when dropped into a harness built for a frontier model. This paper shows that much of the hard work is shared across task instances and can be moved out of the model into the surrounding harness—through clearer instructions, filtered or newly written tools, and simple runtime checks. A meta-agent that reads failure trajectories can automatically discover those harness changes. On seven business-style agent tasks and three small-model families, the adapted harnesses raise performance on 16 of 21 pairings; seven pairings close the gap with the frontier model, and the best small agent recovers about 90% of frontier accuracy at 4% of the cost. The gains are largest when workflows are repetitive and the small model already has decent base skills.

Core claim

For many routine business agent tasks, small language models paired with automatically adapted harnesses can recover most of frontier-LLM performance at a small fraction of the inference cost, because shared task difficulty can be lifted from the model into tailored instructions, tools, and orchestration loops.

What carries the argument

A failure-mode-to-adaptation framework (tool-use, instruction-following, knowledge, long-context, planning mapped onto context, tool, and loop edits) plus a meta-agent harness optimizer that diagnoses trajectories and searches over prompts, tools, hooks, filters, and sub-agents.

Load-bearing premise

A harness tuned on a small, clean, verifiable train/validation split of curated tasks will stay useful in messier real deployments and as models and tools change.

What would settle it

Re-run the same seven tasks with the published optimized harnesses on a larger, messier held-out business workload (or with a new small-model generation) and check whether the reported accuracy and cost ratios collapse toward the unadapted baseline.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that for routine business agentic tasks, small language models (SLMs) can approach frontier-LLM agent performance at roughly an order-of-magnitude lower inference cost if the agent harness (instructions, tools, hooks, orchestration) is specialized rather than reused from an LLM-oriented design. The authors contribute (i) a capability-indexed map from agent failure modes (tool-use, instruction-following, knowledge, long-context, planning) to harness adaptation strategies (context, tools, loops), (ii) a meta-agent harness optimizer over a software-agent-sdk search space that diagnoses failure trajectories and proposes edits, and (iii) an empirical study on seven business-oriented tasks × three open-weight SLMs against a gemini-3.1-pro baseline. Optimized harnesses improve 16/21 task–SLM pairs, close the gap on seven pairs, and the best SLM (gemma-4-26b-a4b) recovers ~89% of LLM average accuracy at ~4% cost (Table II). Follow-up analyses show gains are larger on less diverse workflows (RQ2) and stronger base models (RQ3), and that successful edits mostly add context and reshape tools to address instruction-following and knowledge failures (RQ4).

Significance. If the results hold under broader deployment conditions, this is a practically important systems contribution for cost-efficient agent deployment: it reframes SLM competitiveness as a harness-search problem rather than only a model-scale problem, and supplies both a diagnostic framework and an automated optimizer. Strengths include a multi-task, multi-model design with held-out tests; three optimization runs with best-validation selection and three averaged evaluation runs; an explicit controlled diversity experiment; honest negative findings (e.g., no successful sub-agent adaptations; weak SLMs remain hard to rescue); and released code. The work is well positioned for the agent-engineering and software-engineering communities concerned with production cost, not only peak benchmark scores.

major comments (3)
  1. [Table II / §IV-D RQ1] Table II and the RQ1 headline (abstract; §IV-D): the 89.7%-of-LLM / 4%-cost claim is an unweighted mean over seven tasks. For the best SLM, residual gaps remain large on stock-alert (58.9 vs 86.7), website-management (45.6 vs 76.7), and code-refactoring (65.0 vs 86.1), while near-ceiling results on attendance, budget, anomaly, and playwright carry the average. RQ2 already shows Spearman ρ=−0.96 and a controlled drop from 89.1% to 68.0% as template diversity rises. The central claim is defensible for repetitive workflows, but the main-result narrative should lead with per-task residual gaps and the diversity condition rather than presenting the suite average as the primary takeaway.
  2. [§IV-D Results (RQ1)] §IV-D states that optimized harnesses “significantly improve performance on 16 of 21 task–SLM pairs” and that seven pairs “close the SLM–LLM performance gap,” but the manuscript does not report a statistical procedure (paired tests over the three evaluation runs, confidence intervals, or a pre-specified closing criterion). With n=3 runs and high agent stochasticity (§IV-C), “significantly” and “closing the gap” need an explicit definition and test; otherwise the count of 16/21 and 7 closings is hard to audit.
  3. [§IV-A / §IV-C / RQ2] §IV-A uses a 20/20/60 split and a fixed ~$20 meta-agent budget, with harness search driven by trajectories on 20 training instances. Combined with RQ2’s finding that reusable workflow skeletons are the load-bearing condition, this raises a concrete external-validity risk for the claim that offline-discovered harnesses transfer to messier business deployments (acknowledged in §IV-C but not stress-tested). A minimal addition would help: e.g., sensitivity to train size, or a held-out workflow-template split on the controlled budget/attendance variants, so readers can see how much of the gain depends on seeing the same skeleton in train.
minor comments (5)
  1. [Footnote 1] Footnote 1 has a corrupted URL prefix (“gtbhttps://github.com/...”).
  2. [§IV-C] Section heading “Threats to V alidity” appears to contain a stray space in “Validity.”
  3. [Figure 5] Figure 5 caption and axes are clear, but the left panel’s diversity proxy (average pairwise normalized Levenshtein on LLM tool-call sequences) should be named in the figure caption, not only in the text, so the plot is self-contained.
  4. [Table I / Table II] Table I lists N=50 for website-management vs 100 for others; a one-line note on whether averages in Table II weight tasks equally (they appear to) would avoid ambiguity.
  5. [§VI] Related work (§VI) is appropriate; a slightly sharper contrast with concurrent Life-Harness and Meta-Harness on objective (cost–performance for specialized SLMs vs transfer/general harness evolution) would help readers place the contribution.

Circularity Check

0 steps flagged

Empirical systems paper with held-out test evaluation; no derivation reduces to its inputs by construction.

full rationale

This paper does not present a first-principles derivation or a fitted-parameter-as-prediction chain. Its central claims (Table II: optimized harnesses improve 16/21 task–SLM pairs; best adapted gemma recovers ~89.7% of gemini average accuracy at ~4% cost) are measured outcomes on held-out 60% test splits after train/val search, with objective task metrics (exact match, executable checks, runnable tests). The failure-mode → adaptation framework (Section II, Fig. 2) is a literature-synthesized taxonomy used for post-hoc inspection (RQ4), not a definition that equates success with the proposed edits. Harness selection via GEPA-style Pareto sampling on validation scores is standard optimizer practice and does not force test numbers by construction. Self-citations (e.g., underspecification work sharing authors) are peripheral, not load-bearing uniqueness theorems. No equation, uniqueness claim, or ansatz is smuggled in such that the reported recovery equals the optimization objective by definition. Residual concerns about low-diversity task dominance (RQ2) and small train/val sets are external-validity / correctness issues, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on experimental design choices and domain premises about when difficulty can be offloaded from models to harnesses, not on free physical constants or new ontological entities. The ledger below lists the knobs and assumptions that actually carry the result.

free parameters (4)
  • per-task optimization budget
    Fixed at $20 after early piloting; directly limits search depth and which harnesses can be discovered.
  • train/validation/test split sizes
    20/20/60 instances per task; small train/val sets determine what failure patterns the meta-agent sees and which harness is selected.
  • number of optimization and evaluation repeats
    Three optimization runs (keep best val) and three eval runs averaged; stochasticity is only partially controlled.
  • task-diversity proxy (avg pairwise normalized Levenshtein on LLM tool-call sequences)
    Hand-chosen operationalization of 'diversity' used to support RQ2; alternative diversity measures could reorder tasks.
axioms (5)
  • domain assumption For many routine business agent tasks, substantial difficulty is shared across instances and can be externalized into prompts, tools, hooks, and orchestration rather than remaining in model weights.
    Core premise of abstract/intro and Fig. 1–2; RQ2 tests sensitivity but the deployment claim depends on this holding outside the suite.
  • domain assumption Objective task metrics (exact match, executable checks, tests) are adequate proxies for successful business automation.
    Used throughout §IV task definitions; real deployments may care about partial credit, policy nuance, or human preference.
  • domain assumption A frontier LLM meta-agent inspecting trajectories can propose useful harness edits within a constrained SDK search space (GEPA-style Pareto sampling).
    Method §III; if diagnosis quality is poor, reported gains would not materialize (authors discuss this in §V-A).
  • domain assumption API list prices and measured token usage yield a fair cost comparison across models and harnesses.
    RQ1 cost methodology; ignores self-hosting, batching, caching, and non-token ops costs.
  • standard math Standard empirical ML practice (held-out test after validation selection) supports generalization claims within each task distribution.
    Ordinary train/val/test discipline; not a formal proof of out-of-distribution deployment validity.
invented entities (2)
  • Capability-indexed failure-mode → harness-adaptation map (tool-use, instruction-following, knowledge, long-context, planning mapped to context/tool/loop edits) no independent evidence
    purpose: Organize diagnosis and explain why adapted harnesses help SLMs.
    Synthesized taxonomy from model cards/benchmarks and prior failure taxonomies; useful framing rather than a new physical entity. Independent evidence is the qualitative RQ4 audit, not an external measurement apparatus.
  • Meta-agent harness optimizer over software-agent-sdk components independent evidence
    purpose: Automatically search harness edits from failure trajectories without hand-engineering each task–model pair.
    Method contribution built from existing agent-SDK and search ideas; existence is demonstrated by code/experiments, but it is an engineered system, not a postulated natural object.

pith-pipeline@v1.1.0-grok45 · 22767 in / 3550 out tokens · 54442 ms · 2026-07-13T05:38:52.523478+00:00 · methodology

0 comments
read the original abstract

Frontier LLM agents are automating many business tasks, but their high inference cost makes large-scale deployment unsustainable. Small language models (SLMs) offer a cheaper alternative, yet they typically fall short when swapped into a harness designed for a frontier LLM. We show that for many routine business tasks, SLM agents can match LLM performance at 90% lower cost, when paired with an adapted harness that can be automatically discovered by a meta agent. The key insight is that much of the task difficulty is shared across instances and can be lifted from the model into the harness via tailored instructions, tools, and orchestration loops. To study this systematically, we create a framework that maps agent failure modes to harness adaptation strategies, and build a harness optimizer that automatically discovers effective adaptations from failure trajectories. Across seven business-oriented agentic tasks and three SLM families, we found optimized harnesses significantly improve performance on 16 of 21 task-SLM pairs, with seven pairs closing the SLM-LLM performance gap and the best SLM agent recovering 89.7% of LLM performance at 4% of the cost. Our analysis further shows that adaptation works best for tasks with more repetitive workflows and for SLMs with sufficient base capabilities. Together, these results suggest that harness adaptation can expand the practical deployment range of SLM agents in routine business tasks.

Figures

Figures reproduced from arXiv: 2607.08938 by Chenyang Yang, Christian K\"astner, Tongshuang Wu, Xinran Zhao.

Figure 1
Figure 1. Figure 1: Small language models often perform poorly when naively swapped into an agent harness designed for frontier LLMs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mapping Agent Failure Modes to Harness Adaptations. When model capabilities fall short of task demands, this mismatch manifests as agent failures in trajectories. These gaps can be bridged with harness adaptation, which we group into three categories: adapting contexts, tools, and agent loops. a) tool-use failures: Tool-use failures occur when the model selects the wrong tool, produces a malformed tool cal… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the automated harness optimization loop. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Simplified harness code identified by the meta-agent [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Less capable models are harder to optimize. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 34 linked inside Pith

  1. [1]

    How Cox automotive launched AI agents at scale using Amazon Bedrock AgentCore,

    Amazon Web Services, “How Cox automotive launched AI agents at scale using Amazon Bedrock AgentCore,” https://aws.amazon.com/ko/so lutions/case-studies/cox-auto-case-study/, 2026

    2026

  2. [2]

    Engine provides personalized service to 1 million travelers with Agentforce,

    Salesforce, “Engine provides personalized service to 1 million travelers with Agentforce,” https://www.salesforce.com/customer-stories/engine/, 2026

    2026

  3. [3]

    Anthropic clinches $380 billion valuation after $30 billion funding round,

    Reuters, “Anthropic clinches $380 billion valuation after $30 billion funding round,” https://www.reuters.com/technology/anthropic-valued-3 80-billion-latest-funding-round-2026-02-12/, 2026

    2026

  4. [4]

    Uber CTO shows how claude code can blow up AI budgets,

    L. Bratton, “Uber CTO shows how claude code can blow up AI budgets,” https://www.theinformation.com/newsletters/applied-ai/uber-cto-shows -claude-code-can-blow-ai-budgets, 2026

    2026

  5. [5]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Kimi K2. 5: Visual Agentic Intelligence,

    K. Team, T. Bai, Y . Bai, Y . Bao, S. Cai, Y . Cao, Y . Charles, H. Che, C. Chen, G. Chenet al., “Kimi K2. 5: Visual Agentic Intelligence,”arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Deepseek-v4: Towards highly efficient million-token context intelligence,

    DeepSeek-AI, “Deepseek-v4: Towards highly efficient million-token context intelligence,” 2026

    2026

  8. [8]

    Small language models are the future of agentic ai,

    P. Belcak, G. Heinrich, S. Diao, Y . Fu, X. Dong, S. Muralidharan, Y . C. Lin, and P. Molchanov, “Small language models are the future of agentic ai,”arXiv preprint arXiv:2506.02153, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Introducing the openhands index,

    OpenHands Team, “Introducing the openhands index,” https://openhands. dev/blog/openhands-index, 2026

    2026

  10. [10]

    Darwin godel machine: Open-ended evolution of self-improving agents,

    J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune, “Darwin godel machine: Open-ended evolution of self-improving agents,”arXiv preprint arXiv:2505.22954, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Meta- harness: End-to-end optimization of model harnesses,

    Y . Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn, “Meta- harness: End-to-end optimization of model harnesses,”arXiv preprint arXiv:2603.28052, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Introducing gpt-5.2,

    OpenAI, “Introducing gpt-5.2,”OpenAI, 2025. [Online]. Available: https://openai.com/index/introducing-gpt-5-2/

    2025

  13. [13]

    Introducing claude opus 4.7,

    Anthropic, “Introducing claude opus 4.7,”Anthropic, 2026. [Online]. Available: https://www.anthropic.com/news/claude-opus-4-7

    2026

  14. [14]

    Qwen3.5: Accelerating productivity with native multimodal agents,

    Q. Team, “Qwen3.5: Accelerating productivity with native multimodal agents,” February 2026. [Online]. Available: https://qwen.ai/blog?id=qw en3.5

    2026

  15. [15]

    The Berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models,

    S. G. Patil, H. Mao, C. Cheng-Jie Ji, F. Yan, V . Suresh, I. Stoica, and J. E. Gonzalez, “The Berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models,” inF orty-second International Conference on Machine Learning, 2025

    2025

  16. [16]

    Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers,

    C. Bandi, B. Hertzberg, G. Boo, T. Polakam, J. Da, S. Hassaan, M. Sharma, A. Park, E. Hernandez, D. Rambadoet al., “Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers,” arXiv preprint arXiv:2602.00933, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Instruction-following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Measuring short-form factuality in large language models,

    J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Measuring short-form factuality in large language models,”arXiv preprint arXiv:2411.04368, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,”arXiv preprint arXiv:2403.07974, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks,

    T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P. Fishman, M. Aljubeh, P. Thacker, L. Fauconnetet al., “Gdpval: Evaluating ai model performance on real-world economically valuable tasks,”arXiv preprint arXiv:2510.04374, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchananet al., “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “Webarena: A realistic web environment for building autonomous agents,”arXiv preprint arXiv:2307.13854, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Browsecomp: A simple yet challenging benchmark for browsing agents,

    J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese, “Browsecomp: A simple yet challenging benchmark for browsing agents,”arXiv preprint arXiv:2504.12516, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    What prompts don’t say: Understanding and managing underspecification in llm prompts,

    C. Yang, Y . Shi, Q. Ma, M. X. Liu, C. Kästner, and T. Wu, “What prompts don’t say: Understanding and managing underspecification in llm prompts,”arXiv preprint arXiv:2505.13360, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandranet al., “Why do multi-agent llm systems fail?”arXiv preprint arXiv:2503.13657, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Where llm agents fail and how they can learn from failures,

    K. Zhu, Z. Liu, B. Li, M. Tian, Y . Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhanget al., “Where llm agents fail and how they can learn from failures,”arXiv preprint arXiv:2509.25370, 2025

    arXiv 2025

  27. [27]

    Claude code overview,

    Anthropic, “Claude code overview,” https://code.claude.com/docs/en/ov erview, 2026

    2026

  28. [28]

    OpenAI, “Codex,” https://developers.openai.com/codex, 2026

    2026

  29. [29]

    Agentic context engineering: Evolving contexts for self-improving language models,

    Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V . Kamanuru, J. Rainton, C. Wu, M. Ji, H. Liet al., “Agentic context engineering: Evolving contexts for self-improving language models,”arXiv preprint arXiv:2510.04618, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Evaluating agents. md: Are repository-level context files helpful for coding agents?

    T. Gloaguen, N. Mündler, M. Müller, V . Raychev, and M. Vechev, “Evaluating agents. md: Are repository-level context files helpful for coding agents?”arXiv preprint arXiv:2602.11988, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Skill authoring best practices,

    Anthropic, “Skill authoring best practices,” https://platform.claude.com/ docs/en/agents-and-tools/agent-skills/best-practices, 2026

    2026

  32. [32]

    Recursive language models,

    A. L. Zhang, T. Kraska, and O. Khattab, “Recursive language models,” arXiv preprint arXiv:2512.24601, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Acon: Optimizing context compression for long-horizon llm agents,

    M. Kang, W.-N. Chen, D. Han, H. A. Inan, L. Wutschitz, Y . Chen, R. Sim, and S. Rajmohan, “Acon: Optimizing context compression for long-horizon llm agents,”arXiv preprint arXiv:2510.00615, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Improving the efficiency of llm agent systems through trajectory reduction,

    Y .-A. Xiao, P. Gao, C. Peng, and Y . Xiong, “Improving the efficiency of llm agent systems through trajectory reduction,”arXiv preprint arXiv:2509.23586, 2025

    arXiv 2025

  35. [35]

    Boad: Discovering hierarchical software engineering agents via bandit optimization,

    I. Xu, G. Zeng, Z. He, C. Jin, A. Pareja, D. Gutfreund, C. Gan, and Z.-W. Hong, “Boad: Discovering hierarchical software engineering agents via bandit optimization,”arXiv preprint arXiv:2512.23631, 2025

    arXiv 2025

  36. [36]

    Introducing the model context protocol,

    Anthropic, “Introducing the model context protocol,” https://www.an thropic.com/news/model-context-protocol, Nov. 2024, announcement, accessed 2026-05-02

    2024

  37. [37]

    Inducing programmatic skills for agentic tasks,

    Z. Z. Wang, A. Gandhi, G. Neubig, and D. Fried, “Inducing programmatic skills for agentic tasks,”arXiv preprint arXiv:2504.06821, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Agentoccam: A simple yet strong baseline for llm-based web agents,

    K. Yang, Y . Liu, S. Chaudhary, R. Fakoor, P. Chaudhari, G. Karypis, and H. Rangwala, “Agentoccam: A simple yet strong baseline for llm-based web agents,”arXiv preprint arXiv:2410.13825, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Don’t adapt small language models for tools; adapt tool schemas to the models,

    J. Lee, W. Song, J. Han, H. Pyun, and Y . Jo, “Don’t adapt small language models for tools; adapt tool schemas to the models,”arXiv preprint arXiv:2510.07248, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Intercept and control agent behavior with hooks,

    Anthropic, “Intercept and control agent behavior with hooks,” https: //code.claude.com/docs/en/agent-sdk/hooks, 2026

    2026

  41. [41]

    Symbolic guardrails for domain-specific agents: Stronger safety and security guarantees without sacrificing utility,

    Y . Hong, Y . She, E. Kang, C. S. Timperley, and C. Kästner, “Symbolic guardrails for domain-specific agents: Stronger safety and security guarantees without sacrificing utility,”arXiv preprint arXiv:2604.15579, 2026

    Pith/arXiv arXiv 2026

  42. [42]

    Towards a science of scaling agent systems,

    Y . Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y . Yan, Z. Zhang, Y . Zhuang, Y . Liuet al., “Towards a science of scaling agent systems,”arXiv preprint arXiv:2512.08296, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Magentic-one: A generalist multi-agent system for solving complex tasks,

    A. Fourney, G. Bansal, H. Mozannar, C. Tan, E. Salinas, F. Niedtner, G. Proebsting, G. Bassman, J. Gerrits, J. Alberet al., “Magentic-one: A generalist multi-agent system for solving complex tasks,”arXiv preprint arXiv:2411.04468, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Linet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inThe twelfth international conference on learning representations, 2023

    2023

  45. [45]

    Harness design for long-running application development,

    Anthropic, “Harness design for long-running application development,” https://www.anthropic.com/engineering/harness-design-long-running-a pps, 2026

    2026

  46. [46]

    The bitter lesson,

    R. Sutton, “The bitter lesson,” http://www.incompleteideas.net/IncIdeas /BitterLesson.html, 2019

    2019

  47. [47]

    Live-swe-agent: Can software engineering agents self-evolve on the fly?

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang, “Live-swe-agent: Can software engineering agents self-evolve on the fly?”arXiv preprint arXiv:2511.13646, 2025

    arXiv 2025

  48. [49]

    Gepa: Reflective prompt evolution can outperform reinforcement learning,

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jianget al., “Gepa: Reflective prompt evolution can outperform reinforcement learning,”arXiv preprint arXiv:2507.19457, 2025

    Pith/arXiv arXiv 2025

  49. [50]

    Theagentcompany: Benchmarking llm agents on consequential real world tasks,

    F. F. Xu, Y . Song, B. Li, Y . Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y . Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y . Xie, S. Zhou, and G. Neubig, “Theagentcompany: Benchmarking llm agents on consequential real world tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2412.14161

    Pith/arXiv arXiv 2024

  50. [51]

    Loca-bench: Benchmarking language agents under controllable and extreme context growth,

    W. Zeng, Y . Huang, and J. He, “Loca-bench: Benchmarking language agents under controllable and extreme context growth,”arXiv preprint arXiv:2602.07962, 2026

    arXiv 2026

  51. [52]

    Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch,

    Z. Lu, Y . Yang, H. Ren, H. Hou, H. Xiao, K. Wang, W. Shi, A. Zhou, M. Zhan, and H. Li, “Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch,” 2025. [Online]. Available: https://arxiv.org/abs/2505.03733

    Pith/arXiv arXiv 2025

  52. [53]

    Refactorbench: Evaluating stateful reasoning in language agents through code,

    D. Gautam, S. Garg, J. Jang, N. Sundaresan, and R. Z. Moghaddam, “Refactorbench: Evaluating stateful reasoning in language agents through code,”arXiv preprint arXiv:2503.07832, 2025

    Pith/arXiv arXiv 2025

  53. [54]

    The openhands software agent sdk: A composable and extensible foundation for production agents,

    X. Wang, S. Rosenberg, J. Michelini, C. Smith, H. Tran, E. Nyst, R. Malhotra, X. Zhou, V . Chen, R. Brennan, and G. Neubig, “The openhands software agent sdk: A composable and extensible foundation for production agents,” 2025. [Online]. Available: https: //arxiv.org/abs/2511.03690

    Pith/arXiv arXiv 2025

  54. [55]

    Artificial analysis,

    Artificial Analysis, “Artificial analysis,” https://artificialanalysis.ai/

  55. [56]

    Building effective agents,

    E. S. and B. Zhang, “Building effective agents,”Anthropic Engineering Blog, 2024. [Online]. Available: https://www.anthropic.com/engineering/ building-effective-agents

    2024

  56. [57]

    Effgen: Enabling small language models as capable autonomous agents,

    G. Srivastava, A. Hussain, C. Wang, Y . C. Lin, and X. Wang, “Effgen: Enabling small language models as capable autonomous agents,”arXiv preprint arXiv:2602.00887, 2026

    arXiv 2026

  57. [58]

    Automated design of agentic systems,

    S. Hu, C. Lu, and J. Clune, “Automated design of agentic systems,” arXiv preprint arXiv:2408.08435, 2024

    Pith/arXiv arXiv 2024

  58. [59]

    Agentsquare: Automatic llm agent search in modular design space,

    Y . Shang, Y . Li, K. Zhao, L. Ma, J. Liu, F. Xu, and Y . Li, “Agentsquare: Automatic llm agent search in modular design space,”arXiv preprint arXiv:2410.06153, 2024

    Pith/arXiv arXiv 2024

  59. [60]

    Multi-agent ar- chitecture search via agentic supernet,

    G. Zhang, L. Niu, J. Fang, K. Wang, L. Bai, and X. Wang, “Multi-agent ar- chitecture search via agentic supernet,”arXiv preprint arXiv:2502.04180, 2025

    Pith/arXiv arXiv 2025

  60. [61]

    Multi-agent design: Optimizing agents with better prompts and topologies,

    H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vuli´c, A. Korhonen, and S. Ö. Arık, “Multi-agent design: Optimizing agents with better prompts and topologies,”arXiv preprint arXiv:2502.02533, 2025

    arXiv 2025

  61. [62]

    Adapting the interface, not the model: Runtime harness adaptation for deterministic llm agents,

    T. Xu, H. Wen, and M. Li, “Adapting the interface, not the model: Runtime harness adaptation for deterministic llm agents,”arXiv preprint arXiv:2605.22166, 2026

    Pith/arXiv arXiv 2026