Pith. sign in

REVIEW 3 major objections 6 minor 135 references

About one in six recorded FAIL scores for computer-use agents is wrong, and real failures are mostly about feedback and planning—not clicks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 09:34 UTC pith:CTG3KL5S

load-bearing objection Solid measurement paper: ~15% of public CUA FAIL scores are checker/task faults, and genuine failures skew to verification and planning—not clicks. the 3 major comments →

arxiv 2607.28367 v1 pith:CTG3KL5S submitted 2026-07-30 cs.AI

How Benchmarks Mis-Score Computer-Use Agents

Zihan Dong , Zhiyuan Ma , Zekun Wang , Yunqing Li , Zirou Liu , Ruixuan Deng , Qishi Zhan , Rui Qian This is my paper
classification cs.AI
keywords computer-use agentsbenchmark reliabilityevaluator false negativestrajectory auditfailure taxonomyweb agentsdesktop agentsevaluation pipeline
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Computer-use agents are judged by benchmark scores that come from a four-stage measurement pipeline—task construction, trajectory capture, scoring, and reporting—not from direct observation of skill. This paper argues those pipelines systematically mis-score agents: tasks go stale or break, trajectories omit the visual evidence needed to judge them, scripted oracles reject valid solutions, and a single success rate hides why something failed. Auditing 150 public FAIL trajectories across five web, enterprise, and desktop benchmarks, the authors find 15.3% of FAIL verdicts are incorrect—mostly evaluator false negatives, plus broken tasks. Among genuine failures, verification/feedback errors and planning errors dominate grounding and execution mistakes. The practical claim is that leaderboards and engineering priorities will stay misaligned until benchmarks version tasks and environments, score exactly what the task states, release screenshot-complete trajectories, and report failure locus and process stage instead of a scalar alone.

Core claim

A CUA benchmark score is a pipeline output, and that pipeline is unreliable enough that 15.3% of audited FAIL verdicts should not be trusted as agent failures: 10.7% are evaluator false negatives and 4.7% are broken tasks. When the failure is genuine, a three-tier diagnosis shows verification/feedback failures (39.3%) and planning failures (35.2%) dominate execution/grounding errors (13.9%), so a scalar success rate cannot explain what went wrong or where to fix it.

What carries the argument

A four-stage reliability framework (task construction, trajectory observation, scoring, reporting) plus a two-stage audit: first decide whether a FAIL is a genuine agent failure, evaluator false negative, broken task, or unclear; then, only for genuine failures, assign the earliest decisive break with a three-tier taxonomy (planning/specification, execution/grounding, verification/feedback).

Load-bearing premise

The combined human-and-model labels used as ground truth really do settle whether an open-ended trajectory satisfied the task or the environment was broken.

What would settle it

Independently re-label the same 150 FAIL trajectories with a larger blind human panel (or a PASS-side audit) and check whether the wrong-verdict rate stays near 15.3% and whether verification/planning still dominate grounding; a collapse of those rates or a reverse failure mix would overturn the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Leaderboards that treat every FAIL as an agent deficit will mis-rank systems whenever checkers reject valid alternatives or environments are broken.
  • Engineering effort aimed mainly at finer GUI grounding will miss the larger failure mass in feedback-blind repetition and planning loops.
  • Benchmarks must jointly version tasks, environments, and oracles and pre-release-replay solvability, or stale and unsatisfiable tasks will keep entering scores.
  • Released trajectories need screenshots, tool I/O, and state transitions; without them, both checker audits and process diagnoses become unreliable.
  • Reports should attach failure locus and process-stage labels—and eventually latency/cost per verified success—rather than only a binary success rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If FAIL noise is already ~15% on short public traces, long-horizon hour-scale benchmarks will need even stricter evidence and checker calibration or their rankings will be noisier still.
  • A cheap automatic probe for repeated actions with unchanged screens could triage the largest genuine-failure class before expensive judges are used.
  • Procurement and safety claims that rest on CUA leaderboard gaps should treat those gaps as upper bounds on capability differences until PASS verdicts are audited too.
  • Co-designing task statements with checkers against positive and negative cases is as central as model training for trustworthy computer-use evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that CUA benchmark scores are pipeline outputs subject to task invalidity, incomplete trajectories, fragile oracles, and diagnostically opaque reporting. It organizes these threats into a four-stage reliability framework, then audits 150 public FAIL-scored trajectories from OSWorld-Verified and four AgentRewardBench web/enterprise suites. Using dual vision-LLM judges plus human blind review of disagreements and a stratified agreement sample, it estimates that 15.3% of FAIL verdicts are wrong (10.7% evaluator false negatives, 4.7% broken tasks). Among 122 genuine failures, a three-tier taxonomy adapted from MAST attributes most errors to verification/feedback (39.3%) and planning (35.2%) rather than execution/grounding (13.9%). The paper then derives stage-specific baseline controls and forward-looking design rules for task construction, environments, scoring, and reporting.

Significance. If the audit rates hold under reasonable label uncertainty, the paper supplies concrete, trajectory-level evidence that current CUA leaderboards systematically mis-attribute failures and understate oracle/task faults—an important corrective for a fast-moving evaluation literature that still leans on scripted checkers. Strengths include a documented stratified sample (seed 20260717), dual independent LLM labeling, human review of all LLM disagreements, Wilson intervals, screenshot-ablation evidence that observability changes verdicts, and an explicit FAIL-only scope limitation. The four-stage pipeline framing and the design checklist in Table 3 are reusable contributions beyond the specific 15.3% figure. The work is empirical measurement rather than circular derivation; its main value is diagnostic discipline for CUA evaluation practice.

major comments (3)
  1. [§4.2–4.4, Table 2] §4.2–4.4 and Table 2: The headline 15.3% wrong-FAIL rate treats post-combination Stage-1 labels as ground truth for open-ended task satisfaction. On the 104-row common review set the four annotators’ own wrong-verdict rates already span 7.7–20.2% (mean 13.7%, SD 5.1 pp), and screenshot ablation flips 12–13 LLM verdicts and roughly halves detected EFNs. The paper reports these facts but still presents 15.3% [10.4, 22.0] as the primary population estimate without a label-uncertainty or sensitivity analysis that propagates annotator disagreement into the rate (e.g., bounds under each annotator, or bootstrap over the combination rule). Because this rate is the central empirical claim, the main text should either (i) report a label-robust interval or multi-annotator envelope alongside the Wilson CI, or (ii) clearly demote 15.3% to “combined-label point estimate under our adjudication rule” an
  2. [§4.1, Abstract, Limitations] §4.1 and Limitations: Sampling only released FAIL trajectories is appropriate for false-negative auditing but means the title claim “benchmarks mis-score” is only half-identified—false positives among PASS verdicts are unmeasured. The Limitations section states this, yet the abstract, introduction, and §4.4 implications still generalize to “when a CUA verdict should be distrusted” without consistently restricting to FAIL verdicts. Tighten claim language throughout so “15.3% of FAIL verdicts are wrong” is never elided into a symmetric mis-scoring rate, and flag the missing PASS audit as necessary before using the figure for leaderboard adjustment.
  3. [§4.3–4.4, Table 2] §4.3–4.4, lower panel of Table 2: The claim that verification/feedback and planning dominate execution/grounding is load-bearing for the “scalar success rate cannot explain” thesis, but Stage-2 agreement is only moderate (LLM κ=0.41; four-rater Fleiss’ κ=0.36 on the stress set), with 6.6% ambiguous and 4.9% outside tiers after combination. The earliest-decisive-failure boundary rule is reasonable but under-validated given that disagreement. Either restrict strong distributional claims to Tier-1 aggregate contrasts that survive alternative coding (e.g., dual-coder consensus only), or add a short sensitivity table showing tier shares under each human group and under “any Tier-3 vs primary Tier-3.” As written, the 39.3% / 35.2% / 13.9% split is more fragile than the Stage-1 wrong-verdict finding and should be marked as such in the results discussion.
minor comments (6)
  1. [Table 1] Table 1 is useful but dense; several 2026 arXiv entries are contemporaneous with this manuscript. A one-line note on inclusion cutoff date would help readers interpret coverage.
  2. [§4.2] §4.2: “Claude Sonnet” revision is not pinned while GPT-5.5 / Codex CLI version is. Pin or explicitly mark as unknown for reproducibility.
  3. [Figure 1] Figure 1 is conceptually clear; ensure the camera-ready version keeps stage numbers aligned with the §5/Table 3 numbering (construction / environment / scoring / reporting).
  4. [§6] §6 critical-window slack Swin=(R−r+1)/R is a sensible proposal but unvalidated; the Limitations already call this out—consider moving the formula to a short “proposal” box so it is not read as an evaluated metric.
  5. [Abstract, Table 1] Typos/grammar: abstract “can not” → “cannot”; occasional missing spaces before citations and mixed “V oyager” / “WebV oyager” line-break artifacts in Table 1.
  6. [Appendix B] Appendix B Table 4 mapping from MAST is helpful; a single worked example trajectory labeled under both MAST and the CUA codebook would make the boundary rule easier to apply for replications.

Circularity Check

0 steps flagged

No circular derivation: empirical audit counts external FAIL trajectories under an independent labeling protocol; nothing is predicted from a fitted or self-defined input.

full rationale

This paper is a measurement and design paper, not a first-principles derivation. The headline 15.3% wrong-FAIL rate (10.7% EFN, 4.7% broken tasks) is a direct count over 150 public failure-scored trajectories from AgentRewardBench and OSWorld-Verified, labeled by a two-stage protocol (LLM judges plus human review on disagreements/stress rows). That rate is not obtained by fitting a parameter and re-reporting it as a prediction, nor by defining the target in terms of itself. The three-tier taxonomy is an adapted coding scheme from external MAST work (Cemri et al.), applied after Stage-1 filters out non-agent faults; adapting a codebook is not proving a result from its own definition. Self-citations (e.g., CADWorld) appear only in the landscape/design discussion and do not underwrite the audit numbers. Judge-dependence of Stage-1 ground truth is a validity concern, not circularity under this pass’s criteria. No step reduces a claimed prediction or theorem to its inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The load-bearing content is empirical coding, not a formal derivation. Credibility rests on domain assumptions about what counts as task success, on the adapted MAST taxonomy as an exhaustive-enough failure ontology, and on the sampling frame of public FAIL trajectories with screenshots. No physical constants or fitted scaling laws; free choices are protocol parameters (seed, quotas, judge models) that affect the estimate but are disclosed.

free parameters (2)
  • sample quotas and seed 20260717 = 150 analyzed trajectories; seed 20260717
    Deterministic stratified caps (e.g., 25/25/24/24 web strata; 20 per OSWorld run before exclusions) fix which FAILs enter the 150-set and thus the point estimate.
  • LLM judge pair and human review allocation = 104 human-reviewed rows; 46 two-LLM consensus
    Choice of GPT-5.5 and Claude Sonnet, plus reviewing all 74 disagreements and only 30 agreement rows, determines final labels for the rate.
axioms (4)
  • domain assumption A trajectory satisfies the task iff a careful human grader would accept the observed outcome under the written task statement, independent of the benchmark oracle.
    Stage-1 EFN vs genuine-failure split in §4.2 is defined by this human-acceptability standard.
  • ad hoc to paper Earliest decisive failure in the perception–action loop is the correct single label for a genuine failure (boundary rule between tiers).
    §4.3 codebook rule; different causal attribution would reshuffle the 39.3%/35.2%/13.9% tier shares.
  • domain assumption Public FAIL trajectories with step-level screenshots are adequate evidence to audit oracle verdicts for the five included benchmarks.
    §4.1 scope; paper itself shows removing screenshots flips many judge verdicts.
  • ad hoc to paper MAST multi-agent categories can be mapped into a single-agent CUA taxonomy by dropping inter-agent items and elevating grounding/verification.
    Appendix B mapping table; underwrites Stage-2 diagnoses.
invented entities (2)
  • Four-stage CUA measurement pipeline (task construction, environment/observation, scoring, reporting) no independent evidence
    purpose: Organize where contamination, evaluator fragility, and diagnostic opacity enter a benchmark score.
    Framing device in Figure 1 and §2; useful taxonomy rather than a new physical entity.
  • Three-tier CUA diagnostic taxonomy (T1 planning/spec, T2 execution/grounding, T3 verification/feedback) independent evidence
    purpose: Localize genuine agent failures beyond binary success.
    Adapted from MAST with CUA-specific elevation of grounding and no-op feedback failures; frequencies are the empirical handle.

pith-pipeline@v1.2.0-daily-grok45 · 22245 in / 3606 out tokens · 82140 ms · 2026-07-31T09:34:34.159493+00:00 · methodology

0 comments
read the original abstract

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives and 4.7\% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Figures

Figures reproduced from arXiv: 2607.28367 by Qishi Zhan, Rui Qian, Ruixuan Deng, Yunqing Li, Zekun Wang, Zhiyuan Ma, Zihan Dong, Zirou Liu.

Figure 1
Figure 1. Figure 1: A CUA benchmark score is a pipeline output, not a direct observation of capability. The benchmark projects the execution loop into four measurement stages. Task validity and contamination enter during construction, evaluator fragility enters during scoring, and observability and diagnostic opacity span trajectory observation and reporting. them directly by re-auditing 150 public CUA tra￾jectories (§4). We … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

135 extracted references · 11 linked inside Pith

  1. [1]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    Evaluation and benchmarking of llm agents: A survey , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=

  2. [2]

    The Twelfth International Conference on Learning Representations , year=

    Gaia: a benchmark for general ai assistants , author=. The Twelfth International Conference on Learning Representations , year=

  3. [3]

    2024 , note =

    Liu, Xiao and Yu, Hao and Zhang, Hanchen and Xu, Yifan and Lei, Xuanyu and Lai, Hanyu and Gu, Yu and Ding, Hangliang and Men, Kaiwen and Yang, Kejuan and Zhang, Shudan and Deng, Xiang and Zeng, Aohan and Du, Zhengxiao and Zhang, Chenhui and Shen, Sheng and Zhang, Tianjun and Su, Yu and Sun, Huan and Huang, Minlie and Dong, Yuxiao and Tang, Jie , booktitle...

    2024

  4. [4]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. 2024 , note =

    2024

  5. [5]

    2024 , author =

    Introducing. 2024 , author =

    2024

  6. [6]

    arXiv preprint arXiv:2506.12286 , year=

    The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason , author=. arXiv preprint arXiv:2506.12286 , year=

    arXiv

  7. [7]

    CoRR , year=

    Language Models (Mostly) Know What They Know , author=. CoRR , year=

  8. [8]

    Solved Issues

    Are" Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study , author=. arXiv preprint arXiv:2503.15223 , year=

    Pith/arXiv arXiv

  9. [9]

    Proceedings of the 40th international conference on software engineering , pages=

    Overfitting in semantics-based automated program repair , author=. Proceedings of the 40th international conference on software engineering , pages=

  10. [10]

    Proceedings of the 2017 11th joint meeting on foundations of software engineering , pages=

    Better test cases for better automated program repair , author=. Proceedings of the 2017 11th joint meeting on foundations of software engineering , pages=

    2017

  11. [12]

    Workshop on Scaling Environments for Agents , year=

    WebArena Verified: Reliable Evaluation for Web Agents , author=. Workshop on Scaling Environments for Agents , year=

  12. [13]

    2024 , eprint=

    WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks , author=. 2024 , eprint=

    2024

  13. [14]

    arXiv preprint arXiv:2403.07718 , year=

    Workarena: How capable are web agents at solving common knowledge work tasks? , author=. arXiv preprint arXiv:2403.07718 , year=

    Pith/arXiv arXiv

  14. [15]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  15. [16]

    2025 , month = jan, key =

    Computer-Using Agent , author =. 2025 , month = jan, key =

    2025

  16. [17]

    Introducing

    Pichai, Sundar and Hassabis, Demis and Kavukcuoglu, Koray , howpublished =. Introducing. 2024 , month = dec, key =

    2024

  17. [18]

    arXiv preprint arXiv:2508.13180 , year =

    Search-Time Data Contamination , author =. arXiv preprint arXiv:2508.13180 , year =

    Pith/arXiv arXiv

  18. [19]

    arXiv preprint arXiv:2410.10934 , year =

    Agent-as-a-Judge: Evaluate Agents with Agents , author =. arXiv preprint arXiv:2410.10934 , year =

    Pith/arXiv arXiv

  19. [20]

    arXiv preprint arXiv:2504.08942 , year=

    Agentrewardbench: Evaluating automatic evaluations of web agent trajectories , author=. arXiv preprint arXiv:2504.08942 , year=

    arXiv

  20. [21]

    arXiv preprint arXiv:2510.00415 , year=

    Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigm , author=. arXiv preprint arXiv:2510.00415 , year=

    arXiv

  21. [22]

    arXiv preprint arXiv:2509.23586 , year=

    Improving the efficiency of LLM agent systems through trajectory reduction , author=. arXiv preprint arXiv:2509.23586 , year=

    arXiv

  22. [23]

    Xi, Zhiheng and Ding, Yiwen and others , booktitle =

  23. [24]

    arXiv preprint arXiv:2401.10019 , year=

    R-judge: Benchmarking safety risk awareness for llm agents , author=. arXiv preprint arXiv:2401.10019 , year=

    Pith/arXiv arXiv

  24. [25]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    R-Judge: Benchmarking Safety Risk Awareness for LLM Agents , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  25. [26]

    Zhang, Zhexin and Cui, Shiyao and Lu, Yida and Zhou, Jingzhuo and Yang, Junxiao and Wang, Hongning and Huang, Minlie , journal =. Agent-

  26. [27]

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =

  27. [28]

    ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=

    Why do multiagent systems fail? , author=. ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=

    2025

  28. [29]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  29. [30]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    WebGPT: Browser-assisted Question-Answering with Human Feedback , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  30. [31]

    Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =

    Generative Agents: Interactive Simulacra of Human Behavior , author =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =

  31. [32]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  32. [33]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    AppAgent: Multimodal Agents as Smartphone Users , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  33. [34]

    You, Keen and others , booktitle =. Ferret-

  34. [35]

    Holistic Agent Leaderboard: The Missing Infrastructure for

    Kapoor, Sayash and others , howpublished =. Holistic Agent Leaderboard: The Missing Infrastructure for. 2025 , note =

    2025

  35. [36]

    2025 , note =

    Establishing Best Practices for Building Rigorous Agentic Benchmarks , author =. 2025 , note =

    2025

  36. [37]

    MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and

    Kong, Quyu and others , howpublished =. MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and. 2025 , note =

    2025

  37. [38]

    2024 , note =

    Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration , author =. 2024 , note =

    2024

  38. [39]

    Advances in Neural Information Processing Systems , volume=

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration , author=. Advances in Neural Information Processing Systems , volume=

  39. [40]

    Mobile-Env: Building qualified evaluation benchmarks for

    Zhang, Danyang and others , journal=. Mobile-Env: Building qualified evaluation benchmarks for

  40. [41]

    International Conference on Learning Representations (ICLR) , year =

    SmartPlay: A Benchmark for Large Language Models as Intelligent Agents , author =. International Conference on Learning Representations (ICLR) , year =

  41. [42]

    International Conference on Learning Representations (ICLR) , year =

    Shridhar, Mohit and Yuan, Xingdi and C. International Conference on Learning Representations (ICLR) , year =

  42. [43]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    ScienceWorld: Is Your Agent Smarter than a 5th Grader? , author =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    2022

  43. [44]

    ACM transactions on intelligent systems and technology , volume=

    A survey on evaluation of large language models , author=. ACM transactions on intelligent systems and technology , volume=. 2024 , publisher=

    2024

  44. [45]

    and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E

    Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E. , booktitle =. Gorilla: Large Language Model Connected with Massive

  45. [46]

    Li, Minghao and others , booktitle =

  46. [47]

    Evaluating

    Wang, Zihan and others , booktitle =. Evaluating

  47. [48]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  48. [49]

    Ge, Yingqiang and others , booktitle =

  49. [50]

    Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World

    Chen, Zhi and Jiang, Lingxiao , booktitle =. Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World

  50. [51]

    Proceedings of the International Symposium on Software Testing and Analysis (ISSTA) , year =

    Just, Ren. Proceedings of the International Symposium on Software Testing and Analysis (ISSTA) , year =

  51. [52]

    2020 , pages =

    Widyasari, Ratnadira and Qi, Haodi and Thung, Ferdian and Lo, David and Ouh, Eng Lieh , booktitle =. 2020 , pages =

    2020

  52. [53]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , booktitle =. 2024 , note =

    2024

  53. [54]

    Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) , pages =

    AutoCodeRover: Autonomous Program Improvement , author =. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) , pages =

  54. [55]

    Ma, Yingwei and Cao, Rongyu and Cao, Yan and Zhang, Yuntong and others , journal =. Lingma

  55. [56]

    and Zhang, Alex L

    Yang, John and Jimenez, Carlos E. and Zhang, Alex L. and others , booktitle =

  56. [57]

    2025 , note =

    Zan, Daoguang and others , howpublished =. 2025 , note =

    2025

  57. [58]

    2025 , note =

    Deng, Xiang and Da, Jeff and others , howpublished =. 2025 , note =

    2025

  58. [59]

    2024 , note =

    Xue, Haoran and others , howpublished =. 2024 , note =

    2024

  59. [60]

    and Zhang, Alex L

    Yang, John and Jimenez, Carlos E. and Zhang, Alex L. and Lieret, Kilian and Yang, Joyce and Wu, Xindi and Press, Ori and Muennighoff, Niklas and Synnaeve, Gabriel and Narasimhan, Karthik R. and Yang, Diyi and Wang, Sida I. and Press, Ofir , booktitle =

  60. [61]

    arXiv preprint arXiv:2107.03374 , year =

    Evaluating Large Language Models Trained on Code , author =. arXiv preprint arXiv:2107.03374 , year =

    Pith/arXiv arXiv

  61. [62]

    Morgado-Cardoso, Bernardo and others , journal =. The

  62. [63]

    Measuring Coding Challenge Competence With

    Hendrycks, Dan and Basart, Steven and Kadavath, Saurav and Mazeika, Mantas and Arora, Akul and Guo, Ethan and Burns, Collin and Puranik, Samir and He, Horace and Song, Dawn and Steinhardt, Jacob , booktitle =. Measuring Coding Challenge Competence With

  63. [64]

    arXiv preprint arXiv:2108.07732 , year =

    Program Synthesis with Large Language Models , author =. arXiv preprint arXiv:2108.07732 , year =

    Pith/arXiv arXiv

  64. [65]

    Lu, Shuai and others , journal =

  65. [66]

    Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc , journal =

  66. [67]

    2024 , note =

    Introduction -. 2024 , note =

    2024

  67. [68]

    2025 , key =

    Function calling (tool calling) , author =. 2025 , key =

    2025

  68. [69]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  69. [70]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Self-Refine: Iterative Refinement with Self-Feedback , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  70. [71]

    Nathani, Deepak and Wang, David and Pan, Liangming and Wang, William Yang , booktitle =

  71. [72]

    and others , booktitle =

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and others , booktitle =. Judging

  72. [73]

    International Conference on Learning Representations (ICLR) , year =

    Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations (ICLR) , year =

  73. [74]

    Transactions on Machine Learning Research (TMLR) , year =

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models , author =. Transactions on Machine Learning Research (TMLR) , year =

  74. [75]

    Transactions on Machine Learning Research (TMLR) , year =

    Holistic Evaluation of Language Models , author =. Transactions on Machine Learning Research (TMLR) , year =

  75. [76]

    2023 , note =

    Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le , howpublished =. 2023 , note =

    2023

  76. [77]

    2022 , note =

    Karpas, Ehud and others , howpublished =. 2022 , note =

    2022

  77. [78]

    2023 , note =

    ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases , author =. 2023 , note =

    2023

  78. [79]

    Forty-second International Conference on Machine Learning , year=

    The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models , author=. Forty-second International Conference on Machine Learning , year=

  79. [80]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  80. [81]

    The Thirteenth International Conference on Learning Representations (ICLR) , year =

    ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph , author =. The Thirteenth International Conference on Learning Representations (ICLR) , year =

Showing first 80 references.