Pith. sign in

REVIEW 1 major objections 6 minor 30 references

Partial agent evaluations should report decisions, not scores, because the evidence needed to decide varies from under a fifth of tasks to more than the full run.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:13 UTC pith:7E5A6R7E

load-bearing objection A well-scoped, honest paper that introduces a decision record for partial agent evaluations; the headline 15–25% budgets are conditional on random task order, but the core reporting idea is sound and deserves peer review. the 1 major comments →

arxiv 2608.02444 v1 pith:7E5A6R7E submitted 2026-08-03 cs.AI

ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

Wei-Jung Huang , Bonan Shen This is my paper
classification cs.AI
keywords LLM agentspartial evaluationearly stoppingbenchmark reportingdecision recordscoverage rulebootstrap uncertaintyabstention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a partial LLM-agent evaluation is only meaningful if it records whether the evidence supports a decision, not just a raw score. It introduces ParEvalLayer, a decision layer that takes paired task outcomes and a pre-committed comparison policy, then outputs one of four actions: promote, reject, continue, or abstain. Replaying completed public benchmarks as if runs had stopped early, the paper finds that three benchmarks reach the same conclusion as the full evaluation after 15% to 25% of tasks, while another requires 90% and one never resolves within 95%. This variation is the point: a score alone cannot tell a reader whether a partial run's conclusion would survive the missing tasks. The paper proposes that evaluation reports state the decision rule, coverage status, and how many comparisons remain unresolved.

Core claim

The central discovery is that partial-agent-evaluation reporting is a decision problem, not a score-estimation problem. ParEvalLayer converts a partial run into a decision record by first checking that the observed tasks cover all required benchmark groups (repository, difficulty split, application, domain), then applying a bootstrap tail test to see whether the observed paired margin is far enough from the improvement threshold. Promotes and rejects are only returned when coverage holds and the bootstrap tail is below 5%; otherwise the layer continues or abstains. Replaying completed public records, the paper shows that with a 0 percentage-point threshold, three benchmarks support the same

What carries the argument

The load-bearing object is the decision record produced by ParEvalLayer. It has four actions—promote, reject, continue, abstain—and is generated by a routine that first enforces task-group coverage (every nonempty benchmark group must appear at least once and in proportion to its size) and then applies a bootstrap tail check on the paired differences to decide whether the observed margin is decisive. The sufficiency of a budget is judged by three aggregate targets: conditional false-promotion and false-rejection rates below 5%, coverage failures below 5%, and unresolved comparisons below 25%. These targets make it impossible for a rule to look accurate by deciding only easy pairs.

Load-bearing premise

The early-sufficiency results assume that the tasks observed in a partial run are a uniformly random sample of the full task set; if live task order is fixed or cost-ordered, the observed subset is not random and the same decisions may not transfer.

What would settle it

Take one of the benchmarks that the paper reports as sufficient at 20% of the task budget, run the evaluation with a fixed task order that places the most difficult or most cost-heavy tasks last, and check whether a promote/reject decision made at 20% flips when all tasks are observed. If a flipped decision is found, the early-sufficience claim is falsified for that task order.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Evaluation services can stop runs early and publish a comparison as soon as the decision record says promote or reject, saving wall-clock time and API budget without claiming more evidence than the policy allows.
  • Benchmark reports that include only a score become visibly incomplete: readers can ask for the decision rule, coverage status, and unresolved-comparison count before accepting a partial conclusion.
  • The decision layer can be attached to any existing evaluation pipeline that writes task-level paired outcomes, without changing task execution or scoring.
  • Budget sufficiency numbers are only meaningful with the policy: thresholds, task order, coverage rule, and abstention target all shift the minimum budget, so a task fraction alone is not a property of a benchmark.
  • In live use, a valid sequential or confidence-sequence boundary would be needed for repeated looks; the paper's bootstrap tail supports the replay comparisons but not time-uniform guarantees.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's early-sufficiency results are conditioned on uniform random task order; a natural extension is to test the decision layer on cost-ordered or adversarial task orders, where the paper's own tau-bench check suggests failures can be severe.
  • The 15-25% early-decision benchmarks may have large clusters of nearly homogeneous tasks, so an adaptive, learned task-ordering policy could push even the high-budget benchmarks below 90%; that is an untested consequence of the coverage rule.
  • One could turn the decision record into a live protocol: a benchmark service fixes the policy, then runs tasks in random order and stops at the first budget where the record says promote or reject, with the unresolved rate as a retrospective quality measure.
  • The abstention target (25%) has no statistical grounding; tightening or loosening it changes which budgets count as sufficient, so reporting suites should treat the target as a policy choice, not a universal constant.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes ParEvalLayer, a decision layer for partial LLM-agent evaluations. For a two-system comparison, the layer takes the task outcomes observed so far plus a pre-specified policy (improvement threshold, coverage requirement, uncertainty rule, budget limit) and outputs one of four actions: promote, reject, continue, or abstain. The authors evaluate the layer by replaying completed public benchmark records (SWE-bench Lite/Verified, AppWorld, OSWorld-Verified, tau-bench, and Terminal-Bench) as if the evaluation had stopped at various fractions of the task set or elapsed time, and checking whether promote/reject decisions agree with the completed-record conclusion. Under the primary policy at a 0 percentage-point threshold, the paper reports that AppWorld, OSWorld-Verified, and tau-bench reach sufficiency at 15%, 20%, and 25% task budget, respectively, while SWE-bench Verified requires 90% and SWE-bench Lite has no sufficient budget by 95%. A supplementary cost-aware ordering check on tau-bench shows that non-random cheap-first ordering can produce high coverage failure and wrong decisions, motivating the paper's central recommendation that partial reports should state the decision rule and unresolved-comparison rate rather than only a partial score.

Significance. If the conclusions hold, the paper makes a useful contribution to agent-evaluation reporting. Its strengths are the explicit decision contract (Fig. 2, Eqs. 5-10), the honest replay protocol in which abstentions remain in denominators, the pre-specified policy parameters with no fitted free parameters, and the demonstration that cheap-first task ordering can invalidate a partial comparison. The cross-benchmark variation in sufficient budgets is a valuable negative result: partial-score reporting is not uniformly reliable. The main limitation is that the headline early-budget claims are computed under uniformly random task order, while the paper does not clearly state that randomization is a required part of the policy or evaluate the primary policy under realistic non-random orders with the coverage rule active. This is a scope issue rather than an internal inconsistency, but it affects how the empirical claims can be interpreted and used.

major comments (1)
  1. [IV-B / Table II / Abstract, with V-B] The headline budgets (15-25% for AppWorld, OSWorld-Verified, tau-bench) are produced by sampling task orders per ordered comparison (500 orders, or 2,000 for tau-bench) and averaging over the resulting subsets. Neither the abstract nor the policy specification states that uniform random task order is a required policy element. In a live evaluation with a fixed order (cost-ascending, difficulty-ascending, or grouped by repository/domain), the observed subset at budget b is not a uniform sample of the shared task set. The coverage rule in Eq. (4) checks only per-stratum counts r_g(b), not within-stratum representativeness, and the bootstrap tails in Eqs. (5)-(6) are computed on the observed subset. Section V-B shows that task order can cause 100% coverage failure and wrong decisions on tau-bench, but that check uses forced evaluation without the coverage rule, so it does not establish robu
minor comments (6)
  1. [IV-B] State explicitly that the 500 (2,000 for tau-bench) task orders are sampled uniformly at random, and clarify whether task order is part of the policy itself or only a simulation device. The current wording is ambiguous, especially because Section I lists 'task order' as a fixed policy component.
  2. [III-E and Fig. 2] The coverage-failure target in Eq. (9) is trivially satisfied for the primary policy: Fig. 2 returns CONTINUE or ABSTAIN whenever coverage fails, so the primary policy can never issue a promote/reject before coverage holds. Statements in Section V-A such as 'meets the decision-error and coverage targets' are therefore uninformative for the primary policy. Either define a different coverage-failure quantity (e.g., the rate at which coverage prevents a decision) or state that Eq. (9) is intended only for the non-coverage-aware comparison policies.
  3. [Table II] The column '0 pp unresolved interval' is not defined in the text. Specify at which budget these intervals are computed and how they are derived from directed-pair variability. The tau-bench interval [7.25, 43.84] is particularly wide and needs a footnote explaining that the 12 directed pairs make this a case-study result, not a population estimate.
  4. [V-C] The sentence 'Paired-normal tests give the same sufficient-budget pattern' is too terse. Since the paired-normal rule is one of the decision-rule variants described in IV-B, give its exact criterion (e.g., t-test or z-test on the paired differences, with what variance assumption) so the comparison is reproducible.
  5. [IV-B] The 'small round-budget check' is mentioned but never described or reported. Either state its results or remove the reference, since all headline claims come from the 5-percentage-point sweep.
  6. [Abstract and Conclusion] The phrase 'three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes' should carry the qualifier 'under the stated policy with randomized task order' to avoid over-generalization.

Circularity Check

0 steps flagged

No significant circularity: the replay validation is self-contained, and the policy-dependent definition of sufficiency is explicit rather than hidden.

full rationale

The paper's central claim is an empirical replay result, not a fitted prediction. The protocol in Section IV-B fixes the policy before revealing partial outcomes: "The replay simulation fixes the reporting rule before any partial-run outcomes are computed," and Section III-A states that "Completed-record fields are used only for after-the-fact replay checks, not as inputs to the stopping rule." The partial decision rule uses only the observed subset S_b (Eq. 3) and a bootstrap over observed paired differences (Eqs. 5-6); the completed-record decision (Eq. 2) is used only as the comparison target. Thus the finding that AppWorld, OSWorld-Verified, and tau-bench reach decisions at 15-25% budget is an observed agreement rate, not a parameter fitted to those same rates. There is no equation in which the output is defined in terms of the input. The sufficiency targets in Eqs. (7)-(10) are prespecified reporting thresholds (alpha=0.05, gamma=0.05, eta=0.25), and the paper explicitly frames them as policy choices: "Sufficient budgets are therefore reported relative to the stated policy. Different evaluation settings can choose different targets and thresholds before the run and report those choices in the decision record." This is a transparent reporting convention, not a definition that forces the empirical result. The paper's headline conclusion is that partial evaluations should report decisions and unresolved counts, which is supported by the observed cross-benchmark variation rather than by circular construction. There is also no load-bearing self-citation. The authors do not cite their own prior work; references [1]-[28] point to external benchmark papers, statistical methods, and infrastructure work. No uniqueness theorem or ansatz is imported from a same-author citation. The main vulnerability noted in the reader's take -- that the primary results assume uniform random task order and the coverage rule only checks group counts, not within-group representativeness -- is a correctness and transferability concern about external validity, not circularity. It concerns whether the replayed simulation generalizes to live fixed or cost-biased orders, not whether the derivation reduces to its own inputs. The paper itself acknowledges this limitation in Section VI: "They do not by themselves establish a live stopping guarantee. A live benchmark service would also need task orders and stopping rules specified before execution." The tau-bench cost-ordering failure i

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central sufficiency results rest on five hand-chosen policy targets (δ, a, α, γ, η) and the assumption that the observed partial subset is a uniform random sample of tasks. ParEvalLayer itself is a software design, not a new theoretical entity, so no invented entities are introduced.

free parameters (6)
  • Improvement threshold δ = 0 pp / 5 pp / 10 pp tested
    Hand-chosen threshold for paired improvement; the minimum sufficient budgets in Table II move with δ (e.g., SWE-bench Lite: none at 0 pp, 55% at 5 pp, 35% at 10 pp).
  • Bootstrap tail cutoff a = 0.05
    Default tail probability cutoff in Eq. 5-6; sensitivity check with a=0.02 and 0.10 shifts AppWorld's budget from 25% to 10%.
  • Decision-error target α = 0.05
    Default target for false-promotion and false-rejection rates in Eq. 7-8; chosen without external justification.
  • Coverage-failure target γ = 0.05
    Default target for coverage failures before a final action (Eq. 9).
  • Abstention target η = 0.25
    Default unresolved-comparison target (Eq. 10); directly determines whether a budget is 'sufficient'. No sensitivity analysis for η is provided.
  • Budget grid step = 5% to 95% in 5 pp increments
    Affects the reported minimum sufficient budget granularity and the 'none by 95%' statements.
axioms (5)
  • domain assumption The completed-record decision (Δ_full > δ) is the ground truth for replay evaluation.
    Section III-B defines promote/reject on the shared public overlap; the paper explicitly treats benchmark tasks as a fixed population rather than a sample of a larger task distribution.
  • domain assumption The observed partial subset is a uniform random sample of the full shared task set.
    Section IV-B samples task orders, and the bootstrap and coverage counts in Fig. 2 assume the observed tasks are representative. The paper's own cost-aware tau-bench check shows this fails for non-random orders.
  • domain assumption Benchmark-provided task groups (repository, difficulty, application, domain) are the appropriate strata for coverage.
    Section III-D builds the coverage rule on these public metadata groups; finer behavioral strata would require separate validation.
  • domain assumption Public leaderboard records provide accurate, complete, usable outcomes for the compared systems.
    Section IV-A relies on released task-level outcomes from SWE-bench, AppWorld, OSWorld, tau-bench, and Terminal-Bench; source-specific filtering is used but not independently verified.
  • domain assumption Averaging repeated runs yields a fixed task outcome for a system.
    Section IV-A averages repeated OSWorld-Verified runs before pairing; live runs may be noisier than the averaged record.

pith-pipeline@v1.3.0-daily-deepseek · 12514 in / 14811 out tokens · 145583 ms · 2026-08-04T07:13:56.310237+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision." pith.science (2026). https://pith.science/paper/7E5A6R7E

@misc{pith2026260802444,
  author       = {Pith},
  title        = {Pith review of: ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7E5A6R7E}},
  note         = {Machine review of arXiv:2608.02444}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.

Figures

Figures reproduced from arXiv: 2608.02444 by Bonan Shen, Wei-Jung Huang.

Figure 1
Figure 1. Figure 1: ParEvalLayer reads task outcome rows and a comparison policy chosen in advance, then writes a decision record for the current budget. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Decision routine used for each partial comparison. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Unresolved comparisons at the 0 pp threshold across the 5 percentage point budget grid. Lower is better; budgets above 25% unresolved comparisons [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 15 linked inside Pith

  1. [1]

    Efficient benchmarking of AI agents,

    F. Ndzomga, “Efficient benchmarking of AI agents,”arXiv preprint arXiv:2603.23749, 2026

    arXiv 2026

  2. [2]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inInternational Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum? id=VTF8yNQM66

    2024

  3. [3]

    SWE-bench leaderboards,

    SWE-bench Team, “SWE-bench leaderboards,” 2026, accessed 12 June

    2026

  4. [4]

    AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,

    H. Trivedi, T. Khot, M. Hartmann, R. Manku, V . Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian, “AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,” arXiv preprint arXiv:2407.18901, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y . Liu, Y . Xu, S. Zhou, S. Savarese, C. Xiong, V . Zhong, and T. Yu, “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,”arXiv preprint arXiv:2404.07972, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “ τ-bench: A benchmark for tool-agent-user interaction in real-world domains,”arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    τ 2-Bench: Evaluating conversational agents in a dual-control environment,

    V . Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan, “ τ 2-Bench: Evaluating conversational agents in a dual-control environment,”arXiv preprint arXiv:2506.07982, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y . Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V . Sharma, K. Sun, S. Dillmann, A. Ana...

    Pith/arXiv arXiv 2026

  9. [9]

    Holistic Agent Leaderboard: The missing infrastructure for AI agent evaluation,

    S. Kapoor, B. Stroebl, P. Kirgis, N. Nadgir, Z. S. Siegel, B. Wei, T. Xue, Z. Chen, F. Chen, S. Utpala, F. Ndzomga, D. Oruganty, S. Luskin, K. Liu, B. Yu, A. Arora, D. Hahm, H. Trivedi, H. Sun, J. Lee, T. Jin, Y . Mai, Y . Zhou, Y . Zhu, R. Bommasani, D. Kang, D. Song, P. Henderson, Y . Su, P. Liang, and A. Narayanan, “Holistic Agent Leaderboard: The miss...

    arXiv 2025

  10. [10]

    Sequential tests of statistical hypotheses,

    A. Wald, “Sequential tests of statistical hypotheses,”The Annals of Mathematical Statistics, vol. 16, no. 2, pp. 117–186, 1945

    1945

  11. [11]

    Time-uniform, nonparametric, nonasymptotic confidence sequences,

    S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon, “Time-uniform, nonparametric, nonasymptotic confidence sequences,”The Annals of Statistics, vol. 49, no. 2, pp. 1055–1080, 2021

    2021

  12. [12]

    Selectivenet: A deep neural network with an integrated reject option,

    Y . Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 2151–2159

    2019

  13. [13]

    Consistent estimators for learning to defer to an expert,

    H. Mozannar and D. Sontag, “Consistent estimators for learning to defer to an expert,” inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol

  14. [14]

    Conformal risk control,

    A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,”arXiv preprint arXiv:2208.02814, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    Holistic evaluation of language models,

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. R ´e, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Gu...

    2023

  16. [16]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023

    2023

  17. [17]

    Cer-Eval: Certifiable and cost-efficient evaluation framework for LLMs,

    G. Wang, Z. Chen, B. Li, and H. Xu, “Cer-Eval: Certifiable and cost-efficient evaluation framework for LLMs,”arXiv preprint arXiv:2505.03814, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Efficient evaluation of LLM per- formance with statistical guarantees,

    S. Wu, Y . Nair, and E. J. Cand `es, “Efficient evaluation of LLM per- formance with statistical guarantees,”arXiv preprint arXiv:2601.20251, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Probability inequalities for the sum in sampling without replacement,

    R. J. Serfling, “Probability inequalities for the sum in sampling without replacement,”The Annals of Statistics, vol. 2, no. 1, pp. 39–48, 1974

    1974

  20. [20]

    On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection,

    J. Neyman, “On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection,” Journal of the Royal Statistical Society, vol. 97, no. 4, pp. 558–625, 1934

    1934

  21. [21]

    AI agents that matter,

    S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, “AI agents that matter,”arXiv preprint arXiv:2407.01502, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    What twelve LLM agent benchmark papers disclose about themselves: A pilot audit and an open scoring schema,

    M. N. Moghadasi and F. Ghaderi, “What twelve LLM agent benchmark papers disclose about themselves: A pilot audit and an open scoring schema,”arXiv preprint arXiv:2605.21404, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    General agent evaluation,

    E. Bandel, A. Yehudai, L. Eden, Y . Sagron, Y . Perlitz, E. Venezian, N. Razinkov, N. Ergas, S. Shachor Ifergan, S. Shlomov, M. Jacovi, L. Choshen, L. Ein-Dor, Y . Katz, and M. Shmueli-Scheuer, “General agent evaluation,”arXiv preprint arXiv:2602.22953, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    A2Perf: Real-world autonomous agents benchmark,

    I. Uchendu, J. Jabbour, K. Van den Berghe, J. Runevic, M. Stewart, J. Ma, S. Krishnan, I. Gur, A. Huang, C. Bishop, P. Bailey, W. Jiang, E. M. Songhori, S. Guadarrama, J. Tan, J. K. Terry, A. Faust, and V . J. Reddi, “A2Perf: Real-world autonomous agents benchmark,”arXiv preprint arXiv:2503.03056, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    ProSoftArena: Benchmarking hierarchical capabilities of multimodal agents in professional software environments,

    J. Ai, Y . Feng, F. Zhang, J. Sun, Z. Li, C. Li, Y . Chang, W. Wu, R. Wang, M. Zhai, and K. Zhang, “ProSoftArena: Benchmarking hierarchical capabilities of multimodal agents in professional software environments,” arXiv preprint arXiv:2601.02399, 2025

    arXiv 2025

  26. [26]

    Towards adaptive ML benchmarks: Web-agent-driven construction, domain expansion, and metric optimization,

    H. Jia, Y . Qian, H. Tong, X. Wu, L. Chen, and F. Wei, “Towards adaptive ML benchmarks: Web-agent-driven construction, domain expansion, and metric optimization,”arXiv preprint arXiv:2509.09321, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    AndroidWorld: A dynamic benchmarking environment for autonomous agents,

    C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva, “AndroidWorld: A dynamic benchmarking environment for autonomous agents,”arXiv preprint arXiv:2405.14573, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    ClawTrace: Cost-aware tracing for LLM agent skill distillation,

    B. Yuan, Y . Su, R. Song, S. Yang, and J. Qin, “ClawTrace: Cost-aware tracing for LLM agent skill distillation,”arXiv preprint arXiv:2604.23853, 2026

    Pith/arXiv arXiv 2026

  29. [119]

    7076–7087

    PMLR, 2020, pp. 7076–7087

    2020

  30. [2026]

    Available: https://www.swebench.com/

    [Online]. Available: https://www.swebench.com/