Pith. sign in

REVIEW 4 major objections 6 minor 36 references

GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GDPevo: a benchmark that measures agent self-evolution on real business workflows and finds held-out accuracy gains up to 16.44 percentage points.

desk verdict Genuinely useful evolution benchmark with a real mechanism (rule hybridization), but the headline 'consistently improves' claim is partly a calibrated selection effect and the paper overstates what its own data show. read the letter →

arxiv 2608.03764 v1 pith:4VXRV7TM submitted 2026-08-04 cs.AI

Leijun Zhou , Zhihao Liu , Xiang Qu , Chenxu Liu , Yifei Liu , Yanke Yu , Jingzhe Xu , Xuejun Wu
show 4 more authors
Buyue Qian Xi Chen Yaowei Zheng Junhao Hu
This is my paper
classification cs.AI
keywords agentself-evolutionrulehybridizationenterpriseworkflowbenchmarktrain-testattributionLLMagentsdatacontaminationeconomicallyvaluabletasksskill-basedevolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GDPevo, a benchmark and fully automated pipeline for measuring whether AI agents get better at real business work by reusing experience from earlier tasks. Its central design, rule hybridization, breaks each enterprise workflow into atomic business rules, scatters subsets of those rules across five training tasks, and recombines them in five held-out test tasks, so any test-time improvement has to come from training. The authors evaluate four agents under four supervision types and report that self-evolution consistently raises held-out accuracy, up to 16.44 percentage points, while sometimes also cutting test-time cost. They also estimate a fully informed oracle ceiling of 91.6%, and the best evolved agents stay well below it, suggesting current self-evolution is functional but far from complete.

What carries the argument

Rule hybridization is the mechanism that carries the argument. Each task group's business logic is decomposed into atomic rules, deliberate enterprise-specific conventions absent from the model's world knowledge. The rules are distributed as subsets across five training tasks and recombined in five held-out test tasks within the same shared environment. This design is what makes an accuracy gain after evolution attributable to training experience: a test task can require composing rules never seen together, so only an agent that inferred the rules during training and can combine them succeeds. The deterministic rule-based grader, which converts each rubric point into a code-based test case,

What would settle it

Rerun the pipeline without the calibration thresholds (base accuracy 0.4-0.6, few-shot lift 0.1-0.3, few-shot accuracy below 0.8) and measure whether self-evolution gains persist; if the up-to-16.44-point advantage shrinks or reverses under other bands, the headline result is a selection artifact rather than a general property of self-evolution.

Watch

Extended reading notes

Core claim

The paper's central claim is that rule hybridization makes training-to-test generalization concrete and test-time gains attributable, and that, measured on the resulting benchmark, self-evolution works: every evaluated supervision type improves over the no-evolution base for all four agents, by 2.59 to 16.44 percentage points. At the same time, no evolved agent approaches the fully informed oracle ceiling of 91.6%, which the paper treats as evidence that self-evolution ability is still far from fully realized. The benchmark spans CRM, ERP, finance, healthcare, legal, and data-centric workflows, with 240 tasks in 24 task groups, and the pipeline regenerates fresh versions quickly to counter c

Load-bearing premise

The headline result depends on calibration filters: a task group is kept only when base accuracy is 40-60%, few-shot training lifts it by 10-30 percentage points, and evolved accuracy stays below 80%; those bands select for groups where evolution gains are visible, so the measured 'evolution helps' effect is partly a property of the filtered benchmark.

Editorial extensions

If this is right

  • Evolution-native benchmarks can be constructed from the outset rather than carved out of existing task suites, making the transferable ability explicit and testable.
  • Self-evolution is measurable and reproducible on economically valuable tasks: every tested supervision type improved every agent on held-out tasks.
  • Few-shot supervision behaves like supervised fine-tuning and can overfit its source domain, while reflection-based feedback transfers more robustly across domains.
  • Because all evolved agents remain below the 91.6% oracle ceiling, there is clear headroom; future self-evolution methods can be scored against this bound.
  • The fully automated pipeline can refresh the benchmark as public tasks become exposed, providing a practical response to data contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline 'evolution helps' result is partly a selection effect: calibration keeps only task groups with base accuracy of 40-60%, few-shot lift of 10-30 points, and post-evolution accuracy below 80%; changing those bands could change the measured gains.
  • The 91.6% oracle ceiling is a human-plus-model estimate given full training-side evidence, not an ideal mathematical upper bound, so improvements beyond it are not ruled out.
  • The finding that a minimal 'naive' skill creator matches or beats elaborate creators suggests that, for the skill-based evolution setting tested here, the model's intelligence matters more than the evolution method; testing parameter-updating evolution methods would be a natural extension.
  • Cross-domain transfer results imply that deployment choices about supervision type should weigh cost and negative-transfer risk: reflection-based feedback may be preferable when off-target transfer is expensive, while few-shot supervision is strongest within a domain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces GDPevo, a benchmark and automated pipeline for evaluating self-evolution of AI agents on enterprise workflows. The core proposal is rule hybridization: each workflow is decomposed into atomic business rules, different subsets are planted in five training tasks, and recombinations are used in five held-out test tasks, so that test-time gains can be attributed to training experience. The release contains 240 tasks in 24 groups across six domains (V1+V2). The evaluation covers four harness+model agents and four supervision types (base, self, reflect-3, fewshot), using deterministic rule-based graders and cost metrics. The authors report that every evolved configuration improves over its base by up to 16.44 pp, that fewshot is the most reliable supervision type, that reflect transfers across domains more robustly than fewshot, and that the evolution method matters less than the underlying model. They also report a fully informed oracle ceiling of 91.6%, which the evolved agents remain far below.

Significance. If the benchmark construction is accepted, GDPevo is a useful contribution to the evaluation methodology for agent self-evolution. Its strengths are real: rule hybridization is a concrete mechanism for making train-test transfer attributable; the pipeline is automated and can regenerate tasks to resist contamination; scoring is deterministic and rule-based; the experimental isolation controls are carefully described; and the appendix case studies (Appendix F) provide genuinely diagnostic qualitative evidence, including honest examples of negative transfer. The public release of the pipeline, benchmark, and per-task artifacts is a substantial asset. However, the empirical headline that 'self-evolution consistently improves held-out accuracy' is stronger than the evidence. The calibration procedure in Section 3.4 selects task groups precisely for a 0.1-0.3 fewshot lift, which makes the headline improvement partly a property of the acceptance rule rather than an unbiased measurement. Group-level results in Table 5 and Figure 5 also show negative transfer and many groups with no improvement. These issues are local to the interpretation of the empirical results rather than to the benchm

major comments (4)
  1. [Section 3.4] The calibration acceptance rule states that a task group is kept only if a calibration agent has base accuracy 0.4-0.6, fewshot lift 0.1-0.3, and fewshot accuracy below 0.8. The headline result that every agent improves under fewshot (Section 4.2, Table 3) is therefore partly engineered: the benchmark excludes task groups where the calibration model does not exhibit a 10-30 pp fewshot gain. The observed gains of GPT-5.5 (+15.14 pp) and Opus-4.8 (+16.44 pp) fall inside this band, as expected under the acceptance rule. The paper never reports how the headline changes if the bands are varied, nor how many candidate groups were rejected at each stage. Since the central empirical claim depends on this selection, the manuscript should provide a sensitivity analysis over the calibration thresholds and report rejection rates. Without this, 'self-evolution consistently improves' should be restate
  2. [Tables 3 and 5; Figure 5] The claim that self-evolution 'consistently improves' is contradicted by the paper's own per-group data. Table 5 shows that for TG018 (DeepSeek-V4-Pro-Preview), every evolved supervision type is below base (base 48.36 vs fewshot 39.83, self 41.34, reflect-3 45.71); for TG016 (GLM-5.2), every evolved type is below base. Figure 5(a) states that self improves only 16 of 24 task groups and reflect 20 of 24 for Opus-4.8. The macroscopic statement 'every evolved combination improves over its base' is therefore a statement about macroaverages, not about consistent improvement at the group level. In addition, the paper reports no statistical significance tests. The smallest reported gain, DeepSeek self +2.59 pp, is within one standard deviation of the group-level distribution (STD 7.89 in Table 3) and is not shown to be significant. The authors should report per-group sign tests or confidence in
  3. [Section 4.3] The RQ2 cross-domain transfer experiment uses only one task group per domain (tg02, tg06, tg10) and only two supervision types (fewshot and reflect-3). The claim that 'fewshot overfits its source group and can hurt on others, whereas reflect transfers more robustly' is based on a 3x3 matrix with six off-diagonal cells. With one group per domain, the observed deltas may reflect group-specific properties rather than domain-level or supervision-type-level regularities. The paper should either expand this experiment to more task groups or substantially qualify the conclusion. At minimum, the variance across groups should be reported so the reader can see whether the five negative fewshot cells are robust or driven by the choice of a single group.
  4. [Section 3.4 and Section 4.2] The calibration step is performed with a single calibration agent (Codex with GPT-5.5), and the same model is used as the calibration oracle and as one of the evaluated models. This makes the cross-model claims less independent than they appear. A task group is retained only if that particular model exhibits a fewshot lift; the fact that a different model (e.g., DeepSeek-V4-Pro-Preview) also improves on many groups is not independent evidence of general self-evolution ability, because the task set has been filtered for a property measured through GPT-5.5. The manuscript should disclose this dependency explicitly and, ideally, rerun calibration with at least one other model or report how much of the macro gain is attributable to the filtering. This is a load-bearing issue for the general claim that 'current agents' self-evolve reliably.
minor comments (6)
  1. [Abstract / Introduction] There are typographical and spacing issues such as 'GDPevoaddresses' (Introduction, first paragraph after the limitations) and 'GDPevois' in the same paragraph. These should be corrected in a final pass.
  2. [Section 4.1 / Table 3] The four evaluated 'agents' are not factorial in harness and model: GPT-5.5 is paired only with Codex, while the other three models are paired with Claude Code. This confounds model and harness in the RQ1 comparisons. The paper should state this limitation clearly in the main text rather than only in the appendix, and the comparisons in Table 3 should be read as configurations rather than as isolated model or harness effects.
  3. [Figure 2] Error bars are one standard deviation across task groups, not standard errors. For small gains such as DeepSeek self (+2.59 pp), the error bars suggest that the effect could be zero or negative. Please clarify in the caption what the error bars represent and, ideally, add standard errors or confidence intervals.
  4. [Appendix D] The main text (Section 4.2) describes the oracle as 'a fully informed model' that is given all hidden rules and gold answers, while Appendix D describes a human operator paired with Codex GPT-5.5 who receives the training packages but not the held-out test answer or evaluator. These are not the same protocol. The text should be reconciled so readers understand which procedure produced the 91.6% ceiling.
  5. [Section 4.3 / Figure 3] The heatmap uses only one randomly chosen group per domain, but the randomness is not described (seed, number of draws). Since the conclusion rests on six off-diagonal cells, the selection procedure and its stability should be documented.
  6. [Section 3.1 / Table 2] The claim that V2 was produced 'within two days' would be more convincing with documentation of compute, number of agent runs, and failure/retry counts. This is not a central issue, but it supports the reproducibility of the pipeline.

Circularity Check

1 steps flagged · score 5.0 of 10

Calibration filter partly constructs the headline fewshot-gain result, but most evaluation findings remain independently measured.

  1. fitted input called prediction [Section 3.4 and Section 4.2 (RQ1)]
    "Second, when the agent first learns from the 5 training tasks and evolves before solving the test tasks—here we use fewshot, the supervision type that currently performs best (Section 4)—the test score must improve by about 0.1–0.3; this rejects task groups whose hidden rules either cannot be learned from training or are too easy to learn."

    The acceptance rule for every task group requires a 0.1–0.3 fewshot lift for the calibration agent (Codex+GPT-5.5, per Figure 1). The later RQ1 finding that GPT-5.5 fewshot improves by 15.14 pp—and the abstract's general 'self-evolution consistently improves' claim—partly restates this inclusion criterion for that agent/supervision pair: groups without such a lift were revised or rejected during calibration. Thus the GPT-5.5 fewshot component of the headline is guaranteed by construction rather than independently discovered. Results for other models, other supervision types, transfer heatmaps, and the oracle gap remain measured and are not circular.

full rationale

The paper is transparent about its calibration bands and does not hide the selection rule, so this is not a concealed fit. Still, the benchmark is filtered on the very effect the evaluation then reports for the calibration model, which is a partial selection circularity rather than a derivation. No load-bearing self-citation or imported uniqueness theorem appears: reference [23] is used only for the model-laziness motivation, and the skill-based evolution method is an explicit experimental choice, not an ansatz smuggled in via citation. RQ2/RQ3, cost analyses, per-group transfer (including the negative-transfer case TG018 where DS-V4 fewshot scores 39.83 vs base 48.36), and the 91.6% oracle ceiling are independent measurements, which keeps the paper mostly self-contained. The unqualified 'consistently improves' wording is also stronger than the per-group data support, but that is a consistency/correctness concern rather than circularity. Score 5 reflects one constructed headline component while acknowledging the substantial measured contents.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are postulated; rule hybridization is a data-generation procedure, and SKILL.md is an artifact format, not an invented entity requiring independent evidence.

free parameters (4)
  • Calibration thresholds = base 0.4-0.6, fewshot lift 0.1-0.3, fewshot accuracy < 0.8
    Hand-chosen bands in Section 3.4 that accept or reject task groups; they select for observable evolution gains and define the benchmark's difficulty regime.
  • Tasks per group = 5 training / 5 test
    Design choice in Section 3.3; small per-group sample limits statistical power and forces aggregation across groups.
  • Reflect rounds = 3
    Chosen setting for reflect-3 supervision (Section 4.1); affects the amount of feedback and thus the measured reflect gains.
  • Rubric point weights = {1, 2, 3}
    Weight set in Section 4.1 and Appendix G.3 to keep scoring binary and prevent builder-chosen partial credit.
assumptions (4)
  • domain assumption Business workflows decompose into atomic, independently checkable business rules.
    Central premise of rule hybridization (Section 3.3); the entire train-test design depends on this decomposition being meaningful.
  • domain assumption Rule recombination across test tasks makes test-time gains attributable to training experience.
    Attribution claim of Section 3.3; it is a design assumption, not a proven property, and calibration is used to enforce visible gains.
  • domain assumption LLM-translated rubric points produce deterministic graders that faithfully measure rule compliance.
    Section 4.1; grading quality rests on LLM translation, checked only by automated reviewer agents.
  • ad hoc to paper Calibration bands (base 0.4-0.6, lift 0.1-0.3, fewshot < 0.8) define a meaningful difficulty and learnability regime.
    Section 3.4; thresholds are specific to the Codex + GPT-5.5 calibration agent and are not justified from independent theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks." pith.science (2026). https://pith.science/paper/4VXRV7TM

@misc{pith2026260803764,
  author       = {Pith},
  title        = {Pith review of: GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4VXRV7TM}},
  note         = {Machine review of arXiv:2608.03764}
}
read the original abstract

Agent self-evolution updates an agent's persistent state from prior experience and reuses it to solve related tasks more effectively. Evaluating self-evolution is difficult: existing benchmarks provide limited coverage of economically valuable task domains, do not always design training and test tasks such that test-time gains can be attributed to training experience, and remain vulnerable to data contamination. We present GDPevo, an evolution-native benchmark grounded in GDP-related enterprise workflows, together with the fully automated data pipeline that generates it. Its core mechanism, rule hybridization, decomposes each enterprise workflow into atomic business rules, distributes subsets of these rules across training tasks, and recombines them in held-out test tasks so that test-time gains are attributable. GDPevo spans CRM, ERP, finance, healthcare, legal, and data-centric workflows. Its V1 release contains 120 tasks in 12 groups, with five training and five held-out test tasks per group. Full automation enables the pipeline to expand the suite to 240 tasks in 24 groups (V2) within two days, providing a practical response to contamination. Using GDPevo, we evaluate four agents, each comprising a harness and a model, under four supervision types. Self-evolution consistently improves held-out accuracy by up to 16.44 percentage points. But the best evolved agents remain far below the fully informed oracle ceiling of 91.6%, indicating that the self-evolution ability of current agents remains far from fully realized. We publicly release the pipeline, benchmark, and full evaluation results at https://github.com/Prism-Shadow/GDPevo.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 22 canonical work pages

  1. [1]

    A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence.Transactions on Machine Learning Research, 2026

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang. A survey of se...

    work page 2026

  2. [2]

    A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:5362–5383, 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:5362–5383, 2024

    work page 2024

  3. [3]

    Reflexion: Language 10 agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language 10 agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023

    work page 2023

  4. [4]

    ExpeL: LLM agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. InProceedings of the 38th AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    work page 2024

  5. [5]

    G¨ odel machines: Fully self-referential optimal universal self-improvers

    J¨ urgen Schmidhuber. G¨ odel machines: Fully self-referential optimal universal self-improvers. InArtificial General Intelligence, pages 199–226. 2007

    work page 2007

  6. [6]

    Lange, and Jeff Clune

    Jenny Zhang, Shengran Hu, Cong Lu, Robert T. Lange, and Jeff Clune. Darwin G¨ odel machine: Open-ended evolution of self-improving agents.CoRR, abs/2505.22954, 2025

    arXiv 2025

  7. [7]

    EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    Deyao Zhu, Xin Zhou, Shengling Qin, et al. EdgeBench: Unveiling scaling laws of learning from real-world environments.CoRR, abs/2607.05155, 2026

    work page Pith review arXiv 2026

  8. [8]

    SkillFlow: Benchmarking lifelong skill discovery and evolution for autonomous agents.CoRR, abs/2604.17308, 2026

    Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang, Qisheng Su, Haibo Qiu, Wei Yang, Qingnan Ren, Shun Zou, Wenxuan Huang, Lin Chen, Zehui Chen, and Feng Zhao. SkillFlow: Benchmarking lifelong skill discovery and evolution for autonomous agents.CoRR, abs/2604.17308, 2026

    arXiv 2026

Show all 36 references
  1. [9]

    SEA-Eval: A benchmark for evaluating self-evolving agents beyond episodic assessment.CoRR, abs/2604.08988, 2026

    Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Shisong Chen, Jinghao Zhang, Tianjun Pan, Weijia Zhou, Jiaqing Liang, and Yanghua Xiao. SEA-Eval: A benchmark for evaluating self-evolving agents beyond episodic assessment.CoRR, abs/2604.08988, 2026

    2026 arXiv

  2. [10]

    EvoAgentBench: Benchmarking agent self-evolution via ability transfer

    Xingze Gao, Chuanrui Hu, Hongda Chen, Pengfei Yao, Zhao Wang, Yi Bai, Zhengwei Wu, Yunyun Han, Xiaofeng Cong, Jie Gui, Yafeng Deng, and Teng Li. EvoAgentBench: Benchmarking agent self-evolution via ability transfer. CoRR, abs/2607.05202, 2026

    2026 arXiv

  3. [11]

    BenchTrace: A benchmark for testing reflection ability and controlled evolution in LLM agents.CoRR, abs/2605.29225, 2026

    Jiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, and Akiko Aizawa. BenchTrace: A benchmark for testing reflection ability and controlled evolution in LLM agents.CoRR, abs/2605.29225, 2026

    2026 arXiv

  4. [12]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InProceedings of the 12th International Conference on Learning Representations, 2024

    2024

  5. [13]

    Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, Estefany Kelly Buchanan, et al

    Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, Estefany Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.CoRR, abs/2601.11868, 2026

    2026 arXiv

  6. [14]

    Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Sim´ on Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Ame...

    2025

  7. [15]

    SOP-Bench: Complex industrial SOPs for evaluating LLM agents.CoRR, abs/2506.08119, 2025

    Subhrangshu Nandi, Arghya Datta, Nikhil Vichare, Indranil Bhattacharya, Huzefa Raja, Jing Xu, Shayan Ray, Giuseppe Carenini, Abhi Srivastava, Aaron Chan, Man Ho Woo, Amar Kandola, Brandon Theresa, and Francesco Carbone. SOP-Bench: Complex industrial SOPs for evaluating LLM age...

    2025

  8. [16]

    JobBench: Aligning agent work with human will.CoRR, 2026

    Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma, Kaiyuan Zheng, Fengqing Jiang, Xinghua Sun, Rulin Shao, Zichen Chen, Yue Huang, Xinyang Han, Brian Lee, Kayla Xu, Shenglai Zeng, Hang Hua, Xiangliang Zhang, Basel Alomair, Ranjay Krishna, Luke Zettlemoyer, Pang Wei Koh, Bhaskar ...

    2026

  9. [17]

    LiveBench: A challenging, contamination-limited LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, et al. LiveBench: A challenging, contamination-limited LLM benchmark. In Proceedings of the 13th International Conference on Learning Represe...

    2025

  10. [18]

    AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge

    Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. InProceedings of the 63rd A...

    2025

  11. [19]

    Benchmarking large language models under data contamination: A survey from static to dynamic evaluation

    Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. InProceedings of the 2025 Conference o...

    2025

  12. [20]

    Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    2024

  13. [21]

    Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J

    Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Daniel Klein, Matei Zaharia, and Omar Khattab. GEPA: Ref...

    2025 arXiv

  14. [22]

    RSIBench-data: Benchmarking data-centric research for recursive self-improvement.CoRR, abs/2607.25886, 2026

    Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao, Haocheng Lu, Mengkang Hu, and Michael Qizhe Shieh. RSIBench-data: Benchmarking data-centric research for recursive self-improvement.CoRR, abs/2607.25886, 2026

    2026 arXiv

  15. [23]

    LLMigrate: Transforming “lazy” large language models into efficient source code migrators.CoRR, abs/2503.23791, 2025

    Yuchen Liu, Junhao Hu, Yingdi Shan, Ge Li, Yanzhen Zou, Yihong Dong, and Tao Xie. LLMigrate: Transforming “lazy” large language models into efficient source code migrators.CoRR, abs/2503.23791, 2025

    2025 arXiv

  16. [24]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–...

    1901

  17. [25]

    Reward is enough: LLMs are in-context reinforcement learners.CoRR, abs/2506.06303, 2025

    Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Yanjun Qi, and Shangtong Zhang. Reward is enough: LLMs are in-context reinforcement learners.CoRR, abs/2506.06303, 2025

    2025 arXiv

  18. [26]

    SaaS-Bench: Can computer-use agents leverage real-world SaaS to solve professional workflows?CoRR, abs/2605.15777, 2026

    Kean Shi, Zihang Li, Tianyi Ma, Zengji Tu, Jialong Wu, Xinbo Xu, Qingyao Yang, Ruoyu Wu, Weichu Xie, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, and Baobao Chang. SaaS-Bench: Can computer-use agents leverage real-world SaaS to solve professional wo...

    2026 arXiv

  19. [27]

    Victor Barr` es, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2-Bench: Evaluating conversational agents in a dual-control environment.CoRR, abs/2506.07982, 2025

    2025 arXiv

  20. [28]

    Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, H...

    2026 arXiv

  21. [29]

    Black, Gloria Geng, Danny Park, Andrew Y

    Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, Andrew Y. Ng, and Jonathan H. Chen. MedAgent- Bench: Dataset for benchmarking LLMs as agents in medical applications.CoRR, abs/2501.14654, 2025

    2025 arXiv

  22. [30]

    Pollard, Alistair Johnson, Edward Choi, Yugang Jia, and Jong Ha Lee

    Gyubok Lee, Elea Bach, Eric Yang, Tom J. Pollard, Alistair Johnson, Edward Choi, Yugang Jia, and Jong Ha Lee. FHIR-AgentBench: Benchmarking LLM agents for realistic interoperable EHR question answering.CoRR, abs/2509.19319, 2025

    2025

  23. [31]

    Harvey LAB: The legal agent benchmark

    Harvey AI. Harvey LAB: The legal agent benchmark. GitHub repository, version 1.0, 2026

    2026

  24. [32]

    SpreadsheetBench: Towards challenging real world spreadsheet manipulation

    Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spreadsheet manipulation. InAdvances in Neural Information Processing Systems, 2024

    2024

  25. [33]

    InfiAgent-DABench: Evaluating agents on data analysis tasks

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. InfiAgent-DABench: Evaluating agents on data analysis tasks. InProceedings of the 4...

    2024

  26. [34]

    BIRD-INTERACT: Re-imagining text-to-SQL evaluation for large language models via lens of dynamic interactions.CoRR, abs/2510.05318, 2025

    Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma ¨Ozcan, Yannis Papa...

    2025

  27. [35]

    LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks.https://github.com/bird-bench/livesqlbench, 2025

    BIRD Team. LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks.https://github.com/bird-bench/livesqlbench, 2025

    2025

  28. [36]

    may be better

    Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. WorkBench: A benchmark dataset for agents in a realistic workplace setting.CoRR, abs/2405.00823, 2024. 12 ID Scenario focus Macro domain Train/Test tg01 Marketing lead capture CRM ...

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.