Pith.

Artificial Intelligence · arXiv:2611.23068 v1 · submitted 17 Nov 2026 · 47 references

The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang

show 3 more authors

Yexin Li, Ruo Shan, Kan Ren

The paper says

“Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher.”

more of the abstract

This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 18 Nov 2026 · read the full text (175,395 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A large-scale empirical study argues that agent evolution stalls not because proposers fail to generate better artifacts, but because noisy evaluations cannot reliably identify them. The evidence is substantial and the argument is coherent, but the candidate-level results are limited to one benchmark and the code is not yet released. This line is wrong

8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 175,395 characters. Not checked: figures, code, what the 47 references say.

8 minor objections

The paper says, at Section 3.2 (Experiments)

Pith says: minor objection 1 of 8

The paper states 'over 10M LLM calls and 400K CPU core hours' but does not provide a breakdown of how these resources were distributed across experiments. A table showing the cost per experiment group would be helpful for reproducibility and for readers assessing the feasibility of similar studies.

The paper says, at Table 1 (Variance measurement)

Pith says: minor objection 2 of 8

Table 1 (tab:variance-text) is very dense and difficult to read. Consider splitting it by benchmark or method family, or using a visual representation (e.g., a heatmap) to make the patterns more accessible.

show 6 more

The paper says, at Section 5.2 (Selection) and Table 2

Pith says: minor objection 3 of 8

Table 2 reports pairwise accuracy and false acceptance rates with credible intervals. The false acceptance rate for execution-based selection is 62.1% with a very wide interval (±42.9%). The paper should comment on why this interval is so wide and whether this affects the conclusions.

The paper says, at Appendix D.2 (Per-Benchmark Results)

Pith says: minor objection 4 of 8

The discussion of LiveMathematicianBench is important but somewhat buried in the appendix. The finding that the 'stronger result' option is always correct when it appears is a significant evaluation shortcut that undermines the benchmark. This should be highlighted more prominently, perhaps in the main text.

The paper says, at Section 5.3 (Noise Accumulates Across Rounds)

Pith says: minor objection 5 of 8

Figure 7 (tau-growth) shows run-to-run variance growing over rounds. The paper should clarify whether the noise-corrected estimate can become negative (as noted in Appendix F) and how this affects the interpretation of early rounds.

The paper says, at Section 7 (Limitations)

Pith says: minor objection 6 of 8

The paper lists several limitations but does not discuss the potential for publication bias in the margin survey (Appendix C.3). Papers that find small or negative margins may be less likely to be published, so the reported margins may overestimate typical effect sizes. This should be acknowledged.

The paper says, at Appendix C.3 (Margins Reported in Agent-Evolution Papers)

Pith says: minor objection 7 of 8

The survey of reported margins is useful, but the paper should clarify whether the 'strongest competitor' is always another evolution method or could be a non-evolution baseline. The definition says 'another optimization method (prompt, skill, memory, or harness optimization); zero-shot prompting, human-written prompts or harnesses, and fixed retrieval baselines are not competitors.' This is clear but should be stated in the main text as well.

The paper says, at Section 4.1 (Single Reports Cannot Rank Methods)

Pith says: minor objection 8 of 8

The paper uses the term 'resolution' informally. It might be helpful to define it more precisely, perhaps as the minimum detectable effect size given the variance components and a desired power level. The current definition ('the smallest true improvement it can reliably distinguish from noise') is a good start but could be formalized.

What would settle it

If a future study on a different benchmark or with a different set of methods finds that single-run rankings reliably identify the best method (e.g., the top-ranked method in a single run agrees with the true ranking more than 80% of the time), that would contradict the paper's claim that typical method margins are below single-run resolution.

from Pith's summary of the paper

How the review went

5 passes by deepseek-v4-flash · 18 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic concern withdrawn: the paper already says this 8 s · 52,595 tokens read, 596 written
  3. Referee 8 minor 18 s · 52,881 tokens read, 2,281 written
  4. Check against the paper kept 0 of 5 objections 11 s · 53,876 tokens read, 1,715 written
  5. Editor wrote the verdict above 7 s · 53,153 tokens read, 463 written

Not run on this paper: the simulated authors' defence.

Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (5 the paper already says).

The review in full

The review in brief, machine-written

This paper makes a clear, well-supported argument that the bottleneck in agent evolution is evaluation resolution, not proposal quality. The authors run over 1,000 independent evolution runs and 2.1 million test trajectories across seven methods, six models, and four benchmarks—by a wide margin the largest reproducibility study of agent evolution I have seen. The central claim, the Resolution Hypothesis, is stated plainly and tested at both the method level (where typical margins are smaller than single-run noise) and the candidate level (where selection is near chance). The gain audit is particularly effective: showing that the largest reported gains come from evaluation shortcuts that a single sentence can reproduce and that vanish after repair is a clean demonstration of the hypothesis.

The paper is honest about its limitations. The candidate-level evidence comes from Terminal-Bench v2.1 only, the trajectory re-evaluation covers two methods and two models, and the authors acknowledge that the conclusions may not generalize to larger pools, different generation strategies, or program evolution. The code and data are promised upon acceptance but not yet released, which limits reproducibility today.

A few soft spots worth noting. The candidate pools use n=100 skills; whether the Quality@K curves would continue to climb at larger K is untested. The empirical-Bayes estimator has a free parameter (kappa_max) that affects the separation between observed and null curves, though the authors test sensitivity across a range. The trajectory judges are evaluated on a single model (DeepSeek-V4-Flash) without an ACE training loop, so the null result for LLM judges may not be the last word.

This paper is for researchers working on self-improving agents, evaluation methodology, or reproducibility in AI. It deserves a serious referee and would benefit from code release and replication on additional benchmarks. I recommend peer review with a request for the code and data.

The stress test's concern

Withdrawn: the paper already says this.

The paper says

“The candidate-level evidence is evaluated on Terminal-Bench v2.1 only: the Proposal Pool in Section~\ref{sec:proposals} evaluates $n{=}100$ skills proposed from 10 Terminal-Bench training tasks across 10 disjoint test tasks, the Selection Pool in Section~\ref{sec:selection} evaluates $n{=}100$ skills proposed from Terminal-Bench and DeepSWE trajectories across 11 tasks, and the trajectory re-evaluation in Section~\ref{sec:accumulation} covers two methods and two models. Whether the conclusions hold for larger pools, different generation strategies, or other benchmark domains remains an open question.”

The referee's report

Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.

Pith's reading of the paper (a machine's summary, not the authors')

The paper proposes the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.

What carries the argument. The Resolution Hypothesis: the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them. The paper defines evaluation resolution as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. The paper uses a variance decomposition separating evaluation noise (σ²) from run-to-run variance (τ²) to measure resolution.

Core claim. The paper argues that agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.

The premise it rests on. The paper assumes that the evaluation noise and run-to-run variance measured on the four benchmarks (ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1) generalize to other benchmarks and settings.

If the paper is right (Pith's inferences, not the authors')

If the Resolution Hypothesis is correct, then improving agent evolution requires either enlarging the improvements being proposed or making smaller improvements reliably detectable.

The paper suggests that developing higher-resolution feedback may be as important as developing more powerful evolution algorithms.

The paper recommends publishing both evaluation noise (σ̂) and run-to-run variance (τ̂) alongside point scores so readers can judge whether a claimed margin exceeds the evaluation's resolution.

The paper proposes component-wise evaluation: Quality@K curves for proposers and selected-artifact quality curves for selectors, to split one comparison that falls below the resolution into two that exceed it.

The paper's findings suggest that many published rankings of agent evolution methods may be unreliable, as typical margins are smaller than single-run variance.

The paper's analysis implies that evaluation shortcuts (e.g., adapting to a grader that credits only one answer format) can produce large reported gains that vanish when the evaluation is repaired, calling into question the validity of such gains.

The paper's framework could be applied to other domains where noisy evaluation is used for selection, such as hyperparameter optimization or neural architecture search.

The paper's recommendation of component-wise evaluation (Quality@K for proposers, selected-artifact quality for selectors) could be adopted as standard practice in the field.

Check this review

What was read

The passes were given all 175,395 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.23068

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0053 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0049 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0052 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Check against the paper deepseek-v4-flash · medium effort · $0.0052 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
  5. Editor deepseek-v4-flash · medium effort · $0.0049 · instructions of the Editor, fingerprint e7211f6a3d27…
  6. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0052 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.031 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.0ef1a750a3e3

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 18 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): a44cd811e9219038a8cbf922644aa00ae001a7ce26b47f0dfbb92ebcb3ef66e4

curl -s 'https://pith.science/api/v1/verdicts/2611.23068?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-18

The authors' main claim, as the classifier picked it out 0.90

“Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements.”

How the main claim is stated: hedged possibility 0.96

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.28 · human participants 0.04 · controlled comparison 0.26 · randomized assignment 0.03 · states quantitative result 0.93 · compares with prior results 0.59 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. background 1.00 0.07 0.03 asserted 1.00
This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. background 0.62 0.06 0.02 asserted 0.99
We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. interpretation 0.39 0.32 0.79 asserted 0.96
Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. result 0.64 0.97 0.17 asserted 1.00
At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. result 0.99 0.92 0.12 asserted 1.00
At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. result 0.98 0.87 0.08 asserted 0.93
Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. interpretation 0.96 0.83 0.85 hedged possibility 0.96
This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable. future work 0.92 0.09 0.31 asserted 0.63

What kind of paper the classifier reports this is: empirical study 0.99 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-18: 1 request, 6,267 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 1 of the paper
Figure 1. (a) Four candidate moves on the same landscape, scored under high and low evaluation noise (thought bubble: score relative to the current artifact). Under high noise, a worse candidate artifact scores highest. (b) Agent evolution as a loop: a proposer writes candidate artifacts, and a selector accepts the artifacts with the highest noisy score. P1–P3 are the predictions we test in Section 5. rankings of evolution methods. At the candidate level, the same kind of evaluation decides which artifacts to carry forward to the next round. As a result, the evolution methods themselves…
Figure 2 of the paper
Figure 2. (a) Gap between all-run means of same￾benchmark configuration pairs vs. the standard devia￾tion of a single-run comparison; shaded: gap below one standard deviation. (b) Runs per method to identify the median literature margin (3.4 pp) at 80% power; gray bar: interquartile range 1.2–6.6 pp. Per-method values
Figure 4 of the paper
Figure 4. Headroom in the pool (a) and what selectors capture of it (b) on Terminal-Bench v2.1. (a) Quality@K (solid) vs. matched equal-quality null (dashed) on the 10 disjoint test tasks (n=100 skills, κmax=100 EB), contrast￾ing the 2-hour budget (blue) and original timeouts (char￾coal); shading: 95% intervals. (b) Selected-artifact gain on the 11-task cross-source selection pool (Appendix F.3); Random draws an artifact, and the oracle is clipped to the selector range. Selection Pairwise acc. (%) False accept. (%) Random 50.0 ± 0.0 30.5 ± 14.0 Execution (1 rollout) 49.7 ± 1.9 62.1 ± 42.9…
Figure 5 of the paper
Figure 5. In-loop vs. independently re-evaluated validation gains across rounds (relative to initialization; shading: 95% bootstrap intervals over eight runs). In-loop gains reflect noise exploitation, not true improvement. 0 1 2 3 0 10 DeepSeek-V4-Flash 0 1 2 3 0 10 GPT-5.6 Luna 0 5 10 0 15 30 0 4 8 12 0 30 60 Run-to-run variance (pp 2 ) SkillOpt-Lite GEPA Evolution round Noise-corrected variance Uncorrected variance
Figure 7 of the paper
Figure 7. Distribution of the score a single run would report (Appendix D.1) for the unevolved baseline (Base) and the four text-artifact methods (SOL: SkillOpt-Lite, GEPA: GEPA, RB: ReasoningBank, T2S: Trace2Skill), from 105 resamples of the observed artifacts and evaluations. Dotted line: mean over all runs and evaluations. Ridges share one height; horizontal ranges differ between panels. record how often the top-ranked method and the full ranking agree with those of the means, and average over models. The top-ranked method agrees 67%, 51%, 73%, and 33% of the time on SpreadsheetBench,…
Figure 8 of the paper
Figure 8. Per-benchmark gain audit. Gray: gain reported under the released evaluation. Black tick: unevolved baseline plus one rule. Green: the same artifacts under the repaired evaluation. Light bar with dotted end: headroom after repair (100 minus the repaired baseline). SpreadsheetBench gains and the rule use the released grader; its repaired values use LibreOffice recalculation. the rule removes more of the answers the parser fails to read. On SearchQA the rule recovers 45% to 78% of the gain, but under semantic grading it lowers the score of GPT-5.4 nano by 2.0 pp relative to the…
Figure 9 of the paper
Figure 9. Effect of agent timeout cutoff on Quality@K vs. the matched equal-quality null on the Proposal Pool (10 disjoint Terminal-Bench v2.1 test tasks, n=100 skills, κmax=100 empirical-Bayes estimator). Panels (a)–(e) evaluate the same 16,000 trajectories under progressively relaxed cutoffs: original per-task timeout (900–1800 s), 2×, 3×, 4×, and the full 2-hour budget. Panel (f) overlays the net excess above the matched null across all five cutoffs. Research & Skill Authoring Guidance Your goal is to investigate the training corpus in workspace/.skillopt/corpus/, uncover the concrete…
Figure 10 of the paper
Figure 10. Per-task Quality@K (solid) and matched equal-quality null (dashed) across all 10 disjoint Terminal-Bench v2.1 test tasks of the Proposal Pool under the 2-hour budget (blue) and original per-task timeout (charcoal), using κmax=100 empirical Bayes; each panel uses its own vertical scale. • Account for exact assertions: Repeated runs of a task are not independent tasks, and failing the same test function does not mean failing for the same reason. Check the actual verifier output per run to distinguish early format or missing-file failures from downstream numerical or behavioral…
Figure 11 of the paper
Figure 11. Sensitivity of macro-averaged Quality@K vs. matched equal-quality null under the 2-hour budget to the upper concentration cap κmax ∈ {30, 100, 300,∞} of the empirical-Bayes Beta prior. A tight cap (κmax=30) inflates dispersion in homogeneous null pools, whereas κmax=100 (primary) through unbounded (κmax=∞) behave consistently and cleanly separate observed from null. F.3 Selection Operators and Evaluation Metrics Independent-reference protocol. Each selector sees only the first rollout of each artifact (the native SkillOpt￾Lite and GEPA rows, the first two). Artifact quality is…
Figure 12 of the paper
Figure 12. Per-task selection curves underlying Figure 4b across the 11 tasks of the Selection Pool. Each panel shows selected-artifact success probability under the held-out quality reference (Appendix F.3); the oracle and its credible band are clipped to the selector range. 34

Showing 10 of the paper's 12 figures. Figures are the authors', reproduced from the paper.

References (47)
  1. [1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice.Advances in neural information processing systems, 34:29304–29320, 2021 on Pith
  2. [2] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflecti... on PithPith's reviewarXiv
  3. [3] Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation.Journal of Machine Learning Research, 11(70):2079–2107, 2010. URL https: //jmlr.org/papers/v11/cawley10a.html on Pith
  4. [4] Kaiming Cheng, Laura Wang, Jack Khuu, Mark Saroufim, Wenyuan Chi, Jiannan Wang, and Joe Isaacson. KernelAgent: Hardware-guided GPU kernel optimization via multi-agent orchestration. PyTorch Blog, 2026. URL https://pytorch.org/blog/kernelagent-hardware-guided-gpu-kernel-optimiz... on Pith
  5. [5] Databricks. OfficeQA: A grounded reasoning benchmark. Official dataset card, 2025. URLhttps://huggingfac e.co/datasets/databricks/officeqa. Documents from Hugging Face revision 763a836; splits as released with SkillOpt-Lite (commit 4cb4eee); official grader fromhttps://github.... on Pith
  6. [6] DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https: //arxiv.org/abs/2606.19348 on PitharXiv
  7. [7] Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, and Fan Yang. Agent skills can be harmful: An empirical study of skill-induced failures in LLM agents, 2026. URLhttps://arxiv.org/abs/2608.11888 on PithPith's reviewarXiv
  8. [8] Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. SearchQA: A new Q&A dataset augmented with context from a search engine, 2017. URLhttps://arxiv.org/abs/1704.05179 on PithPith's reviewarXiv
  9. [9] GLM-5-Team. GLM-5: from vibe coding to agentic engineering, 2026. URLhttps://arxiv.org/abs/2602.15763 on PithPith's reviewarXiv
  10. [10] Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. LiveMathematicianBench: A live benchmark for research-level mathematical reasoning with proof sketches, 2026. URLhttps://arxiv.org/abs/2604.01754 on PithPith's reviewarXiv
  11. [11] Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for many-hop fact extraction and claim verification. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460, Online, 2020. Association ... on Pithdoi:10.18653/v1/2020.findings-emnlp.309
  12. [12] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi- armed bandit models.Journal of Machine Learning Research, 17:1–42, 2016. URLhttps://arxiv.org/abs/1407 .4443 on Pith
  13. [13] Kimi Team. Kimi K3: Open frontier intelligence. Technical report, Moonshot AI, 2026. URLhttps://github.c om/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf on Pith
  14. [14] Zhiyi Kuang, Ryan Rong, YuCheng Yuan, and Allen Nie. Learning game-playing agents with generative code optimization. InICML 2025 Workshop on Programmatic Representations for Agent Learning, 2025. URL https://arxiv.org/abs/2508.19506 on PithPith's reviewarXiv
  15. [15] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses, 2026. URLhttps://arxiv.org/abs/2603.28052 on PithPith's reviewarXiv
  16. [16] Siyan Li, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. PAPILLON: Privacy preservation from Internet-based and local language model ensembles. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Co... on Pithdoi:10.18653/v1/2025.naacl-long.173
  17. [17] Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks, 2026. URLhttps://arxiv.org/abs/2602.12670 on PithPith's reviewarXiv
  18. [18] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses, 2026. URLhttps://arxiv.org/abs/2604.25850 on PithPith's reviewarXiv
  19. [19] Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, and Yangqiu Song. Rethinking self-evolving agent skills: Feedback dynamics over multiple rounds, 2026. URLhttps://arxiv.org/abs/2608.02636. 11 on PithPith's reviewarXiv
  20. [20] Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spreadsheet manipulation, 2024. URLhttps: //arxiv.org/abs/2406.14991. arXiv:2406.14991 on PithPith's reviewarXiv
show 27 more
  1. [21] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, on Pith
  2. [22] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastro... on PithPith's reviewarXiv
  3. [23] Daniel Mewes, Nayana Bannur, and Catherine Kim. Beating ARC-AGI-2 with code evolution. Imbue Blog, 2026. URLhttps://imbue.com/blog/2026-02-27-arc-agi-2-evolution. Accessed: 2026-10-09 on Pith
  4. [24] Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transferable agent skills, 2026. URL https://arxiv.org/abs/2603.25158 on PithPith's reviewarXiv
  5. [25] OpenAI. Introducing GPT-5.4 mini and nano, 2026. URLhttps://openai.com/index/introducing-gpt-5-4-min i-and-nano/. Accessed: 2026-10-09 on Pith
  6. [26] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition, 2026. URLhttps://openai.com/index/g pt-5-6/. Accessed: 2026-10-09 on Pith
  7. [27] Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with rea... on Pith
  8. [28] Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. InAdvances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 38,... on PithPith's reviewarXiv
  9. [29] Linfang Shang, Ming Xu, Yiding Sun, Tianle Xia, Lingxiang Hu, Lan Xu, and Ning Zheng. When self-evolution backfires: Pre-commit gating against skill contamination in LLM agents, 2026. URLhttps://arxiv.org/abs/26 08.05810 on Pith
  10. [30] Zayx Shawn. PACE: Anytime-valid acceptance tests for self-evolving agents.arXiv preprint arXiv:2606.08106, 2026 on PithPith's reviewarXiv
  11. [31] Yifei Shen, Bo Li, and Xinjie Zhang. SkillOpt-Lite: Better and faster agent self-evolution via one line of vibe, on Pith
  12. [32] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations, 2021. URLhttps://arxiv.org/abs/2010.03768 on PithPith's reviewarXiv
  13. [33] Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi. Do agent optimizers compound? a continual-learning evaluation on terminal-bench 2.0, 2026. URLhttps://arxiv.org/abs/2607.14004. 12 on PithPith's reviewarXiv
  14. [34] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Noah A. Smith, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents, 2026. URLhttps://arxiv.org/abs/2607.12227 on PithPith's reviewarXiv
  15. [35] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and M... on Pith
  16. [36] Jinghan Xu, Yikai Zhang, Aili Chen, Weiyuan Li, Jiaqing Liang, and Deqing Yang. Verify smarter, evolve further: Efficient harness evolution through behavior-aware verification, 2026. URLhttps://arxiv.org/abs/2608.27311 on PithPith's reviewarXiv
  17. [37] Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt: Executive strategy for self-evolving agent skills, 2026. URLhttps://arxiv.org/abs/2605.23904 on PithPith's reviewarXiv
  18. [38] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pr... on Pithdoi:10.18653/v1/d18-1259
  19. [39] Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, and Chien-Sheng Wu. On the fragility of self-improving agents: Variance, task order, and underspecification, 2026. URL https://arxiv.org/abs/2608.18066v2 . arXiv:2608.18066, version 2, September 6, 2026 on PithPith's reviewarXiv
  20. [40] Zhaochen Yu, Yingcheng Wu, Zhenfei Yin, Kaiyuan Chen, Zhe Zhao, Mengdi Wang, Shuicheng Yan, and Ling Yang. Recursive experiential-working memory evolution for long-horizon agent harnesses, 2026. URL https://arxiv.org/abs/2608.24876 on PithPith's reviewarXiv
  21. [41] Z.ai. GLM-5.2. Model card, 2026. URLhttps://huggingface.co/zai-org/GLM-5.2 on Pith
  22. [42] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin gödel machine: Open-ended evolution of self-improving agents, 2025. URLhttps://arxiv.org/abs/2505.22954. arXiv:2505.22954 on PithPith's reviewarXiv
  23. [43] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models, 2025. URL... on PithPith's reviewarXiv
  24. [44] Congjie Zheng, Chuanyi Xue, Bin Liang, Jun Yang, and Changshui Zhang. SEAGym: An evaluation environment for self-evolving LLM agents, 2026. URLhttps://arxiv.org/abs/2606.17546 on PithPith's reviewarXiv
  25. [45] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Inform... on Pith
  26. [2021] URL https://openaccess.thecvf.com/content/WACV2021/html/Mathew_DocVQA_A_Dataset_for_VQA_on_Doc ument_Images_WACV_2021_paper.html on Pith
  27. [2026] URL https://arxiv.org/abs/2607.03451. Code: https://github.com/EvolvingLMMs-Lab/SkillOpt-Lite, commit 4cb4eee on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment