Artificial Intelligence · arXiv:2611.23068 v1 · submitted 17 Nov 2026 · 47 references
The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
The paper says
“Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher.”
more of the abstract
This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 18 Nov 2026 · read the full text (175,395 characters) · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A large-scale empirical study argues that agent evolution stalls not because proposers fail to generate better artifacts, but because noisy evaluations cannot reliably identify them. The evidence is substantial and the argument is coherent, but the candidate-level results are limited to one benchmark and the code is not yet released. This line is wrong
8 minor
How often objections from reviews like this one are right: not yet measured
The paper, first character to last
The reviewer was given the whole text, 175,395 characters. Not checked: figures, code, what the 47 references say.
8 minor objections
The paper says, at Section 3.2 (Experiments)
Pith says: minor objection 1 of 8
The paper states 'over 10M LLM calls and 400K CPU core hours' but does not provide a breakdown of how these resources were distributed across experiments. A table showing the cost per experiment group would be helpful for reproducibility and for readers assessing the feasibility of similar studies.
The paper says, at Table 1 (Variance measurement)
Pith says: minor objection 2 of 8
Table 1 (tab:variance-text) is very dense and difficult to read. Consider splitting it by benchmark or method family, or using a visual representation (e.g., a heatmap) to make the patterns more accessible.
show 6 more
The paper says, at Section 5.2 (Selection) and Table 2
Pith says: minor objection 3 of 8
Table 2 reports pairwise accuracy and false acceptance rates with credible intervals. The false acceptance rate for execution-based selection is 62.1% with a very wide interval (±42.9%). The paper should comment on why this interval is so wide and whether this affects the conclusions.
The paper says, at Appendix D.2 (Per-Benchmark Results)
Pith says: minor objection 4 of 8
The discussion of LiveMathematicianBench is important but somewhat buried in the appendix. The finding that the 'stronger result' option is always correct when it appears is a significant evaluation shortcut that undermines the benchmark. This should be highlighted more prominently, perhaps in the main text.
The paper says, at Section 5.3 (Noise Accumulates Across Rounds)
Pith says: minor objection 5 of 8
Figure 7 (tau-growth) shows run-to-run variance growing over rounds. The paper should clarify whether the noise-corrected estimate can become negative (as noted in Appendix F) and how this affects the interpretation of early rounds.
The paper says, at Section 7 (Limitations)
Pith says: minor objection 6 of 8
The paper lists several limitations but does not discuss the potential for publication bias in the margin survey (Appendix C.3). Papers that find small or negative margins may be less likely to be published, so the reported margins may overestimate typical effect sizes. This should be acknowledged.
The paper says, at Appendix C.3 (Margins Reported in Agent-Evolution Papers)
Pith says: minor objection 7 of 8
The survey of reported margins is useful, but the paper should clarify whether the 'strongest competitor' is always another evolution method or could be a non-evolution baseline. The definition says 'another optimization method (prompt, skill, memory, or harness optimization); zero-shot prompting, human-written prompts or harnesses, and fixed retrieval baselines are not competitors.' This is clear but should be stated in the main text as well.
The paper says, at Section 4.1 (Single Reports Cannot Rank Methods)
Pith says: minor objection 8 of 8
The paper uses the term 'resolution' informally. It might be helpful to define it more precisely, perhaps as the minimum detectable effect size given the variance components and a desired power level. The current definition ('the smallest true improvement it can reliably distinguish from noise') is a good start but could be formalized.
What would settle it
If a future study on a different benchmark or with a different set of methods finds that single-run rankings reliably identify the best method (e.g., the top-ranked method in a single run agrees with the true ranking more than 80% of the time), that would contradict the paper's claim that typical method margins are below single-run resolution.
from Pith's summary of the paper
How the review went
5 passes by deepseek-v4-flash · 18 Nov 2026
- Reader holds up, with conditions
- Skeptic concern withdrawn: the paper already says this
- Referee 8 minor
- Check against the paper kept 0 of 5 objections
- Editor wrote the verdict above
Not run on this paper: the simulated authors' defence.
Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (5 the paper already says).
The review in full
The review in brief, machine-written
This paper makes a clear, well-supported argument that the bottleneck in agent evolution is evaluation resolution, not proposal quality. The authors run over 1,000 independent evolution runs and 2.1 million test trajectories across seven methods, six models, and four benchmarks—by a wide margin the largest reproducibility study of agent evolution I have seen. The central claim, the Resolution Hypothesis, is stated plainly and tested at both the method level (where typical margins are smaller than single-run noise) and the candidate level (where selection is near chance). The gain audit is particularly effective: showing that the largest reported gains come from evaluation shortcuts that a single sentence can reproduce and that vanish after repair is a clean demonstration of the hypothesis.
The paper is honest about its limitations. The candidate-level evidence comes from Terminal-Bench v2.1 only, the trajectory re-evaluation covers two methods and two models, and the authors acknowledge that the conclusions may not generalize to larger pools, different generation strategies, or program evolution. The code and data are promised upon acceptance but not yet released, which limits reproducibility today.
A few soft spots worth noting. The candidate pools use n=100 skills; whether the Quality@K curves would continue to climb at larger K is untested. The empirical-Bayes estimator has a free parameter (kappa_max) that affects the separation between observed and null curves, though the authors test sensitivity across a range. The trajectory judges are evaluated on a single model (DeepSeek-V4-Flash) without an ACE training loop, so the null result for LLM judges may not be the last word.
This paper is for researchers working on self-improving agents, evaluation methodology, or reproducibility in AI. It deserves a serious referee and would benefit from code release and replication on additional benchmarks. I recommend peer review with a request for the code and data.
The stress test's concern
Withdrawn: the paper already says this.
The paper says
“The candidate-level evidence is evaluated on Terminal-Bench v2.1 only: the Proposal Pool in Section~\ref{sec:proposals} evaluates $n{=}100$ skills proposed from 10 Terminal-Bench training tasks across 10 disjoint test tasks, the Selection Pool in Section~\ref{sec:selection} evaluates $n{=}100$ skills proposed from Terminal-Bench and DeepSWE trajectories across 11 tasks, and the trajectory re-evaluation in Section~\ref{sec:accumulation} covers two methods and two models. Whether the conclusions hold for larger pools, different generation strategies, or other benchmark domains remains an open question.”
The referee's report
Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.
Pith's reading of the paper (a machine's summary, not the authors')
The paper proposes the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.
What carries the argument. The Resolution Hypothesis: the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them. The paper defines evaluation resolution as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. The paper uses a variance decomposition separating evaluation noise (σ²) from run-to-run variance (τ²) to measure resolution.
Core claim. The paper argues that agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.
The premise it rests on. The paper assumes that the evaluation noise and run-to-run variance measured on the four benchmarks (ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1) generalize to other benchmarks and settings.
If the paper is right (Pith's inferences, not the authors')
If the Resolution Hypothesis is correct, then improving agent evolution requires either enlarging the improvements being proposed or making smaller improvements reliably detectable.
The paper suggests that developing higher-resolution feedback may be as important as developing more powerful evolution algorithms.
The paper recommends publishing both evaluation noise (σ̂) and run-to-run variance (τ̂) alongside point scores so readers can judge whether a claimed margin exceeds the evaluation's resolution.
The paper proposes component-wise evaluation: Quality@K curves for proposers and selected-artifact quality curves for selectors, to split one comparison that falls below the resolution into two that exceed it.
The paper's findings suggest that many published rankings of agent evolution methods may be unreliable, as typical margins are smaller than single-run variance.
The paper's analysis implies that evaluation shortcuts (e.g., adapting to a grader that credits only one answer format) can produce large reported gains that vanish when the evaluation is repaired, calling into question the validity of such gains.
The paper's framework could be applied to other domains where noisy evaluation is used for selection, such as hyperparameter optimization or neural architecture search.
The paper's recommendation of component-wise evaluation (Quality@K for proposers, selected-artifact quality for selectors) could be adopted as standard practice in the field.
Check this review
What was read
The passes were given all 175,395 characters of the paper's LaTeX source.
Fingerprint of that text (SHA-256): a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8
Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:
PYTHONPATH=. python scripts/review_input_check.py 2611.23068
The passes, in the order they ran
- Reader deepseek-v4-flash · medium effort · $0.0053 · instructions of the Reader, fingerprint f9ead7300382…
- Skeptic deepseek-v4-flash · medium effort · $0.0049 · instructions of the Skeptic, fingerprint 8f88c0f61488…
- Referee deepseek-v4-flash · medium effort · $0.0052 · instructions of the Referee, fingerprint a852d99aea0d…
- Check against the paper deepseek-v4-flash · medium effort · $0.0052 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
- Editor deepseek-v4-flash · medium effort · $0.0049 · instructions of the Editor, fingerprint e7211f6a3d27…
- Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0052 · instructions of the Summary for the page, fingerprint e4f56bf568e7…
The passes cost $0.031 in model calls together.
Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.0ef1a750a3e3
Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.
curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum
prints the fingerprint of the reader's instructions. The same works for every pass.
The receipt
The review's fingerprint was taken on 18 Nov 2026. The review you are reading still matches it.
Fingerprint of the review (SHA-256): a44cd811e9219038a8cbf922644aa00ae001a7ce26b47f0dfbb92ebcb3ef66e4
curl -s 'https://pith.science/api/v1/verdicts/2611.23068?form=canonical' | sha256sum
prints the fingerprint of the review as it stands now. The rule, in four steps.
A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-18
The authors' main claim, as the classifier picked it out 0.90
“Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements.”
How the main claim is stated: hedged possibility 0.96
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.28 · human participants 0.04 · controlled comparison 0.26 · randomized assignment 0.03 · states quantitative result 0.93 · compares with prior results 0.59 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. | background 1.00 | 0.07 | 0.03 | asserted 1.00 |
| This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. | background 0.62 | 0.06 | 0.02 | asserted 0.99 |
| We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. | interpretation 0.39 | 0.32 | 0.79 | asserted 0.96 |
| Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. | result 0.64 | 0.97 | 0.17 | asserted 1.00 |
| At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. | result 0.99 | 0.92 | 0.12 | asserted 1.00 |
| At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. | result 0.98 | 0.87 | 0.08 | asserted 0.93 |
| Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. | interpretation 0.96 | 0.83 | 0.85 | hedged possibility 0.96 |
| This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable. | future work 0.92 | 0.09 | 0.31 | asserted 0.63 |
What kind of paper the classifier reports this is: empirical study 0.99 · evidence from observation or experiment 0.93
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-18: 1 request, 6,267 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Showing 10 of the paper's 12 figures. Figures are the authors', reproduced from the paper.
References (47)
- [1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice.Advances in neural information processing systems, 34:29304–29320, 2021 on Pith
- [2] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflecti... on PithPith's reviewarXiv
- [3] Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation.Journal of Machine Learning Research, 11(70):2079–2107, 2010. URL https: //jmlr.org/papers/v11/cawley10a.html on Pith
- [4] Kaiming Cheng, Laura Wang, Jack Khuu, Mark Saroufim, Wenyuan Chi, Jiannan Wang, and Joe Isaacson. KernelAgent: Hardware-guided GPU kernel optimization via multi-agent orchestration. PyTorch Blog, 2026. URL https://pytorch.org/blog/kernelagent-hardware-guided-gpu-kernel-optimiz... on Pith
- [5] Databricks. OfficeQA: A grounded reasoning benchmark. Official dataset card, 2025. URLhttps://huggingfac e.co/datasets/databricks/officeqa. Documents from Hugging Face revision 763a836; splits as released with SkillOpt-Lite (commit 4cb4eee); official grader fromhttps://github.... on Pith
- [6] DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https: //arxiv.org/abs/2606.19348 on PitharXiv
- [7] Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, and Fan Yang. Agent skills can be harmful: An empirical study of skill-induced failures in LLM agents, 2026. URLhttps://arxiv.org/abs/2608.11888 on PithPith's reviewarXiv
- [8] Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. SearchQA: A new Q&A dataset augmented with context from a search engine, 2017. URLhttps://arxiv.org/abs/1704.05179 on PithPith's reviewarXiv
- [9] GLM-5-Team. GLM-5: from vibe coding to agentic engineering, 2026. URLhttps://arxiv.org/abs/2602.15763 on PithPith's reviewarXiv
- [10] Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. LiveMathematicianBench: A live benchmark for research-level mathematical reasoning with proof sketches, 2026. URLhttps://arxiv.org/abs/2604.01754 on PithPith's reviewarXiv
- [11] Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for many-hop fact extraction and claim verification. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460, Online, 2020. Association ... on Pithdoi:10.18653/v1/2020.findings-emnlp.309
- [12] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi- armed bandit models.Journal of Machine Learning Research, 17:1–42, 2016. URLhttps://arxiv.org/abs/1407 .4443 on Pith
- [13] Kimi Team. Kimi K3: Open frontier intelligence. Technical report, Moonshot AI, 2026. URLhttps://github.c om/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf on Pith
- [14] Zhiyi Kuang, Ryan Rong, YuCheng Yuan, and Allen Nie. Learning game-playing agents with generative code optimization. InICML 2025 Workshop on Programmatic Representations for Agent Learning, 2025. URL https://arxiv.org/abs/2508.19506 on PithPith's reviewarXiv
- [15] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses, 2026. URLhttps://arxiv.org/abs/2603.28052 on PithPith's reviewarXiv
- [16] Siyan Li, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. PAPILLON: Privacy preservation from Internet-based and local language model ensembles. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Co... on Pithdoi:10.18653/v1/2025.naacl-long.173
- [17] Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks, 2026. URLhttps://arxiv.org/abs/2602.12670 on PithPith's reviewarXiv
- [18] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses, 2026. URLhttps://arxiv.org/abs/2604.25850 on PithPith's reviewarXiv
- [19] Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, and Yangqiu Song. Rethinking self-evolving agent skills: Feedback dynamics over multiple rounds, 2026. URLhttps://arxiv.org/abs/2608.02636. 11 on PithPith's reviewarXiv
- [20] Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spreadsheet manipulation, 2024. URLhttps: //arxiv.org/abs/2406.14991. arXiv:2406.14991 on PithPith's reviewarXiv
show 27 more
- [21] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, on Pith
- [22] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastro... on PithPith's reviewarXiv
- [23] Daniel Mewes, Nayana Bannur, and Catherine Kim. Beating ARC-AGI-2 with code evolution. Imbue Blog, 2026. URLhttps://imbue.com/blog/2026-02-27-arc-agi-2-evolution. Accessed: 2026-10-09 on Pith
- [24] Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transferable agent skills, 2026. URL https://arxiv.org/abs/2603.25158 on PithPith's reviewarXiv
- [25] OpenAI. Introducing GPT-5.4 mini and nano, 2026. URLhttps://openai.com/index/introducing-gpt-5-4-min i-and-nano/. Accessed: 2026-10-09 on Pith
- [26] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition, 2026. URLhttps://openai.com/index/g pt-5-6/. Accessed: 2026-10-09 on Pith
- [27] Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with rea... on Pith
- [28] Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. InAdvances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 38,... on PithPith's reviewarXiv
- [29] Linfang Shang, Ming Xu, Yiding Sun, Tianle Xia, Lingxiang Hu, Lan Xu, and Ning Zheng. When self-evolution backfires: Pre-commit gating against skill contamination in LLM agents, 2026. URLhttps://arxiv.org/abs/26 08.05810 on Pith
- [30] Zayx Shawn. PACE: Anytime-valid acceptance tests for self-evolving agents.arXiv preprint arXiv:2606.08106, 2026 on PithPith's reviewarXiv
- [31] Yifei Shen, Bo Li, and Xinjie Zhang. SkillOpt-Lite: Better and faster agent self-evolution via one line of vibe, on Pith
- [32] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations, 2021. URLhttps://arxiv.org/abs/2010.03768 on PithPith's reviewarXiv
- [33] Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi. Do agent optimizers compound? a continual-learning evaluation on terminal-bench 2.0, 2026. URLhttps://arxiv.org/abs/2607.14004. 12 on PithPith's reviewarXiv
- [34] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Noah A. Smith, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents, 2026. URLhttps://arxiv.org/abs/2607.12227 on PithPith's reviewarXiv
- [35] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and M... on Pith
- [36] Jinghan Xu, Yikai Zhang, Aili Chen, Weiyuan Li, Jiaqing Liang, and Deqing Yang. Verify smarter, evolve further: Efficient harness evolution through behavior-aware verification, 2026. URLhttps://arxiv.org/abs/2608.27311 on PithPith's reviewarXiv
- [37] Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt: Executive strategy for self-evolving agent skills, 2026. URLhttps://arxiv.org/abs/2605.23904 on PithPith's reviewarXiv
- [38] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pr... on Pithdoi:10.18653/v1/d18-1259
- [39] Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, and Chien-Sheng Wu. On the fragility of self-improving agents: Variance, task order, and underspecification, 2026. URL https://arxiv.org/abs/2608.18066v2 . arXiv:2608.18066, version 2, September 6, 2026 on PithPith's reviewarXiv
- [40] Zhaochen Yu, Yingcheng Wu, Zhenfei Yin, Kaiyuan Chen, Zhe Zhao, Mengdi Wang, Shuicheng Yan, and Ling Yang. Recursive experiential-working memory evolution for long-horizon agent harnesses, 2026. URL https://arxiv.org/abs/2608.24876 on PithPith's reviewarXiv
- [41] Z.ai. GLM-5.2. Model card, 2026. URLhttps://huggingface.co/zai-org/GLM-5.2 on Pith
- [42] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin gödel machine: Open-ended evolution of self-improving agents, 2025. URLhttps://arxiv.org/abs/2505.22954. arXiv:2505.22954 on PithPith's reviewarXiv
- [43] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models, 2025. URL... on PithPith's reviewarXiv
- [44] Congjie Zheng, Chuanyi Xue, Bin Liang, Jun Yang, and Changshui Zhang. SEAGym: An evaluation environment for self-evolving LLM agents, 2026. URLhttps://arxiv.org/abs/2606.17546 on PithPith's reviewarXiv
- [45] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Inform... on Pith
- [2021] URL https://openaccess.thecvf.com/content/WACV2021/html/Mathew_DocVQA_A_Dataset_for_VQA_on_Doc ument_Images_WACV_2021_paper.html on Pith
- [2026] URL https://arxiv.org/abs/2607.03451. Code: https://github.com/EvolvingLMMs-Lab/SkillOpt-Lite, commit 4cb4eee on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment