Pith. sign in

REVIEW 4 major objections 5 minor 95 references

ADIAS: Automated Design of Interactive Agentic Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces issue-centric agent optimization, which stores repair progress as a persistent issue state and uses it to control full-code agent revisions, and reports that this outperforms the strongest candidate-centric baseline…

desk verdict Genuinely new control-state idea for automated agent design, with honest ablations and a useful Tau-Bench expansion, but the central mechanism's accuracy is unmeasured and the headline gains rest on subsets and single rollouts. read the letter →

arxiv 2608.06410 v1 pith:G76DMC5A submitted 2026-08-03 cs.AI cs.CLcs.MA

Lekang Jiang , Bohan Tang , Stephan Goetz , Yiwen Guo This is my paper · ORCID
classification cs.AIcs.CLcs.MA
keywords issue-centricagentoptimizationautomateddesignpersistentissuestatefull-coderevisioninteractivebenchmarksharnesslifecycletrackingLLMagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated agent design normally keeps cross-round experience organized around candidate agents, so the optimizer has to reconstruct what went wrong and what has already been tried at every round. This paper argues that repair progress should instead be stored as an explicit persistent issue state, with stable identities, priorities, lifecycle status, supporting evidence, and intervention-outcome histories, and that this state should directly control which code revision comes next. The paper instantiates this as ADIAS, a full-code design method, and reports that it outperforms the strongest baseline by 25.2% on average across five interactive benchmarks and stays ahead across four backbone models. Controlled ablations report drops of up to 40.7% when the persistent state is removed or demoted to mere context. The paper's own limitation is that it does not measure the accuracy of the failure diagnosis and issue association that the state depends on.

What carries the argument

The machinery that carries the argument is the persistent issue state together with the Issue Manager that maintains it. Each issue record $e_t^i = (id_i, q_i^t, s_i^t, B_i^t, U_i^t)$ stores a stable identity, a priority, a lifecycle status, supporting evidence, and an intervention-outcome history, and the state is updated after every round by associating new diagnostic findings with existing identities or creating new ones. Issue-guided optimization then jointly selects target issues and a revision plan, chooses a parent agent, and applies a focused full-code patch, so the state is an operational control signal rather than background context. Supporting components, including a one-shot external prior search, an evidence-querying diagnostic agent, and a trajectory database, feed the state, but the paper's controlled ablations isolate the state itself as the load-bearing difference.

What would settle it

Run ADIAS on one benchmark with issue identities randomly permuted at every round while keeping all other components fixed: if performance stays near the reported level, the persistent issue state is not the operative cause. Alternatively, trace diagnostic reports across rounds and count how often the same underlying failure is assigned a new identity instead of being matched to an existing one; a high mismatch rate would violate the stable-identity assumption the method depends on.

Watch

Extended reading notes

Core claim

The central claim is that in long-horizon interactive environments, the bottleneck in automated agent design is not the expressiveness of the search space but the continuity of the repair objective. ADIAS therefore augments the usual archive of candidates with a persistent issue state $E_t$, in which each issue carries a stable identity $id_i$, priority $q_i^t$, lifecycle status $s_i^t$ (active, tentatively fixed, confirmed fixed, or regressed), supporting evidence $B_i^t$, and intervention-outcome history $U_i^t$. The Issue Manager associates newly diagnosed failures with existing identities, records targeted interventions separately from side effects on other issues, and chooses both the parent candidate and the revision direction from this state. The paper reports a 25.2% average improvement over the strongest baseline across Tau-Bench, ALFWorld, TextCraft, WebShop, and ScienceWorld, consistent gains across four backbone models on Tau-Bench, and ablation drops of up to 40.7% when the issue state is removed or used only as context.

Load-bearing premise

The load-bearing premise is that the failure-detection component reliably identifies distinct problems and that the Issue Manager attaches each newly observed failure to the correct persistent identity across rounds; the paper does not measure this accuracy, so noisy or drifting labels could let the reported gains come from external priors, extra diagnosis, or search budget instead of issue-centric tracking.

Editorial extensions

If this is right

  • Automated agent design becomes a cumulative repair process: later rounds inherit the issue-specific outcome of earlier interventions instead of re-deriving repair intent from raw candidate records.
  • The 25.2% average gain over the strongest baseline across five interactive benchmarks implies that making repair progress explicit is more important than expanding the editable code space alone.
  • Consistent gains across four backbone models on Tau-Bench suggest the issue-centric control state transfers across model families, so the method is not tuned to one language model's failure patterns.
  • Ablation drops of up to 40.7% imply that neither external priors nor extra diagnosis explain the gains; the issue state must actively steer parent selection and revision planning.
  • The expanded Tau-Bench retail result, 81.7 versus 68.7 for the strongest baseline, indicates that the advantage is not confined to the cheaper return-intent subset used in the main experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to measure diagnostic and issue-attribution accuracy explicitly; if identities drift, adding a de-duplication or identity-verification step should improve ADIAS further, a test the paper does not run.
  • The same persistent-issue control could be ported to prompt-level and architecture-level agent search, because those settings face the same repair-targeting bottleneck even without full-code edits.
  • The lifecycle state, active, tentatively fixed, confirmed fixed, or regressed, doubles as a human-auditable record of why each code change was made, which could support inspection and safety review of self-improving agents.
  • Because ADIAS can edit its own issue manager and code improver, its optimization loop is itself evolvable; a focused experiment would check whether diagnostic quality improves over rounds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes issue-centric agent optimization, a paradigm for automated design of interactive agent systems in which cross-round repair progress is stored in an explicit persistent issue state (issue identity, priority, lifecycle status, supporting evidence, and intervention-outcome history) rather than reconstructed from candidate archives. The proposed framework ADIAS combines a diagnostic agent, a one-shot external task-prior search, an Issue Manager that maintains this state, and an issue-guided optimizer that jointly selects repair targets and revision directions for focused full-code modifications. Across five interactive benchmarks (Tau-Bench, ALFWorld, TextCraft, WebShop, ScienceWorld), ADIAS reports an average score of 78.4 versus 62.6 for the strongest baseline (a 25.2% relative improvement), consistent gains across four backbone models on Tau-Bench, and ablations showing performance drops of up to 40.7% when the issue state is removed or decoupled from control. The paper also provides repeated-rollout and full-distribution checks for Tau-Bench.

Significance. If the claims hold, issue-centric state is a potentially valuable design principle for automated agent design, shifting the field from candidate archives to explicit repair-state representation. The paper has notable strengths: the authors ship code, use a matched backbone model for the optimizer and the task agent, run controlled ablations for external priors, round-level diagnosis, archive-wide synthesis, and parent-selection policies, and include an expanded Tau-Bench distribution check. The cross-model results and repeated-rollout analysis on Tau-Bench strengthen robustness. However, the headline improvement is not yet established with statistical confidence across all benchmarks, and the central mechanism—stable issue identities and lifecycle tracking—is not directly measured. The contribution is significant but currently conditional on additional evidence.

major comments (4)
  1. [Section 5.1, Table 1; Section E.1, Table 6] The headline claim of a 25.2% average improvement over the strongest baseline is computed from single-rollout scores in Table 1, and repeated-rollout evidence is provided only for Tau-Bench (Table 6). Because every other benchmark is reported without variance or significance estimates, the cross-benchmark claim is not yet statistically supported. Please add repeated rollouts or error bars for all five benchmarks, or explicitly restrict the claim to the settings with robustness evidence.
  2. [Section 4 (Benchmarks), Appendix D.1] The task subsets are filtered in ways that may be correlated with the proposed mechanism: WebShop retains only instances requiring more than 10 non-redundant interactions, ScienceWorld retains only instances whose reference trajectories complete within 30 steps and whose task types have more than five eligible cases, and the main Tau-Bench setting uses only return-related tasks. The expanded-distribution check in Appendix E.2 is limited to Tau-Bench. These filters could select long-horizon, high-variance tasks that specifically reward explicit issue-state tracking, so the five-benchmark claim should be validated on unfiltered distributions or accompanied by an analysis of how the filter affects each method.
  3. [Section 3.3.1 (Persistent Issue State), Section 6 (Limitations)] The persistent issue state is the paper's central contribution, but the quality of its core inputs is never measured. Section 6 states that failure-label and issue-attribution accuracy are not reported, and Section 3.3.1 defines issue association without any precision, recall, merge/split, or identity-stability statistics. The lifecycle transitions (e.g., CONFIRMED-FIXED after alpha_min = 2 absences) and intervention-outcome histories are built on these unverified labels, so the ablation in Table 3 cannot distinguish the hypothesis that stable issue identities drive improvement from the hypothesis that any structured summarization of the same evidence would help. Please report diagnostic and association accuracy, or run controlled manipulations such as oracle labels, shuffled identities, or no identity persistence.
  4. [Section 3.3.1, Appendix C.1, Table 3] ADIAS is the only method that receives external task-level priors, and Table 3 shows that removing this prior costs up to 30 points on TextCraft (91.0 to 61.0). The main comparison in Table 1 therefore confounds the issue-centric mechanism with an information advantage shared by no baseline. To support the claim that ADIAS outperforms candidate-centric methods because of issue-centric optimization, the external prior should either be given to all baselines or be ablated in the main results, or the authors should quantify the interaction between the prior and the issue-state mechanism.
minor comments (5)
  1. [Section 3.3.1] The lifecycle parameter alpha_min is fixed at 2 with no sensitivity analysis; since the CONFIRMED-FIXED transition is part of the claimed mechanism, please report a small sweep or justify the choice.
  2. [Section E.1, Table 6] The paragraph explaining Table 6 is ambiguous about the relationship between the two individual passes and the Pass 1/Pass@2 aggregates; please clarify that a 'same score' across the two passes does not imply identical per-task outcomes, as the text briefly notes.
  3. [Figure 4] The line styles for Best-Candidate Revision, Latest-Candidate Continuation, and ADIAS are hard to distinguish in small print; please use larger markers and a legend with explicit test scores.
  4. [Appendix D.2] The paper states that AHE and Meta-Harness were adapted for general interactive tasks but gives no details on the adaptation; please document what was changed so readers can assess fairness.
  5. [General] There are minor typographical issues, such as the running title 'AUTOMATEDDESIGN' missing a space; please proofread the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ADIAS's reported gains are empirical benchmark comparisons, not derivations; the unmeasured diagnostic accuracy noted in Section 6 is a stated limitation, not a self-referential reduction.

full rationale

The paper contains no derivation chain whose output is equivalent to its input by construction. Its central claims are empirical: measured test scores in Tables 1, 2, and 7, and controlled ablations in Table 3. ADIAS is not defined in terms of the benchmark scores it predicts; the issue state is constructed from trajectories, diagnoses, and public task priors, and the objective is to maximize held-out test performance after selecting the best validation agent. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation is invoked; the reference list contains no work by the present authors. The closest concern is Section 6, which states: 'we do not separately report the accuracy of failure labels or issue attributions.' That is an honest limitation about whether stable issue identities are the operative cause of the observed gains, not circularity: the ablations could still show that the state matters even if diagnostic quality is unmeasured, and the gains would remain an empirical outcome of the compared systems. Similarly, the external prior search supplies additional input to ADIAS that baselines do not receive; this is an evaluation-fairness question, not a case where the conclusion is presupposed. Under the required standard of exhibiting a specific reduction, no circular step exists.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on assumptions about diagnosis quality, issue identity stability, and evaluation fairness rather than on fitted parameters. The only explicit numeric hyperparameter is alpha_min=2 for lifecycle confirmation. No new physical or theoretical entities are introduced; the persistent issue state is a software construct whose value is asserted through ablations.

free parameters (2)
  • alpha_min (lifecycle confirmation threshold) = 2
    Number of consecutive evaluations without observation required to mark an issue CONFIRMED-FIXED. Chosen by hand (Section 3.3.1); no sensitivity analysis is reported.
  • number of target issues per round
    Section 3.3.2 says a 'small set' of targets is selected, but the size and the prioritization formula are not specified. This is an undeclared design choice that can affect results.
assumptions (4)
  • domain assumption The diagnostic agent accurately extracts issue categories, supporting evidence, and inferred causes from trajectories.
    Section 3.3.1 and the Limitations section explicitly state that diagnostic accuracy is not measured; the entire issue state is built from these labels.
  • domain assumption Issue association using normalized failure category, affected capability, execution context, and evidence yields stable identities across generations.
    Section 3.3.1, 'Issue Association'. If identities are unstable, repair progress cannot accumulate and the method degenerates toward candidate-centric search.
  • domain assumption External prior search retrieves only public task-level information and excludes held-out labels and instance-specific solutions.
    Appendix C.1 asserts this restriction but does not audit the retrieved content; benchmark priors could advantage ADIAS over baselines that do not receive them.
  • domain assumption The filtered benchmark subsets provide an unbiased comparison across methods.
    Section 4 and Appendix D.1 describe subsetting rules (returns-only Tau-Bench, >10-step WebShop, <=30-step ScienceWorld); only Tau-Bench receives a full-distribution check.
invented entities (1)
  • Persistent issue state (issue identity, priority, lifecycle status, evidence, intervention-outcome history) independent evidence
    purpose: Carries repair progress across rounds and guides which issue to repair and from which parent to resume optimization.
    It is the paper's main new construct. Within-paper ablations (Archive-Wide Synthesis, Best/Latest Candidate Revision) provide falsifiable evidence for its causal role, but its reliability depends on the unmeasured diagnosis and association assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADIAS: Automated Design of Interactive Agentic Systems." pith.science (2026). https://pith.science/paper/G76DMC5A

@misc{pith2026260806410,
  author       = {Pith},
  title        = {Pith review of: ADIAS: Automated Design of Interactive Agentic Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G76DMC5A}},
  note         = {Machine review of arXiv:2608.06410}
}
read the original abstract

Automated agent design improves agent harnesses through iterative revision, evaluation, and feedback summarization. Existing methods are largely candidate-centric: cross-round experience is organized around candidate agents, which leaves the repair progress implicit. This causes inefficient repair targeting, slow consolidation of partial progress, and propagation of ineffective interventions across rounds. Therefore, we formulate issue-centric agent optimization, in which repair progress is carried forward as an explicit persistent issue state to guide optimization, rather than re-derived from candidate history in each round. We instantiate the formulation in ADIAS, a framework for automated full-code agent design with two mechanisms. A persistent issue state maintains stable issue identities, lifecycle status, supporting evidence, and intervention-outcome histories. Issue-guided optimization uses this state to jointly propose repair targets and revision directions for subsequent focused full-code modification. Across five interactive benchmarks, ADIAS outperforms the strongest baseline by 25.2% on average and achieves consistent gains across four backbone models. Controlled ablations further show that removing persistent issue state or replacing issue-centric revision with candidate-centric policies leads to performance drops of up to 40.7%.

Figures

Figures reproduced from arXiv: 2608.06410 by the authors.

Figure 1
Figure 1. Comparison between candidate-centric agent optimization and issue-centric optimization. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of ADIAS that consists of two stages: initialization and iterative optimization. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Optimization process on ALFWorld. Cross-Model Robustness. We conduct a cross￾model evaluation on Tau-Bench using four het￾erogeneous language models: DeepSeek-V4- Flash, GLM-5.2, Hy3-Preview, and GPT-5.4. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of optimization processes on the TextCraft benchmark. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Task comparison between single-IO tasks and long-horizon interactive environments. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

95 extracted references · 44 canonical work pages

  1. [1]

    The Thirteenth International Conference on Learning Representations , year=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. The Thirteenth International Conference on Learning Representations , year=

  2. [2]

    International Conference on Learning Representations , year=

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning , author=. International Conference on Learning Representations , year=

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Webshop: Towards scalable real-world web interaction with grounded language agents , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Scienceworld: Is your agent smarter than a 5th grader? , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  5. [5]

    Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    Adapt: As-needed decomposition and planning with language models , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    2024

  6. [6]

    International Conference on Learning Representations (ICLR) , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  7. [10]

    2026 , howpublished =

    2026

  8. [11]

    International Conference on Learning Representations , volume=

    Automated design of agentic systems , author=. International Conference on Learning Representations , volume=

Show all 95 references
  1. [12]

    International Conference on Learning Representations , volume=

    Agentsquare: Automatic llm agent search in modular design space , author=. International Conference on Learning Representations , volume=

  2. [13]

    International Conference on Learning Representations , volume=

    Aflow: Automating agentic workflow generation , author=. International Conference on Learning Representations , volume=

  3. [14]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  4. [16]

    The eleventh international conference on learning representations , year=

    Large language models are human-level prompt engineers , author=. The eleventh international conference on learning representations , year=

  5. [17]

    International Conference on Learning Representations , volume=

    Large language models as optimizers , author=. International Conference on Learning Representations , volume=

  6. [18]

    gradient descent

    Automatic prompt optimization with “gradient descent” and beam search , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  7. [19]

    International Conference on Learning Representations , volume=

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers , author=. International Conference on Learning Representations , volume=

  8. [20]

    Forty-first International Conference on Machine Learning , year=

    Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution , author=. Forty-first International Conference on Machine Learning , year=

  9. [21]

    International Conference on Learning Representations , volume=

    Promptagent: Strategic planning with language models enables expert-level prompt optimization , author=. International Conference on Learning Representations , volume=

  10. [22]

    International Conference on Learning Representations , year=

    Gepa: Reflective prompt evolution can outperform reinforcement learning , author=. International Conference on Learning Representations , year=

  11. [23]

    International Conference on Learning Representations , volume=

    DSPy: compiling declarative language model calls into state-of-the-art pipelines , author=. International Conference on Learning Representations , volume=

  12. [24]

    International Conference on Learning Representations , year=

    Agentic context engineering: Evolving contexts for self-improving language models , author=. International Conference on Learning Representations , year=

  13. [25]

    Forty-first International Conference on Machine Learning , year=

    Gptswarm: Language agents as optimizable graphs , author=. Forty-first International Conference on Machine Learning , year=

  14. [28]

    Yin, Xunjian and Wang, Xinyi and Pan, Liangming and Lin, Li and Wan, Xiaojun and Wang, William Yang , booktitle=. G

  15. [29]

    Darwin G

    Zhang, Jenny and Hu, Shengran and Lu, Cong and Lange, Robert and Clune, Jeff , journal=. Darwin G

  16. [30]

    AutoGen: Enabling Next-Gen

    Qingyun Wu and Gagan Bansal and Jieyu Zhang and Yiran Wu and Beibin Li and Erkang Zhu and Li Jiang and Xiaoyun Zhang and Shaokun Zhang and Jiale Liu and Ahmed Hassan Awadallah and Ryen W White and Doug Burger and Chi Wang , booktitle=. AutoGen: Enabling Next-Gen. 2024 , url=

    2024

  17. [31]

    Frontiers of Computer Science , volume=

    A survey on large language model based autonomous agents , author=. Frontiers of Computer Science , volume=. 2024 , publisher=

    2024

  18. [32]

    Science China Information Sciences , volume=

    The rise and potential of large language model based agents: A survey , author=. Science China Information Sciences , volume=. 2025 , publisher=

    2025

  19. [33]

    , author=

    Large Language Model based Multi-Agents: A Survey of Progress and Challenges. , author=. 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024) , year=

    2024

  20. [34]

    Journal of Artificial Intelligence Research , volume=

    Agentic large language models, a survey , author=. Journal of Artificial Intelligence Research , volume=

  21. [35]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  22. [36]

    arXiv preprint arXiv:2502.01600 , year=

    Reinforcement learning for long-horizon interactive llm agents , author=. arXiv preprint arXiv:2502.01600 , year=

    arXiv

  23. [37]

    International Conference on Learning Representations , volume=

    Webarena: A realistic web environment for building autonomous agents , author=. International Conference on Learning Representations , volume=

  24. [39]

    Transactions on Machine Learning Research , year=

    A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence , author=. Transactions on Machine Learning Research , year=

  25. [42]

    International Conference on Machine Learning , pages=

    Multi-agent Architecture Search via Agentic Supernet , author=. International Conference on Machine Learning , pages=. 2025 , organization=

    2025

  26. [43]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Agentswift: Efficient llm agent design via value-guided hierarchical search , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  27. [52]

    arXiv preprint arXiv:2607.13104 , year=

    Self-Improvements in Modern Agentic Systems: A Survey , author=. arXiv preprint arXiv:2607.13104 , year=

    arXiv

  28. [55]

    arXiv preprint arXiv:2602.16037 , year=

    Optimization Instability in Autonomous Agentic Workflows for Clinical Symptom Detection , author=. arXiv preprint arXiv:2602.16037 , year=

  29. [56]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  30. [58]

    arXiv preprint arXiv:2607.14004 , year=

    Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 , author=. arXiv preprint arXiv:2607.14004 , year=

    arXiv

  31. [60]

    lilianweng.github.io , year =

    Harness Engineering for Self-Improvement , author =. lilianweng.github.io , year =

  32. [63]

    Gepa: Reflective prompt evolution can outperform reinforcement learning

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Repre...

    2026

  33. [64]

    Moss: Self-evolution through source-level rewriting in autonomous agent systems

    Qianshu Cai, Yonggang Zhang, Xianzhang Jia, Huajiang Zheng, Wei Xue, Jun Song, Xinmei Tian, and Yike Guo. Moss: Self-evolution through source-level rewriting in autonomous agent systems. arXiv preprint arXiv:2605.22794, 2026

    2026 arXiv

  34. [65]

    Harnessx: A composable, adaptive, and evolvable agent harness foundry

    Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, et al. Harnessx: A composable, adaptive, and evolvable agent harness foundry. arXiv preprint arXiv:2606.14249, 2026

    2026 arXiv

  35. [66]

    DeepSeek V4 Preview Release

    DeepSeek . DeepSeek V4 Preview Release . https://api-docs.deepseek.com/news/news260424, 2026. Accessed: 2026-07

    2026

  36. [67]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference on Machine Learning, 2024

    2024

  37. [68]

    A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, et al. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence. Transactions on Machine Learn...

    2026

  38. [69]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, volume 2024, pp....

    2024

  39. [70]

    Amine El Hattami, Nicolas Chapados, and Christopher Pal. Skill. nb: Selective formalization and gated execution for durable agent workflows. arXiv preprint arXiv:2606.08049, 2026

    2026 arXiv

  40. [71]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, volume 2025, pp.\ 21344--21377, 2025

    2025

  41. [72]

    Understanding the planning of llm agents: A survey

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716, 2024

    2024 arXiv

  42. [73]

    Memoharness: Agent harnesses that learn from experience

    Yue Huang, Wenjie Wang, Han Bao, Yuchen Ma, Xiaonan Luo, Yi Nian, Haomin Zhuang, Zheyuan Liu, Yue Zhao, and Xiangliang Zhang. Memoharness: Agent harnesses that learn from experience. arXiv preprint arXiv:2607.14159, 2026

    2026 arXiv

  43. [74]

    Dspy: compiling declarative language model calls into state-of-the-art pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al. Dspy: compiling declarative language model calls into state-of-the-art pipelines. In International Conference on Le...

    2024

  44. [75]

    Meta-harness: End-to-end optimization of model harnesses

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026

    2026 arXiv

  45. [76]

    Agentswift: Efficient llm agent design via value-guided hierarchical search

    Yu Li, Lehui Li, Zhihao Wu, Qingmin Liao, Jianye Hao, Kun Shao, and Fengli Xu. Agentswift: Efficient llm agent design via value-guided hierarchical search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp.\ 31843--31851, 2026

    2026

  46. [77]

    Autoflow: Automated workflow generation for large language model agents

    Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. arXiv preprint arXiv:2407.12821, 2024

    2024 arXiv

  47. [78]

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026 a

    2026 arXiv

  48. [79]

    Position: Agentic evolution is the path to evolving llms

    Minhua Lin, Hanqing Lu, Zhan Shi, Bing He, Rui Mao, Zhiwei Zhang, Zongyu Wu, Xianfeng Tang, Hui Liu, Zhenwei Dai, et al. Position: Agentic evolution is the path to evolving llms. arXiv preprint arXiv:2602.00359, 2026 b

    2026

  49. [80]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pp.\ 52989--53046, 2024

    2024

  50. [81]

    Autoharness: improving llm agents by automatically synthesizing a code harness

    Xinghua Lou, Miguel L \'a zaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P Murphy. Autoharness: improving llm agents by automatically synthesizing a code harness. arXiv preprint arXiv:2603.03329, 2026

    2026

  51. [82]

    Automated design of agentic systems: A survey of algorithms for searching, optimizing, and evolving llm agents, workflows, and prompts

    Maksim Madžar and Igor Mekterović. Automated design of agentic systems: A survey of algorithms for searching, optimizing, and evolving llm agents, workflows, and prompts. Preprints, June 2026. doi:10.20944/preprints202606.0238.v1. URL https://doi.org/10.20944/preprints202606.0238.v1

    2026

  52. [83]

    Grasp: Gated regression-aware skill proposer for self-improving llm agents

    Johannes Moll, Jean-Philippe Corbeil, Jiazhen Pan, Martin Hadamitzky, Daniel Rueckert, Lisa Adams, and Keno Bressem. Grasp: Gated regression-aware skill proposer for self-improving llm agents. arXiv preprint arXiv:2605.29668, 2026

    2026 arXiv

  53. [84]

    Tthe: Test-time harness evolution

    Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, and Bo Han. Tthe: Test-time harness evolution. arXiv preprint arXiv:2607.08124, 2026

    2026 arXiv

  54. [85]

    Code as agent harness

    Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, et al. Code as agent harness. arXiv preprint arXiv:2605.18747, 2026

    2026 arXiv

  55. [86]

    GPT-5.4 Model

    OpenAI . GPT-5.4 Model . https://developers.openai.com/api/docs/models/gpt-5.4, 2026. Accessed: 2026-07

    2026

  56. [87]

    Adapt: As-needed decomposition and planning with language models

    Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pp.\ 4226--4252, 2024

    2024

  57. [88]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp.\ 7957--7968, 2023

    2023

  58. [89]

    Archon: An architecture search framework for inference-time techniques

    Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher R \'e , et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254, 2024

    2024 arXiv

  59. [90]

    Agentsquare: Automatic llm agent search in modular design space

    Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic llm agent search in modular design space. In International Conference on Learning Representations, volume 2025, pp.\ 3841--3865, 2025

    2025

  60. [91]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36: 0 8634--8652, 2023

    2023

  61. [92]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021

    2021

  62. [93]

    Introducing Hy3

    Tencent Hy . Introducing Hy3 . https://hy.tencent.com/research/hy3, 2026. Accessed: 2026-07

    2026

  63. [94]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18 0 (6): 0 186345, 2024 a

    2024

  64. [95]

    Harness handbook: Making evolving agent harnesses readable, navigable, and editable

    Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, et al. Harness handbook: Making evolving agent harnesses readable, navigable, and editable. arXiv preprint arXiv:2607.13285, 2026 a

    2026 arXiv

  65. [96]

    Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 11279--11298, 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 11279--11298, 2022

    2022

  66. [97]

    u rgen Schmidhuber. Huxley-g

    Wenyi Wang, Piotr Pi e kos, Li Nanbo, Firas Laakom, Yimeng Chen, Mateusz Ostaszewski, Mingchen Zhuge, and J \"u rgen Schmidhuber. Huxley-g " odel machine: Human-level coding agent development by an approximation of the optimal self-improving machine. arXiv preprint arXiv:2510....

  67. [98]

    Promptagent: Strategic planning with language models enables expert-level prompt optimization

    Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In International Conference on Learning Representations, volume 2024...

    2024

  68. [99]

    Rethinking the evaluation of harness evolution for agents

    Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. arXiv preprint arXiv:2607.12227, 2026 b

    2026 arXiv

  69. [100]

    Harness engineering for self-improvement

    Lilian Weng. Harness engineering for self-improvement. lilianweng.github.io, July 2026. URL https://lilianweng.github.io/posts/2026-07-04-harness/

    2026

  70. [101]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, volume 2024, pp.\ 12028--12068, 2024

    2024

  71. [102]

    Skillopt: Executive strategy for self-evolving agent skills

    Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. Skillopt: Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026

    2026 arXiv

  72. [103]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 0 20744--20757, 2022

    2022

  73. [104]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  74. [105]

    -bench: A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. -bench: A benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  75. [106]

    G \"o del agent: A self-referential agent framework for recursively self-improvement

    Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. G \"o del agent: A self-referential agent framework for recursively self-improvement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2025

  76. [107]

    Do self-evolving agents forget? capability degradation and preservation in lifelong llm agent adaptation

    Ye Yu, Xiaopeng Yuan, Haibo Jin, Heming Liu, Yaoning Yu, and Haohan Wang. Do self-evolving agents forget? capability degradation and preservation in lifelong llm agent adaptation. arXiv preprint arXiv:2605.09315, 2026

    2026 arXiv

  77. [108]

    From static templates to dynamic runtime graphs: a survey of workflow optimization for llm agents

    Ling Yue, Kushal Raj Bhandari, Ching-Yun Ko, Dhaval Patel, Shuxin Lin, Nianjun Zhou, Jianxi Gao, Pin-Yu Chen, and Shaowu Pan. From static templates to dynamic runtime graphs: a survey of workflow optimization for llm agents. arXiv preprint arXiv:2603.22386, 2026

    2026

  78. [109]

    GLM-5.2 Model

    Z.ai . GLM-5.2 Model . https://docs.bigmodel.cn/cn/guide/models/text/glm-5.2, 2026. Accessed: 2026-07

    2026

  79. [110]

    Multi-agent architecture search via agentic supernet

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet. In International Conference on Machine Learning, pp.\ 75834--75852. PMLR, 2025 a

    2025

  80. [111]

    Darwin g \"o del machine: Open-ended evolution of self-improving agents

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin g \"o del machine: Open-ended evolution of self-improving agents. SuperIntelligence-Robotics-Safety & Alignment, 2 0 (3), 2025 b

    2025

  81. [112]

    Hyperagents

    Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. Hyperagents. arXiv preprint arXiv:2603.19461, 2026 a

    2026

  82. [113]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. In International Conference on Learning Representations, volume 2025, pp.\ 34040--34077, 2025 c

    2025

  83. [114]

    Harnessing agentic evolution

    Jiayi Zhang, Yongfeng Gu, Jianhao Ruan, Maojia Song, Yiran Peng, Zhiguang Han, Jinyu Xiang, Zhitao Wang, Caiyin Yang, Yixi Ouyang, et al. Harnessing agentic evolution. arXiv preprint arXiv:2605.13821, 2026 b

    2026 arXiv

  84. [115]

    Agentic context engineering: Evolving contexts for self-improving language models

    Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models. In International Conference on Learning Represent...

    2026

  85. [116]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 19632--19642, 2024

    2024

  86. [117]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2022

    2022

  87. [118]

    Gptswarm: Language agents as optimizable graphs

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J \"u rgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.