Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that current LLM-based autonomous agents complete only about half of executable coding tasks, and that the 104 observed failures split into 19 causes across planning, execution, and response generation.

desk verdict A solid failure taxonomy and a plausible 50% baseline, but the quantitative comparisons rest on single-run, small-sample data and need replication before the numbers can be trusted. read the letter →

arxiv 2508.13143 v1 pith:N5BEKJMH submitted 2025-08-18 cs.AI cs.SE

Ruofan Lu , Yichen Li , Yintong Huo This is my paper · ORCID
classification cs.AIcs.SE
keywords LLMagentsautonomousfailureanalysistaxonomybenchmarktaskplanningcodegenerationself-diagnosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why LLM-based autonomous agents fail at end-to-end programming tasks, rather than only reporting how often they succeed. It builds a benchmark of 34 executable, automatically checkable tasks drawn from web crawling, data analysis, and file operations, and runs three open-source agent frameworks with two LLM backbones. Across 204 runs, only about half of the tasks were completed exactly, and the authors classified the 104 failures into a three-tier taxonomy: planning errors, execution errors, and response-generation errors, with 19 specific causes. If this failure map is right, improving agent frameworks means targeting planning feedback and self-diagnosis, not just adding more iterations or a stronger model.

What carries the argument

The central object is the three-tier failure taxonomy built on the planner–code generator–executor architecture common to the evaluated agents. The taxonomy turns raw execution logs into a structured failure map: each failure is assigned to a task phase and then to one of 19 leaf causes, such as unrealistic planning, incorrect API usage, or exceeding the maximum round limit. This classification is what lets the paper move from a single success-rate number to targeted advice about where agent design should change.

What would settle it

A concrete check: have independent annotators who did not write the paper re-label the published 104 failure logs into the three-tier taxonomy and measure inter-annotator agreement; low agreement would show the taxonomy is not stable. A second check is to run the same frameworks on tasks drawn without the 'at least partially solvable' filter and see whether success rates and failure proportions change sharply.

Watch

Extended reading notes

Core claim

The central discovery is that current agent frameworks do not fail randomly: their failures cluster in predictable phases that line up with the planner, code generator, and executor roles inside the system. Evaluating three popular open-source frameworks with two LLM backbones on 34 tasks, the paper finds roughly a 50% exact-match completion rate, with the smaller backbone sometimes beating the larger one. The 104 failures are then organized into a three-tier taxonomy with 19 causes, from improper task decomposition and failed self-refinement in planning, through tool misuse and flawed code in execution, to context-window loss and formatting errors in response generation. The paper also reports that success improves with more allowed iterations but saturates after about ten, and that a stronger model can hurt performance when its planning steps conflict with safety constraints.

Load-bearing premise

The load-bearing premise is that the 34 benchmark tasks represent real-world programmable tasks, since the authors deliberately kept only tasks that are executable, automatically checkable, and at least partially solvable by an agent; if that curation skews the sample, the 50% rate and the failure taxonomy do not generalize.

Editorial extensions

If this is right

  • Debugging an agent framework becomes a matter of locating which phase produces the failure, rather than treating the whole system as a black box.
  • Because success saturates after roughly ten iterations, adding more retry rounds is a weak lever; redesigning the planning loop should yield larger gains.
  • The smaller backbone occasionally outperforming the larger one implies that model capability and framework behavior interact, so framework evaluations should report multiple backbones.
  • The proposed mitigations, learning from execution feedback during replanning and an early-stop meta-controller that detects repeated failures, give concrete design targets for future agent systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the same three-tier taxonomy could serve as a diagnostic checklist for agent designs beyond the three frameworks tested; a testable extension is whether failure proportions shift when the planner receives explicit executor feedback after each step.
  • The reported 'overthinking' failures of the stronger model suggest a testable hypothesis: safety constraints and planning modules are in conflict, so letting the planner pre-check tool permissions and web-scraping rules may close part of the gap without changing the model.
  • The exact-match success metric is stricter than a human rubric; an editorial inference is that a partial-credit evaluation might reveal how much of the 50% failure rate is deep reasoning failure versus surface formatting failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a benchmark of 34 programmable tasks in three categories (web crawling, data analysis, file operations), evaluates three open-source agent frameworks (TaskWeaver, MetaGPT, AutoGen) with two LLM backbones (GPT-4o and GPT-4o-mini), and reports an overall task completion rate of about 50% across 204 runs. The authors manually inspect the 104 logged failures and construct a three-tier failure taxonomy (task planning, task execution, response generation) with 19 specific causes, illustrate three common failure cases, and propose two design recommendations: learning-from-feedback planning and early-stop/navigation mechanisms. The paper's central contributions are empirical: a baseline success rate, a failure taxonomy, and associated mitigation advice.

Significance. If the headline success rate and the taxonomy are robust, the paper would provide a useful quantitative reference and a structured vocabulary for diagnosing failures in LLM-based agents. Strengths include the executable-task benchmark with human-verified ground truth, pinned framework and model versions, the release of data and evaluation code, and a manual log analysis that grounds the taxonomy in concrete execution traces rather than speculation. However, the statistical support for the headline numbers and, especially, for cross-framework and cross-model comparisons is currently insufficient, and the manual annotation lacks an inter-annotator agreement measure. These issues need to be addressed before the empirical claims can be taken as baseline results.

major comments (4)
  1. [Section IV-A, Tables II and III] All quantitative results are point estimates from one execution per (task, framework, backbone) cell, with no random seeds, temperatures, or replication stated. With only 34 tasks total and per-category cell sizes of 12, 18, and 4 (e.g., a 100% file-operation score is 4/4), the observed differences are within sampling noise. For instance, the claimed GPT-4o-mini advantage for TaskWeaver (58.82% vs 50.00%) corresponds to 20/34 vs 17/34, which is not significant under an exact binomial comparison, and the AutoGen comparison (50.00% vs 38.24%) is similarly fragile. The paper should report exact binomial confidence intervals or, preferably, run multiple seeds per cell and report means and variances; every comparative claim in Section IV-A should be qualified accordingly.
  2. [Section IV-A, 'overthinking' paragraph] The claim that GPT-4o underperforms because of 'overthinking' (a conflict between task-planning processes and built-in safety constraints such as denying web scraping) is based on qualitative inspection of selected logs rather than a systematic count or a controlled comparison. Since the observed performance difference is within sampling noise, this explanation is not currently supported by the evidence presented. Please either quantify the frequency of this pattern across all GPT-4o failures or explicitly frame it as an anecdotal observation.
  3. [Section IV-B1] The failure taxonomy is the paper's second main contribution, but its reliability is not established. The annotations were produced by the three authors with no reported inter-annotator agreement statistic and no independent validation set. Because the 104 failure logs are a single stochastic realization and the leaf-level counts in the taxonomy (e.g., the 'most frequent' cause highlighted in Figure 3) inherit that variability, the paper should at minimum report agreement statistics such as Cohen's kappa, and preferably also assess the sensitivity of the taxonomy to the particular run.
  4. [Section III-A] The task-selection criteria (executable, automatically evaluable, and at least partially solvable by an agent) are transparent, but they mean the benchmark is a curated subset rather than a representative sample of 'programmable tasks.' The 50% figure should therefore be consistently described as benchmark-specific, and the authors should report the size of the initial task pool and how many tasks were excluded at each screening step so that readers can judge selection bias.
minor comments (5)
  1. [Section III-C] Please report the maximum iteration limit, sampling temperature, and any random seed or deterministic settings; these details are needed to interpret Figure 2 and the tables.
  2. [Section IV-B2 and Figure 3] The text says there are three planning causes but the figure lists four planning-level leaves, including 'Infinite loop with same response'; align the numbering with the figure or explain the relationship between 'failed self-refinement' and 'infinite loop.'
  3. [Section IV-A] The sentence 'Both TaskWeaver and AutoGen achieve perfect scores in File Operations' refers to the GPT-4o-mini results in Table III, but as written it appears to describe both tables; clarify which model is being discussed.
  4. [Section IV-A] The sentence 'The superior performance of GPT-4o-mini is consistent with prior research findings [28]' cites a relevance-assessment study that is not directly about agent task completion; either cite evidence from agent benchmarks or temper the claim.
  5. [Figure 2] The x-axis definition ('maximum iteration numbers') and the aggregation across tasks and models should be stated in the caption or text, and no error bars or replication are shown.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported success rates are measured against external ground truth, and the failure taxonomy is a descriptive labeling of execution logs rather than a fitted or self-referential derivation.

full rationale

The paper's central claims are empirical measurements, not derivations. The headline 'approximately 50%' completion rate is computed by executing 34 benchmark tasks across three frameworks and two LLM backbones and comparing outputs to 'human-verified ground-truth labels' under the stated metric that 'a task is deemed successful only if its output exactly matches the ground-truth answer' (Sec. III-A). This is an external benchmark comparison, so the success rates are not defined in terms of the paper's own conclusions. The three-tier failure taxonomy (planning, execution, response generation) is an annotation of the 104 recorded failure logs (Sec. IV-B1) and is used to describe and illustrate failure modes, not to predict them; no fitted parameter is fed back into the measurements, and no quantity is 'predicted' from the taxonomy. The benchmark curation criterion that tasks must be 'at least partially solvable by an agent' is a design filter, not an equation that forces the measured rates. There is no load-bearing self-citation chain: references are to the evaluated frameworks (TaskWeaver, MetaGPT, AutoGen), external benchmarks such as DABench, and prior work on success-rate metrics; none of these are invoked as an unverified uniqueness theorem or as a substitute for the experiments. Statistical concerns about single-seed runs and small per-category sample sizes are validity issues, not circularity: they do not make any claimed result equivalent to its inputs by construction. Accordingly, the paper is self-contained against external ground truth and warrants a circularity score of 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

This is an empirical evaluation paper with no fitted constants or invented entities. The central claims rest on domain assumptions about task selection, the evaluation metric, and the reliability of manual annotation. The maximum iteration limit is an unreported experimental parameter that affects the headline results.

free parameters (1)
  • Maximum iteration limit = not explicitly reported
    The reported success rates in Tables II and III depend on the maximum number of iterations allowed, which is not stated in the paper; Figure 2 illustrates the effect of varying this threshold.
assumptions (3)
  • domain assumption The 'exact match' output criterion is a valid measure of task success.
    Section III-A defines success as exact output match, which treats minor formatting differences as failures and may undercount near-correct solutions.
  • domain assumption Manual log review by the three authors is unbiased and reliable.
    Section IV-B1 describes annotation by the authors without inter-annotator agreement metrics; the taxonomy derives entirely from this subjective review.
  • domain assumption The 34 selected tasks are representative of common programmable tasks.
    Section III-A selection criteria (executable, automatically evaluable, partially solvable) may introduce selection bias limiting generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks." pith.science (2026). https://pith.science/paper/N5BEKJMH

@misc{pith2026250813143,
  author       = {Pith},
  title        = {Pith review of: Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N5BEKJMH}},
  note         = {Machine review of arXiv:2508.13143}
}
read the original abstract

Autonomous agent systems powered by Large Language Models (LLMs) have demonstrated promising capabilities in automating complex tasks. However, current evaluations largely rely on success rates without systematically analyzing the interactions, communication mechanisms, and failure causes within these systems. To bridge this gap, we present a benchmark of 34 representative programmable tasks designed to rigorously assess autonomous agents. Using this benchmark, we evaluate three popular open-source agent frameworks combined with two LLM backbones, observing a task completion rate of approximately 50%. Through in-depth failure analysis, we develop a three-tier taxonomy of failure causes aligned with task phases, highlighting planning errors, task execution issues, and incorrect response generation. Based on these insights, we propose actionable improvements to enhance agent planning and self-diagnosis capabilities. Our failure taxonomy, together with mitigation advice, provides an empirical foundation for developing more robust and effective autonomous agent systems in the future.

Figures

Figures reproduced from arXiv: 2508.13143 by the authors.

Figure 1
Figure 1. The basic framework of an autonomous agent system. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The success rate concerning max iteration numbers. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The agent failure taxonomy, where the most frequent failure is highlighted and illustrated in Sec. IV-B3. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Three most common failure types. often due to a lack of online or tool-use knowledge. (2) the generator agent produces flawed code with syntax er￾rors, functionality errors (executable but deviating from the intended output), incorrect API usage with wrong parameters, …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Across 1,794 CLI agent trajectories, failures are mostly epistemic, start by median step 7, and often stay silent until after lock-in.

  2. SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    SkillTV-Bench provides a multi-domain, skill-aware trajectory verification benchmark, and SkillTV-Evolve's evolved JudgeSkill improves an agent judge's accuracy by 14.8 points on a held-out set.

Reference graph

Works this paper leans on

33 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    Taskweaver: A code-first agent framework,

    B. Qiao, L. Li, X. Zhang, S. He, Y . Kang, C. Zhang, F. Yang, H. Dong, J. Zhang, L. Wang et al., “Taskweaver: A code-first agent framework,” arXiv preprint arXiv:2311.17541 , 2023

    arXiv 2023

  2. [2]

    Autogen: Enabling next-gen LLM applications via multi- agent conversations,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, “Autogen: Enabling next-gen LLM applications via multi- agent conversations,” in First Conference on Language Modeling, 2024

    work page 2024

  3. [3]

    CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. ...

    work page 2024

  4. [4]

    Executable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” in Proceedings of the 41st International Conference on Machine Learning , ser. ICML’24. JMLR.org, 2024

    work page 2024

  5. [5]

    Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification,

    A. Zhou, K. Wang, Z. Lu, W. Shi, S. Luo, Z. Qin, S. Lu, A. Jia, L. Song, M. Zhan, and H. Li, “Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification,” in International Conference on Representation Learning , vol. 2024, 2024, pp. 4468– 4494

    work page 2024

  6. [6]

    Cibench: Evaluating your llms with a code interpreter plugin,

    C. Zhang, S. Zhang, Y . Hu, H. Shen, K. Liu, Z. Ma, F. Zhou, W. Zhang, X. He, D. Lin et al. , “Cibench: Evaluating your llms with a code interpreter plugin,” arXiv preprint arXiv:2407.10499 , 2024

    arXiv 2024

  7. [7]

    Data dialogue with chatgpt: Using code interpreter to simulate and analyse experimental data,

    A. Low and Z. Y . Kalender, “Data dialogue with chatgpt: Using code interpreter to simulate and analyse experimental data,” arXiv preprint arXiv:2311.12415, 2023

    arXiv 2023

  8. [8]

    Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration,

    Y . Zhang, Z. Ma, Y . Ma, Z. Han, Y . Wu, and V . Tresp, “Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 22, 2025, pp. 23 378–23 386

    work page 2025

Show all 33 references
  1. [9]

    Redcode: Risky code execution and generation benchmark for code agents,

    C. Guo, X. Liu, C. Xie, A. Zhou, Y . Zeng, Z. Lin, D. Song, and B. Li, “Redcode: Risky code execution and generation benchmark for code agents,” Advances in Neural Information Processing Systems , vol. 37, pp. 106 190–106 236, 2024

    2024

  2. [10]

    Large language model-based agents for software engineering: A sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A sur- vey,” arXiv preprint arXiv:2409.02977 , 2024

    2024 arXiv

  3. [11]

    Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,” ACM Transactions on Software Engineering and Methodology , vol. 34, no. 5, pp. 1–30, 2025

    2025

  4. [12]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,” Proc. ACM Softw. Eng. , vol. 2, no. FSE, Jun. 2025

    2025

  5. [13]

    Self-collaboration code generation via chatgpt,

    Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt,” ACM Transactions on Software Engineering and Method- ology, vol. 33, no. 7, pp. 1–38, 2024

    2024

  6. [14]

    Mare: Multi-agents col- laboration framework for requirements engineering,

    D. Jin, Z. Jin, X. Chen, and C. Wang, “Mare: Multi-agents col- laboration framework for requirements engineering,” arXiv preprint arXiv:2405.03256, 2024

    2024 arXiv

  7. [15]

    Requirements are all you need: From requirements to code with llms,

    B. Wei, “Requirements are all you need: From requirements to code with llms,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 416–422

    2024

  8. [16]

    Marscode agent: Ai-native automated bug fixing,

    Y . Liu, P. Gao, X. Wang, J. Liu, Y . Shi, Z. Zhang, and C. Peng, “Marscode agent: Ai-native automated bug fixing,” arXiv preprint arXiv:2409.00899, 2024

    2024 arXiv

  9. [17]

    Cycle: Learning to self- refine the code generation,

    Y . Ding, M. J. Min, G. Kaiser, and B. Ray, “Cycle: Learning to self- refine the code generation,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA1, pp. 392–418, 2024

    2024

  10. [18]

    Self-planning code generation with large language models,

    X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 7, pp. 1–30, 2024

    2024

  11. [19]

    Swt-bench: Testing and validating real-world bug-fixes with code agents,

    N. M ¨undler, M. M ¨uller, J. He, and M. Vechev, “Swt-bench: Testing and validating real-world bug-fixes with code agents,” Advances in Neural Information Processing Systems , vol. 37, pp. 81 857–81 887, 2024

    2024

  12. [20]

    Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ...

    2024

  13. [21]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    2024

  14. [22]

    Infiagent-dabench: evaluating agents on data analysis tasks,

    X. Hu, Z. Zhao, S. Wei, Z. Chai, Q. Ma, G. Wang, X. Wang, J. Su, J. Xu, M. Zhu, Y . Cheng, J. Yuan, J. Li, K. Kuang, Y . Yang, H. Yang, and F. Wu, “Infiagent-dabench: evaluating agents on data analysis tasks,” in Proceedings of the 41st International Conference on Machine Lear...

    2024

  15. [23]

    Super: Evaluating agents on setting up and executing tasks from research repositories,

    B. Bogin, K. Yang, S. Gupta, K. Richardson, E. Bransom, P. Clark, A. Sabharwal, and T. Khot, “Super: Evaluating agents on setting up and executing tasks from research repositories,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 202...

    2024

  16. [24]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  17. [25]

    MetaGPT: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “MetaGPT: Meta programming for a multi-agent collaborative framework,” in The Twelfth International Conference on Learning Re...

    2024

  18. [26]

    OpenAI, “Gpt-4o,” 2024, https://openai.com/index/hello-gpt-4o/

    2024

  19. [27]

    Gpt-4o mini,

    ——, “Gpt-4o mini,” 2024, https://openai.com/index/gpt-4o-mini- advancing-cost-efficient-intelligence/

    2024

  20. [28]

    Multi- stage large language model pipelines can outperform gpt-4o in relevance assessment,

    J. A. Schnabel, J. R. Trippas, F. Scholer, and D. Hettiachchi, “Multi- stage large language model pipelines can outperform gpt-4o in relevance assessment,” in Companion Proceedings of the ACM on Web Conference 2025, 2025, pp. 1288–1292

    2025

  21. [29]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR) , 2023

    2023

  22. [30]

    Reasoning with language model is planning with world model,

    S. Hao, Y . Gu, H. Ma, J. Hong, Z. Wang, D. Wang, and Z. Hu, “Reasoning with language model is planning with world model,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing . Singapore: Association for Computational Linguistics, Dec. 202...

    2023

  23. [31]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 819–831

    2024

  24. [32]

    Perfcodegen: Improving performance of llm generated code with execution feedback,

    Y . Peng, A. D. Gotmare, M. R. Lyu, C. Xiong, S. Savarese, and D. Sahoo, “Perfcodegen: Improving performance of llm generated code with execution feedback,” in 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 1–13

    2025

  25. [33]

    Self-edit: Fault-aware code editor for code generation,

    K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin, “Self-edit: Fault-aware code editor for code generation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 769–787

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.