arxiv: 2604.05278 · v1 · submitted 2026-04-07 · 💻 cs.SE · cs.AI· cs.MA

Recognition: 2 theorem links

· Lean Theorem

Spec Kit Agents: Context-Grounded Agentic Workflows

Pardis Taghavi , Santosh Bhavani

Authors on Pith no claims yet

Pith reviewed 2026-05-10 20:04 UTC · model grok-4.3

classification 💻 cs.SE cs.AIcs.MA

keywords spec-driven developmentAI coding agentscontext groundingmulti-agent workflowssoftware engineeringSWE-benchhallucination mitigationcode quality evaluation

0 comments

The pith

Context-grounding hooks in a multi-agent spec-driven pipeline improve LLM-judged code quality by 0.15 points while preserving 99.7-100 percent test compatibility.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Spec Kit Agents is a multi-agent system for spec-driven development that inserts phase-level context-grounding hooks to counter agents becoming context blind in large, evolving repositories. The hooks consist of read-only probing to tie each stage to actual repository evidence and validation steps to check intermediate outputs against the live environment. Evaluations on 128 runs covering 32 features in five repositories show these additions produce a statistically significant quality lift on a composite LLM-as-judge score while leaving repository-level test compatibility essentially unchanged. The same framework also raises Pass@1 on SWE-bench Lite from baseline to 58.2 percent. A reader would care because the work supplies a concrete, integrable mechanism for reducing API hallucinations and architectural violations without requiring full repository rewrites or loss of existing tests.

Core claim

The paper claims that inserting read-only probing hooks and validation hooks at each phase of a PM-developer multi-agent SDD pipeline grounds agent decisions in repository evidence, thereby reducing hallucinations and violations. This produces a +0.15 gain on the 1-5 composite LLM-as-judge score with p < 0.05 under Wilcoxon testing, maintains 99.7-100 percent test compatibility across 128 runs, and yields a 1.7 percent absolute improvement to 58.2 percent Pass@1 on SWE-bench Lite.

What carries the argument

Phase-level context-grounding hooks that combine read-only repository probing with environment validation at the Specify, Plan, Tasks, and Implement stages of the multi-agent workflow.

If this is right

The pipeline supports 32 distinct features across five separate repositories without dropping below 99.7 percent test compatibility.
Quality gains reach statistical significance under a non-parametric test while benchmark performance improves by 1.7 percent absolute.
Role separation between project-management and developer agents remains compatible with the added grounding steps.
The same hooks can be applied to existing agent baselines on SWE-bench Lite without architectural overhaul.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same style of read-only probing and validation could be inserted into non-coding multi-agent systems that operate over large, changing knowledge bases.
Because the evaluation relies on an LLM judge, follow-up work with blinded human raters would be required to confirm the quality lift holds outside automated scoring.
The approach points toward a general pattern for making agents less context-blind in any domain where partial observability of a large state space produces hallucinations.

Load-bearing premise

The composite LLM-as-judge score accurately measures real code quality and that measured gains are caused by the context-grounding hooks rather than other pipeline or prompt differences.

What would settle it

A side-by-side human expert rating study on the same set of tasks that finds no quality difference between outputs generated with and without the context-grounding hooks.

Figures

Figures reproduced from arXiv: 2604.05278 by Pardis Taghavi, Santosh Bhavani.

read the original abstract

Spec-driven development (SDD) with AI coding agents provides a structured workflow, but agents often remain "context blind" in large, evolving repositories, leading to hallucinated APIs and architectural violations. We present Spec Kit Agents, a multi-agent SDD pipeline (with PM and developer roles) that adds phase-level, context-grounding hooks. Read-only probing hooks ground each stage (Specify, Plan, Tasks, Implement) in repository evidence, while validation hooks check intermediate artifacts against the environment. We evaluate 128 runs covering 32 features across five repositories. Context-grounding hooks improve judged quality by +0.15 on a 1-5 composite LLM-as-judge score (+3.0 percent of the full score; Wilcoxon signed-rank, p < 0.05) while maintaining 99.7-100 percent repository-level test compatibility. We further evaluate the framework on SWE-bench Lite, where augmentation hooks improve baseline by 1.7 percent, achieving 58.2 percent Pass@1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Spec Kit Agents adds phase-level read-only probing and validation hooks to a multi-agent SDD pipeline and reports small gains on an LLM judge plus SWE-bench Lite.

read the letter

Spec Kit Agents adds phase-level read-only probing and validation hooks to a multi-agent SDD pipeline and reports small gains on an LLM judge plus SWE-bench Lite. The hooks sit inside a PM-developer role structure and are meant to stop agents from hallucinating APIs or breaking architecture in real repositories. They run at Specify, Plan, Tasks, and Implement stages, then check the outputs against the actual environment. That combination is the concrete addition over plain agentic workflows. The evaluation uses 128 runs on 32 features across five repositories and keeps repository-level test compatibility at 99.7-100 percent. On SWE-bench Lite the hooks lift the baseline by 1.7 points to 58.2 percent Pass@1. The paper also shows a +0.15 shift on a 1-5 composite LLM-as-judge score with a Wilcoxon test at p<0.05. These numbers give the work some external grounding instead of pure self-report. The approach is straightforward to understand and the hooks look like a practical engineering move that others could copy. The main soft spot is the judge metric itself. The abstract gives no calibration against human ratings, no inter-rater numbers, and no ablation that holds every other prompt and pipeline element fixed while toggling only the hooks. The effect size is modest and the 128 runs are grouped across 32 features, so judge bias or uncontrolled differences remain possible explanations. The paper would benefit from more detail on exactly how the baselines were constructed and whether the judge score tracks real code quality. This is aimed at people building or studying AI coding agents for software engineering. Anyone already working with multi-agent setups or spec-driven workflows will see the hook pattern as something they can test quickly. The work has enough concrete description and benchmark results to go to peer review, though the evaluation section will need close attention on the metric and controls.

Referee Report

4 major / 2 minor

Summary. The manuscript introduces Spec Kit Agents, a multi-agent spec-driven development (SDD) pipeline with PM and developer roles that augments each workflow phase (Specify, Plan, Tasks, Implement) with context-grounding hooks: read-only repository probing to ground decisions in evidence and validation hooks to check artifacts against the environment. It reports results from 128 runs across 32 features in five repositories, claiming that the hooks yield a +0.15 improvement on a 1-5 composite LLM-as-judge quality score (Wilcoxon signed-rank, p<0.05; +3% of scale) while preserving 99.7-100% repository-level test compatibility, plus a 1.7% gain to 58.2% Pass@1 on SWE-bench Lite.

Significance. If the reported gains prove robust and causally attributable to the hooks, the work offers a practical, phase-structured approach to reducing context blindness and hallucinations in repository-scale agentic coding, which could meaningfully improve reliability in AI-assisted software engineering workflows. The dual use of custom feature tasks and the external SWE-bench Lite benchmark supplies independent grounding that strengthens the evaluation relative to purely self-referential tests.

major comments (4)

[Evaluation] Evaluation (128-run custom benchmark): The central +0.15 quality gain is attributed to context-grounding hooks, yet the manuscript provides no human calibration, inter-rater reliability, or correlation analysis showing that the composite LLM-as-judge score tracks objective code quality (e.g., defect density, architectural compliance, or maintainability). Without this, the small effect size (+3% of scale) cannot be confidently linked to the claimed mechanism rather than judge bias or uncontrolled prompt variation.
[Method] Method and Evaluation: No ablation is described that holds all other pipeline elements (prompt templates, agent roles, task decomposition) fixed while toggling only the read-only probing and validation hooks. The reported improvement therefore cannot be isolated from other systematic differences between conditions.
[Results] Results (128 runs across 32 features): The statistical test aggregates 128 runs grouped over only 32 features; per-feature breakdowns, variance estimates, or power analysis are not reported. Combined with the modest effect size, this raises the possibility that a few outlier features or judge inconsistencies drive the Wilcoxon result.
[SWE-bench evaluation] SWE-bench Lite evaluation: The 1.7% Pass@1 improvement to 58.2% is presented without an explicit definition of the baseline agent configuration or confirmation that the augmentation hooks constitute the sole controlled difference; exact baseline numbers, prompt variants, and statistical significance for this increment are also omitted.

minor comments (2)

[Evaluation] The five repositories used for the 128 runs are not named, hindering reproducibility and external assessment of generalizability.
[Evaluation] The composite LLM-as-judge scoring rubric (dimensions and weighting) is not fully specified, making it difficult to interpret or replicate the +0.15 delta.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the constructive and detailed review. We address each major comment below with clarifications from the manuscript and commit to revisions that improve transparency and robustness where the current version is insufficient.

read point-by-point responses

Referee: [Evaluation] Evaluation (128-run custom benchmark): The central +0.15 quality gain is attributed to context-grounding hooks, yet the manuscript provides no human calibration, inter-rater reliability, or correlation analysis showing that the composite LLM-as-judge score tracks objective code quality (e.g., defect density, architectural compliance, or maintainability). Without this, the small effect size (+3% of scale) cannot be confidently linked to the claimed mechanism rather than judge bias or uncontrolled prompt variation.

Authors: We acknowledge that the manuscript does not include human calibration, inter-rater reliability statistics, or explicit correlation analysis linking the composite LLM-as-judge score to objective measures such as defect density. The 1-5 composite is formed from multiple dimension-specific LLM prompts, consistent with evaluation practices in recent agentic coding papers, and the reported Wilcoxon result is supported by consistent test compatibility across repositories. However, we agree this constitutes a limitation for a modest effect size. In revision we will expand the Evaluation section to discuss potential judge bias, report run-to-run consistency metrics already available from the 128 runs, and explicitly list the absence of human validation as a limitation with directions for future work. revision: yes
Referee: [Method] Method and Evaluation: No ablation is described that holds all other pipeline elements (prompt templates, agent roles, task decomposition) fixed while toggling only the read-only probing and validation hooks. The reported improvement therefore cannot be isolated from other systematic differences between conditions.

Authors: The evaluation compares the identical multi-agent SDD pipeline (PM and developer roles, four phases, task decomposition structure, and base prompt templates) with and without the context-grounding hooks; the hooks are the sole controlled difference. We will revise the Method section to state this controlled comparison explicitly, list any hook-specific prompt adaptations, and add a dedicated ablation paragraph that isolates the read-only probing and validation components. revision: yes
Referee: [Results] Results (128 runs across 32 features): The statistical test aggregates 128 runs grouped over only 32 features; per-feature breakdowns, variance estimates, or power analysis are not reported. Combined with the modest effect size, this raises the possibility that a few outlier features or judge inconsistencies drive the Wilcoxon result.

Authors: We will add per-feature quality-score tables, run-level variance estimates, and a post-hoc power analysis to the Results section. These additions will allow direct inspection of effect consistency across the 32 features and will be computed from the existing 128-run data. revision: yes
Referee: [SWE-bench evaluation] SWE-bench Lite evaluation: The 1.7% Pass@1 improvement to 58.2% is presented without an explicit definition of the baseline agent configuration or confirmation that the augmentation hooks constitute the sole controlled difference; exact baseline numbers, prompt variants, and statistical significance for this increment are also omitted.

Authors: We will revise the SWE-bench Lite subsection to define the baseline as the multi-agent SDD pipeline without context-grounding hooks, confirm that the hooks are the only difference, report the precise baseline Pass@1 value, document any prompt variants, and include a statistical significance test for the 1.7 percentage-point gain. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical claims grounded in external benchmarks

full rationale

The paper describes an engineering framework (Spec Kit Agents) and reports empirical results from 128 runs on five repositories plus SWE-bench Lite. The central claims rest on measured differences in LLM-as-judge scores and Pass@1 rates against external test suites and baselines. No equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The evaluation chain is self-contained against independent benchmarks rather than reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on abstract only; the claim rests on the assumption that LLM judge scores track real quality and that hook attribution is clean.

axioms (1)

domain assumption LLM-as-judge composite score is a reliable proxy for code quality
Central to the +0.15 quality improvement claim.

pith-pipeline@v0.9.0 · 5476 in / 1209 out tokens · 88846 ms · 2026-05-10T20:04:18.785347+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Spec Kit Agents augments the Spec Kit stages with a context-grounding layer: (i) discovery hooks that perform read-only probing before each stage... and (ii) validation hooks that check intermediate artifacts
IndisputableMonolith/Cost/FunctionalEquation washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Context-grounding hooks improve judged quality by +0.15 on a 1-5 composite LLM-as-judge score

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

41 extracted references · 15 canonical work pages · 12 internal anchors

[1]

Vaibhav Aggarwal, Ojasv Kamal, Abhinav Japesh, Zhijing Jin, and Bernhard Schölkopf. 2025. DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal. InProceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers)

2025
[2]

Aider. 2024. How aider scored SOTA 26.3% on SWE Bench Lite. https://aider. chat/2024/05/22/swe-bench-lite.html

2024
[3]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations

2023
[4]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073(2022)

work page internal anchor Pith review Pith/arXiv arXiv 2022
[5]

Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information pro- cessing systems37 (2024), 74325–74362

2024
[6]

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. 2023. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. InThe Twelfth International Conference on Learning Representations

2023
[7]

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128(2023)

work page internal anchor Pith review arXiv 2023
[8]

Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al. 2025. Multi-agent collaboration via evolving orchestration.arXiv preprint arXiv:2505.19591(2025)

work page arXiv 2025
[9]

GitHub. 2026. Spec-Driven Development with Spec Kit. https://github.com/ github/spec-kit/blob/main/spec-driven.md Accessed March 12, 2026

2026
[10]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing.arXiv preprint arXiv:2305.11738(2023)

work page internal anchor Pith review arXiv 2023
[11]

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680 (2024)

work page internal anchor Pith review arXiv 2024
[12]

Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al
[13]

InThe twelfth international conference on learning representations

MetaGPT: Meta programming for a multi-agent collaborative framework. InThe twelfth international conference on learning representations
[14]

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

work page internal anchor Pith review Pith/arXiv arXiv 2023
[15]

Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. 2025. ReVeal: Self-Evolving Code Agents via Reliable Self-Verification. arXiv preprint arXiv:2506.11442(2025)

work page arXiv 2025
[16]

Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al. 2022. MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.arXiv preprint arXiv:2205.00445(2022)

work page internal anchor Pith review arXiv 2022
[17]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems33 (2020), 9459–9474

2020
[18]

Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society.Advances in neural information processing systems36 (2023), 51991–52008

2023
[19]

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. InProceedings of the 2023 conference on empirical methods in natural language processing. 3102–3116

2023
[20]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688(2023)

work page internal anchor Pith review arXiv 2023
[21]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al
[22]

Self-refine: Iterative refinement with self-feedback.Advances in neural information processing systems36 (2023), 46534–46594

2023
[23]

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al
[24]

Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332(2021)

work page internal anchor Pith review Pith/arXiv arXiv 2021
[25]

Albert Örwall. 2024. Moatless Tools. https://github.com/aorwall/moatless-tools

2024
[26]

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Go- rilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems37 (2024), 126544–126565

2024
[27]

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023. 5687–5711

2023
[28]

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789(2023)

work page internal anchor Pith review Pith/arXiv arXiv 2023
[29]

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems36 (2023), 68539–68551

2023
[30]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems36 (2023), 8634–8652

2023
[31]

Anubhav Shrimal, Stanley Kanagaraj, Kriti Biswas, Swarnalatha Raghuraman, Anish Nediyanchath, Yi Zhang, and Promod Yenigalla. 2024. MARCO: Multi-agent real-time chat orchestration. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 1381–1392

2024
[32]

Haoyu Wang, Christopher M Poskitt, and Jun Sun. 2025. Agentspec: Cus- tomizable runtime enforcement for safe and reliable llm agents.arXiv preprint arXiv:2503.18666(2025)

work page internal anchor Pith review arXiv 2025
[33]

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A Survey on Large Language Model Based Autonomous Agents.Frontiers of Computer Science18, 6 (2024), 186345

2024
[34]

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741(2024)

work page internal anchor Pith review arXiv 2024
[35]

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. InFirst conference on language modeling

2024
[36]

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489(2024)

work page internal anchor Pith review arXiv 2024
[37]

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

2024
[38]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems36 (2023), 11809–11822

2023
[39]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations

2022
[40]

Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2025. AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent (TEA) Protocol.arXiv preprint arXiv:2506.12508(2025)

work page arXiv 2025
[41]

Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604. A Representative Task Set Table 6 provides a representative subset of the task set used in the custom repository evaluation. These ...

2024