A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

Chun Yong Chong; David Lo; Guowen Yuan; Jeongjin Ju; Jounghoon Kim; Kisub Kim; Rui Shu; Seongmin Ju; Taein Yoon; Xin Zhou

REVIEW 3 major objections 4 minor 53 references

A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces SWE-RPG, a repository-level benchmark that scores coding agents on requirement clarification, planning, and code generation, and finds the main failure stage is recovery of implicit requirements, with average…

desk verdict A genuinely useful benchmark for coding agents, but the headline bottleneck claim rests on an LLM judge whose validation is too thin to carry it. read the letter →

arxiv 2608.09072 v1 pith:I3FTJMHQ submitted 2026-08-10 cs.SE cs.AI

Xin Zhou , Chun Yong Chong , Kisub Kim , Yun Peng , Rui Shu , Zihan Wu , Xu Han , Guowen Yuan

show 6 more authors

Zeyang Zhuang Jounghoon Kim Jeongjin Ju Seongmin Ju Taein Yoon David Lo

This is my paper · ORCID

classification cs.SEcs.AI

keywords codingagentsrepository-levelbenchmarkrequirementclarificationimplementationplanningcodegenerationimplicitrequirementsstage-awareevaluationSWE-RPG

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SWE-RPG, a repository-level benchmark that evaluates coding agents not only on whether their final patch passes tests, but also on the two reasoning stages before code: recovering the request's implicit requirements and forming an implementation plan. The central claim is that this design makes it possible to attribute an unsuccessful run to a specific stage, and the paper's measurements on 163 real issue tasks across 31 Python and Java repositories find that agents resolve only 31.5% of tasks on average, with implicit-requirement recovery the dominant failure stage in 24.5% to 46.0% of runs. The reason this would matter is that current benchmarks typically report only pass or fail, which cannot show whether an agent misunderstood the request, planned poorly, or wrote faulty code; a benchmark with validated intermediate references can point developers toward the capability that actually needs work.

What carries the argument

The machinery that carries the argument is the stage-attributed diagnosis protocol built on two kinds of validated ground-truth references. Clarification GTs state, in question-answer form, the implementation-critical requirements left implicit in the original request, organized into six categories: functional intent, business semantics, technical context, interface and protocol specifications, code structure and conventions, and data-structure semantics. Plan GTs give ordered, actionable steps whose reproducibility is verified by generating a patch from each step and checking semantic equivalence with the corresponding gold subpatch. An LLM judge then compares each agent trajectory with these references to measure pre-code coverage and to attribute unresolved runs to requirement, planning, code-generation, or other stages, while executable fail-to-pass and pass-to-pass tests from the merged pull request serve as the final correctness oracle.

What would settle it

A second annotation team, independent of the paper's authors, could reconstruct Clarification and Plan ground truths and stage labels for a stratified sample of the 163 tasks and rerun the judge with only those references; if the dominant failure stage shifts from requirement to planning or code generation, the bottleneck conclusion is a measurement artifact. A lighter check is to compare the reported 92% judge-human agreement on stage labels with an inter-annotator agreement for the reference construction itself, which the paper does not report.

Watch

Extended reading notes

Core claim

SWE-RPG is presented as the first repository-level benchmark to jointly evaluate requirement clarification, implementation planning, and code generation. Each of its 163 tasks supplies an executable patch oracle plus two validated intermediate references: Clarification ground truths, which spell out implementation-critical implicit requirements in six practitioner-derived categories, and Plan ground truths, which provide step-level implementation guidance checked for plan-to-code reproducibility. The evaluation protocol aligns each agent trajectory with these references and assigns every unresolved run to the earliest deviating stage using an LLM judge. On this benchmark, three agent frameworks paired with six language models resolve 31.5% of tasks on average; stage attribution identifies implicit-requirement recovery as the main bottleneck (24.5% to 46.0% of runs), ahead of code generation (7.4% to 37.4%) and planning (5.5% to 17.8%).

Load-bearing premise

The conclusion that implicit requirements are the main bottleneck depends on the accuracy and neutrality of the LLM-generated Clarification and Plan ground truths and of the LLM judge's stage attributions; if those references misstate what a task truly requires or the judge systematically favors one failure stage, the bottleneck ranking is an artifact of measurement rather than a property of the agents.

Editorial extensions

If this is right

If the benchmark is right, an end-to-end resolve rate alone is a misleading summary: similar resolve rates can accompany very different stage-level failure profiles, so intermediate references are needed to know what to improve.
The dominant failure stage, missed implicit requirements, points to a concrete target: agents should be evaluated and improved on recovering unstated constraints before they plan or edit code.
Planning coverage falls off from target location to implementation approach to constraints and validation, so plan quality should be measured on whether constraints and validation obligations are captured, not just where to edit.
The validated Clarification and Plan references can support more than scoring: they can serve as few-shot demonstrations or training data for the clarification and planning stages themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: a direct test of the bottleneck claim is to supply the Clarification GTs to the same agents at test time and measure the resolved rate; the paper does not run this condition, and if the rate does not rise, requirement recovery is necessary but not sufficient.
Editorial inference: if implicit-requirement recovery is the main bottleneck, trajectory-level signals should show it: agents that spend more effort on repository exploration or clarifying questions before editing should correlate with higher resolve rates on the released runs.
Editorial inference: the six-category clarification taxonomy could be reused as a standalone rubric for requirement-elicitation systems, turning clarification quality into a per-category score; that use is beyond the paper's stated scope.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

SWE-RPG is a real step forward for coding-agent evaluation. It delivers something prior benchmarks don't: validated intermediate ground truths for implicit requirement clarification and implementation planning, plus a test of whether each plan step can actually reproduce its gold subpatch. The construction pipeline is serious — 2,000+ candidates filtered down to 163 with reproducible Docker environments and stable tests, and every resulting GT was manually reviewed by two authors. Compared to RACE-bench and Dialogue SWE-Bench, the distinction is genuinely novel: RACE doesn't provide references for naturally implicit requirements, and Dialogue SWE-Bench creates synthetic omissions rather than mining real ones. The evaluation of 3 agents × 6 LLMs with cost/time data is also solid, and the differences in stage profiles between similarly-performing pairings (e.g., Codex–DSV4 vs. Claude–DSV4) show that outcome-only evaluation is missing signal.

The soft spot is exactly where the reader and the stress test put it. The central diagnostic finding — that implicit requirement recovery is the dominant bottleneck, at 24.5–46% of runs — rests on attribution labels from a single LLM judge (GPT-5.6-Sol). The only validation reported is 92% exact match on 50 unresolved runs. That's not enough. There's no per-stage confusion matrix, no inter-annotator agreement, and the 'earliest deviating stage' rule likely inflates requirement failures by counting any early omission, even one that never became consequential. The judge also evaluates against the same kind of LLM-synthesized GTs it's being validated with, so anchoring is a real possibility. The paper's own Limitations section concedes that the alignment labels depend on LLM-assisted judging and lists inter-annotator agreement as future work — which is honest, but means the headline claim is not yet fully supported.

None of this kills the benchmark itself. The resolved rates (31.5% average) stand on executable tests, not on the judge. The clarification and plan coverage scores are also interesting, though they too rely on the LLM judge. What needs fixing is the measurement validity section: publish judge prompts and trajectory pre-processing, release raw run data and per-stage confusion, add inter-annotator statistics, and ideally run an ablation with a different judge or no GTs. That's a revision, not a rewrite.

Bottom line: this deserves a serious referee and should go to review, but with a request for major revision on the stage-attribution validation. I'd bring it to a reading group and would probably cite it for the benchmark artifacts, with a caveat about the diagnostics.

Referee Report

3 major / 4 minor

Summary. The paper introduces SWE-RPG, a repository-level benchmark of 163 tasks from 31 Python and Java repositories that combines executable patch evaluation with ground-truth references for requirement clarification and implementation planning. The authors evaluate three coding-agent frameworks paired with six LLM backends, reporting an average resolved rate of 31.5%. Using a GPT-5.6-Sol judge to attribute unresolved runs to failure stages, they conclude that implicit requirement recovery is the main bottleneck, accounting for 24.5%–46.0% of agent runs. The benchmark also reports pre-code coverage rates of clarification categories and planning dimensions for the three agents.

Significance. If the measurements are valid, this is a valuable contribution: SWE-RPG is, to my knowledge, the first repository-level benchmark that jointly evaluates requirement clarification, planning, and code generation with validated intermediate references, and the stage-attributed failure analysis goes beyond the pass/fail verdicts of SWE-bench-style benchmarks. The authors provide a reproducible pipeline, executable environments, and public code and data, which are concrete strengths. The paper also offers actionable hypotheses, such as agents being weakest on interface, structure, and data-semantics requirements and on planning constraints/validation. However, the central diagnostic claims rest on LLM-generated ground truths and LLM-judged trajectory labels whose validity is only partially demonstrated; the reported validation samples are small and lack per-stage and per-category breakdowns. The resource itself is likely to be useful regardless, but the headline bottleneck conclusion needs stronger measurement evidence before it can be accepted.

major comments (3)

[Section 4, 'Failure Attribution to Stages'] The paper's central diagnostic claim—that implicit requirement recovery is the main bottleneck, accounting for 24.5%–46.0% of runs (Figure 6 and Section 5.2)—rests entirely on labels produced by the GPT-5.6-Sol judge, yet the only reported validation is exact-match agreement of 92% on a stratified sample of 50 unresolved runs. This is insufficient because exact-match accuracy on n=50 has a wide confidence interval, per-stage error rates are not reported, and a judge can be 92% accurate overall while systematically overcounting one stage if its errors concentrate there. In addition, the judge is given the same LLM-synthesized Clarification and Plan GTs (Section 3.2) that it evaluates against, with no blind condition or ablation to rule out anchoring toward 'requirement failure' when trajectory evidence is thin; the 'earliest deviating stage' protocol also counts any early omission as a requirement failure even if it never became consequential. The Limitations section itself concedes that 'retrospective alignment labels depend on LLM-assisted construction and judging' and lists inter-annotator agreement as future work. Please provide per-stage confusion matrices, precision/recall with confidence intervals, a second human annotator, and a sensitivity analysis that re-attributes failures under an alternative protocol such as 'most consequential deviation.'
[Section 3.2, 'Ground-Truth Construction for Req. Clarification' and 'Ground-Truth Construction for Planning'] The Clarification and Plan GTs are synthesized by GPT-5.4 and then 'two authors independently review every resulting reference for evidential support and cross-stage consistency,' but no quantitative inter-annotator agreement is reported, and the paper does not specify how 'evidential support' was operationalized or whether reviewers could detect a synthesized QA pair that is plausible but not actually necessary for the task. Because the diagnostic judge in Section 4 compares trajectories to these GTs, any invented or misweighted implicit requirement directly shifts the stage-attribution distribution. Please report annotator agreement on the GTs, provide evidence-provenance traces showing which repository artifact supports each QA pair and plan step, and, if feasible, run a sensitivity analysis that re-runs diagnosis with a subset of GTs removed or with an independently constructed GT set.
[Section 4, 'Pre-code Stage Evaluation'] The 96% agreement figure for the judge's binary coverage decisions is based on 'a manually annotated sample of 50 information-point assessments,' but no per-category breakdown is given. Figure 7's finding that agents are weakest on C4–C6 (interface, structure, data semantics) and on planning constraints/validation could be an artifact of judge bias on exactly those categories if the judge is more likely to mark an item uncovered for categories with less textual evidence in the trajectory. Please report per-category agreement, per-category error rates, and inter-annotator reliability for the coverage judgments; otherwise the capability-ranking claims in Section 5.3 are not sufficiently supported.

minor comments (4)

[Section 5.1, 'Effectiveness of Agents, LLMs, and Pairings'] The statement 'We run each agent-LLM configuration twice on every task and report run-level aggregates over the two runs' is ambiguous; please state explicitly whether the reported resolve rate is per run or per task (e.g., a task counts as resolved if either run succeeds), since all reported percentages depend on this choice.
[Table 3] The column header 'Plan GT Statistic' appears to label rows that are task-level statistics (Requirement Words, Codebase LOC), while the plan-specific rows follow below; consider separating task-level and GT-level statistics or adjusting the header to avoid confusion.
[Figure 6] The stacked bars contain many dense percentage labels that are hard to read; consider presenting the numerical values in a table as well, and define the 'Others' category in the caption (the legend already mentions timeouts and patch format, but the caption should be self-contained).
[Section 3.2, 'Synthesis and Validation'] The validation agent is said to check for 'absence of implementation detail leakage,' but the paper does not define what counts as leakage or how the check was enforced; please clarify this criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline numbers are empirical measurements grounded in executable tests and human-reviewed references; the LLM-judge dependence is a disclosed validity limitation, not a by-construction reduction.

full rationale

SWE-RPG's central claims are empirical benchmark measurements rather than derivations, and none reduces to its inputs by construction. The average resolved rate of 31.5% is computed from executable fail-to-pass and pass-to-pass tests following the SWE-bench protocol, an external oracle independent of the paper's ground-truth references. The bottleneck claim (implicit requirement recovery accounting for 24.5-46.0% of runs) is not logically entailed by the Clarification and Planning GTs: a trajectory could cover all GT information points and still fail at code generation, or omit a GT point and still produce a correct patch; the judge's earliest-deviating-stage protocol is an explicit operational rule, not a hidden tautology. The GTs are synthesized by GPT-5.4 but then evidence-checked against repository, PR, and tests, and every reference is independently reviewed by two authors; planning GTs are additionally validated through plan-to-code execution. The LLM judge (GPT-5.6-Sol) is checked against human consensus (92% exact match on 50 stratified unresolved runs; 96% on 50 coverage assessments), providing an external, if small, anchor. The only legitimate concern is measurement validity: the judge and GTs are LLM-assisted, the agreement sample is small, and per-stage confusion is unreported, so the bottleneck ranking could be biased. The paper itself flags this in Limitations: 'Its intermediate GTs and retrospective alignment labels depend on LLM-assisted construction and judging, although all GTs are manually validated. Future work should ... strengthen annotation audits, including inter-annotator agreement.' That is a disclosed limitation of the measuring instrument, not a by-construction equivalence, a fitted-input prediction, or a self-citation chain. No load-bearing self-citation, imported uniqueness theorem, or renamed-known-result pattern is present. Thus the paper is not circular; any weakness belongs to correctness/robustness risk.

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

The benchmark construction introduces two key invented artifacts: the clarification taxonomy and the plan GT. The central measurement depends on the assumption that LLM-generated references, after manual review, are correct and unbiased. There are no fitted numeric constants, but the construction choices (sample sizes, taxonomy) act as modeling parameters that shape all reported numbers. The paper has no formal proofs; its validation is empirical and LLM-assisted.

free parameters (2)

Taxonomy of six clarification categories
The taxonomy (C1-C6) is derived from interviews with 10 engineers and then used to structure all clarification GTs. It is a modeling choice that determines what counts as an implicit requirement, so it shapes the main bottleneck result, but it is not a fitted numeric parameter.
Agreement samples (n=50 for judge; n=50 for coverage) = 92% and 96%
The judge reliability is only established on small stratified samples. The samples are not free parameters in the derivation sense, but the reported agreement rates are used to support the validity of the main measurement. The size of these samples is a choice that limits the strength of the validation.

assumptions (3)

domain assumption Ground truths generated by GPT-5.4 and judged by GPT-5.6-Sol are sufficiently accurate for stage attribution after human review.
The entire diagnostic protocol in Sections 3.2 and 4 assumes LLM-generated references and LLM-judged attributions are reliable. The paper provides human review and small-sample agreement rates, but the core measurement rests on this assumption.
domain assumption The merged PR's developer patch is the correct gold patch for each task.
The code-generation GT is the merged PR itself (Section 3.2). This assumes that merged PRs contain correct, complete implementations of the stated issue, which is the standard SWE-bench assumption but nevertheless an assumption.
domain assumption Fail-to-pass and pass-to-pass test selection correctly identifies tests that directly exercise the target behavior.
Task filtering in Section 3.2 relies on manual verification that tests target the intended behavior. Missing or mislabeled tests would make the patch oracle inaccurate.

invented entities (2)

Implementation-sufficient plan ground truth (Plan GT)
purpose: Serves as a validated intermediate reference for evaluating agent planning and for diagnosing planning failures.
The plan GT is a new benchmark artifact constructed by LLM synthesis plus manual review. Its validity is only internal to this paper, not independently falsifiable outside it.
Six-dimensional clarification taxonomy (C1-C6)
purpose: Defines the categories of implicit requirements that agents are evaluated on.
The taxonomy is derived from interviews with engineers and is used throughout the benchmark construction. It is not independently validated, and the paper does not report inter-annotator agreement on its application.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents." pith.science (2026). https://pith.science/paper/I3FTJMHQ

@misc{pith2026260809072,
  author       = {Pith},
  title        = {Pith review of: A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3FTJMHQ}},
  note         = {Machine review of arXiv:2608.09072}
}

read the original abstract

Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.

Figures

Figures reproduced from arXiv: 2608.09072 by the authors.

**Figure 2.** Ground-truths Construction for Requirement Clarification and Planning. Using the original requirement, repository, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** An illustrative SWE-RPG instance. executable, reproducible patch evaluation. Together, these artifacts support stage-aware analysis and evaluation. 3.2 Benchmark Construction Repository and Task Selection. SWE-RPG collects merged issue–PR pairs from mature, actively maintained Python and Java repositories. Each pair must include reproducible tests, involve a recent and traceable change, and not overlap with existin… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Cost and execution time versus resolve rate. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Resolve rates across agent–LLM combinations. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Failure diagnoses across agent–LLM configurations. Green denotes resolved runs; the remaining segments show the [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Clarification-category and planning-dimension coverage by agent, averaged across six LLMs. Boxes mark the lowest [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 35 canonical work pages

[1]

arXiv preprint arXiv:2303.08774 , year =

Achiam, Josh and others , title =. arXiv preprint arXiv:2303.08774 , year =

arXiv
[2]

arXiv preprint arXiv:2107.03374 , year =

Chen, Mark and others , title =. arXiv preprint arXiv:2107.03374 , year =

arXiv
[3]

and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , title =. International Conference on Learning Representations , year =

work page
[4]

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

Li, Wei and Zhang, Xin and Guo, Zhongxin and Mao, Shaoguang and Luo, Wen and Peng, Guangyue and Huang, Yangyu and Wang, Houfeng and Li, Scarlett , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2025 , doi =

work page 2025
[5]

arXiv preprint arXiv:2108.07732 , year =

Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others , title =. arXiv preprint arXiv:2108.07732 , year =

arXiv
[6]

arXiv preprint arXiv:2105.09938 , year =

Hendrycks, Dan and Basart, Steven and Kadavath, Saurav and Mazeika, Mantas and Arora, Akul and Guo, Ethan and Burns, Collin and Puranik, Samir and He, Horace and Song, Dawn and others , title =. arXiv preprint arXiv:2105.09938 , year =

arXiv
[7]

arXiv preprint arXiv:2406.15877 , year =

Zhuo, Terry Yue and Vu, Minh Chien and Chim, Jenny and Hu, Han and Yu, Wenhao and Widyasari, Ratnadira and Yusuf, Imam Nur Bani and Zhan, Haolan and He, Junda and Paul, Indraneil and others , title =. arXiv preprint arXiv:2406.15877 , year =

arXiv
[8]

Advances in Neural Information Processing Systems , year =

Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , title =. Advances in Neural Information Processing Systems , year =

work page

Show all 53 references

[9]

and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =

Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

2024
[10]

Trustworthy Agents in Practice , year =
[11]

arXiv preprint arXiv:2605.27922 , year =

Yao, Yilun and Tan, Xinyu and Liu, Chao-Hsuan and Li, Yaoming and Wang, Zhengyang and Yu, Wenhan and Tan, Zhewen and Tian, Yuxuan and Zhao, Guangxiang and Sun, Lin and Zhang, Xiangzheng and Yang, Tong , title =. arXiv preprint arXiv:2605.27922 , year =

arXiv
[12]

Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , year =

Zhang, Yuntong and Ruan, Haifeng and Fan, Zhiyu and Roychoudhury, Abhik , title =. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , year =
[13]

and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and others , title =

Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and others , title =. International Conference on Learning Representations , year =
[14]

and Nguyen, Phong X

Phan, Huy Nhat and Nguyen, Tien N. and Nguyen, Phong X. and Bui, Nghi D. Q. , title =. arXiv preprint arXiv:2409.16299 , year =

arXiv
[15]

Proceedings of the ACM on Software Engineering , year =

Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming , title =. Proceedings of the ACM on Software Engineering , year =
[16]

2025 , eprint =

Antoniades, Antonis and. 2025 , eprint =

2025
[17]

2024 , eprint =

Pan, Jiayi and Wang, Xingyao and Neubig, Graham and Jaitly, Navdeep and Ji, Heng and Suhr, Alane and Zhang, Yizhe , title =. 2024 , eprint =

2024
[18]

arXiv preprint arXiv:2507.23370 , year =

arXiv
[19]

arXiv preprint arXiv:2511.13646 , year =

Xia, Chunqiu Steven and Wang, Zhe and Yang, Yan and Wei, Yuxiang and Zhang, Lingming , title =. arXiv preprint arXiv:2511.13646 , year =
[20]

Systems Engineering Research Center Conference , year =

Tan, Thomas and Boehm, Barry and Clark, Brad , title =. Systems Engineering Research Center Conference , year =
[21]

C., Vageesh and Iyer, Arun and Parthasarathy, Suresh and Rajamani, Sriram and Ashok, B

Bairi, Ramakrishna and Sonwane, Atharv and Kanade, Aditya and D. C., Vageesh and Iyer, Arun and Parthasarathy, Suresh and Rajamani, Sriram and Ashok, B. and Shet, Shashank , title =. Proceedings of the ACM on Software Engineering , volume =. 2024 , doi =

2024
[22]

20th IEEE International Requirements Engineering Conference , year =

Langenfeld, Vincent and Post, Amalinda and Podelski, Andreas , title =. 20th IEEE International Requirements Engineering Conference , year =
[23]

Choose a Permission Mode , year =
[24]

Best Practices for Codex , year =
[25]

2024 , howpublished =

Introducing. 2024 , howpublished =

2024
[26]

and Zhang, Alex L

Yang, John and Jimenez, Carlos E. and Zhang, Alex L. and Lieret, Kilian and Yang, Joyce and Wu, Xindi and Press, Ori and Muennighoff, Niklas and Synnaeve, Gabriel and Narasimhan, Karthik R. and Yang, Diyi and Wang, Sida I. and Press, Ofir , title =. The Thirteenth Internationa...
[27]

Advances in Neural Information Processing Systems , volume =

Zan, Daoguang and Huang, Zhirong and Liu, Wei and Chen, Hanwu and Zhang, Linhao and Xin, Shulin and Chen, Lu and Liu, Qi and Zhong, Xiaojian and Li, Aoyan and others , title =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

2025
[28]

2025 , eprint =

Rashid, Muhammad Shihab and Bock, Christian and Zhuang, Yuan and Buchholz, Alexander and Esler, Tim and Valentin, Simon and Franceschi, Luca and Wistuba, Martin and Sivaprasad, Prabhu Teja and Kim, Woo Jung and Deoras, Anoop and Zappella, Giovanni and Callot, Laurent , title =...

2025
[29]

and Wettig, Alexander and Khandpur, Kabir and Zhang, Yanzhe and Hui, Binyuan and Press, Ofir and Schmidt, Ludwig and Yang, Diyi , title =

Yang, John and Lieret, Kilian and Jimenez, Carlos E. and Wettig, Alexander and Khandpur, Kabir and Zhang, Yanzhe and Hui, Binyuan and Press, Ofir and Schmidt, Ludwig and Yang, Diyi , title =. 2025 , eprint =

2025
[30]

and Wang, Zifan and Zhang, Chen Bo Calvin and Jacobson, Noah and Liu, Bing and Kenstler, Brad , title =

Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Yiming and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Pasari, Nitin and Rane, Chetan and Sampath, Karmini and Krishnan, Maya and Kundurthy, Srivatsa and Hendryx, Sean M. and Wang, Zifan and Zhang...
[31]

Advances in Neural Information Processing Systems , volume =

Badertdinov, Ibragim and Golubev, Alexander and Nekrashevich, Maksim and Shevtsov, Anton and Karasik, Simon and Andriushchenko, Andrei and Trofimova, Maria and Litvintseva, Daria and Yangel, Boris , title =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

2025
[32]

Advances in Neural Information Processing Systems , volume =

Zhang, Linghao and He, Shilin and Zhang, Chaoyun and Kang, Yu and Li, Bowen and Xie, Chengxing and Wang, Junhao and Wang, Maoquan and Huang, Yufan and Fu, Shengyu and Nallipogu, Elsie and Lin, Qingwei and Dang, Yingnong and Rajmohan, Saravan and Zhang, Dongmei , title =. Advan...

2025
[33]

Proceedings of the 43rd International Conference on Machine Learning , year =

Xu, Jingxuan and Deng, Ken and Li, Weihao and Yu, Songwei and Tang, Huaixi and Huang, Haoyang and Lai, Zhiyi and Zhan, Zizheng and Wu, Yanan and Zhang, Chenchen and others , title =. Proceedings of the 43rd International Conference on Machine Learning , year =
[34]

and Tang, Xiangru and Wang, Xingyao , title =

Deng, Gangda and Chen, Zhaoling and Yu, Zhongming and Fan, Haoyang and Liu, Yuhong and Yang, Yuxin and Parikh, Dhruv and Kannan, Rajgopal and Cong, Le and Wang, Mengdi and Zhang, Qian and Prasanna, Viktor K. and Tang, Xiangru and Wang, Xingyao , title =. Proceedings of the 43r...
[35]

arXiv preprint arXiv:2606.07297 , year =

Zhang, Shaoqiu and Wang, Yuhang and Liang, Jialiang and Shi, Yuling and Zeng, Wenhao and Wang, Maoquan and He, Shilin and Xu, Ningyuan and Ye, Siyu and Cai, Kai and Gu, Xiaodong , title =. arXiv preprint arXiv:2606.07297 , year =. doi:10.48550/arXiv.2606.07297 , url =

doi arXiv
[36]

arXiv preprint arXiv:2606.13995 , year =

King, Brendan and Flanigan, Jeffrey , title =. arXiv preprint arXiv:2606.13995 , year =. doi:10.48550/arXiv.2606.13995 , url =

doi
[37]

arXiv preprint arXiv:2603.26337 , year =

Liu, Shuhan and Zhao, Zhiyi and Hu, Xing and Liu, Kui and Yang, Xiaohu and Xia, Xin , title =. arXiv preprint arXiv:2603.26337 , year =

arXiv
[38]

Proceedings of the 43rd International Conference on Machine Learning , year =

Badertdinov, Ibragim and Nekrashevich, Maksim and Shevtsov, Anton and Golubev, Alexander , title =. Proceedings of the 43rd International Conference on Machine Learning , year =
[39]

Thai, Minh V. T. and Le, Tue and Manh, Dung Nguyen and Nhat, Huy Phan and Bui, Nghi D. Q. , title =. arXiv preprint arXiv:2512.18470 , year =

arXiv
[40]

and Cardie, Claire and Gall

Zhao, Wenting and Jiang, Nan and Lee, Celine and Chiu, Justin T. and Cardie, Claire and Gall. Commit0: Library Generation from Scratch , booktitle =
[41]

arXiv preprint arXiv:2512.12730 , year =

Ding, Jingzhe and Long, Shengda and Pu, Changxin and Zhou, Huan and Gao, Hongwan and Gao, Xiang and He, Chao and Hou, Yue and Hu, Fei and Li, Zhaojian and others , title =. arXiv preprint arXiv:2512.12730 , year =
[42]

Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , pages =

Bo, Lili and Ji, Wangjie and Sun, Xiaobing and Zhang, Ting and Wu, Xiaoxue and Wei, Ying , title =. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , pages =. 2024 , publisher =

2024
[43]

Requirements Engineering , volume =

Knauss, Eric and Damian, Daniela and Cleland-Huang, Jane and Helms, Remko , title =. Requirements Engineering , volume =. 2015 , doi =

2015
[44]

Proceedings of the ACM on Software Engineering , volume =

Mu, Fangwen and Shi, Lin and Wang, Song and Yu, Zhuohao and Zhang, Binquan and Wang, Chenxue and Liu, Shichao and Wang, Qing , title =. Proceedings of the ACM on Software Engineering , volume =. 2024 , doi =

2024
[45]

Requirements Engineering , volume =

Franch, Xavier and Palomares, Cristina and Quer, Carme and Chatzipetrou, Panagiota and Gorschek, Tony , title =. Requirements Engineering , volume =. 2023 , doi =

2023
[46]

2026 , howpublished =

2026
[47]

2025 , howpublished =

2025
[48]

2026 , month = mar, howpublished =

Introducing. 2026 , month = mar, howpublished =

2026
[49]

2026 , month = jun, howpublished =

Introducing. 2026 , month = jun, howpublished =

2026
[50]

2026 , month = jun, howpublished =

2026
[51]

2026 , month = jul, howpublished =

2026
[52]

Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering , pages =

Zhou, Xin and Kim, Kisub and Zhang, Ting and Weyssow, Martin and Gomes, Lu. Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering , pages =
[53]

, title =

Jiang, Hongchao and Chen, Yiming and Cao, Yushi and Lee, Hung-yi and Tan, Robby T. , title =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics , pages =. 2026 , doi =

2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2303.08774 , year =

Achiam, Josh and others , title =. arXiv preprint arXiv:2303.08774 , year =

arXiv

[2] [2]

arXiv preprint arXiv:2107.03374 , year =

Chen, Mark and others , title =. arXiv preprint arXiv:2107.03374 , year =

arXiv

[3] [3]

and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , title =. International Conference on Learning Representations , year =

work page

[4] [4]

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

Li, Wei and Zhang, Xin and Guo, Zhongxin and Mao, Shaoguang and Luo, Wen and Peng, Guangyue and Huang, Yangyu and Wang, Houfeng and Li, Scarlett , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2025 , doi =

work page 2025

[5] [5]

arXiv preprint arXiv:2108.07732 , year =

Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others , title =. arXiv preprint arXiv:2108.07732 , year =

arXiv

[6] [6]

arXiv preprint arXiv:2105.09938 , year =

Hendrycks, Dan and Basart, Steven and Kadavath, Saurav and Mazeika, Mantas and Arora, Akul and Guo, Ethan and Burns, Collin and Puranik, Samir and He, Horace and Song, Dawn and others , title =. arXiv preprint arXiv:2105.09938 , year =

arXiv

[7] [7]

arXiv preprint arXiv:2406.15877 , year =

Zhuo, Terry Yue and Vu, Minh Chien and Chim, Jenny and Hu, Han and Yu, Wenhao and Widyasari, Ratnadira and Yusuf, Imam Nur Bani and Zhan, Haolan and He, Junda and Paul, Indraneil and others , title =. arXiv preprint arXiv:2406.15877 , year =

arXiv

[8] [8]

Advances in Neural Information Processing Systems , year =

Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , title =. Advances in Neural Information Processing Systems , year =

work page