Pith. sign in

REVIEW 4 major objections 5 minor 64 references

ICAE-Bench shows that even the strongest coding agents pass only 38.2% of interactive project-building tasks when starting from vague product requirements.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:06 UTC pith:BCT5MM4J

load-bearing objection A genuinely new interactive repository-building benchmark with a strong host-verified evaluation core, but the paper never audits whether the User Agent router actually surfaces every hidden constraint, so the headline pass rates could partly reflect routing misses; still deserves serious refereeing. the 4 major comments →

arxiv 2607.21217 v1 pith:BCT5MM4J submitted 2026-07-23 cs.AI

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Zhongyuan Peng , Dan Huang , Chuyu Zhang , Caijun Xu , Changyi Xiao , Shibo Hong , David Lo , Lin Qiu
show 3 more authors
Xuezhi Cao Jiyuan He Yixin Cao
This is my paper
classification cs.AI
keywords coding agentsbenchmark constructionvibe codingfuzzy requirementsinteractive clarificationrepository generationblack-box evaluationrequirement ambiguity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ICAE-Bench is a benchmark for the emerging "vibe coding" setting where an agent receives a fuzzy product request, asks a simulated user for clarification, and must build a runnable repository. The paper's central claim is that this interactive-project-building ability can be measured reliably: tasks are derived from real open-source repositories, hidden requirements are stored in grounded records rather than invented on the fly, and final repositories are scored with black-box tests. On the full set, the best model reaches a 38.2% overall pass rate, and agents generally reproduce visible behavior while missing hidden constraints and edge cases. The paper also establishes two empirical findings: giving agents the complete specification (GroundPRD) remains a strong upper bound, and recovering more constraints through dialogue does not automatically translate into a higher pass rate. The reason to care is that it isolates a distinct "information-to-execution" gap that existing benchmarks do not measure.

Core claim

The paper introduces ICAE-Bench, arguing that coding-agent evaluation should move from static, fully specified tasks to interactive project building. Each of 480 tasks starts from a fuzzy PRD, with selected constraints hidden and stored as User Agent Data; a grounded User Agent answers only from those records, and evaluation uses Public/Native/Enhanced black-box cases plus structural and interaction diagnostics. The paper claims that this design validly measures the whole pipeline, and that experiments show interaction recovers only part of the gap to GroundPRD, while higher constraint coverage does not predict correct repositories. In the authors' framing, the bottleneck is not asking quest

What carries the argument

The central mechanism is a three-part construction: (1) GroundPRD, a complete specification synthesized from a real repository's executable behavior; (2) fuzzification levels L1-L3 that hide selected constraints while storing them as User Agent Data, with trigger phrases, grounded responses, and fallback replies; and (3) an ultimate image—a provisioned runtime stripped of golden code and tests—where the agent builds, plus black-box test cases (Native and Enhanced, Public and Hidden) for host-verified scoring. The User Agent router is the load-bearing design: it restricts answers to benchmark-authored records so that clarification is reproducible and cannot leak implementation artifacts.

Load-bearing premise

The benchmark assumes that every hidden constraint that matters can be recovered through the user agent's keyword-triggered answers; if a real edge case cannot be elicited by any question an agent might reasonably ask, the measured pass rate will understate the agent's true ability.

What would settle it

Run an agent on a task with an unlimited query budget; if the pass rate stays near the 16-query result, the ceiling is not information access. Alternatively, replace the user-agent router with a dump of all hidden constraint records and measure pass rate; if removing retrieval does not change scores, then the benchmark is not actually testing interactive clarification.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, existing leaderboards that measure function-level or fully specified tasks systematically overstate how ready agents are for real product-building work.
  • Making hidden constraints executable—placing public test files in the workspace instead of requiring the agent to reconstruct them from dialogue—can raise pass rates sharply (from 37.4% to 61.8% in one controlled run), so evaluation should report verification scaffolding separately.
  • Expanding the interaction budget beyond a saturation point can lower correctness even while constraint coverage rises; the benchmark implies that question budgets should be treated as a real resource, not a free good.
  • Because rankings change when the agent scaffold changes, any claim about which model is best is only meaningful relative to a specific harness; benchmark results should be reported across frameworks.
  • Language-agnostic black-box tests make cross-language reimplementation feasible, so the benchmark can measure whether an agent understands task semantics independently of the target language.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I infer from the information-to-execution gap that the next performance lever is not better clarification but better working memory: agents that explicitly log clarified constraints and re-read them during implementation should outperform agents that only maintain a dialogue.
  • The benchmark's sensitivity to the user-simulator's routing suggests scores partly measure the router, not just the agent; a testable extension would run tasks with the full hidden records dumped verbatim to see how much of the gap remains.
  • The executable-scaffolding result hints that future coding agents should be rewarded for writing their own test harnesses early; a natural extension would add an intermediate metric counting whether agents create runnable self-checks before their final submission.
  • Since richer provisioned environments actually lowered pass rates in one comparison, a follow-up could investigate whether dependency availability creates a decision-paralysis effect under fixed reasoning budgets—testable by varying both environment richness and thinking budget in a grid.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ICAE-Bench, a benchmark for evaluating coding agents in an interactive, from-scratch repository-building setting. Each task starts from a fuzzy product requirement derived from a real open-source repository, allows the agent to clarify ambiguities through a grounded User Agent backed by benchmark-authored records, and evaluates the resulting repository with black-box functional tests plus agentic, structural, and interaction diagnostics. The benchmark contains 480 tasks in 12 languages and a 50-task Lite subset. The authors evaluate six coding models under two agent frameworks, reporting a best Overall pass rate of 38.2% on the full benchmark and showing that interaction and richer initial specifications recover only part of the gap to the fully specified GroundPRD. The paper also includes ablations of frameworks, User Agent backbones, interaction budgets, thinking configuration, execution environment, and cross-lingual reimplementation.

Significance. If the measurement validity concerns are resolved, ICAE-Bench is a timely and substantial contribution. Its core functional evaluation is host-verified through fresh-container replay against golden implementations, which avoids many of the reliability problems of LLM-judged benchmarks. The grounded User Agent design is a principled way to make interactive clarification reproducible while preventing hallucinated requirements. The three fuzzy levels, the multi-language coverage, and the controlled environment ablations provide a rich experimental resource. The paper is also honest in reporting the human-validation results for its subjective metrics and in acknowledging that ICAE-Bench-Lite is not sufficient for definitive model ranking. The strengths—fresh-container replay, grounded interaction data, and broad ablation coverage—make the benchmark potentially reusable by the community.

major comments (4)
  1. [§III-E, §III-C, Appendix H] The validity of the hidden-constraint design rests on every omitted constraint being recoverable through a well-formed clarification question, but the paper never measures this. The verification in §III-C checks semantic equivalence of the fuzzy PRD + User Agent Data against GroundPRD, not whether DeepSeek-V3.2’s router actually returns the grounded record when a precisely targeted question is asked. The constraint coverage and fallback rates in §IV are properties of each tested agent’s actual questions, not a conditional recall metric over well-formed questions. If a nontrivial fraction of hidden constraints is systematically unreachable, the RQ1/RQ2 conclusions—interaction recovers only part of the gap—would be confounded. Please add an audit in which an oracle questioner asks a maximally precise grounded question for each hidden constraint (e.g., using trigger-keyword paraphrases not
  2. [§III-C, Appendices D–E] The paper states that the fuzzy PRD and User Agent Data are validated by two complementary checks and that the prompts are provided in Appendices D and E. But Appendix D is the semantic-equivalence judge and Appendix E is the Fuzzy L3 construction prompt; there is no prompt or protocol for the black-box execution-equivalence check described in §III-C. This is a load-bearing reproducibility gap: the claim that the task artifacts preserve the executable contract cannot be independently verified without this protocol. Please provide the missing prompt/instructions or correct the appendix reference.
  3. [Table X vs. Tables VII and XIII] Table X (agent-framework comparison) reports GLM-5.1 under Claude Code at 37.4% Overall, which equals the think-8k configuration in Table XIII, whereas the main default adaptive configuration gives 40.0% on ICAE-Bench-Lite (Table VII). If the framework comparison used a non-default thinking setting for GLM-5.1, the framework effect for that model is confounded. Please state the configuration used in Table X and, if it is not the default, rerun or clearly relabel.
  4. [§V-B, Table XVI] The human validation of the subjective metrics shows low agreement: Pearson correlations of 0.372, 0.381, and 0.471 and ICC(C,1) values of 0.240, 0.320, and 0.367 for semantic, API, and design scores. The paper acknowledges this in §V-B, but the main-results section (§IV-B) still uses these scores to support claims such as “similar functional scores conceal substantial differences in artifact design” and to draw model-level distinctions. These diagnostical claims should be presented as exploratory, and the central conclusions should rest primarily on the host-verified functional and interaction metrics.
minor comments (5)
  1. [Abstract/Introduction] Typos: “an benchmark” and “starts from” should be “a benchmark” and “start from”.
  2. [Figure 7] Figure 7 is unclear: the token totals are labeled as “PRD” but appear to include User Agent Data; the caption says User Agent Data is accessible only through interaction, yet the bars seem to include it in the totals. Please clarify what the stacked bars represent.
  3. [Figure 6 / RQ2] The GroundPRD and RecoveredPRD results are shown only in a figure without numeric labels. Since RQ2’s central claim is that GroundPRD remains a strong upper bound and interaction recovers only part of the gap, please report the numeric Overall pass rates for all models and fuzzy-level conditions in a table.
  4. [References] References [10] and [11] duplicate [8] and [9]; references [2] and [37] are also duplicates. Please consolidate.
  5. [Appendix C vs. Table V] Appendix C specifies a 150–250 word target for the fuzzy PRD, while Table V reports token lengths. Clarify the relation between words and tokens, or report both consistently.

Circularity Check

0 steps flagged

No circularity: evaluation target is external, validity checks are runnable, and self-citations are related-work only.

full rationale

ICAE-Bench's claimed derivation chain is not circular. The evaluation target is external: each task is anchored to a real open-source repository whose original tests pass in Docker, and final scoring uses host-verified black-box cases (Native and Enhanced) that were required to pass against the golden implementation before acceptance (Sections III-B and III-C). The fuzzy PRD and User Agent Data are validated against GroundPRD through both an LLM semantic-equivalence judge and black-box execution-equivalence checks, so the validity claim rests on independently runnable artifacts rather than on the benchmark's own conclusions. The self-citations to PRDBench [5] and RepoGenesis [21] appear only in the related-work comparison and are not load-bearing for any of ICAE-Bench's premises, uniqueness claims, or predicted rankings. The only noteworthy concern — GPT-5.5 serving both as the semantic-equivalence judge and as a tested coding agent — is a potential evaluation-design conflict, but it is not a reduction of any measured quantity to its input by construction: pass rates are computed from independent black-box execution against golden-validated cases. Similarly, the unverified recall of the trigger-keyword router (Appendix H) is a validity or coverage limitation, not circularity, because the router is not fitted to the reported pass rates and no reported 'prediction' is defined in terms of the router's own matching behavior. No step satisfies the requirement of exhibiting a specific reduction (Eq. X = Eq. Y by construction, or a fitted parameter renamed as a prediction), so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted to data; the three listed configuration choices are hand-picked and affect the measured interaction-recovery claims. The User Agent and ultimate image are benchmark infrastructure, not newly postulated scientific entities. The benchmark's target behavior comes from external real repositories, so the circularity burden is low.

free parameters (3)
  • semantic equivalence threshold = 0.90
    Chosen in Appendix D to define task_semantic_equivalent; the claim that Fuzzy L1/L3 are equivalent to GroundPRD for all 50 Lite repos rests on this hand-picked threshold.
  • User Agent query budget = 16
    Default interaction budget (Section III-E); the conclusion that '16 queries provide sufficient nominal retrieval capacity' and the observed performance plateau depend on this choice; the ablation shows 24 queries lowers Overall to 34.4%.
  • matched-technical-points cap per reply = 3
    Each User Agent reply addresses at most three matched points (Section III-E); with about 20+ ambiguity points per task, this constrains how quickly constraints can be recovered and interacts with router quality.
axioms (5)
  • domain assumption Original tests that pass in Docker define the true behavioral target of each task
    Section III-B: a repository is retained only if all original tests pass; this assumes the test suite adequately captures the intended product behavior.
  • domain assumption Black-box refactoring preserves external behavior without leaking implementation details
    Appendix A: only externally observable behavior is kept, exceptions are normalized; assumes the dispatcher and test.sh faithfully reproduce golden behavior for all cases.
  • domain assumption The User Agent's grounded records and trigger matching can surface every hidden constraint that a precise question targets
    Section III-E: routing is restricted to benchmark-authored records; if a well-posed question fails to match a trigger keyword, the constraint is unrecoverable and failure could be misattributed to the coding agent.
  • domain assumption GPT-5.5 semantic-equivalence judgment at 0.90 threshold reliably reflects task-level equivalence
    Section III-C and Appendix D: the semantic-equivalence check is the primary validation that Fuzzy PRD + User Agent Data equals GroundPRD; the execution-equivalence check is described only briefly.
  • domain assumption Fresh-container evaluation replay accurately reproduces the agent's development environment
    Appendix F requires reproducible install steps in rcb_tests/test.sh; assumes no hidden state from development survives into evaluation.

pith-pipeline@v1.3.0-alltime-deepseek · 22925 in / 13810 out tokens · 144405 ms · 2026-08-01T08:06:41.828295+00:00 · methodology

0 comments
read the original abstract

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

Figures

Figures reproduced from arXiv: 2607.21217 by Caijun Xu, Changyi Xiao, Chuyu Zhang, Dan Huang, David Lo, Jiyuan He, Lin Qiu, Shibo Hong, Xuezhi Cao, Yixin Cao, Zhongyuan Peng.

Figure 1
Figure 1. Figure 1: Overview of the interactive requirement clarification setting of ICAE-Bench. Given an ambiguous requirement, vibe coding resolves missing design decisions through dialogue between a human and a coding agent. Our benchmark simulates this process with a user agent that answers clarification questions and guides the coding agent toward a complete, testable specification, which is then used to build a working … view at source ↗
Figure 2
Figure 2. Figure 2: ICAE-Bench framework. A task begins with a fuzzy PRD and an ultimate image rather than a complete implementation contract. The coding agent can develop, execute, debug, and ask the User Agent for clarifications. Final repositories are evaluated using source-based Native and Enhanced black-box cases; the Public subset is also reported as a visibility-based view. by U under a fixed budget, and produces a rep… view at source ↗
Figure 3
Figure 3. Figure 3: Initial-requirement hierarchy. Information available before interaction [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task category distribution in ICAE-Bench (total=480). [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task golden-code LOC distribution in ICAE-Bench (total=480). [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overall pass rate across PRD ambiguity settings on ICAE-Bench-Lite. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Requirement-token comparison on ICAE-Bench-Lite. User Agent Data [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evaluator-visible outcomes on full ICAE-Bench main runs. Failure [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of ambiguity categories across Fuzzy L1–L3 in ICAE [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Interaction-budget ablation on ICAE-Bench-Lite with GLM-5.1 [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Runtime and token usage for main Claude Code runs on ICAE [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 12 linked inside Pith

  1. [1]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  2. [3]

    Commit0: Library generation from scratch,

    W. Zhao, N. Jiang, C. Lee, J. Chiu, C. Cardie, M. Gall ´e, and A. Rush, “Commit0: Library generation from scratch,” inInternational Confer- ence on Learning Representations, vol. 2025, 2025, pp. 12 061–12 076

    2025

  3. [4]

    Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents,

    J. Ding, S. Long, C. Pu, H. Zhou, H. Gao, X. Gao, C. He, Y . Hou, F. Hu, Z. Liet al., “Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents,”arXiv preprint arXiv:2512.12730, 2025

    arXiv 2025

  4. [5]

    Automatically benchmarking llm code agents through agent-driven annotation and evaluation,

    L. Fu, B. Zhang, H. Guan, Y . Zhu, L. Qiu, W. Liu, X. Cao, X. Cai, W. Zhang, and Y . Yu, “Automatically benchmarking llm code agents through agent-driven annotation and evaluation,”arXiv preprint arXiv:2510.24358, 2025

    arXiv 2025

  5. [6]

    Programbench: Can language models rebuild programs from scratch?

    J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeveet al., “Programbench: Can language models rebuild programs from scratch?”arXiv preprint arXiv:2605.03546, 2026

    Pith/arXiv arXiv 2026

  6. [7]

    Realbench: A repo-level code generation benchmark aligned with real-world software development practices,

    J. Li, H. Deng, Y . Zhang, K. Zhang, T. Shao, T. Zhao, W. Wang, Z. Jin, G. Li, Y . Liuet al., “Realbench: A repo-level code generation benchmark aligned with real-world software development practices,”arXiv preprint arXiv:2604.22659, 2026

    Pith/arXiv arXiv 2026

  7. [10]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  8. [11]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  9. [12]

    Repobench: Benchmarking repository- level code auto-completion systems,

    T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repository- level code auto-completion systems,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 47 832–47 850

    2024

  10. [13]

    M2rc-eval: Massively multilingual repository-level code completion evaluation,

    J. Liu, K. Deng, C. Liu, J. Yang, S. Liu, H. Zhu, P. Zhao, L. Chai, Y . Wu, J. JinKeet al., “M2rc-eval: Massively multilingual repository-level code completion evaluation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 15 661–15 684

    2025

  11. [14]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    2024

  12. [15]

    Multi-swe-bench: A multilingual benchmark for issue resolving,

    D. Zan, Z. Huang, W. Liu, H. Chen, S. Xin, L. Zhang, Q. Liu, L. Aoyan, L. Chen, X. Zhonget al., “Multi-swe-bench: A multilingual benchmark for issue resolving,”Advances in Neural Information Processing Sys- tems, vol. 38, 2026

    2026

  13. [16]

    Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Raneet al., “Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?”arXiv preprint arXiv:2509.16941, 2025

    Pith/arXiv arXiv 2025

  14. [17]

    Swe-compass: Towards unified evaluation of agentic coding abilities for large language models,

    J. Xu, K. Deng, W. Li, S. Yu, H. Tang, H. Huang, Z. Lai, Z. Zhan, Y . Wu, C. Zhanget al., “Swe-compass: Towards unified evaluation of agentic coding abilities for large language models,”arXiv preprint arXiv:2511.05459, 2025

    arXiv 2025

  15. [18]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Transactions on Software Engineering and Methodology, vol. 35, no. 2, pp. 1–72, 2026

    2026

  16. [19]

    Large language model-based agents for software engineering: A sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A sur- vey,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  17. [20]

    A survey on code generation with llm-based agents,

    Y . Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, “A survey on code generation with llm-based agents,”arXiv preprint arXiv:2508.00083, 2025

    Pith/arXiv arXiv 2025

  18. [21]

    Repogenesis: Benchmarking end-to-end microservice generation from readme to repository,

    Z. Peng, X. Yin, P. Zhao, F. Yang, L. Wang, R. Jia, X. Chen, Q. Lin, S. Rajmohan, and D. Zhang, “Repogenesis: Benchmarking end-to-end microservice generation from readme to repository,”arXiv preprint arXiv:2601.13943, 2026

    Pith/arXiv arXiv 2026

  19. [22]

    Projecteval: A benchmark for programming agents automated evaluation on project- level code generation,

    K. Liu, Y . Pan, Y . Xiang, D. He, J. Li, Y . Du, and T. Gao, “Projecteval: A benchmark for programming agents automated evaluation on project- level code generation,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 20 205–20 221

    2025

  20. [23]

    Reqelicit- gym: An evaluation environment for interview competence in conversa- tional requirements elicitation,

    D. Jin, Z. Jin, Z. Fang, L. Li, X. Yang, Y . He, and X. Chen, “Reqelicit- gym: An evaluation environment for interview competence in conversa- tional requirements elicitation,”arXiv preprint arXiv:2602.18306, 2026

    arXiv 2026

  21. [24]

    Python code generation by asking clarification questions,

    H.-S. X. Li, M. Mesgar, A. F. Martins, and I. Gurevych, “Python code generation by asking clarification questions,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 14 287–14 306

    2023

  22. [25]

    Llm- based test-driven interactive code generation: User study and empirical evaluation,

    S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri, “Llm- based test-driven interactive code generation: User study and empirical evaluation,”IEEE Transactions on Software Engineering, vol. 50, no. 9, pp. 2254–2268, 2024

    2024

  23. [26]

    Humanevalcomm: Benchmarking the commu- nication competence of code generation for llms and llm agents,

    J. J. Wu and F. H. Fard, “Humanevalcomm: Benchmarking the commu- nication competence of code generation for llms and llm agents,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–42, 2025

    2025

  24. [27]

    Assessing the impact of requirement ambiguity on llm-based function-level code generation,

    D. Yang, X. Xie, X. Yang, M. Hu, Y . Huang, Y . Zhang, W. Miao, T. Su, C. Wan, and G. Pu, “Assessing the impact of requirement ambiguity on llm-based function-level code generation,”arXiv preprint arXiv:2604.21505, 2026

    Pith/arXiv arXiv 2026

  25. [28]

    When benchmarks talk: Re-evaluating code llms with interactive feedback,

    J. Pan, R. Shar, J. Pfau, A. Talwalkar, H. He, and V . Chen, “When benchmarks talk: Re-evaluating code llms with interactive feedback,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 24 672–24 700

    2025

  26. [29]

    Ask or assume? uncertainty- aware clarification-seeking in coding agents,

    N. Edwards and S. Schuster, “Ask or assume? uncertainty- aware clarification-seeking in coding agents,”arXiv preprint arXiv:2603.26233, 2026

    Pith/arXiv arXiv 2026

  27. [30]

    claude-agent-sdk-python,

    Anthropic, “claude-agent-sdk-python,” https://github.com/anthropics/ claude-agent-sdk-python, 2026

    2026

  28. [31]

    Openai gpt-5 system card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthramet al., “Openai gpt-5 system card,”arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  29. [32]

    Claude opus 4.8,

    Anthropic, “Claude opus 4.8,” https://www.anthropic.com/news/ claude-opus-4-8, 5 2026, accessed: 2026-06-29

    2026

  30. [33]

    Introducing Claude Sonnet 4.6,

    ——, “Introducing Claude Sonnet 4.6,” https://www.anthropic.com/ news/claude-sonnet-4-6, 2 2026, accessed: 2026-06-29

    2026

  31. [34]

    Glm-5: from vibe coding to agentic engineer- ing,

    A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xieet al., “Glm-5: from vibe coding to agentic engineer- ing,”arXiv preprint arXiv:2602.15763, 2026

    Pith/arXiv arXiv 2026

  32. [35]

    Gemini 3.1 pro - model card,

    Google DeepMind, “Gemini 3.1 pro - model card,” https://deepmind. google/models/model-cards/gemini-3-1-pro/, 2 2026, accessed: 2026- 06-29

    2026

  33. [36]

    The minimax-m2 series: Mini activations un- leashing max real-world intelligence,

    A. Chen, A. Li, B. Zhou, B. Gong, B. Jiang, B. Dan, C. Yu, C. Wang, C. Ma, C. Zhonget al., “The minimax-m2 series: Mini activations un- leashing max real-world intelligence,”arXiv preprint arXiv:2605.26494, 2026

    Pith/arXiv arXiv 2026

  34. [37]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 65 882–65 919

    2025

  35. [38]

    Openclaw: Personal ai assistant,

    P. Steinberger and contributors, “Openclaw: Personal ai assistant,” https: //github.com/openclaw/openclaw, 2026. APPENDIXA PROMPT FORTESTREFACTORING ANDGROUNDPRD CONSTRUCTION GroundPRD Construction & Test Refactoring ## Task Description You are given an original runnable Docker environment containing the repository, its runtime environment, and its origin...

    2026

  36. [39]

    Read the source code and the original repository tests

  37. [40]

    Keep those tests; discard white-box or implementation-specific assertions

    For each original test, decide whether it checks externally visible behavior that is independent of the implemen- tation choice. Keep those tests; discard white-box or implementation-specific assertions

  38. [41]

    The PRD must be self-contained: each leaf feature block should be understandable from its text and embedded example cases alone

    Write the PRD and generatetest_cases: each retained test is converted into JSON cases, and the PRD describes the behavior in pure input/output terms. The PRD must be self-contained: each leaf feature block should be understandable from its text and embedded example cases alone

  39. [42]

    The result must be zero FAIL

    Runbash rcb_tests/test.sh --cases-dir test_casesinside the original Docker image. The result must be zero FAIL. If a case fails, either the dispatcher is inconsistent with the golden implementation or the case should not have been retained. ## Requirements

  40. [43]

    The only files you may add or change are the test data,test.sh, and the Task-1 dispatcher/adapter

    Do not modify the repository’s functional code. The only files you may add or change are the test data,test.sh, and the Task-1 dispatcher/adapter. 2)bash rcb_tests/test.shmust support--cases-dir <subdir>(defaulttest_cases). For every case it writes the raw stdout torcb_tests/stdout/<cases-dir>/{stem}@{idx:03d}.txt. Different case directories must never ov...

  41. [44]

    One leaf fea- ture corresponds to one JSON file

    Each JSON file uses schema{description, cases:[{input, expected_output}]}. One leaf fea- ture corresponds to one JSON file. 4)test_casesmust remain a subset of the original repository tests. Do not invent new behavior at this stage. 5)inputmust be the program’s real input data, not source-level method names or class names. 6)expected_outputmust be the pro...

  42. [45]

    8)descriptionmust let a reader understand the scenario together withinputandexpected_output, but it must not copy the concrete input values or exact expected output verbatim

    Errors must be normalized into language-neutral black-box contracts rather than leaking host-language exception names or runtime-specific formatting. 8)descriptionmust let a reader understand the scenario together withinputandexpected_output, but it must not copy the concrete input values or exact expected output verbatim

  43. [46]

    The PRD must not leak original repository identifiers or implementation details such as repository names, author names, original class or method names, namespaces, or language-locked framework labels

  44. [47]

    Every leaf feature corresponds to one independent functionality point and one JSON file

    Use multi-level features when necessary. Every leaf feature corresponds to one independent functionality point and one JSON file. ## PRD Template The PRD should contain a neutral function-based title, a project goal, background and problem statement, architecture and engineering constraints, and a core-features section in which each leaf feature includes:...

  45. [48]

    A maintainable core system description reflected in the PRD

  46. [49]

    An execution/test adapter that reads JSON from stdin, invokes the core behavior, and prints the required stdout contract

  47. [50]

    APPENDIXB PROMPT FORENHANCEDTESTCASEGENERATION Enhanced Test Cases Generation ## Role You are an expert QA automation agent

    An automated harnessbash rcb_tests/test.shthat runs all cases from a selected case directory and records per-case stdout files exactly as specified. APPENDIXB PROMPT FORENHANCEDTESTCASEGENERATION Enhanced Test Cases Generation ## Role You are an expert QA automation agent. Your goal is to analyze a programming problem, read the existing test data, and sup...

  48. [51]

    Do not modifystart.md

    Read the content from the input paths above. Do not modifystart.md

  49. [52]

    Step 2: Supplementation (enhanced = NEW cases only)

    Analyze missing dimensions in the existing cases, including boundary values, special constraints, performance- related stress, and logic-specific paths. Step 2: Supplementation (enhanced = NEW cases only)

  50. [53]

    Identify gaps between the existingtest_casesand your multidimensional analysis

  51. [54]

    Generate additional cases in JSON format that strictly follow the same schema as the existing ones

  52. [55]

    Do not copy or merge native cases into the enhanced suite

    Writeonlythe newly generated cases toenhanced_test_cases. Do not copy or merge native cases into the enhanced suite

  53. [56]

    Any case whose(input, expected_output) exactly matches a case already present intest_casesmust be excluded

    The enhanced suite must be disjoint from the native suite. Any case whose(input, expected_output) exactly matches a case already present intest_casesmust be excluded

  54. [57]

    Step 3: Verification

    After writing, run the deduplication pass to remove accidental overlap with the native suite and delete any empty files. Step 3: Verification

  55. [58]

    Run the evaluation script against the reference solution

  56. [59]

    Check that all enhanced cases pass against the golden implementation

  57. [60]

    fuzzy_prd

    If there are logic or JSON-format errors, fix them and rerun until successful. ## Final Output Requirement •A short summary of the additional edge cases and test dimensions covered. •The generated enhanced JSON test cases saved to the output target path. •The execution log confirming that the enhanced cases pass against the reference solution. APPENDIXC P...

  58. [61]

    Select at most three constraints to hide, restricted to the most unguessable literals such as deliberate typos, obscure magic constants, and non-obvious sentinel values

  59. [62]

    For each hidden constraint, provide a vague replacement phrase, exact PRD strings to replace, at least five trigger keywords, and a precise oracle answer

  60. [63]

    hide": [{...}],

    Pick one or two non-hidden constraints and rewrite them as vague internal references (vague_refs). Do not hide action names, parameter names, default values, error categories, or feature descriptions. Return only JSON in the form: { "hide": [{...}], "vague_refs": [{...}] } APPENDIXF CODINGAGENTTASKINSTRUCTION Coding Agent Task Instruction ## Task A Docker...

  61. [64]

    Downloading the source code of the target repository from the internet, or directly calling its existing API, is prohibited

    You MUST implement the full functionality yourself. Downloading the source code of the target repository from the internet, or directly calling its existing API, is prohibited

  62. [65]

    Clarification

    First read{workdir}/{prd_name}thoroughly. If the PRD contains a “Clarification” interaction section, the requirement isintentionally incomplete and vague— you shouldfirst ask the User Agent questions as instructedto clarify the hidden interface contracts, feature details, and test cases, andonly start coding once things are clear. Anything left unclarifie...

  63. [66]

    You need to design the dispatcher / runner /rcb_tests/test.shyourself, and turn recovered examples intorcb_tests/public_test_cases/ *.jsonfor self-checking

    Any Public example cases made available in the PRD or recovered through User Agent interaction define the dispatcher’s input/output contract. You need to design the dispatcher / runner /rcb_tests/test.shyourself, and turn recovered examples intorcb_tests/public_test_cases/ *.jsonfor self-checking. 4)bash rcb_tests/test.shMUST support--cases-dir <subdir>to...

  64. [67]

    append_id

    System packages count too. Anything you install withaptmust also be installed byrcb_tests/test.sh idempotently, because the evaluation container starts clean. 9)Do not hard-code absolute paths.Use relative paths or environment variables only. 10)Rebuild from a clean state before self-checking: delete language-specific build caches such astarget/, bin/,obj...