Pith. sign in

REVIEW 4 major objections 6 minor 92 references

The paper claims that a test-driven agent can move from structured requirements to a whole runnable web system, not just code snippets, and that this 'requirement compilation' beats direct LLM coding: 84.93% vs 48.93% average GUI test pass

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:25 UTC pith:7UGYWZQA

load-bearing objection A credible framework for requirement-to-repo compilation, but the evidence for the headline gap is weaker than the abstract makes it look. the 4 major comments →

arxiv 2602.13723 v5 pith:7UGYWZQA submitted 2026-02-14 cs.SE

Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective

Weiyu Kong , Yun Lin , Xiwen Teoh , Duc-Minh Nguyen , Ruofei Ren , Jiaxin Chang , Haoxu Hu , Haoyu Chen This is my paper
classification cs.SE
keywords agentic software engineeringrequirement compilationdomain-specific languagetest-driven developmentcode generationLLM agentsweb system generationtraceability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ARC aims to show that an LLM-based agent can 'compile' a large, multi-modal requirement document — one written in a light graph-based DSL and containing hundreds of scenarios — into a runnable web system, rather than merely generating code snippets. The paper argues that by first deriving testable UI, API, and database interfaces from the requirement graph and then implementing bottom-up against those tests, requirement drift and omission are controlled. If the claim holds, the practical unit of AI-assisted development shifts from prompting for code to authoring structured requirements, with generated systems accompanied by tests and traceability. The evidence is a six-system benchmark where ARC's generated systems pass 84.93% of GUI tests on average versus 48.93% for the best baseline, plus a 21-participant study in which novices wrote DSL documents for up to 174 scenarios.

Core claim

The paper's central claim is that requirement compilation is achievable: a DSL-based requirement graph, with each node carrying multi-modal descriptions, dependencies, scenarios, and Gherkin-like steps, can serve as the source program, and an agent can deterministically compile it into a modular web system with UI, API, and DB layers. ARC's bidirectional test-driven loop is the mechanism: a top-down phase synthesizes interface signatures and test suites before any application logic exists; a bottom-up phase generates code node-by-node, running each module against its tests until the tests pass, while a traceability record links every requirement to its interfaces, tests, and implementation.

What carries the argument

The key mechanism is a requirement graph written in a lightweight graph-based DSL, where nodes are features, scenarios are workflows with Given/When/Then steps, and dependencies form a DAG. ARC traverses this graph with a depth-first strategy: in the top-down phase, each node is decomposed into UI, API, and database interfaces, each with synthesized tests (unit, integration, and end-to-end); in the bottom-up phase, an agent implements leaf modules first and moves upward, executing tests as gates in a reactive loop. The traceability record maps each requirement to its interfaces, tests, and implementation, enabling interface reuse and non-regression. The formal problem it solves is stated as

Load-bearing premise

The load-bearing premise is that the manually authored DSL requirement graph completely and faithfully captures all intended behavior: ARC treats the graph as the source of truth, so any scenario not written down (e.g., duplicate passenger IDs on the ticket-booking system) will not be implemented, no matter how well the compilation loop runs.

What would settle it

Prepare a fresh requirement document for a mature system with two test oracles: the DSL-derived test suite that ARC synthesizes, and an independent hand-written GUI acceptance suite that includes edge cases deliberately omitted from the DSL. If ARC's system passes the derived tests but fails the independent suite at rates close to baseline level, the reported compilation advantage is an artifact of test/requirement alignment rather than general compilation ability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If ARC works as claimed, repository-scale system generation no longer requires a human to write code directly: authoring a structured requirement document can yield a runnable, tested system.
  • Requirement drift and regression are controlled because each feature's interface and tests are fixed before coding, so later modules cannot silently break earlier behavior.
  • The generated systems come with maintainability artifacts — traceability from requirement to interface, test, and code — which the paper argues supports software evolution beyond the initial build.
  • ARC's overhead grows predictably with system size, while the baselines keep roughly constant effort and degrade in quality; this suggests structured compilation is the more reliable strategy as projects scale.
  • Non-functional properties such as security, performance, and reliability are not handled by ARC; the paper explicitly scopes its claim to functional correctness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because ARC's tests are generated from the same DSL that drives implementation, the reported pass rate is partly a measure of internal consistency; an independent oracle written from the real system's behavior would reveal how much of the 84.93% is genuine functional fidelity.
  • Editorial extension: the framework implicitly turns 'writing requirements' into a programming activity — the DSL is the source language, tests are the type checker, and traceability is the debugger — which suggests the next bottleneck is not code generation but requirement-authoring tooling and validation of the DSL itself.
  • Editorial extension: if the DSL's dependency graph is kept as a first-class artifact, the same compilation machinery should support incremental recompilation when a requirement changes, which is precisely the n-to-n-plus-one software-evolution scenario the paper flags as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ARC, a framework that treats a DSL-based multi-modal requirement document as a high-level 'source' and compiles it into a runnable web system. The DSL models requirements as a DAG of Requirement Nodes, each containing Scenarios decomposed into Given/When/Then Steps. ARC runs a two-phase, test-driven agentic loop: a top-down phase synthesizes UI/API/DB interfaces and test suites from the requirement graph, and a bottom-up phase implements code bottom-up, gating each module by its pre-defined tests, while maintaining a traceability record across requirements, interfaces, tests, and code. The evaluation reports GUI test pass rates on six web systems (BookStack, Keep, Stack Overflow, PrestaShop, 12306, Ctrip), comparing ARC with MetaGPT, OpenHands, Cursor, and Copilot; an ablation; and a user study with 21 participants. The abstract additionally claims results on the AppForge benchmark of 101 mobile app generation tasks, which does not appear in the body.

Significance. If the reported results are substantiated, ARC would be a meaningful step beyond snippet-level code generation and a concrete instantiation of test-driven, repository-scale requirement compilation. The paper's strengths are its explicit DSL meta-model, a clear algorithmic description (Algorithms 1–3), detailed agent prompts in Appendix A, and a frank discussion of failure modes such as the 12306 duplicate-passenger-ID case and limited test feedback in §5.2. The paper also ships an anonymous artifact/website, which supports reproducibility. However, the empirical evidence is currently under-specified in ways that are load-bearing for the central claim: baseline inputs are not defined, no variance or trial-level data are reported, and the AppForge claim in the abstract is absent from the body. The framework itself is plausible and worth revising; the evaluation reporting must be strengthened before the headline comparison can be accepted.

major comments (4)
  1. [§5.1.2, Table 2] The baseline comparison is not reproducible as reported. Baselines are described only as performing 'vibe-coding' from 'natural language descriptions and supplementary images' (§5.1.2), but the paper never states whether those natural-language descriptions contained the same scenarios, dependencies, and constraints that ARC receives in its DSL graph, nor does it give the exact prompts or configurations used for MetaGPT, OpenHands, Cursor, and Copilot. Appendix A documents ARC's prompts only. If the baselines were given a much shorter or lossy specification, the 36-percentage-point average advantage in Table 2 could reflect information asymmetry rather than ARC's test-driven compilation loop. Please report the full baseline input construction, all baseline prompts/configurations, and ideally give baselines the same structured information in a form each tool can consume.
  2. [§5.2, Table 2; abstract] The central 'consistently outperforms' claim is not supported at the reported granularity. Table 2 reports only point estimates, with no per-trial results, standard deviations, confidence intervals, or significance tests; Table 4 has the same issue. Since ARC and the baselines are stochastic LLM agents, a single favorable run could produce the reported gap. Moreover, the abstract claims results 'across three independent trials,' but the body never presents trial-level data. Please report per-run results and variance for Tables 2 and 4 (and the user study), state explicitly how many trials underlie each number, and either reconcile the abstract's 'three independent trials' with the body or correct the abstract.
  3. [Abstract vs. Sections 5.1–5.7] The abstract claims 'AppForge benchmark of 101 mobile app generation tasks' and a '68.3% test pass rate on AppForge,' but no AppForge methodology, baseline comparison, metric definition, or results appear in the body. This is a load-bearing scope claim: if AppForge is part of the evaluation, it must be described fully; if it is not, the abstract should not cite it. Please add the AppForge setup and results, or remove the claim from the abstract.
  4. [§5.1.1, §5.2] The benchmark construction has a shared-source validity threat. The requirement graphs and the ground-truth GUI test suites were both manually prepared by the same research group after exploring the same six systems, and the paper does not report separation between DSL authors and test authors, inter-annotator agreement, or a procedure for handling omissions. Since ARC treats the requirements graph as the source of truth (the paper states this explicitly in §5.2), the evaluation primarily measures how well ARC satisfies the DSL/test pair, not how well it would handle an arbitrary real-world requirement document. The 12306 duplicate-ID example is an honest acknowledgment of this boundary, but the benchmark section should clarify the independence of the test oracle from the DSL, publish the DSLs and test suites as benchmark artifacts, and measure sensitivity to requirement omissions (e.g.,
minor comments (6)
  1. [Abstract, §5.2] The abstract says ARC achieves '50.6% more GUI tests passed on average,' while §5.2 reports an average absolute improvement of 36 percentage points and an average relative improvement of 50.4%. Please make the metric explicit and use consistent wording throughout.
  2. [§5.3, Table 3] The evaluation metrics text calls the runtime 'Time (s)' while Table 3's header says 'Time (min)' and the values are clearly minutes. Please unify the unit names.
  3. [Table 5] Row 3 of Table 5 appears garbled: '3969.8 4 174' should likely be '96 / 9.8 / 4 / 174' but the current formatting makes the values ambiguous. Please verify the table layout, including group 5 and group 7.
  4. [§5.5] The paper says participants were organized into 7 groups of 3, and Table 5 has a 'Preferred Method' column, but it is unclear what 'Preferred Method' means for groups that were asked to use ARC (e.g., group 6 is listed as 'Vibe-coding'). Please clarify whether participants chose a method or were assigned one, and how that affects the survey result that '6 out of 7 groups agreed ARC was more effective.'
  5. [Table 1] The column header 'Link Type' contains URLs (e.g., demo.bookstackapp.com), which is a confusing label. Presumably it refers to the reference system's demo link, not to a 'link type.' Please rename the column.
  6. [§5.1.1] The benchmark selection criteria ('at least 100 million users' or 'top 100 revenue') are vague and not tied to a verifiable source. Please provide explicit criteria and justifications for each selected system, especially Keep and BookStack.

Circularity Check

0 steps flagged

No significant circularity: the evaluation oracle is a separately hand-written GUI suite, not a restatement of the DSL, and the central empirical claim is not forced by the framework's own test-generation loop.

full rationale

The paper's central claim is empirical: ARC takes a manually authored DSL requirement graph, generates interfaces and tests, implements code against those tests, and is then evaluated on a separately prepared GUI test suite. Section 5.1.1 says the benchmark provides both '(1) a multi-modal requirement document G_i^r that specifies the intended functionalities, and (2) a GUI test suite T_i used to validate the correctness of the implemented S_i.' The GUI suite is described as 'sophisticated end-to-end GUI test cases for each feature in a subject web system,' prepared from the user perspective, not as a re-export of the DSL. ARC's internal test generation (Section 4.2.2) does derive tests from DSL steps, but the evaluation oracle is not identical to that internal gate: Section 5.2 explicitly reports a failure on 12306 duplicate passenger IDs, a condition that could not appear in a GUI test that merely restated the DSL. Thus the reported pass rates are not a tautology. The formal problem statement defines T as 'strictly derived from the expectations specified in V_r' and then proposes to 'Maximize the structural consistency between the requirement space V_r and the test space T'; that is a definitional tautology as a formal objective, but it is descriptive rather than load-bearing, and it is not the basis of the headline comparison. Self-citations (e.g., CoedPilot, Trae Agent, intention-driven test generation) appear only in related work and do not support any load-bearing inference. The main concerns with the headline result are evaluation-validity issues such as unspecified baseline inputs and missing variance, which are threats to empirical validity rather than derivation-level circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 3 invented entities

The ledger reflects that ARC's contribution is mostly architectural: it assumes the DSL spec is complete, assumes the GUI tests are a valid oracle, and assumes the base LLM can code once constrained. No free numerical parameters are fitted to data; the main hand-chosen knob is the iteration budget b in Algorithm 3. The framework's invented entities have only in-paper evidence.

free parameters (1)
  • max iteration budget b
    Algorithm 3 uses a hand-chosen cap on test-fix attempts; when exhausted ARC keeps the last implementation. Affects pass rates and overhead substantially, but no value or sensitivity analysis is reported.
axioms (5)
  • domain assumption The DSL requirement graph completely specifies all intended behavior; ARC does not infer implicit scenarios.
    Section 5.2 'When can ARC fail...' and Section 5.7 Internal Validity state missing/implicit scenarios are not automatically inferred; this determines the upper bound of correctness.
  • domain assumption Hand-written GUI test suites are a valid oracle for functional correctness.
    Section 5.1.1 defines them as ground truth; pass rate is the sole effectiveness metric, and test coverage/quality is acknowledged as limited in Section 5.6.
  • domain assumption The underlying LLM (Gemini Pro 3) is capable of producing working code once constrained by interfaces and tests.
    Section 5.7 discusses transferability and says the approach relies on already capable models; no model-agnostic guarantee.
  • domain assumption UI/API/DB events with JSON payloads can capture the behavior of the target web systems.
    Section 4.2.1 defines the interface/event model; the claim that this is sufficient for repository-scale apps is assumed, not proven.
  • domain assumption The six selected systems are representative of web applications generally.
    Section 5.1.1 selection criteria and Section 5.7 External Validity acknowledge exclusion of streaming, real-time, and single-page apps.
invented entities (3)
  • Requirement DSL meta-model (Requirement Node/Scenario/Step DAG) no independent evidence
    purpose: Structured source language for requirements; enables decomposition into tests and prevents requirement drift.
    Introduced by the paper; no external evidence beyond the paper's own experiments.
  • Interface event model e=(name,payload) with UI/API/DB interface types no independent evidence
    purpose: Formal contracts that gate code generation and enable cross-layer data-flow tracking.
    Invented abstraction internal to ARC; its sufficiency for full-stack systems is assumed.
  • Traceability Record M no independent evidence
    purpose: Provenance mapping from requirements to interfaces, tests, and code; supports reuse and maintenance.
    Formal artifact of the framework; the claimed maintenance benefit rests on self-reported user-study feedback.

pith-pipeline@v1.3.0-alltime-deepseek · 24147 in / 13334 out tokens · 120507 ms · 2026-08-02T23:25:49.375921+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have significantly improved programming efficiency by translating natural language into code, yet their performance deteriorates when handling large-scale, multi-modal requirement documents containing hundreds of scenarios, often producing incorrect implementations or missing critical constraints. To address this challenge, we propose ARC (Agentic Requirement Compilation), a framework that compiles DSL-based requirement documents into runnable web systems while automatically generating modular software architecture, comprehensive test suites, and traceability across requirements, design, and code. ARC adopts a bidirectional test-driven agentic workflow, combining a top-down architecture design phase with a bottom-up implementation phase to ensure that generated code satisfies synthesized tests. We evaluate ARC on six runnable web system benchmarks and the AppForge benchmark of 101 mobile app generation tasks. Across three independent trials, ARC consistently outperforms state-of-the-art LLM-based baselines, achieving 50.6% more GUI tests passed on average for web systems, a 100% compilation success rate, and a 68.3% test pass rate on AppForge. A user study with 21 participants further shows that users with limited programming experience can write DSL-based requirement documents containing up to 174 scenarios within an average of 5.6 hours to generate maintainable runnable systems, including a real-world ticket-booking application of approximately 10K lines of code.

Figures

Figures reproduced from arXiv: 2602.13723 by Duc-Minh Nguyen, Haoxu Hu, Haoyu Chen, Jiaxin Chang, Ruofei Ren, Weiyu Kong, Xiwen Teoh, Yun Lin.

Figure 1
Figure 1. Figure 1: The meta-model (or schema) of the multi-modal requirement of ARC. Each requirement node is [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An instance of multi-modal requirement document conforming to our DSL. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An overview of ARC to parse a multi-modal requirement into a runnable web system. In addition [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Screenshots of the generated BookStack system. [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Screenshots of the generated Keep system. [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Screenshots of the generated Stack Overflow system. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Screenshots of the generated PrestaShop system. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Screenshots of the generated 12306 system. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Screenshots of the generated Ctrip system. [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 14 linked inside Pith

  1. [1]

    https://www.12306.cn/en/index.html

    12306 China Railway 2011. https://www.12306.cn/en/index.html

    2011

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. (2025)

    2025

  4. [4]

    Miltiadis Allamanis and Charles Sutton. 2014. Mining idioms from source code. InProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering(Hong Kong, China)(FSE 2014). Association for Computing Machinery, New York, NY, USA, 472–483. doi:10.1145/2635868.2635901

    arXiv 2014

  5. [5]

    Andrea Arcuri and Xin Yao. 2008. Search based software testing of object-oriented containers.Information Sciences 178, 15 (2008), 3075–3095

    2008

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  7. [7]

    C., Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D. C., Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. 2024. CodePlan: Repository-Level Coding using LLMs and Planning.Proc. ACM Softw. Eng.1, FSE, Article 31 (July 2024), 24 pages. doi:10.1145/3643757

    doi:10.1145/3643757 2024

  8. [8]

    R. Balzer. 1985. A 15 Year Perspective on Automatic Programming.IEEE Transactions on Software EngineeringSE-11, 11 (1985), 1257–1268. doi:10.1109/TSE.1985.231877

    arXiv 1985

  9. [9]

    https://www.bookstackapp.com/

    BookStack 2015. https://www.bookstackapp.com/

    2015

  10. [10]

    Pietro Braione, Giovanni Denaro, Andrea Mattavelli, and Mauro Pezzè. 2017. Combining symbolic execution and search-based testing for programs with complex heap inputs. InProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 90–101

    2017

  11. [11]

    Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. InOSDI, Vol. 8. 209–224

    2008

  12. [12]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  13. [13]

    Trevor Cohn, Phil Blunsom, and Sharon Goldwater. 2010. Inducing Tree-Substitution Grammars.J. Mach. Learn. Res. 11 (Dec. 2010), 3053–3096

    2010

  14. [14]

    https://github.com/features/copilot

    Copilot 2023. https://github.com/features/copilot

    2023

  15. [15]

    https://trip.com/

    Ctrip 2017. https://trip.com/

    2017

  16. [16]

    https://cursor.com/

    Cursor 2023. https://cursor.com/

    2023

  17. [17]

    Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. InInternational conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340

    2008

  18. [18]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering. 2130–2141

    2022

  19. [19]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration Code Generation via ChatGPT.ACM Trans. Softw. Eng. Methodol.33, 7 (2024), 189:1–189:38. , Vol. 1, No. 1, Article . Publication date: March 2026. ARC: Compiling Hundreds of Requirement Scenarios into A Runnable Web System 21

    2024

  20. [20]

    Kevin Forsberg and Harold Mooz. 1991. The relationship of system engineering to the project cycle.Center for Systems Management5333 (1991), 4–6

    1991

  21. [21]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  22. [22]

    Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. 2025. Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation. arXiv preprint arXiv:2510.24358(2025)

    arXiv 2025

  23. [23]

    Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. InProceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation. 213–223

    2005

  24. [24]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Van- schoren and S. Yeung (Eds.), Vol. 1. https...

    2021

  26. [26]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. InThe Twelfth International Conference on Learning Representations

    2024

  27. [27]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  28. [28]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id= chfJJYC3iL

    2025

  29. [29]

    Susmit Jha, Sumit Gulwani, Sanjit A Seshia, and Ashish Tiwari. 2010. Oracle-guided component-based program synthesis. InProceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. 215–224

    2010

  30. [30]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  31. [31]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2312–2323

    2023

  33. [33]

    https://keep.google.com/

    Keep 2013. https://keep.google.com/

    2013

  34. [34]

    Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. 2024. CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=vYhglxSj8j

    2024

  35. [35]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931

    2023

  36. [36]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushm...

    2022

  37. [37]

    Chenyan Liu, Yufan Cai, Yun Lin, Yuhuan Huang, Yunrui Pei, Bo Jiang, Ping Yang, Jin Song Dong, and Hong Mei. 2024. Coedpilot: Recommending code edits with learned prior edit relevance, project-wise awareness, and interactive nature. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 466–478

    2024

  38. [38]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics 12 (2024), 157–173. , Vol. 1, No. 1, Article . Publication date: March 2026. 22 Trovato et al

    2024

  39. [39]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code Auto- Completion Systems. InThe Twelfth International Conference on Learning Representations. https://openreview.net/ forum?id=pPjZIOuQuF

    2024

  40. [41]

    Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot’s code suggestions. InProceedings of the 19th International Conference on Mining Software Repositories. 1–5

    2022

  41. [42]

    Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2111–2123

    2023

  42. [43]

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2025. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=dw9VUsSHGB

    2025

  43. [44]

    Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. InCompanion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion. 815–816

    2007

  44. [45]

    Huy Nhat Phan, Phong X Nguyen, and Nghi DQ Bui. 2024. HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale.arXiv preprint arXiv:2406.11912(2024)

    Pith/arXiv arXiv 2024

  45. [46]

    https://prestashop.com/

    PrestaShop 2007. https://prestashop.com/

    2007

  46. [47]

    https://anonymous.4open.science/r/ARC-temp-C192/

    Project Repository (Anonymized) 2026. https://anonymous.4open.science/r/ARC-temp-C192/

    2026

  47. [48]

    https://anonymous-8h5ynlrxqovd.github.io/anonymous-repo- TsxwHr3NGLQNR3r4

    Project Website (Anonymized) 2026. https://anonymous-8h5ynlrxqovd.github.io/anonymous-repo- TsxwHr3NGLQNR3r4

    2026

  48. [49]

    Binhang Qi, Yun Lin, Xinyi Weng, Yuhuan Huang, Chenyan Liu, Hailong Sun, Zhi Jin, and Jin Song Dong. 2025. Intention-driven generation of project-specific test cases.arXiv preprint arXiv:2507.20619(2025)

    Pith/arXiv arXiv 2025

  49. [50]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15174–15186

    2024

  50. [51]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ChatDev: Communicative Agents for Software Development.arXiv preprint arXiv:2307.07924(2023). https://arxiv.org/abs/2307.07924

    Pith/arXiv arXiv 2023

  51. [52]

    Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A concolic unit testing engine for C.ACM SIGSOFT Software Engineering Notes30, 5 (2005), 263–272

    2005

  52. [53]

    Aditya S Shethiya. 2024. Engineering with Intelligence: How Generative AI and LLMs Are Shaping the Next Era of Software Systems.Spectrum of Research4, 1 (2024)

    2024

  53. [54]

    Jiho Shin, Sepehr Hashtroudi, Hadi Hemmati, and Song Wang. 2024. Domain Adaptation for Code Model-Based Unit Test Case Generation. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1211–1222

    2024

  54. [55]

    Vanshika Singh, Caglar Korlu, and Wesley KG Assunção. 2025. Experiences on Using Large Language Models to Re-Engineer a Legacy System at Volvo Group. In2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 102–112

    2025

  55. [56]

    https://stackoverflow.com/

    Stack Overflow 2008. https://stackoverflow.com/

    2008

  56. [57]

    Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. 2024. ML-Bench: Evaluating Large Language Models an...

    Pith/arXiv arXiv 2024

  57. [58]

    Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling. arXiv:2507.23370 [cs.SE] https://arxiv.org/abs/2507.23370

    Pith/arXiv arXiv 2025

  58. [59]

    Valerio Terragni, Partha Roop, and Kelly Blincoe. 2024. The Future of Software Engineering in an AI-Driven World. arXiv:2406.07737 [cs.SE] https://arxiv.org/abs/2406.07737

    arXiv 2024

  59. [60]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context.arXiv preprint arXiv:2009.05617(2020)

    Pith/arXiv arXiv 2020

  60. [61]

    Waldinger and Richard C

    Richard J. Waldinger and Richard C. T. Lee. 1969. PROW: a step toward automatic program writing. InProceedings of the 1st International Joint Conference on Artificial Intelligence(Washington, DC)(IJCAI’69). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 241–252

    1969

  61. [62]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741(2024). , Vol. 1, No. 1, Article . Publication date: March 2026. ARC: Compiling Hundreds of Requirement Scena...

    Pith/arXiv arXiv 2024

  62. [63]

    Jin Wen, Qiang Hu, Yuejun Guo, Maxime Cordy, and Yves Le Traon. 2025. Variable Renaming-Based Adversarial Test Generation for Code Model: Benchmark and Enhancement.ACM Transactions on Software Engineering and Methodology (2025)

    2025

  63. [64]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155 [cs.AI] https://arxiv.org/abs/2308.08155

    Pith/arXiv arXiv 2023

  64. [65]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The rise and potential of large language model based agents: A survey.Science China Information Sciences68, 2 (2025), 121101

    2025

  65. [66]

    Yingfei Xiong, Jie Wang, Runfa Yan, Jiachen Zhang, Shi Han, Gang Huang, and Lu Zhang. 2017. Precise Condition Synthesis for Program Repair. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 416–426. doi:10.1109/ICSE.2017.45

    doi:10.1109/icse.2017.45 2017

  66. [67]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  67. [68]

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. 2024. Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859(2024)

    Pith/arXiv arXiv 2024

  68. [69]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation.Proc. ACM Softw. Eng.1, FSE, Article 76 (jul 2024), 24 pages. doi:10.1145/3660783

    doi:10.1145/3660783 2024

  69. [70]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen

  70. [71]

    Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. 2025. Commit0: Library Generation from Scratch. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=MMwaQEVsAg

    2025

  71. [72]

    Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chenjie Xu, Chao Ji, and Wenyun Zhao. 2018. Benchmarking microservice systems for software engineering research. InProceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. 323–324. A Prompts This section presents the core prompts used in ARC. A.1 System Prompt Listing 1. Syst...

    2018

  72. [74]

    Do not check if it exists first

    ** NO GUESSING **: If a file path is provided in the context , write to it directly . Do not check if it exists first

  73. [75]

    Never run the full suite unless explicitly asked

    ** SURGICAL TESTING **: Only run tests relevant to the current Task ID . Never run the full suite unless explicitly asked . ** Don't run test in RED phase **. # Part 2. Core Workflow ( AUTONOMOUS LOOP ) ## Phase 1: Initialization & Startup ( Execute ONCE )

  74. [76]

    ## Phase 2: The Loop ( Repeated ) ### Step 1: Fetch Mission

    Call`init_project`to prepare the backlog . ## Phase 2: The Loop ( Repeated ) ### Step 1: Fetch Mission

  75. [77]

    All requirements completed

    Call`pop_next_mission`. If`pop_next_mission`returns " All requirements completed " , then stop

  76. [78]

    - * Note : The system scheduler decides the order

    ** CRITICAL :** Read the output to identify your ** Phase ** (`RED`or`GREEN`) and ** Task ID **. - * Note : The system scheduler decides the order . You might Design A -> Design B -> Implement B . Trust the scheduler .*

  77. [79]

    User Management Module

    ** FAST - TRACK CHECK ( Management Nodes ) **: - Check the output . If the requirement has : - NO`Acceptance Scenarios`AND - NO`UI Descriptions`AND - NO`User Stories`/`Description`( or just a generic container name like " User Management Module ") , can just skip and call`save_progress`with message " Feat ( REQ - ID ) : [ RED / GREEN ] management node " #...

  78. [80]

    - ** Note **: Some entity data in the ** UI Description ** may be ** sample data **

    ** Analyze Full Requirements **: - meticulous read the ** Requirement Description ** , ** Frontend Description ** , ** Acceptance Scenarios ** , and ** Architecture Context ** ( Parent Constraints ) . - ** Note **: Some entity data in the ** UI Description ** may be ** sample data **. You need to disign or modify data table to just support the functionali...

  79. [81]

    - ** Backend **: Define API Route signatures and Service Function skeletons

    ** Design System & Schema **: - ** Frontend **: Define UI Component skeletons and basic CSS styles ( consistent with UI description ) . - ** Backend **: Define API Route signatures and Service Function skeletons . - ** Database Evolution **: Design or modify table structures in`metadata . md` and`init_db . js`. - ** Idempotent DDL **: In`init_db . js`, st...

  80. [82]

    * ** Data Prerequisite **:`run_test`ALWAYS resets the database ( deletes` backend / database

    ** Write Failing Tests **: * ** Analyze Sources **: Combine`Current Requirement`,`UI Description`, and` Acceptance Scenarios`to understand the user flow . * ** Data Prerequisite **:`run_test`ALWAYS resets the database ( deletes` backend / database . db`and runs DB init ) for every run . Therefore every test file MUST re - import / init DB and insert seed ...

    2026

Showing first 80 references.