Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WebGen-Bench measures whether AI agents can build working websites from scratch, and a fine-tuned 32B model beats every proprietary rival tested.

desk verdict A genuinely useful from-scratch website benchmark with careful test-case construction, but the headline fine-tuning result should not be trusted until the automated evaluator is validated on the exact model configuration. read the letter →

arxiv 2505.03733 v2 pith:H56A4BZI submitted 2025-05-06 cs.CL

Zimu Lu , Yunqiao Yang , Houxing Ren , Haotian Hou , Han Xiao , Ke Wang , Weikang Shi , Aojun Zhou
show 2 more authors
Mingjie Zhan Hongsheng Li
This is my paper · ORCID
classification cs.CL
keywords LLMagentswebsitegenerationbenchmarkautomatedUItestingwebnavigationagentfine-tuningcodefrom-scratchcodebase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WebGen-Bench is a benchmark built to measure whether an LLM-powered code agent can take a natural-language request and produce a working multi-file website from scratch. It consists of 101 curated instructions and 647 atomic test cases, each pairing an operation with an expected outcome; a web-navigation agent executes the operation on the generated site and scores it as yes, partial, or no. The paper reports that the best general configuration reaches only 27.8% accuracy, which it takes as evidence that current agents remain far from reliable at this task. It then claims that fine-tuning a 32B open-weight coder model on website-building trajectories produces WebGen-LM-32B at 38.2%, above every proprietary model tested. If the measurement is sound, the benchmark gives researchers a reproducible yardstick for a capability that previously had none.

What carries the argument

The load-bearing piece is the automated test-case harness: every instruction is tied to atomic test cases that specify both an operation and the expected result, so open-ended generation becomes a set of objective checks that can be run automatically. A web-navigation agent executes each operation against the live website and answers YES, PARTIAL, or NO, and the paper's accuracy formula counts partial as half a pass. This harness makes evaluation cheap and reproducible. The second mechanism is the training pipeline: WebGen-Instruct provides 6,667 decontaminated instructions, trajectories are collected with a code-agent framework, rejection sampling keeps only trajectories whose outputs score at least 3 out of 5 on appearance, and supervised fine-tuning on those trajectories creates the WebGen-LM model family.

What would settle it

Have human testers score all 647 test cases for WebGen-LM-32B and for the best proprietary configuration, then compare their verdicts with the automated agent's; if the 38.2% versus 27.8% gap shrinks or reverses under manual scoring, the central ranking claim would rest on evaluator error rather than website quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a reproducible way to grade from-scratch website generation. Instead of asking human testers to judge each site, the authors decompose each instruction into atomic, checkable test cases and rely on a web-navigation agent to interact with the live site and return YES, PARTIAL, or NO; accuracy is computed with partial credit counting half. Under this measure, no existing general model combination passes even 30% of the test cases, while the authors' fine-tuned 32B model reaches 38.2% and beats the best proprietary baseline by 10.4 percentage points. The paper also reports category-level results showing that design-validation checks are easier for agents than functional tests, and that accuracy improves as more training trajectories are used.

Load-bearing premise

The load-bearing premise is that the automated web-navigation agent's yes/partial/no judgments are accurate enough to rank the models, but the paper checks this against human testers on only three of the sixteen evaluated configurations, with agreement rates between 86.1% and 94.4%.

Editorial extensions

If this is right

  • If WebGen-Bench measures what it claims, researchers get an automated, shared yardstick for website-generation agents, replacing slow and expensive human evaluation.
  • The 27.8% ceiling for general configurations means there is wide-open headroom: progress on this task should be directly visible as accuracy rises.
  • The 38.2% fine-tuned result implies trajectory-level supervision on a code-agent framework transfers into the model itself, making open-weight specialization a competitive route against proprietary models.
  • The monotonic accuracy gains from 150 to 300 to 600 training trajectories point to data scaling as a reliable way to push the score higher.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper validates its automated judge against human testers on only three of sixteen evaluated configurations, with 86.1% to 94.4% agreement; the remaining thirteen orderings, including some of the finer gaps, are unverified and could shift under manual scoring.
  • Because the same automated judge scores every model, any systematic judge bias would affect all results; running the evaluation with a stronger or ensembled judge, or human-scoring all configurations, would show whether the headline 38.2% versus 27.8% gap persists.
  • The benchmark's instructions and test cases are generated with substantial GPT-4o involvement; if such generation injects a model-specific bias, models of similar style may be advantaged, and a fully human-authored subset would test that.
  • Beyond websites, the same trajectory fine-tuning recipe could transfer to other from-scratch codebase-generation tasks, but that extension is not in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces WebGen-Bench, a benchmark for evaluating LLM-based agents' ability to generate multi-file websites from scratch. It comprises 101 human-curated instructions and 647 human-refined test cases, plus an automated pipeline in which a WebVoyager UI agent powered by Qwen2.5-VL-32B-Instruct executes each test case and issues YES/NO/PARTIAL verdicts, while GPT-4o grades visual appearance. The authors evaluate Bolt.diy, OpenHands, and Aider with several proprietary and open-source LLMs; the best general configuration, Bolt.diy with DeepSeek-R1, reaches 27.8% accuracy. They also construct WebGen-Instruct (6,667 instructions) and fine-tune Qwen2.5-Coder-32B-Instruct on 600 rejection-sampled Bolt.diy trajectories, obtaining WebGen-LM-32B with 38.2% accuracy, which they claim surpasses all proprietary models tested. Code, data, and model weights are released.

Significance. If the reported results are reliable, WebGen-Bench fills a genuine gap: it is the first benchmark, to my knowledge, that measures from-scratch website generation with atomic, human-validated functional test cases rather than patch-based repository benchmarks. The data curation is careful: instructions are derived from 10,152 human-written project descriptions, test cases are independently refined by two annotators, and a manual decontamination check is reported. The release of code, data, and model weights is a clear strength, as is the rejection-sampling training recipe with an honest scaling analysis. The central weakness is that the headline model comparison depends on an automated UI-agent evaluator whose agreement with human testers is validated on only three of sixteen configurations, not on the winning configuration, and whose stochasticity is not quantified. These issues are addressable and do not, in my judgment, invalidate the benchmark's usefulness, but they require additional evidence before the comparative claim can be accepted.

major comments (3)
  1. [§4.2, Table 5] The paper's central claim—that WebGen-LM-32B surpasses the best proprietary model by 10.4 points (38.2% vs. 27.8%)—rests entirely on the WebVoyager/Qwen2.5-VL UI agent's verdicts, but the agent's alignment with human testers is measured only for Bolt.diy with Claude-3.5-Sonnet, DeepSeek-R1, and DeepSeek-V3 (Table 5: 86.1–94.4% alignment). The winning WebGen-LM-32B configuration is not among the validated ones, and no argument is given that the evaluator's error rate transfers to that configuration. Because WebGen-LM-32B is trained on rejection-sampled Bolt.diy trajectories, its websites may contain more conventional, easily navigable layouts, which could make them systematically easier for the UI agent to judge as YES even when their functional completeness is comparable. Please report human-validated alignment for a sample of WebGen-LM-32B test cases (and ideally for the other unvalidated configurations) before claiming superiority.
  2. [§4.2 and checklist item 7] All accuracy numbers in Tables 3 and 4 come from a single run of a stochastic, LLM-based evaluation pipeline; no error bars, confidence intervals, or repeated runs are reported. The checklist confirms this ('Error bars are not reported because it would be too computationally expensive'). Given that the automated evaluator agrees with human judges only 86–94% of the time, and that the head-to-head margin is 10.4 points, the reported ranking could be within evaluator noise. Please provide at least a bootstrap over the 647 test cases (treating the fixed verdicts as a sample) or repeated evaluations on a subset, and state the resulting uncertainty around the headline accuracies.
  3. [Table 8 vs. §3.2] Table 8 reports a 'Total' of 667 test cases, while the text and all other tables consistently state that WebGen-Bench contains 647 test cases. If the rows count subcategory memberships and therefore overlap, the total should be removed or explicitly labeled as the sum of overlapping memberships; as printed, the 667/647 discrepancy makes the benchmark statistics internally inconsistent.
minor comments (5)
  1. [Appendix L] The appendix contains the sentence 'We will add this discussion to the revised paper,' indicating that the appearance-score reliability analysis is presented as a plan rather than a completed result. Please integrate this discussion into the paper or clearly mark it as a proposed addition; as written, it reads as an unfinished part of the manuscript.
  2. [Section 3.1] The subsection 'Web Development Project Descriptions Collection' appears twice, with identical text, which appears to be a duplication error.
  3. [§G and §H figure references] The text in §G states that Fig. 10 is the start-testing prompt and Fig. 9 is the limit-reached prompt, but the figures are labeled in the opposite order; likewise, §H refers to Fig. 10 for the appearance-grading prompt while the prompt appears in Fig. 11. Please correct the cross-references.
  4. [Table 8] If the subcategory counts in Table 8 are not mutually exclusive (a test case can belong to several subcategories), the table should state this explicitly next to the numbers; the current layout invites a sum-of-parts reading that does not match the main-category totals.
  5. [Appendix N] The heading 'Examples of Websites with Different errors or flaws' contains inconsistent capitalization; please use title case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark construction is externally validated and the fine-tuning result is an empirical observation, not a fitted prediction.

full rationale

The paper does not derive any result from an input that already contains it. WebGen-Bench is externally anchored: instructions are human-authored project descriptions converted by GPT-4o, and the 647 test cases are independently reviewed and refined by two Ph.D. students to ensure they cover the instruction requirements. The automated UI-agent evaluator is checked against human annotations for three configurations in Table 5, and the appearance grader is additionally cross-checked with o3, Claude-3.5-Sonnet, and human graders in Appendix L. The fine-tuning claim is empirical: WebGen-LM-32B is trained on 600 Bolt.diy trajectories from WebGen-Instruct, which is decontaminated from the test set via Jaccard and embedding similarity, and its 38.2% accuracy is measured on the separate 101-instruction/647-test-case benchmark. The ablation in Table 6 shows accuracy rising with training sample size (150->25.1%, 300->31.9%, 600->38.2%), so the result is not a fitted artifact. No parameter is fitted to the test-set accuracy, and no self-citation carries a load-bearing uniqueness argument. The concern that the automated evaluator is validated on only three of sixteen configurations is a measurement-validity and statistical-significance limitation—the paper itself says error bars are not reported—but it is not circular: the evaluator is an independent instrument, not a quantity defined in terms of the models' outputs.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on manually curated instructions, GPT-4o-generated and human-validated test cases, and the assumption that the automated UI agent is an accurate proxy for human testers. The training result depends on several hand-chosen thresholds and a budget-limited sample size of 600 trajectories.

free parameters (7)
  • Jaccard similarity threshold for decontamination = 0.6
    Chosen by the authors to filter training instructions that overlap with test instructions; affects the composition of WebGen-Instruct and hence the fine-tuning result.
  • Cosine similarity threshold for semantic deduplication = 0.55
    Settled on after experimenting with various threshold settings; controls how aggressively near-duplicate training instructions are removed.
  • Appearance score filter for rejection sampling = >= 3
    Trajectories are kept only if the generated website receives a GPT-4o appearance grade of at least 3; this selection criterion shapes the fine-tuning data and the resulting WebGen-LM behavior.
  • Number of training trajectories used = 600
    The subset of WebGen-Instruct used for fine-tuning was capped at 600 trajectories due to API budget constraints; accuracy increases with sample count (150, 300, 600), so this budget affects the reported 38.2%.
  • Fine-tuning learning rate = 4e-5
    Standard hyperparameter chosen for the supervised fine-tuning; influences the final model performance on the benchmark.
  • Fine-tuning number of epochs = 2
    Training duration chosen for the fine-tuning runs; affects convergence and final accuracy.
  • Fine-tuning batch size = 32
    Standard training hyperparameter; chosen along with the number of GPUs for the fine-tuning experiments.
assumptions (5)
  • domain assumption Manual curation of the 10,152 project descriptions and the 101 test instructions produces a representative sample of real-world website development requests.
    Section 3.1: the benchmark's diversity and practicality claims rest entirely on this manual curation process, which involved forty PhD students and discussions among the authors.
  • domain assumption GPT-4o-generated test cases, after manual filtering and adjustment, correctly cover all instruction requirements and are atomic.
    Section 3.2: the validity of the benchmark depends on the assumption that the 647 test cases are accurate and comprehensively aligned with the 101 instructions.
  • domain assumption WebVoyager with Qwen2.5-VL-32B-Instruct is an accurate proxy for human testers across all test cases.
    Section 3.2 and Table 5: alignment is only measured on three configurations and ranges from 86.1% to 94.4%, yet the assumption is applied to all sixteen evaluated configurations.
  • domain assumption The decontamination procedure prevents semantic leakage from WebGen-Instruct into WebGen-Bench.
    Appendix D: the 5-gram Jaccard and Sentence-BERT cosine similarity thresholds are used, but manual inspection covers only the top three matches per test instruction, so the guarantee of no contamination is approximate.
  • domain assumption The 20 application categories derived from Upwork, Freelancer, and Proginn cover nearly all important types of web applications.
    Section 3.1 and Appendix E: this category list is used as the seed set for instruction generation, and the paper claims the categories encompass nearly all important web application types.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch." pith.science (2026). https://pith.science/paper/H56A4BZI

@misc{pith2026250503733,
  author       = {Pith},
  title        = {Pith review of: WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H56A4BZI}},
  note         = {Machine review of arXiv:2505.03733}
}
read the original abstract

LLM-based agents have demonstrated great potential in generating and managing code within complex codebases. In this paper, we introduce WebGen-Bench, a novel benchmark designed to measure an LLM-based agent's ability to create multi-file website codebases from scratch. It contains diverse instructions for website generation, created through the combined efforts of human annotators and GPT-4o. These instructions span three major categories and thirteen minor categories, encompassing nearly all important types of web applications. To assess the quality of the generated websites, we use GPT-4o to generate test cases targeting each functionality described in the instructions, and then manually filter, adjust, and organize them to ensure accuracy, resulting in 647 test cases. Each test case specifies an operation to be performed on the website and the expected result after the operation. To automate testing and improve reproducibility, we employ a powerful web-navigation agent to execute tests on the generated websites and determine whether the observed responses align with the expected results. We evaluate three high-performance code-agent frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and open-source LLMs as engines. The best-performing combination, Bolt.diy powered by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting the challenging nature of our benchmark. Additionally, we construct WebGen-Instruct, a training set consisting of 6,667 website-generation instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories generated from a subset of this training set achieves an accuracy of 38.2\%, surpassing the performance of the best proprietary model.

Figures

Figures reproduced from arXiv: 2505.03733 by the authors.

Figure 1
Figure 1. The data-curation and automatic-testing pipeline. (a) depicts the process for curating website-generation instructions together with their corresponding test cases. (b) presents the testing pipeline for verifying whether the generated websites meet functionality and design requirements with the WebVoyager UI agent, and for analyzing their aesthetic quality using GPT-4o. provides a comprehensive assessment of overall… view at source ↗
Figure 2
Figure 2. The examples of website screenshots at each appearance score. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Accuracy vs. sample number. 5 Conclusion In this paper, we introduce WebGen-Bench, a novel benchmark for evaluating the ability of LLM￾based agents to generate websites from scratch. The benchmark requires agents to construct and organize multi-file codebases while satisfying various functional and visual constraints. We evaluate three code-agent frameworks using both proprietary and open-source LLMs. The best-perfo… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: The prompt for deriving instructions from human annotated descriptions. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Top semantic matches for the first test instruction in WebGen-Bench with similarity scores. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Top semantic matches for the second test instruction in WebGen-Bench with similarity [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Top semantic matches for the third test instruction in WebGen-Bench with similarity scores. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The prompt for deriving test cases that covers all the functional and appearance require [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: The prompt for starting the operation of a test case, where {task} is replaced with the [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: The prompt for inducing an answer when the limit of the number of website interactions [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: The prompt for grading the appearance of the webpage. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: The prompt for testing OpenHands paired with CodeActAgent on WebGen-Bench. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: The prompt for aider websites generation. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: The distribution of the task case categories. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Examples of the screenshots of websites of different appearance scores. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: The examples of errors or flaws that generated webs may include. [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: The examples of UI agent testing processes resulting in YES. [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: The examples of UI agent testing processes resulting in YES. [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: The examples of UI agent testing processes resulting in YES. [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]
Figure 20
Figure 20. Figure 20: The examples of UI agent testing processes resulting in YES. [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: The examples of UI agent testing processes resulting in YES. [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: The examples of UI agent testing processes resulting in PARTIAL. [PITH_FULL_IMAGE:figures/full_fig_p031_22.png]
Figure 23
Figure 23. Figure 23: The examples of UI agent testing processes resulting in PARTIAL. [PITH_FULL_IMAGE:figures/full_fig_p032_23.png]
Figure 24
Figure 24. Figure 24: The examples of UI agent testing processes resulting in PARTIAL. [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]
Figure 25
Figure 25. Figure 25: The examples of UI agent testing processes resulting in NO. [PITH_FULL_IMAGE:figures/full_fig_p034_25.png]
Figure 26
Figure 26. Figure 26: The examples of UI agent testing processes resulting in NO. [PITH_FULL_IMAGE:figures/full_fig_p035_26.png]
Figure 27
Figure 27. Figure 27: The examples of UI agent testing processes resulting in NO. [PITH_FULL_IMAGE:figures/full_fig_p036_27.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing

    cs.SE 2025-06 conditional novelty 6.0 of 10

    GUIRepair, a cross-modal LLM pipeline that converts issue screenshots into reproduction code and rendered patch screenshots into validation feedback, resolves 157/517 SWE-bench M instances with GPT-4o and 175 with o4-mini.

  2. Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    Automatically adapted agent harnesses let capable SLMs recover most frontier-LLM accuracy on repetitive business tasks at a few percent of the inference cost.

  3. Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software

    cs.CR 2026-02 reject novelty 5.0 of 10

    Frontend features of LLM-generated apps can predict hidden backend vulnerabilities that a given model tends to reproduce, enabling black-box attack triage.

  4. You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation

    cs.SE 2025-08 unverdicted novelty 5.0 of 10

    RealDevWorld, an automated GUI-testing evaluation system, matches human expert judgments (0.92 accuracy, 0.85 correlation) on assessing LLM-generated production-ready software.

Reference graph

Works this paper leans on

70 extracted references · 37 canonical work pages · cited by 4 Pith papers

  1. [1]

    Ai pair programming in your terminal, 2024

    Aider-AI. Ai pair programming in your terminal, 2024. Accessed: 2025-04-22

    work page 2024

  2. [2]

    Swe-bench+: Enhanced coding benchmark for llms

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992, 2024

    arXiv 2024

  3. [3]

    Introducing claude 3.5 sonnet, 2024

    Anthropic. Introducing claude 3.5 sonnet, 2024. Accessed: 2025-04-22

    work page 2024

  4. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  5. [5]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [6]

    Github copilot, 2024

    GitHub Copilot. Github copilot, 2024. Accessed: 2025-04-22

    work page 2024

  7. [7]

    Cursor: The ai code editor, 2024

    Cursor. Cursor: The ai code editor, 2024. Accessed: 2025-04-22

    work page 2024

  8. [8]

    Automating gui testing with image-based deep reinforcement learning

    Juha Eskonen, Julen Kahles, and Joel Reijonen. Automating gui testing with image-based deep reinforcement learning. In 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS), pages 160–167, 2020

    work page 2020

Show all 70 references
  1. [9]

    Fernandes, Manuel Lopes, and Rui Prada

    Pedro M. Fernandes, Manuel Lopes, and Rui Prada. Agents for automated user experience testing. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 247–253, 2021

    2021

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [11]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  4. [12]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021

    2021 arXiv

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  8. [16]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  9. [17]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  10. [18]

    Repobench: Benchmarking repository-level code auto-completion systems

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023. 11

    2023 arXiv

  11. [19]

    Uxagent: An llm agent-based usability testing framework for web design

    Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Laurence Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. Uxagent: An llm agent-based usability testing framework for web design. arXiv preprint arXiv:2502.12561, 2025

    2025 arXiv

  12. [20]

    Lingma swe-gpt: An open development-process-centric language model for automated software improvement

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. Lingma swe-gpt: An open development-process-centric language model for automated software improvement. arXiv preprint arXiv:2411.00622, 2024

    2024 arXiv

  13. [21]

    Repository structure-aware training makes slms better issue resolver

    Zexiong Ma, Shengnan An, Zeqi Lin, Yanzhen Zou, and Bing Xie. Repository structure-aware training makes slms better issue resolver. arXiv preprint arXiv:2412.19031, 2024

    2024 arXiv

  14. [22]

    Sorft: Issue resolving with subtask-oriented reinforced fine-tuning

    Zexiong Ma, Chao Peng, Pengfei Gao, Xiangxin Meng, Yanzhen Zou, and Bing Xie. Sorft: Issue resolving with subtask-oriented reinforced fine-tuning. arXiv preprint arXiv:2502.20127, 2025

    2025 arXiv

  15. [23]

    Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    2025 arXiv

  16. [24]

    Octopack: Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro V on Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Fo...

    2023

  17. [25]

    Openai o3-mini, 2025

    OpenAI. Openai o3-mini, 2025. Accessed: 2025-04-22

    2025

  18. [26]

    Training software engineering agents and verifiers with swe-gym

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024

    2024 arXiv

  19. [27]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  20. [28]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

  21. [29]

    Specrover: Code intent extraction via llms

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. Specrover: Code intent extraction via llms. arXiv preprint arXiv:2408.02232, 2024

    2024 arXiv

  22. [30]

    bolt.diy, 2024

    stackblitz labs. bolt.diy, 2024. Accessed: 2025-04-22

    2024

  23. [31]

    Stahlke, Atiya Nova, and Pejman Mirza-Babaei

    Samantha . Stahlke, Atiya Nova, and Pejman Mirza-Babaei. Artificial playfulness: A tool for automated agent-based playtesting. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, page 1–6, New York, NY , USA, 2019. Association fo...

    2019

  24. [32]

    Table b-3

    US Bureau of Labor Statistics. Table b-3. average hourly and weekly earnings of all employees on private nonfarm payrolls by industry sector, seasonally adjusted., 2024

    2024

  25. [33]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representa...

    2024

  26. [34]

    Introducing devin, the first ai software engineer, 2024

    Scott Wu. Introducing devin, the first ai software engineer, 2024. Accessed: 2025-04-22

    2024

  27. [35]

    Agentless: Demystifying llm-based software engineering agents

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  28. [36]

    Swe-fixer: Training open-source llms for effective and efficient github issue resolution

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution. arXiv preprint arXiv:2501.05040, 2025. 12

    2025 arXiv

  29. [37]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  30. [38]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  31. [39]

    Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024

    arXiv

  32. [40]

    Scaling relationship on learning mathematical reasoning with large language models

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  33. [41]

    Repocoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023

    2023 arXiv

  34. [42]

    Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts

    Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. arXiv preprint arXiv:2405.04520, 2024

    2024 arXiv

  35. [43]

    Autocoderover: Au- tonomous program improvement

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Au- tonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604, 2024

    2024

  36. [44]

    Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15...

    2024 arXiv

  37. [45]

    Your output should align with the main objective of the website and expand upon the requirements

  38. [46]

    You should not specify any technical details in the instructions

  39. [47]

    You should not refer to any outside applications in your instructions

  40. [48]

    </important> The following is an example: <example> Objective: A hotel and travel ticket distribution website

    You should not output any additional comments. </important> The following is an example: <example> Objective: A hotel and travel ticket distribution website. Other requirements:

  41. [49]

    Order tickets and hotels

  42. [50]

    Browse tickets and hotels

  43. [51]

    The website should have functionalities for placing, canceling, and verifying orders

    Light blue background and dark olive green component Converted Instruction: Please implement a distribution website for travel and ticketing that sells products such as tickets and hotels. The website should have functionalities for placing, canceling, and verifying orders. Us...

  44. [52]

    Direct a UI agent to perform a single, atomic task. 2. Include validation criteria

  45. [53]

    Align with the goals and features described in the original prompt

  46. [54]

    Wheel of Fortune

    Ensure each task is atomic (tests one function at a time) and avoids combining multiple sub-tasks. Structure each instruction as: Task: Clear, singular task for the UI agent. Expected Result: Specific outcome to confirm success. Original prompt: {orig prompt} Focus on testing:...

  47. [55]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: As shown in Abstract and Introduction. Guidelines: • The answer NA means that the abstract and introduction do not inc...

  48. [56]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: As shown in Limitations and Future Work. 37 Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the pap...

  49. [57]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines: • The answer NA means that the pa...

  50. [58]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  51. [59]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We open-source all our code...

  52. [60]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: As shown in Experiments. Guid...

  53. [61]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Error bars are not reported because it would be too...

  54. [62]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: As detailed in Experime...

  55. [63]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: As explained in Ethics Statement in Appendix. Guidelines: • The answer NA means...

  56. [64]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: As explained in Ethics Statement in Appendix. Guidelines: • The answer NA means that there is no societal...

  57. [65]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: As explained i...

  58. [66]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: As explained in ...

  59. [67]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: As shown in the released code and data. Guidelines: • The answer NA means that the paper does not release new assets...

  60. [68]

    We used authors and student volunteers instead

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  61. [69]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  62. [70]

    Answer: [Yes] Justification: As detailed in Method

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.