{"id":"01040157-8e84-4d39-9141-cccfea27c429","arxiv_id":"2505.03733","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"WebGen-Bench provides 101 website-building tasks with 647 atomic test cases, and a trained Qwen2.5-Coder-32B model reaches 38.2% accuracy, outperforming DeepSeek-R1's 27.8%.","lead":"This paper introduces a benchmark, WebGen-Bench, that tests how well AI agents can build complete interactive websites from a short written request. It also shows that fine-tuning an open-source model on example website-building sessions can beat the best proprietary models on that benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The automated UI-agent evaluator is human-validated on only 3 of 16 configurations, but the headline WebGen-LM-32B result is not among them, so its 38.2% vs 27.8% lead may reflect evaluator bias rather than true quality.","rationale":"The reader's strongest claim and weakest assumption identify evaluator reliability as the crux. I agree and sharpen it: the missing human validation is not merely for 'remaining configurations' in general but specifically for the fine-tuned model that produces the paper's most surprising result. Because WebGen-LM-32B was trained with appearance-score rejection sampling, its outputs are likely to be more standard and navigable, which can bias an LLM-based UI evaluator toward positive verdicts. This is a plausible mechanism, not an accusation. A human check on 150 test cases would settle whether the 38.2% is real. Other concerns, such as the 667 vs 647 test-case total discrepancy in Table 8, the tight decontamination thresholds at cosine similarity around 0.55, and the absence of error bars, are secondary and do not change the conditional verdict. The paper otherwise provides open code and data and a reasonable benchmark design, so the appropriate verdict remains CONDITIONAL pending evaluator validation on the headline configuration.","tokens_in":29486,"tokens_out":7376,"duration_ms":74440,"concrete_test":"Randomly sample 150 test cases from the Bolt.diy evaluation of WebGen-LM-32B. Have three human testers independently label each as YES/NO/PARTIAL using the same protocol as Section 4.3 (with a fourth adjudicator for disagreements), and compare with the UI agent's labels. Compute human-evaluated accuracy for those 150 cases and compare against the human-evaluated 30.1% already reported for DeepSeek-R1 in Table 5. If WebGen-LM-32B's human-evaluated accuracy remains at least 8-10 points above DeepSeek-R1's, the headline ranking survives; if it drops to within a few points, the reported superiority is an artifact of evaluator bias and the benchmark's model ordering is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that WebGen-LM-32B surpasses all proprietary models on WebGen-Bench (38.2% vs DeepSeek-R1's 27.8%). This rests entirely on the automated WebVoyager/Qwen2.5-VL evaluator's YES/NO/PARTIAL judgments. Table 5 validates that evaluator against human testers for only three Bolt.diy configurations (Claude-3.5-Sonnet, DeepSeek-R1, DeepSeek-V3), with alignment rates of 86.1-94.4%. The headline configuration, WebGen-LM-32B, is not among them. The fine-tuned model is trained on Bolt.diy trajectories selected by appearance score, so it likely produces cleaner, more conventional layouts with explicit UI affordances; such websites are systematically easier for the UI agent to navigate and to judge as YES, even if their functional depth is comparable to that of general models. This could inflate WebGen-LM-32B's accuracy by more than the 10.4-point margin. No error bars or repeated evaluator runs are reported, so the stochasticity of the evaluation is unquantified. The concern is not that the authors acted improperly; it is that the single instrument used for the paper's headline comparison is unverified on that exact comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebGen-Bench, a benchmark for evaluating LLM-based agents' ability to generate multi-file websites from scratch. It comprises 101 human-curated instructions and 647 human-refined test cases, plus an automated pipeline in which a WebVoyager UI agent powered by Qwen2.5-VL-32B-Instruct executes each test case and issues YES/NO/PARTIAL verdicts, while GPT-4o grades visual appearance. The authors evaluate Bolt.diy, OpenHands, and Aider with several proprietary and open-source LLMs; the best general configuration, Bolt.diy with DeepSeek-R1, reaches 27.8% accuracy. They also construct WebGen-Instruct (6,667 instructions) and fine-tune Qwen2.5-Coder-32B-Instruct on 600 rejection-sampled Bolt.diy trajectories, obtaining WebGen-LM-32B with 38.2% accuracy, which they claim surpasses all proprietary models tested. Code, data, and model weights are released.","tokens_in":29826,"tokens_out":6099,"duration_ms":53471,"significance":"If the reported results are reliable, WebGen-Bench fills a genuine gap: it is the first benchmark, to my knowledge, that measures from-scratch website generation with atomic, human-validated functional test cases rather than patch-based repository benchmarks. The data curation is careful: instructions are derived from 10,152 human-written project descriptions, test cases are independently refined by two annotators, and a manual decontamination check is reported. The release of code, data, and model weights is a clear strength, as is the rejection-sampling training recipe with an honest scaling analysis. The central weakness is that the headline model comparison depends on an automated UI-agent evaluator whose agreement with human testers is validated on only three of sixteen configurations, not on the winning configuration, and whose stochasticity is not quantified. These issues are addressable and do not, in my judgment, invalidate the benchmark's usefulness, but they require additional evidence before the comparative claim can be accepted.","major_comments":[{"comment":"The paper's central claim—that WebGen-LM-32B surpasses the best proprietary model by 10.4 points (38.2% vs. 27.8%)—rests entirely on the WebVoyager/Qwen2.5-VL UI agent's verdicts, but the agent's alignment with human testers is measured only for Bolt.diy with Claude-3.5-Sonnet, DeepSeek-R1, and DeepSeek-V3 (Table 5: 86.1–94.4% alignment). The winning WebGen-LM-32B configuration is not among the validated ones, and no argument is given that the evaluator's error rate transfers to that configuration. Because WebGen-LM-32B is trained on rejection-sampled Bolt.diy trajectories, its websites may contain more conventional, easily navigable layouts, which could make them systematically easier for the UI agent to judge as YES even when their functional completeness is comparable. Please report human-validated alignment for a sample of WebGen-LM-32B test cases (and ideally for the other unvalidated configurations) before claiming superiority.","section":"§4.2, Table 5"},{"comment":"All accuracy numbers in Tables 3 and 4 come from a single run of a stochastic, LLM-based evaluation pipeline; no error bars, confidence intervals, or repeated runs are reported. The checklist confirms this ('Error bars are not reported because it would be too computationally expensive'). Given that the automated evaluator agrees with human judges only 86–94% of the time, and that the head-to-head margin is 10.4 points, the reported ranking could be within evaluator noise. Please provide at least a bootstrap over the 647 test cases (treating the fixed verdicts as a sample) or repeated evaluations on a subset, and state the resulting uncertainty around the headline accuracies.","section":"§4.2 and checklist item 7"},{"comment":"Table 8 reports a 'Total' of 667 test cases, while the text and all other tables consistently state that WebGen-Bench contains 647 test cases. If the rows count subcategory memberships and therefore overlap, the total should be removed or explicitly labeled as the sum of overlapping memberships; as printed, the 667/647 discrepancy makes the benchmark statistics internally inconsistent.","section":"Table 8 vs. §3.2"}],"minor_comments":[{"comment":"The appendix contains the sentence 'We will add this discussion to the revised paper,' indicating that the appearance-score reliability analysis is presented as a plan rather than a completed result. Please integrate this discussion into the paper or clearly mark it as a proposed addition; as written, it reads as an unfinished part of the manuscript.","section":"Appendix L"},{"comment":"The subsection 'Web Development Project Descriptions Collection' appears twice, with identical text, which appears to be a duplication error.","section":"Section 3.1"},{"comment":"The text in §G states that Fig. 10 is the start-testing prompt and Fig. 9 is the limit-reached prompt, but the figures are labeled in the opposite order; likewise, §H refers to Fig. 10 for the appearance-grading prompt while the prompt appears in Fig. 11. Please correct the cross-references.","section":"§G and §H figure references"},{"comment":"If the subcategory counts in Table 8 are not mutually exclusive (a test case can belong to several subcategories), the table should state this explicitly next to the numbers; the current layout invites a sum-of-parts reading that does not match the main-category totals.","section":"Table 8"},{"comment":"The heading 'Examples of Websites with Different errors or flaws' contains inconsistent capitalization; please use title case.","section":"Appendix N"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unvalidated use of the UI-agent evaluator on the headline configuration. I recommend asking the authors to add human validation for WebGen-LM-32B and at least bootstrap confidence intervals, as these are feasible within the scope of the paper. I also note the manuscript appears to be a draft (the 'We will add this discussion to the revised paper' sentence and the duplicated paragraph), so the authors should be asked to polish it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is the first benchmark I know of that requires agents to build complete multi-file websites from scratch, and the construction pipeline is genuinely careful. The 101 instructions are human brainstormed, GPT-4o turns them into testable tasks, and two Ph.D. students manually filter and refine the 647 test cases. The decontamination of WebGen-Instruct from the test set is described concretely (Jaccard plus semantic thresholds), and the authors release code, data, and model weights. That is real, reproducible asset-building work, and the field needs it.\n\nThe paper's headline claim is that fine-tuning Qwen2.5-Coder-32B on 600 appearance-filtered Bolt.diy trajectories yields 38.2% accuracy, beating DeepSeek-R1's 27.8%. That is a striking result, but the automated UI agent that produces every number in Table 3 is validated against human testers on only three Bolt.diy configurations (Claude-3.5-Sonnet, DeepSeek-R1, DeepSeek-V3), with alignment between 86% and 94%. WebGen-LM-32B is not among them. Because the fine-tuning data was selected by appearance score, the fine-tuned model likely generates cleaner, more standard layouts that a vision-language navigation agent finds easier to operate and judge. That could plausibly account for more than the 10.4-point margin. No error bars or repeated evaluator runs are reported. These concerns do not sink the benchmark, but they do mean the fine-tuning result is not established yet.\n\nThere are also small textual issues: Table 8 says 667 test cases total, while the text says 647; Section 3.1 repeats a paragraph verbatim; Appendix H references the wrong figure number; and Appendix L contains a stray \"We will add this discussion to the revised paper\" line. Minor, but sloppy. The categorical breakdown and the appearance-score cross-check with o3/Claude/human in Appendix L are good and partially answer concerns about GPT-4o-as-judge.\n\nOverall: the benchmark design is sound and the dataset is a contribution worth having. The evaluator-validation gap should be fixed before publication — validate the UI agent on WebGen-LM-32B (and ideally a couple more configs), report variance from repeated runs or at least bootstrap estimates, and clean up the inconsistencies. I would take this to peer review and would read a revised version. For anyone building website-generation agents, this is a useful evaluation target to cite.","headline":"A genuinely useful from-scratch website benchmark with careful test-case construction, but the headline fine-tuning result should not be trusted until the automated evaluator is validated on the exact model configuration.","tokens_in":30350,"tokens_out":2225,"would_cite":true,"duration_ms":22294,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebGen-Bench measures whether AI agents can build working websites from scratch, and a fine-tuned 32B model beats every proprietary rival tested.","keywords":["LLM agents","website generation","benchmark","automated UI testing","web navigation agent","fine-tuning","code generation","from-scratch codebase generation"],"falsifier":"Have human testers score all 647 test cases for WebGen-LM-32B and for the best proprietary configuration, then compare their verdicts with the automated agent's; if the 38.2% versus 27.8% gap shrinks or reverses under manual scoring, the central ranking claim would rest on evaluator error rather than website quality.","tokens_in":29334,"feed_emoji":"🌐","tokens_out":12641,"duration_ms":113547,"temperature":0.7,"pith_summary":"WebGen-Bench is a benchmark built to measure whether an LLM-powered code agent can take a natural-language request and produce a working multi-file website from scratch. It consists of 101 curated instructions and 647 atomic test cases, each pairing an operation with an expected outcome; a web-navigation agent executes the operation on the generated site and scores it as yes, partial, or no. The paper reports that the best general configuration reaches only 27.8% accuracy, which it takes as evidence that current agents remain far from reliable at this task. It then claims that fine-tuning a 32B open-weight coder model on website-building trajectories produces WebGen-LM-32B at 38.2%, above every proprietary model tested. If the measurement is sound, the benchmark gives researchers a reproducible yardstick for a capability that previously had none.","feed_headline":"Fine-tuned 32B model beats all proprietary models on website benchmark","feed_subtitle":"Its 38.2% pass rate tops the best proprietary model's 27.8%, drawing a clear line for future site-building agents.","key_machinery":"The load-bearing piece is the automated test-case harness: every instruction is tied to atomic test cases that specify both an operation and the expected result, so open-ended generation becomes a set of objective checks that can be run automatically. A web-navigation agent executes each operation against the live website and answers YES, PARTIAL, or NO, and the paper's accuracy formula counts partial as half a pass. This harness makes evaluation cheap and reproducible. The second mechanism is the training pipeline: WebGen-Instruct provides 6,667 decontaminated instructions, trajectories are collected with a code-agent framework, rejection sampling keeps only trajectories whose outputs score at least 3 out of 5 on appearance, and supervised fine-tuning on those trajectories creates the WebGen-LM model family.","core_discovery":"On the paper's own terms, the central discovery is a reproducible way to grade from-scratch website generation. Instead of asking human testers to judge each site, the authors decompose each instruction into atomic, checkable test cases and rely on a web-navigation agent to interact with the live site and return YES, PARTIAL, or NO; accuracy is computed with partial credit counting half. Under this measure, no existing general model combination passes even 30% of the test cases, while the authors' fine-tuned 32B model reaches 38.2% and beats the best proprietary baseline by 10.4 percentage points. The paper also reports category-level results showing that design-validation checks are easier for agents than functional tests, and that accuracy improves as more training trajectories are used.","pith_inferences":["The paper validates its automated judge against human testers on only three of sixteen evaluated configurations, with 86.1% to 94.4% agreement; the remaining thirteen orderings, including some of the finer gaps, are unverified and could shift under manual scoring.","Because the same automated judge scores every model, any systematic judge bias would affect all results; running the evaluation with a stronger or ensembled judge, or human-scoring all configurations, would show whether the headline 38.2% versus 27.8% gap persists.","The benchmark's instructions and test cases are generated with substantial GPT-4o involvement; if such generation injects a model-specific bias, models of similar style may be advantaged, and a fully human-authored subset would test that.","Beyond websites, the same trajectory fine-tuning recipe could transfer to other from-scratch codebase-generation tasks, but that extension is not in the paper."],"forward_implications":["If WebGen-Bench measures what it claims, researchers get an automated, shared yardstick for website-generation agents, replacing slow and expensive human evaluation.","The 27.8% ceiling for general configurations means there is wide-open headroom: progress on this task should be directly visible as accuracy rises.","The 38.2% fine-tuned result implies trajectory-level supervision on a code-agent framework transfers into the model itself, making open-weight specialization a competitive route against proprietary models.","The monotonic accuracy gains from 150 to 300 to 600 training trajectories point to data scaling as a reliable way to push the score higher."],"supporting_citations":[{"why":"Supplies the web-navigation agent that executes test cases and issues the YES/PARTIAL/NO judgments that define the benchmark's scores.","marker":"[11]"},{"why":"Prior work that motivates using an automated agent for usability testing, shaping the paper's evaluation design.","marker":"[19]"},{"why":"Supplies the code-agent framework used for the best-performing general configurations and for generating the training trajectories.","marker":"[30]"},{"why":"One of the three code-agent frameworks the benchmark evaluates, providing a generalist baseline.","marker":"[33]"},{"why":"The third code-agent framework compared, used with the same engine models to test framework effects.","marker":"[1]"},{"why":"Provides the sentence embeddings used to remove training instructions that are semantically close to benchmark instructions, underpinning the decontamination claim.","marker":"[28]"},{"why":"The open-weight coder model that is fine-tuned into the WebGen-LM family, carrying the training result.","marker":"[13]"},{"why":"The engine used to generate the training trajectories from the instruction set, with rejection sampling on appearance scores.","marker":"[17]"},{"why":"The proprietary baseline reported as best among general models, which WebGen-LM-32B is claimed to surpass by 10.4 points.","marker":"[10]"}],"fun_headline_variants":["Website bench: tuned 32B outdoes all proprietary","New benchmark shows LLMs struggle with website gen","From-scratch site builder: tuned model beats top AI","LLM website test: tuned 32B bests all proprietary","WebGen-Bench: fine-tuned 32B beats proprietary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated web-navigation agent's yes/partial/no judgments are accurate enough to rank the models, but the paper checks this against human testers on only three of the sixteen evaluated configurations, with agreement rates between 86.1% and 94.4%.","fun_headline_variants_meta":{"raw":{"variants":["Website bench: tuned 32B outdoes all proprietary","New benchmark shows LLMs struggle with website gen","From-scratch site builder: tuned model beats top AI","LLM website test: tuned 32B bests all proprietary","WebGen-Bench: fine-tuned 32B beats proprietary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3086,"prompt_tokens":1018,"completion_tokens":2068,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1984}},"tokens_in":634,"tokens_out":2068,"duration_ms":14937,"temperature":1.0,"reasoning_tokens":1984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:43:20.149924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human testers score all 647 test cases for WebGen-LM-32B and for the best proprietary configuration, then compare their verdicts with the automated agent's; if the 38.2% versus 27.8% gap shrinks or reverses under manual scoring, the central ranking claim would rest on evaluator error rather than website quality.","supporting_citations":[{"cited_title":"bolt.diy, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the code-agent framework used for the best-performing general configurations and for generating the training trajectories."},{"cited_title":"Ai pair programming in your terminal, 2024","cited_arxiv_id":null,"evidence_quote":"The third code-agent framework compared, used with the same engine models to test framework effects."}],"review_version":1}