Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

WebCoderBench claims that web app generation can be evaluated without ground-truth implementations, using 1,572 real user requirements and 24 automated metrics, and that no single model dominates across all perspectives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:41 UTC pith:HF2KUOY5

load-bearing objection Genuinely useful dataset and metric suite for web-app generation, but the ranking table mixes apples and oranges and several metrics are effectively dead weight — worth reviewing with fixes. the 4 major comments →

arxiv 2601.02430 v3 pith:HF2KUOY5 submitted 2026-01-05 cs.SE cs.AI

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Chenxu Liu , Yingjie Fu , Wei Yang , Ying Zhang , Tao Xie This is my paper
classification cs.SE cs.AI
keywords benchmarkweb application generationlarge language modelsLLM-as-a-judgeevaluation metricshuman preference weightingreal-world requirementsinterpretable evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that LLM-generated web applications can be reliably benchmarked without needing ground-truth code or test cases, by scoring generated pages against 24 fine-grained, partly LLM-judged metrics. The benchmark is built from 1,572 real user requirements, and its scores are combined using human-preference weights so the overall ranking reflects what real users care about. The central empirical claim is that no tested model dominates all metrics, and that the per-metric results give developers actionable, targeted signals for improvement. A sympathetic reader would care because this offers a way to compare and improve web-app-generating LLMs in an open-ended, realistic setting.

Core claim

The paper introduces WebCoderBench and argues it is the first real-world-collected, generalizable, and interpretable benchmark for web app generation. It provides 24 metrics across 9 quality perspectives, combining rule-based checks with LLM-as-a-judge scoring, and weights them by preferences elicited from 141 survey respondents using Borda Count. Evaluating 12 LLMs and 2 LLM-based agents, the paper finds that GPT-5-High has the highest overall score, GLM-4.5 is the best open-source model, and no model or agent is dominant across all 24 metrics. It also reports that models perform better on vague requirements than clear ones, that image inputs degrade performance, and that agentic systems al

What carries the argument

The central mechanism is the combination of three ground-truth checklists — functional, visual, and content — extracted per requirement by three LLMs and validated by human annotators, then used by a judge LLM to measure alignment. Around this sit 24 automated metrics (rule-based and LLM-as-a-judge) that yield z-scores, which are combined into an overall score using Borda Count weights derived from user preference rankings. The checklists carry the largest combined weight and are what makes requirement-level alignment measurable without a reference implementation.

Load-bearing premise

The alignment scores — the largest component of the overall score — depend on ground-truth checklists that are generated by LLMs and validated by humans, and the paper reports no agreement statistics between annotators; if those checklists are biased or noisy, the alignment metrics may not reflect what users actually want.

What would settle it

Take a sample of requirements with independently verified ground-truth checklists (e.g., written by experts without LLM assistance), generate artifacts, and check whether WebCoderBench alignment scores track expert-checked satisfaction; alternatively, compute inter-annotator agreement on the checklist-validation step — if agreement is near chance, the alignment metric loses its meaning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Model developers can use the per-metric scores to target specific weaknesses rather than relying on a single aggregate rank.
  • The benchmark provides a fully automated evaluation pipeline that could be applied to new models without manual grading or predefined test cases.
  • The finding that no model dominates all perspectives suggests that hybrid or ensemble systems might outperform any single model on overall web app quality.
  • The observation that vague requirements yield higher scores than clear ones implies that current LLMs are better at creative interpretation than at strict specification compliance.
  • The closing performance gap between open-source and closed-source models suggests that open-source web-app generators are becoming competitive enough for practical deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The three alignment metrics — the most heavily weighted — may be measuring consistency between the LLMs that generated the checklists and the LLM that judges them, rather than genuine user satisfaction, if the checklists omit or hallucinate requirement points.
  • The human-preference weights could be personalized per user persona, since the paper's own data shows designers, legal staff, and operations personnel rank quality perspectives differently.
  • A testable extension would be to run the same benchmark on a second independently collected set of requirements to verify that model rankings remain stable across datasets.
  • The framework could be extended to full-stack or framework-based applications (e.g., React) once automated evaluation for those formats matures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. WebCoderBench introduces a benchmark of 1,572 real-world user requirements for web application generation, together with 24 automated evaluation metrics across 9 perspectives. The metrics combine rule-based checks with LLM-as-a-judge scoring, and the paper derives an overall score by weighting metric z-scores according to a 141-response human survey processed with Borda Count. The authors evaluate 12 LLMs and 2 LLM-based agents, report per-perspective and per-metric z-scores, and conclude that no single model dominates all metrics and that the benchmark provides interpretable, human-aligned evaluation.

Significance. If the evaluation design is sound, WebCoderBench would be a useful contribution: it uses authentic user requirements, avoids reliance on ground-truth implementations, and attempts to make the overall score interpretable through human-preference weights. The dataset size, modality coverage, and breadth of metrics are genuine strengths, as is the explicit effort to test judge stability with the Mann–Whitney U test. However, the benchmark's central claims depend on the validity of LLM-generated checklists, the comparability of models evaluated on different sample sizes, and the treatment of largely unscorable metrics. These issues need to be addressed before the benchmark can be considered a reliable measurement instrument.

major comments (4)
  1. [Table 4 and §5.1] Rows 13–14 (Manus, MiniMax Agent) are ranked against rows 1–12 using scores averaged over only 165 requirements, while all LLMs are averaged over 1,572 requirements. Because each per-metric z-score in Eq. (1) is normalized using means and standard deviations computed over the full sample pool, and because no confidence intervals or significance tests are reported for the differences in Table 4, the current presentation conflates sampling variability with model-level differences. This directly affects the ‘no single model dominates’ claim. Please report the head-to-head comparison on the same 165-requirement subset for all models, or provide bootstrap/standard-error intervals for the overall scores.
  2. [Appendix C, Table 11] Several metrics are unscorable for the overwhelming majority of artifacts. For example, for DeepSeek-R1-0528, Icon Style Consistency (metric 8) is unscorable for 1,455/1,572 cases and Media Quality (metric 13) for 1,543/1,572 cases; similar counts appear for many other model rows. The appendix states that these cases are excluded from averaging rather than scored as zero. As a result, the z-score normalization in Eq. (1) for these metrics is estimated on a small, non-representative subset of artifacts, and the ‘24-metric comprehensive evaluation’ claim is weakened. Please report the scorable sample size for each metric, test sensitivity by imputing a neutral score or treating the metric as missing in a multi-level model, and state whether rankings change.
  3. [§3.1, §E.3, and App. F.3] The ground-truth checklists that drive Functional, Visual, and Content Alignment (combined weight approximately 36%) are generated by LLMs and then human-validated. The prompt in Fig. 7 explicitly instructs the checklist generator to add a ‘minimal functional set’ for abstract objectives—e.g., a file-sharing platform is expanded to ‘Upload files’, ‘Generate sharing link’, ‘Access link to download’. The human validation guidelines in §E.3 retain such inferred items if they are ‘reasonable’ and only delete items that are both unmentioned and unreasonable. Since the alignment judge is GPT-5-chat, the same model family as one of the checklist generators, the three alignment metrics may reward the most conventional LLM interpretation of the requirement rather than fidelity to the user's explicit request. The paper reports no inter-annotator agreement for checklist validation and no statistics
  4. [Limitations and §1] The paper states that the dataset and the implementation of the evaluation metrics cannot be released due to legal constraints. For a benchmark paper, this is a serious reproducibility limitation: readers cannot verify the data-collection pipeline, re-run the checklist generation and human validation, or independently check the rule-based metric implementations. The public leaderboard and planned future open batch are helpful, but they do not substitute for a reproducible evaluation artifact. Please clarify exactly what will be released (e.g., checklists, per-sample scores, evaluation code on a synthetic subset) and how the current findings can be independently audited.
minor comments (5)
  1. [Eq. (1)] The z-score formula should be written with explicit parentheses: z_{i,j} = (x_{i,j} - mu_j) / sigma_j. As typeset, it is ambiguous.
  2. [Table 2] There is a typo in the application category list: ‘Personal Webite’ should be ‘Personal Website’.
  3. [Table 5] The caption says scores are ‘averaged over models and questions’, but the values look like weighted z-scores. Please define the aggregation explicitly and state the unit.
  4. [§5.1] The 165-requirement subset used for agents should be described in more detail: how it was sampled, whether the distribution over complexity/clarity/category matches the full dataset, and why this sample size was chosen.
  5. [Appendix C] The text says the number of unscorable cases ‘does not differ greatly across metrics’ but Table 11 shows large variation (e.g., 0 to over 1,500). Please soften or remove that claim.

Circularity Check

0 steps flagged

No significant circularity: the evaluation pipeline is constructed from external inputs (real user requirements, human-validated checklists, independent survey weights) rather than reducing to its own outputs.

full rationale

WebCoderBench's claimed derivation is a benchmark construction, not a mathematical derivation. The 1,572 requirements come from 'anonymized and filtered real-world online data from our industrial partner' (§3.1); the ground-truth checklists are produced by three LLMs and then 'validated and modified by human annotators' (App. E.3), so the reference standard is not defined solely by the judge. The overall-score weights come from a separate human survey with Borda Count (§4.3), an external input rather than a fitted parameter. The paper makes no load-bearing self-citation: comparisons in Table 1 are to external benchmarks, and the cited checklist and LLM-judge works are not by the present authors. The closest concern is that GPT-5-Chat appears both as one of the three checklist generators (§3.1) and as the Alignment-judge LLM (§4.2), and that checklist prompts allow 'implicit completion' of abstract objectives (App. F.3). This creates a potential bias in the alignment metrics, but it is not a reduction by construction: human validation intervenes, the alignment score is not an algebraic identity with any fitted parameter, and the headline ranking also depends on 17 rule-based metrics and independent preference weights. The Limitations section itself flags metric-correctness and annotation-reliability risks. Those are validity threats, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The benchmark rests on six external assumptions, most importantly the faithfulness of LLM-generated checklists and the validity of LLM judges. The free parameters are hand-chosen scoring constants in individual metrics, not fit to the headline results. No new physical or conceptual entities are introduced.

free parameters (6)
  • errors_per_1k penalty factor = 20
    Used in Runtime Console Errors and Static Syntax Checking ('score = 100 - (errors_per_1k * 20)', capped at 0); chosen by hand, no external calibration.
  • Comment Rate score mapping = sqrt(comment_rate)*10+60
    Appendix A metric 21; arbitrary baseline and scaling to keep scores in 60-100 range.
  • Visual Harmony weights = hue 0.30, brightness 0.25, saturation 0.20, diversity 0.15, temperature 0.10
    Appendix A metric 11; hand-chosen blend.
  • Layout Sparsity grayscale tolerance = 80
    Appendix A metric 10; default threshold for blank-region detection.
  • Survey completion-time filter = >2 minutes
    Section 4.3; retains 141 of 899 responses; threshold chosen by authors.
  • Agent evaluation subset size = 165/1572
    Section 5.1; agents scored on ~10% subset but ranked against full-dataset models.
axioms (6)
  • domain assumption The LLM-generated, human-validated ground-truth checklists are complete and faithful.
    Section 3.1 and Appendix E.3; if checklists omit or hallucinate requirement points, the three alignment metrics (weights ~36%) are invalid.
  • domain assumption LLM-as-a-judge scores approximate human judgments for web app quality.
    Section 4 uses Gemini-2.5-pro and GPT-5-chat as judges; only run-to-run stability tested (Mann-Whitney U), not agreement with humans.
  • domain assumption Excluding unscorable cases does not bias per-metric averages.
    Appendix C states unscorable cases are dropped rather than scored zero; with >90% unscorable for metrics 8 and 13, the remaining subset may not represent the population.
  • domain assumption One week of anonymized traffic from one industrial partner represents the distribution of real-world web app requirements.
    Section 3.1.
  • domain assumption Native HTML-only artifacts cover most real user needs.
    Limitations: benchmark restricted to front-end native HTML; framework/backend requests filtered out, limiting generality.
  • standard math Standard statistical tools are valid at the stated sample sizes.
    Borda count (§4.3), z-scores (§4.3), MinHash/MiniLM dedup (§3.1) are standard.

pith-pipeline@v1.3.0-alltime-deepseek · 34603 in / 11888 out tokens · 109021 ms · 2026-08-03T12:41:22.149072+00:00 · methodology

0 comments
read the original abstract

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.

Figures

Figures reproduced from arXiv: 2601.02430 by Chenxu Liu, Tao Xie, Wei Yang, Yingjie Fu, Ying Zhang.

Figure 1
Figure 1. Figure 1: The dataset construction process of WebCoder [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example user requirement with its corre [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The weight assignment and evaluation work [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The weight proportion of each perspective [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The detailed raw scores of 24 evaluation met [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompts for Generating Classification Labels. [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompts for Generating Ground-Truth Checklists. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompts for Generating Ground-Truth Checklists Continue. [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompts for the Evaluation Metric of General Functionality Correctness. [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompts for the Evaluation Metric of General Functionality Correctness Continue. [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompts for the Evaluation Metric of General Visual Experience. [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompts for the Evaluation Metric of General Visual Experience Continue. [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompts for the Evaluation Metric of Functional Alignment. [PITH_FULL_IMAGE:figures/full_fig_p032_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompts for the Evaluation Metric of Visual Alignment. [PITH_FULL_IMAGE:figures/full_fig_p033_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompts for the Evaluation Metric of Content Alignment. [PITH_FULL_IMAGE:figures/full_fig_p034_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis

    cs.AI 2026-06 unverdicted novelty 7.0

    Introduces WorldCoder-Bench and StateProbe for evaluating LLM-generated physically grounded 3D browser worlds, with frontier models reaching at most 27.8% verification coverage.

  2. Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    Cookie-Bench is a reference-free 1,000-query web development benchmark paired with Cookie-Frame, a metacognition-inspired three-stage framework (static perception, agent interaction, dynamic scoring) that aligns with ...

  3. WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

    cs.AI 2026-05 unverdicted novelty 7.0

    WebGameBench is a benchmark that evaluates coding agents by having them generate browser-native games from specifications, then running those games in a real browser to assign EXCELLENT, USABLE, or UNUSABLE labels, wi...

  4. WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

    cs.AI 2026-05 unverdicted novelty 7.0

    WebGameBench is a new benchmark that evaluates coding agents on building browser-native games from frozen specifications, with runtime browser evaluation showing best agents reach 76.9% usable rate but only 20.2% exce...

  5. SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

    cs.MA 2026-05 conditional novelty 7.0

    SWE-WebDevBench finds that AI app builders commonly fail at translating business needs into complete, secure, production-ready software due to specification bottlenecks, frontend-backend decoupling, low engineering qu...

  6. I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applications

    cs.CL 2026-05 unverdicted novelty 5.0

    A Paper-to-Interactive-System Agent and I-WebGenBench benchmark with 19 papers enable converting scientific PDFs into executable interactive web systems, with PaperVoyager framework shown to improve quality.

  7. HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML

    cs.SE 2026-05 unverdicted novelty 5.0

    HTMLCure uses browser-executed interaction trajectories to diagnose and repair LLM HTML outputs, expanding 97K prompts into a 40K refined SFT set that lifts a 27B model to 50.6 on HTMLBench-400 and 81.2 on MiniAppBench.

Reference graph

Works this paper leans on

66 extracted references · 5 linked inside Pith · cited by 6 Pith papers

  1. [1]

    - Visual Quality: whether the webpage appears aes- thetically pleasing and professional, and whether it adopts appropriate color combinations

    When using a model to generate web applications, which perspectives do you care about the most? Please rank the following dimensions according to your level of impor- tance. - Visual Quality: whether the webpage appears aes- thetically pleasing and professional, and whether it adopts appropriate color combinations. - Code Quality: whether the webpage code...

  2. [2]

    - General Visual Experience: assessing the overall visual experience when using the webpage, emphasizing aesthetics, design quality, and consistency between style and content

    When using a model to generate web applications, which aspects of visual aesthetics do you care about the most? Please rank the following dimensions by importance. - General Visual Experience: assessing the overall visual experience when using the webpage, emphasizing aesthetics, design quality, and consistency between style and content. - Component Style...

  3. [3]

    LMArena.ai

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437. LMArena.ai. 2025. WebDev Arena. Lovable. 2025. Lovable. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2025. WebGen- Bench: Evaluating LLMs on generating interactive and functional websites from scratch.arXiv preprint...

    Pith/arXiv arXiv 2025

  4. [4]

    When using a model to generate web applications, which aspects of content and media resources do you care about the most? Please rank the following dimensions by impor- tance. - Copywriting Quality: assessing the overall quality of the text accuracy, clarity, brevity, contextual consistency, user-friendliness, and compliance with UX and industry standards...

  5. [5]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others

    Web-Bench: A LLM code benchmark based on web standards and frameworks.arXiv preprint arXiv:2505.07473. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others

    Pith/arXiv arXiv

  6. [6]

    Purpose” column describes the rationale of each metric and the aspect each metric aims to evaluate. The “Implementation De- tail

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. H Peyton Young. 1974. An axiomatization of Borda’s rule.Journal of economic theory. Sukmin Yun, Rusiru Thushara, Mohammad Bhat, Yongxin Wang, Mingkai Deng, Jinhong Wang, Tian- hua Tao, Junbo Li, Haonan Li, Preslav Nakov, and 1 others. 2024. Web2Code: A large-scale webpage- to-code dataset and evaluat...

    Pith/arXiv arXiv 1974

  7. [7]

    **Security**: Includes input validation, output encoding, prevention of injection attacks (XSS/CSRF), and dependency risk control to ensure the code is free from common security vulnerabilities. **Scoring Guide**: **10 points**: Input validation is complete, output encoding is strict, prevention against injection/XSS/CSRF vulnera- bilities is comprehensiv...

  8. [8]

    **Branch & State Coverage**: Ensures ‘if/else/switch/ternary‘ structures comprehensively cover critical paths and boundary cases, and handle early returns/exception branches; ensures proper management of variables, loading states, disabled states, error states, and empty states. **Scoring Guide**: **10 points**: All branch paths (including ‘if/else/switch...

  9. [9]

    - General Functionality Correctness: evaluating whether the implemented functions and business logic are correct

    When using a model to generate web applications, which aspects of code quality do you care about the most? Please rank the following dimensions by importance. - General Functionality Correctness: evaluating whether the implemented functions and business logic are correct. - Best Practices: evaluating adherence to web de- velopment best practices, includin...

  10. [10]

    score": [1,2,3,4,5,6,7,8,9,10],

    **Asynchronous Operations & Error Handling**: ‘fetch/Promise/async‘ and other asynchronous operations have complete error handling, timeout control, and are designed with fallback mechanisms and user-friendly error messages. **Scoring Guide**: **10 points**: All asynchronous operations (‘fetch/Promise/async‘) handle exceptions and timeouts, and have compl...

  11. [11]

    When using a model to generate web applications, which aspects of accessibility do you care about the most? Please rank the following dimensions by importance. - Accessibility Core Metrics: evaluating support for users with visual impairments and other groups, checking whether the readable description text is included, the key- board navigation is support...

  12. [12]

    not usable

    When using a model to generate web applications, which aspects of code maintainability do you care about the most? Please rank the following dimensions by importance. - Code Redundancy Rate: assessing how much un- necessary or unused code is present, leading to tedious code. - Comment Rate: evaluating whether natural-language comments are sufficient to ai...

  13. [13]

    infinite block map

    Ambiguous or Logically Incoherent Require- ments:The requirement is unclear or logically in- consistent. Examples include: • The user only uploads an HTML file without specifying any requirements. • Requesting an “infinite block map” in HTML without further explanation

  14. [14]

    Please generate a military training commemo- ration website with sections: 1. Title 2. Photo Wall (use only uploaded reference images, no external resources)

    Missing Supplementary Data:The user’s require- ment lacks essential supplementary materials, such as images or links required to fulfill the request. Ex- ample: • “Please generate a military training commemo- ration website with sections: 1. Title 2. Photo Wall (use only uploaded reference images, no external resources).” In this case, the required refere...

  15. [15]

    Example: • Using ESP32-S3 with a ST7789V display and FT6236U touch panel to create an interactive demo

    Non-native Web Scenarios or Non-Web Imple- mentation Languages / Frameworks:The require- ment specifies a context that is not native to standard web apps, or requires implementation in languages / frameworks outside of native HTML / CSS / JS. Example: • Using ESP32-S3 with a ST7789V display and FT6236U touch panel to create an interactive demo

  16. [16]

    Toilet Man

    Difficult-to-understand Requirements:The re- quirement cannot be quickly interpreted to identify the main functional requirements by an expert with front-end development experience. Examples in- clude: • Not suitable: “Toilet Man” (ambiguous). • Suitable: “Implement a Tetris game” or “Gen- erate a Bomberman-style mini-game.” E.3 Building Ground-Truth Chec...

  17. [17]

    Add XX functionality

    Determine — except for the first turn — whether each subsequent turn is: Functionality Addition (e.g., “Add XX functionality”), Functionality Fix (e.g., “Fix XX issue”), Non-functional Description (e.g., “Confirm requirement”, “Start generation”, “Continue”). Keep only the turns of the Functionality Addition type, and exclude all others

  18. [18]

    Merge the first turn with the subsequent turns that are Functionality Addition, keeping the original description intact as much as possible, including any typos, without altering the original content — only performing a simple merge

  19. [19]

    Between merged sentences, you may add or slightly modify a few words or sentences to make the text coherent and free of obvious merge traces

  20. [20]

    Only output the merged content

    If the original content contains JSON, only modify the value of the “text” field. Only output the merged content. Do not provide any expla- nations or additional text. Now, please merge based on the following multi-turn con- versation content: {User Requirements} F.2 Prompts for Generating Classification Labels To analyze the statistics of our dataset, we...

  21. [21]

    Your output should only include the code itself, with no additional explanations

  22. [22]

    Prompts for generating artifacts using LLM-based agents: You are a professional web front -end application engi- neer and designer

    You may only use native front -end languages (HTML, JS, CSS) to build the page. Prompts for generating artifacts using LLM-based agents: You are a professional web front -end application engi- neer and designer. You will receive user requirements for front-end web pages and write web page code to fulfill those requirements. ##Delivery Requirements

  23. [23]

    You must implement the requirements using only native front-end languages (HTML, JS, CSS)

  24. [24]

    If the implementation can be done in a single file, then you may deliver only one HTML file

  25. [25]

    If the implementation requires multiple files, then you may deliver only three types of files: HTML files, CSS files, and JS files

  26. [26]

    The use of frameworks such as React is strictly prohibited

  27. [27]

    Dimensions & Rubrics

    Only front -end functionality needs to be imple- mented; no database or backend connections are required. If backend-related functionality is involved, use mock data to simulate it. F.5 Prompts for Evaluation Metrics Among all of our 24 evaluation metrics, 5 of them follow the LLM-as-a-judge paradigm. We show their prompts in Figure 9 and 10 (General Func...

  28. [28]

    Assign a level from L1 to L5 for each

    Assess the following three sub-dimensions separately: Functional Complexity, Business Logic, User Interaction. Assign a level from L1 to L5 for each

  29. [29]

    Highest Level Principle

    Determine the final overall complexity level according to the “Highest Level Principle” — the sub-dimension with the highest level determines the final rating. Level Functional Complexity Business Logic User Interaction L1 Single, stateless functionality; no data storage. Linear direct logic (input→output). Minimal interface (single page; basic components...

  30. [30]

    Clarity: Choose from C1, C2, C3

  31. [31]

    Expression Style: Choose from S1, S2, S3, S4

  32. [32]

    application_complexity

    The final result must include both labels. **2.1 Clarity** C1: Clear & Specific — Explicit, detailed requirements akin to a small specifications document. C2: Goal-Oriented — Defines clear objectives but omits implementation details. C3: Vague & Exploratory — Expresses a broad idea or open-ended question only. **2.2 Expression Style** S1: Instructional/Te...

  33. [33]

    **Explicit mention first**: If requirement includes operational verbs (e.g., *upload, play, share*), directly split into requirement points

  34. [34]

    create a file sharing platform

    **Implicit completion**: If requirement is an abstract objective (*e.g., “create a file sharing platform”*), extract the minimal functional set to achieve it: - Upload files - Generate sharing link - Access link to download

  35. [35]

    points system

    **No divergence**: Do not infer features not mentioned (*e.g., “points system”, “admin dashboard”*)

  36. [36]

    **Consistent granularity**: Requirement point should be independently developable and testable (includes input, processing, output)

  37. [37]

    must use Vue framework

    Do not include programming language or framework requirements (*e.g., “must use Vue framework”*). **Example:** > “The system should be able to display product videos online.”→[Upload video], [Play video] > “Users can upload and share files.”→[Upload file], [Generate sharing link], [Download via link] — ### **Visual Dimension** (Visual) **Goal:** Identify ...

  38. [38]

    blue and white

    **Explicit mention first**: If requirement mentions colors or theme description (*e.g., “blue and white”, “industrial style”*), extract as requirement point

  39. [39]

    Focus on **theme colors, responsive layout, animation effects**

  40. [40]

    Do not mention basic UI elements (buttons, input boxes, tables, etc.) unless explicitly stated

  41. [41]

    Overall theme should be blue and white

    If mentions *style*, *brand colors*, *animation effects*, *adaptation for mobile/PC*, it is considered a visual element. **Example:** > “Overall theme should be blue and white”→[Theme color: blue & white] > “Interface must adapt for both mobile and desktop”→[Responsive layout] > “Page transitions must have fade-in/out effects”→[Animation: fade-in/out] — F...

  42. [42]

    today’s news

    **Explicit mention first**: If requirement lists specific media or content (*e.g., “today’s news”, “images”*), directly extract. If content (images, text, etc.) is provided by user, emphasize “provided by user”

  43. [43]

    Extract page language type, images, videos, audio, music, text copy, data sources, etc

  44. [44]

    Display company promotional video and background music

    Exclude logical text (*e.g., prompts, error messages, guiding instructions*). **Example:** > “Display company promotional video and background music.”→[Video: Company promo], [Music: Background track] — ## **[Output Format]** The output must be in JSON format (not Markdown JSON) with the following structure: { "functionals": [ { "type": "functional", "nam...

  45. [45]

    **Scoring Guide**: **10 points**: All business functions are fully implemented, logic aligns with business requirements, and static data conforms to scientific common sense

    **Functional Completeness & Business Logic**: Based on business requirements, ensure all functions are implemented without omission, the logic aligns with business specifications, and check that static data conforms to scientific and business common sense. **Scoring Guide**: **10 points**: All business functions are fully implemented, logic aligns with bu...

  46. [46]

    This includes checking value outputs and system calls, verifying for logical errors, missing, or duplicate output content

    **Output Validation**: Following the code execution flow, evaluate if the output is correct. This includes checking value outputs and system calls, verifying for logical errors, missing, or duplicate output content. It’s especially important to verify that UI updates and state changes reflect business logic changes. **Scoring Guide**: **10 points**: All o...

  47. [47]

    **Forms & Critical Path Flows**: Includes pre-validation, disabled states, protection against duplicate submissions, success/failure notifications, and redirects. Ensures important flows like payments and bookings are idempotent, have state rollback mechanisms, and provide clear error messages, covering industry constraints (e.g., time windows, quantity l...

  48. [48]

    Check the accuracy of data processing methods, avoiding hard-coded values or illogical data assumptions

    **Data Science Logic Validation**: Verify that static data and business logic within the code are reasonable, ensuring data conforms to scientific principles and industry standards. Check the accuracy of data processing methods, avoiding hard-coded values or illogical data assumptions. **Scoring Guide**: **10 points**: All static data is reasonable, and d...

  49. [49]

    **Scoring Guide**: **10 points**: List/card display is perfect

    **List/Card Display**: Check the state management and interactive behavior of list and card components, ensuring that empty data placeholders and loading skeletons are implemented correctly, and error states are handled effectively with user notifications. **Scoring Guide**: **10 points**: List/card display is perfect. Empty data placeholders, loading ske...

  50. [50]

    **Correctness & Boundary Conditions**: Covers all boundary conditions, null/type checks, ensures resources are released correctly, avoids concurrency/race condition issues, and ensures functionality remains reliable under various extreme circumstances. **Scoring Guide**: **10 points**: The function performs perfectly under all boundary conditions, correct...

  51. [53]

    **Data Consistency & Flow Management**: Ensures DOM updates are consistent with the state, avoids race conditions or dirty data issues caused by global variables and closures, and reduces data flow conflicts. **Scoring Guide**: **10 points**: Data flow management is perfect, DOM updates are always consistent with the state, and there are no race condition...

  52. [55]

    Trust your first intuition

  53. [56]

    Don’t try to rationalize why you like or dislike it

  54. [57]

    React authentically like a regular user

  55. [58]

    There are no right answers in aesthetic judgment; trust your own feelings

  56. [59]

    Provide both a sentimental evaluation and a rational analysis of key elements like style, hierarchy, and consistency

  57. [60]

    functional_requirement

    Remain objective and friendly; when pointing out issues, offer direction rather than criticism Figure 12: Prompts for the Evaluation Metric of General Visual Experience Continue. 20 Prompts for the evaluation metric of Functional Alignment: **Role-play:** You are a senior requirements evaluator, project manager, and user of web front-end applications. You...

  58. [61]

    myButton

    **Web page code:** <button id="myButton">Click me</button> <div id="message"></div> <script> document.getElementById("myButton").addEventListener("click", function() { document.getElementById("message").textContent = "The button was clicked!"; }); </script>

  59. [62]

    After clicking a button, the message ‘The button was clicked!’ will be displayed on the page

    **Requirement(s):** [“After clicking a button, the message ‘The button was clicked!’ will be displayed on the page.”] **Expected Output (JSON format):** Note: The final output should only contain the JSON content format, do not wrap it in a Markdown JSON block. [{ "functional_requirement": "After clicking a button, the message ’The button was clicked!’ is...

  60. [63]

    **Web page code:** .primary-button { background-color: #007bff; color: white; font-size: 16px; padding: 10px 20px; border-radius: 5px; }

  61. [64]

    A button with a blue background, white text, a font size of 16 pixels, and a 5-pixel border- radius

    **Requirement Points:** [“A button with a blue background, white text, a font size of 16 pixels, and a 5-pixel border- radius.”] **Expected Output (JSON format):** Note: The final output should only contain the JSON content, without being wrapped in Markdown’s JSON format. [{ "visual_requirement": "A button with a blue background, white text, a font size ...

  62. [65]

    logo.png

    **Code Snippet:** <div>Welcome</div> <img src="logo.png"> <a href="/home"></a>

  63. [66]

    Display a welcome text

    **Requirement Points:** ["Display a welcome text.", "Display a company logo, using logo.png.", "Provide a link to return to the homepage."] **Expected Output (JSON format):** Note: The final output should only contain the JSON content, do not wrap it in Markdown’s JSON format. [{ "content_requirement": "Display a welcome text.", "code_snippet": "<div>Welc...

  64. [2006]

    Anthropic

    Data mining: A preprocessing engine.Journal of Computer Science. Anthropic. 2025a. Claude Code. Anthropic. 2025b. Overview | Claude. Anysphere. 2025. Cursor. Design Arena. 2025. Design Arena: World’s largest crowdsourced benchmark for AI-generated design. Tony Beltramelli. 2018. Pix2Code: Generating code from a graphical user interface screenshot. InPro- ...

    Pith/arXiv arXiv 2025

  65. [2024]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others

    Vision2ui: A real-world dataset with layout for code generation from UI designs.arXiv preprint arXiv:2404.06369. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. DeepSeek- Coder: When the large language model meets pro- gramming – the rise of code intelligence.arXiv preprint ar...

    Pith/arXiv arXiv 2024

  66. [2025]

    Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng

    DesignBench: A comprehensive benchmark for MLLM-based front-end code generation.arXiv preprint arXiv:2506.06251. Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng

    arXiv