Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

Jie Tang; Mingdao Liu; Wenyi Hong; Xiaotao Gu; Zehai He; Zhen Yang; Ziyang Pan

REVIEW 4 major objections 4 minor 2 cited by

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision2Web grades AI website-building from a single page to a full-stack app, and finds every current model falls off sharply as tasks grow.

desk verdict Vision2Web is a genuinely useful benchmark for multimodal coding agents, but the headline rankings lean on a VLM judge whose validation is thinner than the paper claims. read the letter →

arxiv 2603.26648 v3 pith:7RUUPOAI submitted 2026-03-27 cs.SE cs.AI

Zehai He , Wenyi Hong , Zhen Yang , Ziyang Pan , Mingdao Liu , Xiaotao Gu , Jie Tang This is my paper · ORCID

classification cs.SEcs.AI

keywords benchmarkvisualwebsitedevelopmentUI-to-codegenerationfull-stackcodingagentsmultimodalevaluationworkflow-basedverificationfidelity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision2Web is an attempt to measure something existing benchmarks do not: whether AI coding agents can build real websites from visual prototypes, not just edit code or reproduce a single page. The paper organizes the task into three escalating levels — reproduce a static page from screenshots, build an interactive multi-page frontend, and deliver a full-stack site from a requirements document — covering 193 real-world websites with 918 prototype images and 1,255 test cases. To make end-to-end evaluation reproducible, it introduces a workflow-based agent verification paradigm in which a GUI agent executes expert-designed test workflows while a vision-language judge scores visual fidelity against prototypes. Running eight state-of-the-art models under two coding-agent frameworks, the paper finds substantial, consistent performance gaps: scores drop by tens of points from static to full-stack tasks, and even the strongest model stays below the halfway mark on full-stack visual fidelity. The paper offers this as the field's common yardstick — a reproducible way to compare agents — along with a diagnosis of what is missing: long-horizon planning, cross-page coordination, and persistent state reasoning.

What carries the argument

The central mechanism is the workflow-based agent verification paradigm. End-to-end testing is formalized as a directed dependency graph whose nodes are self-contained verification sub-procedures — interactions that bring the app to a target state, then a check — instantiated as agent-executable test workflows covering all nodes. Two complementary verifiers operate at the nodes: a GUI agent checks functional correctness from a tuple of objective, guided actions, and validation criteria (the guided actions constrain the agent, preventing stray exploration), and a vision-language judge scores visual fidelity by segmenting the rendered page into logical blocks and rating each against the protot

What would settle it

Recompute all Visual Scores using a different judge model (for example, a strong open-weights vision-language model or human ratings on a stratified sample), then compare per-model scores and rankings with the published ones. If the two judges disagree on model ordering, or if each model scores systematically higher when it is the judge, the paper's central comparison is not stable. A cheaper first check: measure the judge's agreement with humans on the full benchmark rather than the 100 sampled prototypes — if average Spearman correlation falls near or below 0.5, the visual scores are too noi

Watch

Extended reading notes

Core claim

The paper's central claim is that visual website development can be evaluated end-to-end, and doing so exposes a capability cliff. Vision2Web contains 193 real-world websites in three levels — static UI-to-code (100 tasks), interactive frontends (66), and full-stack construction (27) — so failures can be attributed to a stage. Evaluation uses workflow-based agent verification: a GUI agent checks functional correctness on expert-designed workflows, while a vision-language judge scores visual fidelity against prototypes. Every model degrades sharply as complexity grows, with state management, CRUD, and file handling the weakest areas. Conclusion: strong isolated-task performance does not trans

Load-bearing premise

The load-bearing premise is that the vision-language judge's visual scores are trustworthy: if the judge is biased toward its own outputs or loses agreement with human raters beyond the 100-prototype validation sample (0.66 correlation, below the 0.78 human-human agreement), every model ranking and every claimed capability gap in the paper comes undone.

Editorial extensions

If this is right

If the benchmark's verdict holds, the field has a reproducible, implementation-agnostic yardstick for comparing coding agents across the whole software lifecycle, not just isolated edits or single pages.
The consistent device-dependent drop — tablet and mobile scores run 10–20% below desktop — implies that visual grounding in current agents is fragile enough to matter in responsive design.
The weakest functional categories — state management, CRUD operations, file and media handling — all require persistent state tracking; the paper's findings point to state reasoning as a central bottleneck.
Because framework choice materially changed most models' results, future progress will likely come from optimizing models and their agent frameworks jointly rather than either alone.
The category analysis ranks SaaS platforms as the hardest real-world target, so an agent that handles multi-page, interaction-heavy SaaS builds would be a meaningful milestone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the reported rankings rest on visual scores from a single judge model that is itself one of the evaluated models, validated on only 100 sampled prototypes (Spearman correlation 0.66 vs human, below the human inter-annotator agreement of 0.78). Those rankings should be treated as provisional until judge agreement is measured across the full 193-task benchmark.
A direct test the paper leaves implicit: swap the judge backbone and re-score all outputs. If rankings shift materially, or if each model scores higher when it judges itself, self-preference bias is present and the headline comparisons need re-weighting.
The workflow abstraction is not limited to websites; the same dependency-graph-plus-guided-actions recipe could be applied to desktop or mobile application testing, where reproducibility is equally hard.
The paper's failure analysis blames full-stack breakage partly on agents lacking self-verification; this implies a testable remedy — giving agents the same verifier tools during development might narrow the gap between isolated and end-to-end performance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this paper deserves a serious referee, but it has one real soft spot you should know about before citing its numbers. Vision2Web is a solid new benchmark for end-to-end visual website development, with a three-level hierarchy (static, interactive, full-stack), 193 tasks built from real sites, and a workflow-based verification scheme that pairs a GUI agent for functional tests with a VLM judge for visual fidelity. That combination is genuinely new, and the paper does a good job of documenting dataset construction, prompts, and the agent frameworks. The task design sensibly disentangles capabilities, and the experiments span eight models and two frameworks. The GUI verifier validation (87.2% node-level agreement with human annotations on a sample) is credible.

The load-bearing weakness is the VLM judge. Section 4.4 validates it on 100 prototypes with an average Spearman correlation of 0.66 against human preferences, below the human inter-annotator agreement of 0.78. Spearman measures rank-order agreement only; it says nothing about whether the judge's absolute scores or score differences are calibrated. Yet Table 3 and the main findings use average score gaps (desktop 63.3 vs mobile 48.3, full-stack VS 11.7) as quantitative evidence. The conflict is compounded because the judge is Gemini-3-Pro-Preview, the same model as one of the top evaluated agents. No analysis rules out self-preference. This should be fixed or at least discussed; the main direction (performance drops with complexity and device size) would likely survive, but the cross-model rankings are not clearly unbiased.

Two smaller issues: Table 3 lists GPT-5's static average under OpenHands as 49.6, but (49.0+44.6+40.5)/3 is 44.7. That's a plain arithmetic error and it makes you double-check every other number. And the "contamination-free" claim oversells what sourcing from C4 guarantees; the dataset also lacks a committed release hash. Both are minor and fixable.

My read: this is a useful contribution, not a flawless one. The workflow-based verification paradigm is worth adopting as a template, and the hierarchical task design fills a real gap between single-page UI benchmarks and full-stack agent evaluation. The judge validation is the part to press the authors on. I'd definitely send it to peer review, and I'd ask for a response on the self-preference issue and a corrected Table 3 before trusting the specific rankings.

Referee Report

4 major / 4 minor

Summary. The paper introduces Vision2Web, a three-level benchmark for visual website development: static UI-to-code reproduction, interactive multi-page frontend generation, and full-stack website construction. The benchmark contains 193 tasks, 918 prototype images, and 1,255 test cases built from C4 validation-set websites. Evaluation uses a workflow-based agent verification paradigm with two components: a GUI agent verifier for functional correctness and a VLM-based judge for visual fidelity. Eight multimodal models are evaluated under two coding-agent frameworks, yielding findings about performance drops across task levels, device form factors, model families, and frameworks. The central claim is that Vision2Web provides a systematic, reproducible, implementation-agnostic evaluation of visual website development agents.

Significance. If the evaluation methodology is trustworthy, this is a valuable contribution: it moves beyond static UI-to-code benchmarks and text-only end-to-end benchmarks, provides realistic hierarchical tasks with expert-annotated workflows, and includes human validation of both verifiers. The dataset construction pipeline, task taxonomy, and workflow formalization are concrete strengths that would make the benchmark broadly useful. However, the reliability of the VLM judge is not established at the level needed to support the headline quantitative comparisons. Because the judge is also one of the evaluated models and is validated only by rank correlation on 100 prototypes, the reported visual scores, model rankings, and Findings 1–5 must be regarded as provisional until calibration and bias analyses are added.

major comments (4)

[§4.4, Table 3] The VLM judge is validated only by average Spearman rank correlation ρ=0.66 (median 0.80) against human judgments on 100 sampled prototypes, with human inter-annotator agreement 0.78. Spearman measures rank consistency and is invariant to monotone rescoring; it does not validate the absolute Visual Scores or score differences used throughout Table 3 and Findings 1–3, e.g., desktop 63.3 vs. mobile 48.3 for Gemini-3-Pro-Preview, or full-stack VS 11.7. A judge can rank pages correctly while systematically over-scoring one model family or compressing/expanding score gaps. Please report calibration/absolute-error metrics, per-model agreement with human scores, and at least one held-out sample covering outputs from all eight evaluated agents.
[§4.1, §4.4] The VLM judge is instantiated with Gemini-3-Pro-Preview, which is also one of the top evaluated agents. All visual scores for Gemini-3-Pro-Preview are therefore produced by the same model being scored, creating a self-preference risk that is not addressed by the current validation. The 100-sample rank-correlation check does not test whether the judge assigns higher scores to its own outputs than human raters would. Please add a self-preference analysis comparing judge-vs-human agreement on Gemini outputs versus other models' outputs, or rerun scoring with an independent judge model for a robustness check.
[Table 3] There is a clear numerical error in the static webpage averages for GPT-5 under OpenHands: the row reports Average 49.6, but (49.0 + 44.6 + 40.5)/3 = 44.7. The same inconsistency appears in the GPT-5 row's placement among the other OpenHands averages. This is a central results table, so all averaged columns should be recomputed and verified before acceptance; similar rounding or arithmetic checks are needed for the full-stack and frontend averages.
[§3.2, §4.4] The GUI agent verifier validation reports node-level agreement of 87.2% on 250 sampled nodes, but gives no confidence intervals, no breakdown by task level, and no error taxonomy. Since Functional Scores drive Findings 3, 5, and 6, the statement that 'residual inaccuracies are largely attributable to model-intrinsic reasoning hallucinations' is unsupported without such an analysis. Please quantify failure modes and report per-level or per-category agreement so the reader can judge where functional scores are most reliable.

minor comments (4)

[Figure 8 caption] Typo: 'Interative frontend task example' should be 'Interactive frontend task example'.
[§2.3] 'comprising a total of 21516 input files' is hard to parse; format as 21,516 and verify whether this count is consistent with the sum of prototype images, test cases, and resource files.
[Table 2] The header 'Avg Text Tokens(103)' should be rendered with a superscript or explanatory note (e.g., ×10³) to avoid confusion with the value 103.
[Findings 1 and 3] Finding 1 says Gemini-3-Pro-Preview is strongest on static webpages under OpenHands, while Finding 3 says Claude-Opus-4.5 consistently achieves the strongest overall performance. These are not contradictory, but they should be explicitly qualified by task level and framework to avoid apparent tension.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark construction and agent verification are empirically grounded, not derived from their own outputs.

full rationale

Vision2Web is an empirical benchmark paper rather than a derivation chain, so there is no load-bearing step that reduces to its own inputs. The task hierarchy is a task-formulation choice; the dataset is constructed from C4 with filtering and manual review; functional and visual scores are measurements produced by a GUI agent and a VLM judge, not fitted parameters. The VLM judge's visual scores are externally anchored by validation against human annotations (average Spearman 0.66 vs. human inter-annotator 0.78), so the central evaluation claim does not reduce to the judge's own definition. The fact that Gemini-3-Pro-Preview serves both as an evaluated agent and as the judge is a legitimate validity/conflict concern, but it is not a circularity: the paper does not define the score in terms of the model's self-preference, and the reported rankings are not derived by construction from the benchmark's inputs. Similarly, the use of GLM-4.6V as the GUI agent verifier is an implementation choice citing prior work by overlapping authors, but the verifier's correctness is independently assessed against human annotations and is not the basis of the benchmark's main claims. All headline findings are empirical observations from the evaluation runs, so no self-definitional, fitted-prediction, or self-citation-load-bearing circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a chain of domain assumptions about the data source, filtering pipeline, and verifier reliability. The paper provides partial validation for the verifiers but not for the data-contamination assumption or the filtering thresholds. No new physical or theoretical entities are introduced.

free parameters (3)

Structural assessment thresholds
Hand-chosen DOM depth, tag count, and token length cutoffs in Section 2.2 filter C4 candidates to 63,515 websites; these are design choices, not fit to a target result.
Content screening VLM threshold
VLM-based scoring retains 7,391 pages in Section 2.2; the pass threshold is manually set and not externally validated.
VLM judge scoring rubric = 0 / 0.25 / 0.5 / 0.75 / 1
Appendix A.2.2 defines a discrete component score scale; this is a modeling choice by the authors to quantify visual fidelity.

assumptions (4)

domain assumption C4 validation set is representative and contamination-free
Section 2.2 sources all tasks from C4 validation set to avoid leakage, but C4 is a public Common Crawl snapshot likely present in LLM training corpora; absence of a specific contamination audit is an unverified assumption.
domain assumption VLM-based scoring can reliably filter content quality
The second filtering stage (§2.2) uses a VLM to score functional richness and visual coherence; the paper provides no validation of this filter's accuracy against human judgment.
domain assumption GUI agent and VLM judge provide reliable verification
Section 4.4 validates on a sample: 87.2% node agreement for the GUI agent and Spearman 0.66 for the VLM judge, but full-benchmark reliability is not established.
domain assumption Human annotations are ground truth
Verifier validation treats human annotator judgments as the reference; inter-annotator agreement for visual preferences is only 0.78, leaving ambiguity in the ground truth.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification." pith.science (2026). https://pith.science/paper/7RUUPOAI

@misc{pith2026260326648,
  author       = {Pith},
  title        = {Pith review of: Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RUUPOAI}},
  note         = {Machine review of arXiv:2603.26648}
}

read the original abstract

Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited. To address this gap, we introduce Vision2Web, a hierarchical benchmark for visual website development, spanning from static UI-to-code generation, interactive multi-page frontend reproduction, to long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. To support flexible, thorough and reliable evaluation, we propose workflow-based agent verification paradigm based on two complementary components: a GUI agent verifier and a VLM-based judge. We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels, with state-of-the-art models still struggling on full-stack development.

Figures

Figures reproduced from arXiv: 2603.26648 by the authors.

**Figure 1.** Overview of Vision2Web, a hierarchical benchmark for visual website development. Tasks span three levels—static webpages, interactive frontends, and full-stack websites—requiring agents to integrate visual prototypes with textual specifications. Evaluation is performed via a workflow-based agent verification paradigm, measuring functional correctness and visual fidelity [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Task distribution of Vision2Web across four major categories and 16 subcategories [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distribution of test cases across website-level tasks in Vision2Web. 4. Experiments Employing Vision2Web, we evaluate state-of-the-art multimodal models across coding agent frameworks and task levels to reveal their limitations in visual website development. 4.1. Settings We evaluate eight state-of-the-art multimodal models, including Claude-Opus-4.5, Claude-Sonnet-4.5 (Anthropic, a), Gemini-3-Pro-Preview, Gemini… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Distribution of Visual Scores (VS) across prototype heights for representative models under the OpenHands framework. Finding 2: Agent performance systematically degrades when adapting to smaller device form factors and more visually complex prototype images. Notably,…

**Figure 6.** Figure 6: Comparison of task complexity metrics between Design2Code-Hard and Webpage tasks of Vision2Web. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Cross-device responsive static webpage task example. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Interative frontend task example. Simplified Textual Description Example for Interactive Frontend Tasks I want to build a Blackbelt AC & Electrical website, a multi-page service company site with a global navigation bar (Home, About Us, Services with dropdown, Reviews,…

**Figure 9.** Figure 9: Full-stack website task example. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: failure cases of static webpage tasks (a) prototype:blog (b) claude-opus-4.5 (c) prototype:calendar (d) claude-opus-4.5 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: failure cases of interactive frontend tasks 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: failure cases of full-stack website tasks 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification
cs.CV 2026-08 conditional novelty 7.0 of 10

MT-Web2Code is a 102-page, 16-domain multi-turn benchmark that measures how coding agents reconstruct missing web regions and fix localized defects while preserving the surrounding page.
JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents
cs.CV 2026-07 conditional novelty 5.0 of 10

JarvisHub open-sources a three-layer canvas-state, protocol-bridge, and agent-runtime harness so multimodal creative agents can inspect and update a shared editable project graph over long workflows.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith · cited by 2 Pith papers

[1]

Product Background The Airbnb Community Center is a forum for hosts and community members to exchange experiences and seek advice on topics like property management, booking support, and hosting services
[2]

Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons

Business Workflows 2.1 Page Navigation Flow Pages: Home, Login/Signup, Topic, Post Detail, Post Creation, User Profile. Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons. Supports login-dependent and unauthenticated flows. 2.2 Core Workflows •User Authentication:Login via phone number→Validate→Return home •Browse...
[3]

Workflow-Based Agent Verification Implementation Details A.2.1

Requirements Specification 3.1 User Authentication • Phone number login with validation • Login state persistence and avatar display 3.2 Home Portal • Top nav with logo, search, Post button, language toggle • Featured conversations & popular post list • Sidebar category navigation • Post creation button with permission control 3.3 Topic Category • Breadcr...

1920
[4]

Actions:

Click the Post button in the navigation bar Validations:None •Prototype:post • Objective:Verify that the post creation form accepts valid input for title, topic selection, and sub-topic selection fields. Actions:
[5]

Click the Log in button in the navigation bar
[6]

12345678

Type “12345678” into the Phone number field
[7]

Click the Continue button
[9]

Test topic

Type “Test topic” into the Enter the topic here field

Show all 15 references

[10]

Select “Homes” from the topic dropdown
[11]

Advice on your space

Click the sub-topic card labeled “Advice on your space” Validations: –The title field displays “Test topic” –The topic dropdown displays “Homes” –The sub-topic card “Advice on your space” is visually selected • Objective:Verify that post submission fails when the content field...
[12]

Actions:

Click the Submit button Validations: –Post submission fails and does not proceed –An error message is displayed indicating that the content field is required • Objective:Verify that post creation succeeds when all required fields are filled with valid data, and the user is nav...
[13]

It’s a test content

Type “It’s a test content” into the content field
[14]

Test topic

Click the Submit button Validations: –Post submission succeeds 15 Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification –The page navigates to the newly created post detail page –The post title displays “Test topic” –The post content displ...

2025
[15]

/bin/bash

Product Requirement Document( /workspace/prd.md): Contains overview, business logic, and detailed requirements. 2.Prototype Images(provided above, not from files): Define UI layout, style, interactions, and visual fidelity. 3.Resource Files(/workspace/resources/ **/*): Assets ...
[2023]

Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al

GitHub repository. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al. A survey on llm-as-a- judge.The Innovation, 2024. Gui, Y ., Li, Z., Wan, Y ., Shi, Y ., Zhang, H., Su, Y ., Dong, S., Zhou, X., and Jiang, W. Vision2ui: A real-...

2024 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Product Background The Airbnb Community Center is a forum for hosts and community members to exchange experiences and seek advice on topics like property management, booking support, and hosting services

[2] [2]

Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons

Business Workflows 2.1 Page Navigation Flow Pages: Home, Login/Signup, Topic, Post Detail, Post Creation, User Profile. Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons. Supports login-dependent and unauthenticated flows. 2.2 Core Workflows •User Authentication:Login via phone number→Validate→Return home •Browse...

[3] [3]

Workflow-Based Agent Verification Implementation Details A.2.1

Requirements Specification 3.1 User Authentication • Phone number login with validation • Login state persistence and avatar display 3.2 Home Portal • Top nav with logo, search, Post button, language toggle • Featured conversations & popular post list • Sidebar category navigation • Post creation button with permission control 3.3 Topic Category • Breadcr...

1920

[4] [4]

Actions:

Click the Post button in the navigation bar Validations:None •Prototype:post • Objective:Verify that the post creation form accepts valid input for title, topic selection, and sub-topic selection fields. Actions:

[5] [5]

Click the Log in button in the navigation bar

[6] [6]

12345678

Type “12345678” into the Phone number field

[7] [7]

Click the Continue button

[8] [9]

Test topic

Type “Test topic” into the Enter the topic here field