Pith. sign in

REVIEW 4 major objections 4 minor 8 cited by

Vision2Web grades AI website-building from a single page to a full-stack app, and finds every current model falls off sharply as tasks grow.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:13 UTC pith:7RUUPOAI

load-bearing objection Vision2Web is a genuinely useful benchmark for multimodal coding agents, but the headline rankings lean on a VLM judge whose validation is thinner than the paper claims. the 4 major comments →

arxiv 2603.26648 v3 pith:7RUUPOAI submitted 2026-03-27 cs.SE cs.AI

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

Zehai He , Wenyi Hong , Zhen Yang , Ziyang Pan , Mingdao Liu , Xiaotao Gu , Jie Tang This is my paper
classification cs.SE cs.AI
keywords benchmarkvisual website developmentUI-to-code generationfull-stack developmentcoding agentsmultimodal evaluationworkflow-based verificationvisual fidelity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision2Web is an attempt to measure something existing benchmarks do not: whether AI coding agents can build real websites from visual prototypes, not just edit code or reproduce a single page. The paper organizes the task into three escalating levels — reproduce a static page from screenshots, build an interactive multi-page frontend, and deliver a full-stack site from a requirements document — covering 193 real-world websites with 918 prototype images and 1,255 test cases. To make end-to-end evaluation reproducible, it introduces a workflow-based agent verification paradigm in which a GUI agent executes expert-designed test workflows while a vision-language judge scores visual fidelity against prototypes. Running eight state-of-the-art models under two coding-agent frameworks, the paper finds substantial, consistent performance gaps: scores drop by tens of points from static to full-stack tasks, and even the strongest model stays below the halfway mark on full-stack visual fidelity. The paper offers this as the field's common yardstick — a reproducible way to compare agents — along with a diagnosis of what is missing: long-horizon planning, cross-page coordination, and persistent state reasoning.

Core claim

The paper's central claim is that visual website development can be evaluated end-to-end, and doing so exposes a capability cliff. Vision2Web contains 193 real-world websites in three levels — static UI-to-code (100 tasks), interactive frontends (66), and full-stack construction (27) — so failures can be attributed to a stage. Evaluation uses workflow-based agent verification: a GUI agent checks functional correctness on expert-designed workflows, while a vision-language judge scores visual fidelity against prototypes. Every model degrades sharply as complexity grows, with state management, CRUD, and file handling the weakest areas. Conclusion: strong isolated-task performance does not trans

What carries the argument

The central mechanism is the workflow-based agent verification paradigm. End-to-end testing is formalized as a directed dependency graph whose nodes are self-contained verification sub-procedures — interactions that bring the app to a target state, then a check — instantiated as agent-executable test workflows covering all nodes. Two complementary verifiers operate at the nodes: a GUI agent checks functional correctness from a tuple of objective, guided actions, and validation criteria (the guided actions constrain the agent, preventing stray exploration), and a vision-language judge scores visual fidelity by segmenting the rendered page into logical blocks and rating each against the protot

Load-bearing premise

The load-bearing premise is that the vision-language judge's visual scores are trustworthy: if the judge is biased toward its own outputs or loses agreement with human raters beyond the 100-prototype validation sample (0.66 correlation, below the 0.78 human-human agreement), every model ranking and every claimed capability gap in the paper comes undone.

What would settle it

Recompute all Visual Scores using a different judge model (for example, a strong open-weights vision-language model or human ratings on a stratified sample), then compare per-model scores and rankings with the published ones. If the two judges disagree on model ordering, or if each model scores systematically higher when it is the judge, the paper's central comparison is not stable. A cheaper first check: measure the judge's agreement with humans on the full benchmark rather than the 100 sampled prototypes — if average Spearman correlation falls near or below 0.5, the visual scores are too noi

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark's verdict holds, the field has a reproducible, implementation-agnostic yardstick for comparing coding agents across the whole software lifecycle, not just isolated edits or single pages.
  • The consistent device-dependent drop — tablet and mobile scores run 10–20% below desktop — implies that visual grounding in current agents is fragile enough to matter in responsive design.
  • The weakest functional categories — state management, CRUD operations, file and media handling — all require persistent state tracking; the paper's findings point to state reasoning as a central bottleneck.
  • Because framework choice materially changed most models' results, future progress will likely come from optimizing models and their agent frameworks jointly rather than either alone.
  • The category analysis ranks SaaS platforms as the hardest real-world target, so an agent that handles multi-page, interaction-heavy SaaS builds would be a meaningful milestone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported rankings rest on visual scores from a single judge model that is itself one of the evaluated models, validated on only 100 sampled prototypes (Spearman correlation 0.66 vs human, below the human inter-annotator agreement of 0.78). Those rankings should be treated as provisional until judge agreement is measured across the full 193-task benchmark.
  • A direct test the paper leaves implicit: swap the judge backbone and re-score all outputs. If rankings shift materially, or if each model scores higher when it judges itself, self-preference bias is present and the headline comparisons need re-weighting.
  • The workflow abstraction is not limited to websites; the same dependency-graph-plus-guided-actions recipe could be applied to desktop or mobile application testing, where reproducibility is equally hard.
  • The paper's failure analysis blames full-stack breakage partly on agents lacking self-verification; this implies a testable remedy — giving agents the same verifier tools during development might narrow the gap between isolated and end-to-end performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Vision2Web, a three-level benchmark for visual website development: static UI-to-code reproduction, interactive multi-page frontend generation, and full-stack website construction. The benchmark contains 193 tasks, 918 prototype images, and 1,255 test cases built from C4 validation-set websites. Evaluation uses a workflow-based agent verification paradigm with two components: a GUI agent verifier for functional correctness and a VLM-based judge for visual fidelity. Eight multimodal models are evaluated under two coding-agent frameworks, yielding findings about performance drops across task levels, device form factors, model families, and frameworks. The central claim is that Vision2Web provides a systematic, reproducible, implementation-agnostic evaluation of visual website development agents.

Significance. If the evaluation methodology is trustworthy, this is a valuable contribution: it moves beyond static UI-to-code benchmarks and text-only end-to-end benchmarks, provides realistic hierarchical tasks with expert-annotated workflows, and includes human validation of both verifiers. The dataset construction pipeline, task taxonomy, and workflow formalization are concrete strengths that would make the benchmark broadly useful. However, the reliability of the VLM judge is not established at the level needed to support the headline quantitative comparisons. Because the judge is also one of the evaluated models and is validated only by rank correlation on 100 prototypes, the reported visual scores, model rankings, and Findings 1–5 must be regarded as provisional until calibration and bias analyses are added.

major comments (4)
  1. [§4.4, Table 3] The VLM judge is validated only by average Spearman rank correlation ρ=0.66 (median 0.80) against human judgments on 100 sampled prototypes, with human inter-annotator agreement 0.78. Spearman measures rank consistency and is invariant to monotone rescoring; it does not validate the absolute Visual Scores or score differences used throughout Table 3 and Findings 1–3, e.g., desktop 63.3 vs. mobile 48.3 for Gemini-3-Pro-Preview, or full-stack VS 11.7. A judge can rank pages correctly while systematically over-scoring one model family or compressing/expanding score gaps. Please report calibration/absolute-error metrics, per-model agreement with human scores, and at least one held-out sample covering outputs from all eight evaluated agents.
  2. [§4.1, §4.4] The VLM judge is instantiated with Gemini-3-Pro-Preview, which is also one of the top evaluated agents. All visual scores for Gemini-3-Pro-Preview are therefore produced by the same model being scored, creating a self-preference risk that is not addressed by the current validation. The 100-sample rank-correlation check does not test whether the judge assigns higher scores to its own outputs than human raters would. Please add a self-preference analysis comparing judge-vs-human agreement on Gemini outputs versus other models' outputs, or rerun scoring with an independent judge model for a robustness check.
  3. [Table 3] There is a clear numerical error in the static webpage averages for GPT-5 under OpenHands: the row reports Average 49.6, but (49.0 + 44.6 + 40.5)/3 = 44.7. The same inconsistency appears in the GPT-5 row's placement among the other OpenHands averages. This is a central results table, so all averaged columns should be recomputed and verified before acceptance; similar rounding or arithmetic checks are needed for the full-stack and frontend averages.
  4. [§3.2, §4.4] The GUI agent verifier validation reports node-level agreement of 87.2% on 250 sampled nodes, but gives no confidence intervals, no breakdown by task level, and no error taxonomy. Since Functional Scores drive Findings 3, 5, and 6, the statement that 'residual inaccuracies are largely attributable to model-intrinsic reasoning hallucinations' is unsupported without such an analysis. Please quantify failure modes and report per-level or per-category agreement so the reader can judge where functional scores are most reliable.
minor comments (4)
  1. [Figure 8 caption] Typo: 'Interative frontend task example' should be 'Interactive frontend task example'.
  2. [§2.3] 'comprising a total of 21516 input files' is hard to parse; format as 21,516 and verify whether this count is consistent with the sum of prototype images, test cases, and resource files.
  3. [Table 2] The header 'Avg Text Tokens(103)' should be rendered with a superscript or explanatory note (e.g., ×10³) to avoid confusion with the value 103.
  4. [Findings 1 and 3] Finding 1 says Gemini-3-Pro-Preview is strongest on static webpages under OpenHands, while Finding 3 says Claude-Opus-4.5 consistently achieves the strongest overall performance. These are not contradictory, but they should be explicitly qualified by task level and framework to avoid apparent tension.

Circularity Check

0 steps flagged

No significant circularity: benchmark construction and agent verification are empirically grounded, not derived from their own outputs.

full rationale

Vision2Web is an empirical benchmark paper rather than a derivation chain, so there is no load-bearing step that reduces to its own inputs. The task hierarchy is a task-formulation choice; the dataset is constructed from C4 with filtering and manual review; functional and visual scores are measurements produced by a GUI agent and a VLM judge, not fitted parameters. The VLM judge's visual scores are externally anchored by validation against human annotations (average Spearman 0.66 vs. human inter-annotator 0.78), so the central evaluation claim does not reduce to the judge's own definition. The fact that Gemini-3-Pro-Preview serves both as an evaluated agent and as the judge is a legitimate validity/conflict concern, but it is not a circularity: the paper does not define the score in terms of the model's self-preference, and the reported rankings are not derived by construction from the benchmark's inputs. Similarly, the use of GLM-4.6V as the GUI agent verifier is an implementation choice citing prior work by overlapping authors, but the verifier's correctness is independently assessed against human annotations and is not the basis of the benchmark's main claims. All headline findings are empirical observations from the evaluation runs, so no self-definitional, fitted-prediction, or self-citation-load-bearing circularity is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on a chain of domain assumptions about the data source, filtering pipeline, and verifier reliability. The paper provides partial validation for the verifiers but not for the data-contamination assumption or the filtering thresholds. No new physical or theoretical entities are introduced.

free parameters (3)
  • Structural assessment thresholds
    Hand-chosen DOM depth, tag count, and token length cutoffs in Section 2.2 filter C4 candidates to 63,515 websites; these are design choices, not fit to a target result.
  • Content screening VLM threshold
    VLM-based scoring retains 7,391 pages in Section 2.2; the pass threshold is manually set and not externally validated.
  • VLM judge scoring rubric = 0 / 0.25 / 0.5 / 0.75 / 1
    Appendix A.2.2 defines a discrete component score scale; this is a modeling choice by the authors to quantify visual fidelity.
axioms (4)
  • domain assumption C4 validation set is representative and contamination-free
    Section 2.2 sources all tasks from C4 validation set to avoid leakage, but C4 is a public Common Crawl snapshot likely present in LLM training corpora; absence of a specific contamination audit is an unverified assumption.
  • domain assumption VLM-based scoring can reliably filter content quality
    The second filtering stage (§2.2) uses a VLM to score functional richness and visual coherence; the paper provides no validation of this filter's accuracy against human judgment.
  • domain assumption GUI agent and VLM judge provide reliable verification
    Section 4.4 validates on a sample: 87.2% node agreement for the GUI agent and Spearman 0.66 for the VLM judge, but full-benchmark reliability is not established.
  • domain assumption Human annotations are ground truth
    Verifier validation treats human annotator judgments as the reference; inter-annotator agreement for visual preferences is only 0.78, leaving ambiguity in the ground truth.

pith-pipeline@v1.3.0-alltime-deepseek · 15360 in / 10207 out tokens · 92583 ms · 2026-08-02T17:13:27.163040+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited. To address this gap, we introduce Vision2Web, a hierarchical benchmark for visual website development, spanning from static UI-to-code generation, interactive multi-page frontend reproduction, to long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. To support flexible, thorough and reliable evaluation, we propose workflow-based agent verification paradigm based on two complementary components: a GUI agent verifier and a VLM-based judge. We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels, with state-of-the-art models still struggling on full-stack development.

Figures

Figures reproduced from arXiv: 2603.26648 by Jie Tang, Mingdao Liu, Wenyi Hong, Xiaotao Gu, Zehai He, Zhen Yang, Ziyang Pan.

Figure 1
Figure 1. Figure 1: Overview of Vision2Web, a hierarchical benchmark for visual website development. Tasks span three levels—static webpages, interactive frontends, and full-stack websites—requiring agents to integrate visual prototypes with textual specifications. Evaluation is performed via a workflow-based agent verification paradigm, measuring functional correctness and visual fidelity [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 2
Figure 2. Figure 2: Task distribution of Vision2Web across four major cate￾gories and 16 subcategories [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of test cases across website-level tasks in Vision2Web. 4. Experiments Employing Vision2Web, we evaluate state-of-the-art mul￾timodal models across coding agent frameworks and task levels to reveal their limitations in visual website develop￾ment. 4.1. Settings We evaluate eight state-of-the-art multimodal models, in￾cluding Claude-Opus-4.5, Claude-Sonnet-4.5 (Anthropic, a), Gemini-3-Pro-Previ… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of Visual Scores (VS) across prototype heights for representative models under the OpenHands frame￾work. Finding 2: Agent performance systematically degrades when adapting to smaller device form factors and more vi￾sually complex prototype images. Notably, static webpage tasks reveal consistent device￾dependent drops: desktop interfaces achieve the highest fidelity, while tablet and mobile lay… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of task complexity metrics between Design2Code-Hard and Webpage tasks of Vision2Web. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cross-device responsive static webpage task example. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Interative frontend task example. Simplified Textual Description Example for Interactive Frontend Tasks I want to build a Blackbelt AC & Electrical website, a multi-page service company site with a global navigation bar (Home, About Us, Services with dropdown, Reviews, Blog, Contact Us) and footer quick links. The homepage features a hero section with call-to-action buttons and service showcase cards, whil… view at source ↗
Figure 9
Figure 9. Figure 9: Full-stack website task example. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: failure cases of static webpage tasks (a) prototype:blog (b) claude-opus-4.5 (c) prototype:calendar (d) claude-opus-4.5 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: failure cases of interactive frontend tasks 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: failure cases of full-stack website tasks 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

    cs.SE 2026-07 accept novelty 7.0

    UI2App introduces a benchmark showing that vision-language models can reconstruct web page visuals but largely fail to infer the underlying interaction logic from screenshots alone.

  2. Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    Cookie-Bench is a reference-free 1,000-query web development benchmark paired with Cookie-Frame, a metacognition-inspired three-stage framework (static perception, agent interaction, dynamic scoring) that aligns with ...

  3. JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

    cs.CV 2026-07 conditional novelty 5.0

    JarvisHub open-sources a three-layer canvas-state, protocol-bridge, and agent-runtime harness so multimodal creative agents can inspect and update a shared editable project graph over long workflows.

  4. HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML

    cs.SE 2026-05 unverdicted novelty 5.0

    HTMLCure uses browser-executed interaction trajectories to diagnose and repair LLM HTML outputs, expanding 97K prompts into a 40K refined SFT set that lifts a 27B model to 50.6 on HTMLBench-400 and 81.2 on MiniAppBench.

  5. GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

    cs.CV 2026-04 unverdicted novelty 5.0

    GLM-5V-Turbo integrates multimodal perception as a core part of reasoning and execution for agentic tasks, reporting strong results in visual tool use and multimodal coding while keeping text-only performance competitive.

  6. Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability

    cs.CL 2026-05 unverdicted novelty 4.0

    The paper develops a unified framework that organizes computer-use agent reliability around perception-decision-execution layers and creation-deployment-operation-maintenance stages to map security and alignment inter...

  7. GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

    cs.CV 2026-04 unverdicted novelty 4.0

    GLM-5V-Turbo integrates multimodal perception directly into reasoning, planning, tool use, and execution for agents, yielding strong results in multimodal coding and framework-based tasks while keeping text coding com...

  8. GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

    cs.CV 2026-04 unverdicted novelty 4.0

    GLM-5V-Turbo integrates multimodal perception directly into reasoning and agent workflows, reporting strong results on visual tool use, multimodal coding, and framework-based agent tasks while keeping text coding competitive.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith · cited by 6 Pith papers

  1. [1]

    Product Background The Airbnb Community Center is a forum for hosts and community members to exchange experiences and seek advice on topics like property management, booking support, and hosting services

  2. [2]

    Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons

    Business Workflows 2.1 Page Navigation Flow Pages: Home, Login/Signup, Topic, Post Detail, Post Creation, User Profile. Navigation via top bar, sidebar categories, featured posts, user avatars, and action buttons. Supports login-dependent and unauthenticated flows. 2.2 Core Workflows •User Authentication:Login via phone number→Validate→Return home •Browse...

  3. [3]

    Workflow-Based Agent Verification Implementation Details A.2.1

    Requirements Specification 3.1 User Authentication • Phone number login with validation • Login state persistence and avatar display 3.2 Home Portal • Top nav with logo, search, Post button, language toggle • Featured conversations & popular post list • Sidebar category navigation • Post creation button with permission control 3.3 Topic Category • Breadcr...

    1920

  4. [4]

    Actions:

    Click the Post button in the navigation bar Validations:None •Prototype:post • Objective:Verify that the post creation form accepts valid input for title, topic selection, and sub-topic selection fields. Actions:

  5. [5]

    Click the Log in button in the navigation bar

  6. [6]

    12345678

    Type “12345678” into the Phone number field

  7. [7]

    Click the Continue button

  8. [9]

    Test topic

    Type “Test topic” into the Enter the topic here field

  9. [10]

    Select “Homes” from the topic dropdown

  10. [11]

    Advice on your space

    Click the sub-topic card labeled “Advice on your space” Validations: –The title field displays “Test topic” –The topic dropdown displays “Homes” –The sub-topic card “Advice on your space” is visually selected • Objective:Verify that post submission fails when the content field is empty, demonstrating a negative test scenario for required field validation....

  11. [12]

    Actions:

    Click the Submit button Validations: –Post submission fails and does not proceed –An error message is displayed indicating that the content field is required • Objective:Verify that post creation succeeds when all required fields are filled with valid data, and the user is navigated to the newly created post detail page with matching content. Actions:

  12. [13]

    It’s a test content

    Type “It’s a test content” into the content field

  13. [14]

    Test topic

    Click the Submit button Validations: –Post submission succeeds 15 Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification –The page navigates to the newly created post detail page –The post title displays “Test topic” –The post content displays “It’s a test content” A.2.2. AGENTCONFIGURATION The GUI agent and the VLM ju...

    2025

  14. [15]

    /bin/bash

    Product Requirement Document( /workspace/prd.md): Contains overview, business logic, and detailed requirements. 2.Prototype Images(provided above, not from files): Define UI layout, style, interactions, and visual fidelity. 3.Resource Files(/workspace/resources/ **/*): Assets including images, videos, audio, icons, etc. II. Mandatory Full-Stack Workflow 1...

  15. [2023]

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al

    GitHub repository. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al. A survey on llm-as-a- judge.The Innovation, 2024. Gui, Y ., Li, Z., Wan, Y ., Shi, Y ., Zhang, H., Su, Y ., Dong, S., Zhou, X., and Jiang, W. Vision2ui: A real-world dataset with layout for code generation from ui designs.CoRR, 2024. Gui, Y...

    Pith/arXiv arXiv 2024