{"id":"ec317443-890d-4f80-acb1-15122c5f1443","arxiv_id":"2603.26648","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Vision2Web is a 193-task benchmark for visual website development with a workflow-based agent-verification paradigm, showing that even top multimodal agents struggle on full-stack tasks.","lead":"This paper introduces Vision2Web, a benchmark of 193 tasks that tests AI coding agents on building websites from images — from static pages to full interactive full-stack applications. It also proposes a workflow-based automated verification system using a GUI agent and a VLM judge, and reports large performance gaps across eight current models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VLM judge validation is insufficient: Spearman rank correlation on 100 prototypes does not validate the absolute visual scores driving all headline comparisons, especially since the judge is the same model (Gemini-3-Pro-Preview) as a top evaluated agent.","rationale":"The central claim is that Vision2Web provides a reliable, implementation-agnostic evaluation paradigm, and the paper's headline findings are quantitative comparisons of visual fidelity across models, devices, and task levels. The weakest link is the VLM-based judge: all visual scores come from one model, Gemini-3-Pro-Preview, which is also one of the evaluated agents. The paper's validation of this judge is limited to rank correlation on 100 prototypes, and rank correlation does not validate the absolute score magnitudes or score gaps that the findings rely on. Even setting aside self-preference, a judge with good rank alignment could still systematically over-score or under-score particular models or compress differences, changing the reported gaps. The same-model conflict makes this concern concrete rather than hypothetical. I do not see an internal inconsistency or a fatal flaw; the benchmark design, hierarchical tasks, and GUI verifier validation are genuine contributions. But the visual-score conclusions are not yet demonstrated to be robust, so conditional acceptance is appropriate. The proposed re-scoring test would resolve the concern without discarding the benchmark.","tokens_in":15791,"tokens_out":5183,"duration_ms":57449,"concrete_test":"Re-score a stratified random sample of actual agent outputs (e.g., 20 tasks × 8 models = 160 pages, including desktop/tablet/mobile and frontend/full-stack states) with (a) the same Gemini judge, (b) an independent non-Gemini judge using the identical rubric (e.g., Claude-Opus-4.5 or GPT-5), and (c) three human annotators. Compute per-model mean VS and rank correlations. If the independent judge or human mean scores shift any model's rank or change any headline gap by more than the reported human inter-annotator margin, the visual-score conclusions are not robust; if they agree within that margin, the self-preference/calibration concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All visual fidelity conclusions—device-wise drops, complexity gaps, model rankings—depend on the Gemini-3-Pro-Preview VLM judge's numeric Visual Scores. Section 4.4 validates the judge only by average Spearman ρ=0.66 (median 0.80) against human judgments on 100 sampled prototypes, below human inter-annotator ρ=0.78. Spearman measures rank-order agreement only; it is invariant to monotone rescoring and says nothing about whether the judge's absolute scores or score differences are calibrated. Yet Table 3 and Findings 1–3 use average score gaps (e.g., desktop 63.3 vs mobile 48.3; full-stack VS 11.7) as quantitative evidence. A judge could rank pages correctly while systematically over-scoring one model family or compressing/expanding score differences. The conflict is compounded because the judge is the same model (Gemini-3-Pro-Preview) as one of the top evaluated agents; no analysis rules out self-preference. As reported, the validation also does not establish that agreement holds on the actual outputs of all eight evaluated agents, so the reported model rankings are not demonstrably unbiased.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Vision2Web, a three-level benchmark for visual website development: static UI-to-code reproduction, interactive multi-page frontend generation, and full-stack website construction. The benchmark contains 193 tasks, 918 prototype images, and 1,255 test cases built from C4 validation-set websites. Evaluation uses a workflow-based agent verification paradigm with two components: a GUI agent verifier for functional correctness and a VLM-based judge for visual fidelity. Eight multimodal models are evaluated under two coding-agent frameworks, yielding findings about performance drops across task levels, device form factors, model families, and frameworks. The central claim is that Vision2Web provides a systematic, reproducible, implementation-agnostic evaluation of visual website development agents.","tokens_in":16024,"tokens_out":5070,"duration_ms":53487,"significance":"If the evaluation methodology is trustworthy, this is a valuable contribution: it moves beyond static UI-to-code benchmarks and text-only end-to-end benchmarks, provides realistic hierarchical tasks with expert-annotated workflows, and includes human validation of both verifiers. The dataset construction pipeline, task taxonomy, and workflow formalization are concrete strengths that would make the benchmark broadly useful. However, the reliability of the VLM judge is not established at the level needed to support the headline quantitative comparisons. Because the judge is also one of the evaluated models and is validated only by rank correlation on 100 prototypes, the reported visual scores, model rankings, and Findings 1–5 must be regarded as provisional until calibration and bias analyses are added.","major_comments":[{"comment":"The VLM judge is validated only by average Spearman rank correlation ρ=0.66 (median 0.80) against human judgments on 100 sampled prototypes, with human inter-annotator agreement 0.78. Spearman measures rank consistency and is invariant to monotone rescoring; it does not validate the absolute Visual Scores or score differences used throughout Table 3 and Findings 1–3, e.g., desktop 63.3 vs. mobile 48.3 for Gemini-3-Pro-Preview, or full-stack VS 11.7. A judge can rank pages correctly while systematically over-scoring one model family or compressing/expanding score gaps. Please report calibration/absolute-error metrics, per-model agreement with human scores, and at least one held-out sample covering outputs from all eight evaluated agents.","section":"§4.4, Table 3"},{"comment":"The VLM judge is instantiated with Gemini-3-Pro-Preview, which is also one of the top evaluated agents. All visual scores for Gemini-3-Pro-Preview are therefore produced by the same model being scored, creating a self-preference risk that is not addressed by the current validation. The 100-sample rank-correlation check does not test whether the judge assigns higher scores to its own outputs than human raters would. Please add a self-preference analysis comparing judge-vs-human agreement on Gemini outputs versus other models' outputs, or rerun scoring with an independent judge model for a robustness check.","section":"§4.1, §4.4"},{"comment":"There is a clear numerical error in the static webpage averages for GPT-5 under OpenHands: the row reports Average 49.6, but (49.0 + 44.6 + 40.5)/3 = 44.7. The same inconsistency appears in the GPT-5 row's placement among the other OpenHands averages. This is a central results table, so all averaged columns should be recomputed and verified before acceptance; similar rounding or arithmetic checks are needed for the full-stack and frontend averages.","section":"Table 3"},{"comment":"The GUI agent verifier validation reports node-level agreement of 87.2% on 250 sampled nodes, but gives no confidence intervals, no breakdown by task level, and no error taxonomy. Since Functional Scores drive Findings 3, 5, and 6, the statement that 'residual inaccuracies are largely attributable to model-intrinsic reasoning hallucinations' is unsupported without such an analysis. Please quantify failure modes and report per-level or per-category agreement so the reader can judge where functional scores are most reliable.","section":"§3.2, §4.4"}],"minor_comments":[{"comment":"Typo: 'Interative frontend task example' should be 'Interactive frontend task example'.","section":"Figure 8 caption"},{"comment":"'comprising a total of 21516 input files' is hard to parse; format as 21,516 and verify whether this count is consistent with the sum of prototype images, test cases, and resource files.","section":"§2.3"},{"comment":"The header 'Avg Text Tokens(103)' should be rendered with a superscript or explanatory note (e.g., ×10³) to avoid confusion with the value 103.","section":"Table 2"},{"comment":"Finding 1 says Gemini-3-Pro-Preview is strongest on static webpages under OpenHands, while Finding 3 says Claude-Opus-4.5 consistently achieves the strongest overall performance. These are not contradictory, but they should be explicitly qualified by task level and framework to avoid apparent tension.","section":"Findings 1 and 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset and workflow-verification machinery are useful and well-documented, and the paper is worth publishing after the evaluation validity concerns are addressed. In addition to the requested calibration and self-preference analyses, the editor may wish to ensure that the use of a vendor model (Gemini) as judge and a co-affiliation model (GLM-4.6V) as functional verifier is clearly disclosed and tested for sensitivity; neither choice is disqualifying, but the current validation does not rule out evaluator bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper deserves a serious referee, but it has one real soft spot you should know about before citing its numbers. Vision2Web is a solid new benchmark for end-to-end visual website development, with a three-level hierarchy (static, interactive, full-stack), 193 tasks built from real sites, and a workflow-based verification scheme that pairs a GUI agent for functional tests with a VLM judge for visual fidelity. That combination is genuinely new, and the paper does a good job of documenting dataset construction, prompts, and the agent frameworks. The task design sensibly disentangles capabilities, and the experiments span eight models and two frameworks. The GUI verifier validation (87.2% node-level agreement with human annotations on a sample) is credible.\n\nThe load-bearing weakness is the VLM judge. Section 4.4 validates it on 100 prototypes with an average Spearman correlation of 0.66 against human preferences, below the human inter-annotator agreement of 0.78. Spearman measures rank-order agreement only; it says nothing about whether the judge's absolute scores or score differences are calibrated. Yet Table 3 and the main findings use average score gaps (desktop 63.3 vs mobile 48.3, full-stack VS 11.7) as quantitative evidence. The conflict is compounded because the judge is Gemini-3-Pro-Preview, the same model as one of the top evaluated agents. No analysis rules out self-preference. This should be fixed or at least discussed; the main direction (performance drops with complexity and device size) would likely survive, but the cross-model rankings are not clearly unbiased.\n\nTwo smaller issues: Table 3 lists GPT-5's static average under OpenHands as 49.6, but (49.0+44.6+40.5)/3 is 44.7. That's a plain arithmetic error and it makes you double-check every other number. And the \"contamination-free\" claim oversells what sourcing from C4 guarantees; the dataset also lacks a committed release hash. Both are minor and fixable.\n\nMy read: this is a useful contribution, not a flawless one. The workflow-based verification paradigm is worth adopting as a template, and the hierarchical task design fills a real gap between single-page UI benchmarks and full-stack agent evaluation. The judge validation is the part to press the authors on. I'd definitely send it to peer review, and I'd ask for a response on the self-preference issue and a corrected Table 3 before trusting the specific rankings.","headline":"Vision2Web is a genuinely useful benchmark for multimodal coding agents, but the headline rankings lean on a VLM judge whose validation is thinner than the paper claims.","tokens_in":16593,"tokens_out":2317,"would_cite":true,"duration_ms":25501,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision2Web grades AI website-building from a single page to a full-stack app, and finds every current model falls off sharply as tasks grow.","keywords":["benchmark","visual website development","UI-to-code generation","full-stack development","coding agents","multimodal evaluation","workflow-based verification","visual fidelity"],"falsifier":"Recompute all Visual Scores using a different judge model (for example, a strong open-weights vision-language model or human ratings on a stratified sample), then compare per-model scores and rankings with the published ones. If the two judges disagree on model ordering, or if each model scores systematically higher when it is the judge, the paper's central comparison is not stable. A cheaper first check: measure the judge's agreement with humans on the full benchmark rather than the 100 sampled prototypes — if average Spearman correlation falls near or below 0.5, the visual scores are too noi","tokens_in":15619,"feed_emoji":"🧪","tokens_out":13261,"duration_ms":119922,"temperature":0.7,"pith_summary":"Vision2Web is an attempt to measure something existing benchmarks do not: whether AI coding agents can build real websites from visual prototypes, not just edit code or reproduce a single page. The paper organizes the task into three escalating levels — reproduce a static page from screenshots, build an interactive multi-page frontend, and deliver a full-stack site from a requirements document — covering 193 real-world websites with 918 prototype images and 1,255 test cases. To make end-to-end evaluation reproducible, it introduces a workflow-based agent verification paradigm in which a GUI agent executes expert-designed test workflows while a vision-language judge scores visual fidelity against prototypes. Running eight state-of-the-art models under two coding-agent frameworks, the paper finds substantial, consistent performance gaps: scores drop by tens of points from static to full-stack tasks, and even the strongest model stays below the halfway mark on full-stack visual fidelity. The paper offers this as the field's common yardstick — a reproducible way to compare agents — along with a diagnosis of what is missing: long-horizon planning, cross-page coordination, and persistent state reasoning.","feed_headline":"Even the best AI coders score under 40 on full-stack sites","feed_subtitle":"A 193-task benchmark grades agents from static pages to full apps and shows where today's models fall short.","key_machinery":"The central mechanism is the workflow-based agent verification paradigm. End-to-end testing is formalized as a directed dependency graph whose nodes are self-contained verification sub-procedures — interactions that bring the app to a target state, then a check — instantiated as agent-executable test workflows covering all nodes. Two complementary verifiers operate at the nodes: a GUI agent checks functional correctness from a tuple of objective, guided actions, and validation criteria (the guided actions constrain the agent, preventing stray exploration), and a vision-language judge scores visual fidelity by segmenting the rendered page into logical blocks and rating each against the protot","core_discovery":"The paper's central claim is that visual website development can be evaluated end-to-end, and doing so exposes a capability cliff. Vision2Web contains 193 real-world websites in three levels — static UI-to-code (100 tasks), interactive frontends (66), and full-stack construction (27) — so failures can be attributed to a stage. Evaluation uses workflow-based agent verification: a GUI agent checks functional correctness on expert-designed workflows, while a vision-language judge scores visual fidelity against prototypes. Every model degrades sharply as complexity grows, with state management, CRUD, and file handling the weakest areas. Conclusion: strong isolated-task performance does not trans","pith_inferences":["Editorial inference: the reported rankings rest on visual scores from a single judge model that is itself one of the evaluated models, validated on only 100 sampled prototypes (Spearman correlation 0.66 vs human, below the human inter-annotator agreement of 0.78). Those rankings should be treated as provisional until judge agreement is measured across the full 193-task benchmark.","A direct test the paper leaves implicit: swap the judge backbone and re-score all outputs. If rankings shift materially, or if each model scores higher when it judges itself, self-preference bias is present and the headline comparisons need re-weighting.","The workflow abstraction is not limited to websites; the same dependency-graph-plus-guided-actions recipe could be applied to desktop or mobile application testing, where reproducibility is equally hard.","The paper's failure analysis blames full-stack breakage partly on agents lacking self-verification; this implies a testable remedy — giving agents the same verifier tools during development might narrow the gap between isolated and end-to-end performance."],"forward_implications":["If the benchmark's verdict holds, the field has a reproducible, implementation-agnostic yardstick for comparing coding agents across the whole software lifecycle, not just isolated edits or single pages.","The consistent device-dependent drop — tablet and mobile scores run 10–20% below desktop — implies that visual grounding in current agents is fragile enough to matter in responsive design.","The weakest functional categories — state management, CRUD operations, file and media handling — all require persistent state tracking; the paper's findings point to state reasoning as a central bottleneck.","Because framework choice materially changed most models' results, future progress will likely come from optimizing models and their agent frameworks jointly rather than either alone.","The category analysis ranks SaaS platforms as the hardest real-world target, so an agent that handles multi-page, interaction-heavy SaaS builds would be a meaningful milestone."],"fun_headline_variants":["AI coders ace static pages but flop on full-stack","Full-stack web dev still stumps top AI agents","New benchmark: AI agents fall off a cliff on complex sites","AI coding agents: great at single pages, weak at full apps","Vision2Web: the benchmark that exposes AI's full-stack gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the vision-language judge's visual scores are trustworthy: if the judge is biased toward its own outputs or loses agreement with human raters beyond the 100-prototype validation sample (0.66 correlation, below the 0.78 human-human agreement), every model ranking and every claimed capability gap in the paper comes undone.","fun_headline_variants_meta":{"raw":{"variants":["AI coders ace static pages but flop on full-stack","Full-stack web dev still stumps top AI agents","New benchmark: AI agents fall off a cliff on complex sites","AI coding agents: great at single pages, weak at full apps","Vision2Web: the benchmark that exposes AI's full-stack gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2081,"prompt_tokens":672,"completion_tokens":1409,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":1320}},"tokens_in":416,"tokens_out":1409,"duration_ms":9913,"temperature":1.0,"reasoning_tokens":1320,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T17:13:27.163040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute all Visual Scores using a different judge model (for example, a strong open-weights vision-language model or human ratings on a stratified sample), then compare per-model scores and rankings with the published ones. If the two judges disagree on model ordering, or if each model scores systematically higher when it is the judge, the paper's central comparison is not stable. A cheaper first check: measure the judge's agreement with humans on the full benchmark rather than the 100 sampled prototypes — if average Spearman correlation falls near or below 0.5, the visual scores are too noi","supporting_citations":[],"review_version":1}