The reviewed record of science sign in
Pith

arxiv: 2607.06306 · v1 · pith:XAH4F3PJ · submitted 2026-07-07 · cs.SE · cs.AI· cs.CV

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Grace Man Chen , Litao Guo , Yifan Wu , Yiyu Chen , Yenchi Tseng , Sicheng Liu , Yuyu Luo , Ying-Cong Chen This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 10:11 UTCglm-5.2pith:XAH4F3PJrecord.jsonopen to challenge →

classification cs.SE cs.AIcs.CV
keywords interaction inferencevisual fidelityvision-language modelsweb application generationbenchmarkcross-route statefrozen facadescreenshot-to-code
0
0 comments X

The pith

Looks right, does nothing: VLMs fail interaction inference

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces UI2App, a benchmark that tests whether vision-language models can infer the interactive behavior of a web application from static screenshots alone — no text instructions, no action demonstrations, just images. The central finding is a sharp decoupling between visual fidelity and interaction inference: the model that best reconstructs what a page looks like (Gemini 3.1 Pro Preview, VFS 78.1) is nearly the worst at making it actually work (IIS 7.5), trailing the interaction leader (Claude Sonnet 4.6, IIS 39.3) by 5.2×. The paper shows that current frontier models can produce pixel-faithful but behaviorally inert 'frozen façades' — apps where every button is a no-op and no state persists across page navigation. Cross-route state persistence (S3-scope interactions) is the hardest sub-capability: three of six models score exactly zero on it, and even the best reaches only 21.6 out of 100. The paper constructs this finding through a four-metric evaluation protocol (executability, navigation reachability, visual fidelity, and interaction inference score) applied to 327 screenshots organized into 45 multi-route web applications, with IIS grounded in a seven-category interaction taxonomy tiered by state-management complexity (S1 component-local, S2 data-state, S3 cross-route persistence).

Core claim

The paper's central discovery is that visual reconstruction and interaction realization are independent capabilities in current vision-language models. A model can produce a pixel-perfect rendering of every page in a multi-route web application while simultaneously implementing zero functional interactions — buttons without handlers, cart pages that never receive data, forms that accept no input. This is invisible to any benchmark that measures only visual fidelity, because the rendered output at idle is identical whether or not the underlying interaction logic exists. The divergence surfaces only under live behavioral testing: clicking 'Add to Cart' on a product page and then navigating to

What carries the argument

The Interaction Inference Score (IIS) is the paper's central mechanism. It evaluates generated applications along a seven-category interaction taxonomy (toggle, expand/collapse, list operations, data CRUD, form validation, notification, cross-route state) and three scope tiers of increasing state-management complexity (S1: component-local UI state; S2: shared data-state across components; S3: state persisting across route changes). Each interaction is scored by human annotators on a three-level outcome rubric (working/partial/failed) rather than against a single reference implementation, accommodating the fact that static screenshots underdetermine behavior and admit multiple valid real

If this is right

  • Benchmarks that measure only visual fidelity of generated UIs are measuring a capability orthogonal to the one that matters for producing functional software from design artifacts.
  • Cross-route state persistence is a distinct and currently unsolved sub-problem: models that can manage component-local state still fail when state must survive route unmounting, suggesting the bottleneck is architectural reasoning about application-level state stores, not interaction logic per se.
  • The closed-vs-open model quality hierarchy observed on visual tasks does not transfer to interaction inference: two open-weight models outscore two closed-frontier models on IIS despite the reverse ordering on VFS, indicating that interaction inference may depend on different training signals or capabilities than visual reconstruction.
  • Self-debug (feeding build errors back for repair) improves executability but does not improve interaction inference, suggesting that the interaction gap is not a code-quality problem but a reasoning problem about what behavior the screenshots imply.
  • Within-family scaling shows a phase transition between 32B and 72B parameters for basic app buildability, but even at 72B, 37.8% of generations fail to build, indicating that parameter scaling alone does not close the interaction-inference gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If visual fidelity and interaction inference are genuinely independent axes, then training recipes that optimize for screenshot-to-code visual matching may actively trade against interaction reasoning capacity — a model trained to reproduce pixels might learn to prioritize static DOM structure over behavioral architecture, producing the frozen-façade pattern as a systematic training artifact rathe
  • The seven-category interaction taxonomy could serve as a diagnostic training signal: if models are fine-tuned with explicit interaction-inference objectives organized by scope tier, the S3 bottleneck might decompose into a data-management architecture problem (when to introduce a global store) separable from an interaction-detection problem (what the screenshots imply should happen).
  • The finding that different models excel at different latent-affordance inferences (Kimi infers audio runtimes, Claude infers state machines) suggests interaction inference may not be a single capability but a family of domain-specific reasoning patterns, and a model's IIS profile may be more diagnostic than its aggregate score for predicting where targeted improvement will occur.

Load-bearing premise

The benchmark assumes that its seven-category interaction taxonomy and rubric-based human annotation protocol comprehensively and fairly capture what 'interaction inference' means. If significant interaction classes fall outside the taxonomy, or if the human annotators introduce systematic bias in how they judge partial versus working implementations, the IIS scores may not reflect true interaction-inference ability.

What would settle it

If a model were trained to specifically optimize for the seven IIS categories without gaining general interaction-inference ability, IIS scores would rise without real progress on the underlying capability the benchmark intends to measure.

Figures

Figures reproduced from arXiv: 2607.06306 by Grace Man Chen, Litao Guo, Sicheng Liu, Yenchi Tseng, Yifan Wu, Ying-Cong Chen, Yiyu Chen, Yuyu Luo.

Figure 1
Figure 1. Figure 1: UI2App task overview: visual fidelity and interaction inference are independent capabilities. Same e-commerce screenshots, same action sequence (Add to Cart twice, then navigate to /cart). Claude Sonnet 4.6 reconstructs a fully dynamic application (cart counter increments, cross-route state propagates); Gemini 3.1 Pro Preview produces a higher-fidelity but frozen façade in which every interaction is a no-o… view at source ↗
Figure 2
Figure 2. Figure 2: UI2App dataset diversity. Source pool and filtering. We curate UI2App’s reference screenshots from open￾source GitHub projects. To obtain a broad initial pool, we use 24 archetype-aware GitHub search queries spanning 12 appli￾cation categories, yielding 2,013 candidate repositories. We then apply a four-stage automated pipeline that checks permissive licensing, structural validity, buildability, and authen… view at source ↗
Figure 3
Figure 3. Figure 3: The evaluation protocol of UI2App. The protocol decomposes evaluation into a sequence of automated stages for reproducibility, reserving a final human-annotation stage for behavioral judgments that cannot be reliably captured by runtime automated probes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: IIS per-cell heatmap. Per-(category, model) success rate (×100, capped, zero-imputation) over 7 categories × 6 baselines; rows grouped by typical scope (S1/S2/S3); columns ordered by overall IIS [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qwen2.5-VL scaling on UI2App (3B → 72B). a: EXEC@1 (circles), EXEC@3 (squares), and VFS (triangles, dashed); the shaded band marks the 32B → 72B phase transition. b: per-app build-failure category, stacked to 100%. Categories: syntax, inconsistency, hallucination, other, pass. Build-failure composition shifts with size, not just rate. The dominant build-error category shifts monotonically with size (Figure… view at source ↗
Figure 6
Figure 6. Figure 6: Interaction-inference failure on 48_shopco-ecommerce. Identical input screenshots and identical click sequence; the divergence is the Broken shared state pattern. VFS on the static cart-route capture cannot distinguish the two outputs (both render an empty cart at protocol time), whereas IIS flags the missing S3 affordance. G.6 Case studies: latent-affordance inference The eight case studies below illustra… view at source ↗
Figure 7
Figure 7. Figure 7: Input set for 31_genshin-music (9 screenshots, one per route). The audio capa￾bility must be inferred from these static cues alone: note pads with musical glyphs and MIDI key labels (Q–U/A–J/Z–M) on /zen-keyboard and /keybinds, a composer timeline (/composer, /vsrg-composer), and a full-screen player (/vsrg-player). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Latent-affordance inference on 31_genshin-music. Four of six baselines emit (b)-style code: visually-faithful pads with no audio. Only Kimi K2.5 instantiates the (c)-style Web-Audio signal chain, AudioContext → OscillatorNode → GainNode with a 500 ms exponential decay, and a MIDI-to-frequency converter f = 440·2 (n−69)/12. VFS cannot read this gap from the rendered DOM; the missing audio runtime is a laten… view at source ↗
Figure 9
Figure 9. Figure 9: State-machine inference on 48_shopco-ecommerce: before/after live capture across an identical click sequence. Each row is one model, before/after an identical add-to-cart→cart￾navigation sequence (SPA <Link>, no hard reload, so in-memory state must survive). Left column (before): all three carts are empty baselines. Right column (after): (a2) Sonnet’s CartContext propagates end-to-end: GRADIENT GRAPHIC T-S… view at source ↗
Figure 10
Figure 10. Figure 10: Cross-session state persistence on 39_maciekt07-todoapp. (a) the /purge route on a fresh context shows “no tasks to purge” and disabled action buttons; (b) after seeding three tasks (one completed) into localStorage['taskapp_data'] and triggering a hard reload, the same route reports “3 total, 0 selected, 1 completed” and re-enables the PURGE DONE/PURGE ALL TASKS buttons, proving that Kimi’s AppContext re… view at source ↗
Figure 11
Figure 11. Figure 11: HTML5 DnD primitive reach on 39_maciekt07-todoapp. Kimi/Sonnet bind onDragOver+onDrop to a real file-import flow; Gemini produces the same rectangle as decora￾tion. Identical visual targets, divergent event topology. Pairs with the audio case ( [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Reference-subset rendering on 62_tokyo-mui-dashboard/crypto. Pictorial counter￾part to §E. GLM-4.6V’s rendered DOM is a strict subset of the reference dashboard’s DOM: the top section is reproduced faithfully, while the reference’s subsequent WALLETS+SECURITY blocks have no counterpart in the generated DOM. We verify the subset relation, not just a viewport screenshot: a full-page Playwright capture acros… view at source ↗
Figure 14
Figure 14. Figure 14: Factuality vs instruction-compliance failures. C1 (left) is a symbol-existence hallucina￾tion that the prompt does not pre-empt. C2 (right) violates clause (iv) of the plan-step prompt. The orthogonality of the two error modes motivates reporting an instruction-compliance rate alongside EXEC@3 (§G.1). Functional quiz engine (61_react-duolingo). The gamified learning app’s 10-screenshot input set ( [PITH_… view at source ↗
Figure 16
Figure 16. Figure 16: Functional quiz engine on 61_react-duolingo (before/after the same select￾answer→CHECK sequence). (a,b) Claude validates the answer and shows Correct! feedback, advancing to CONTINUE; (c) Gemini’s CHECK has no handler, so the same clicks leave the question unchanged. (d,e) the correct answer also awards XP, filling Claude’s daily quest 0/10 → 10/10. Only Claude wires the full engine (Kimi partial; the oth… view at source ↗
read the original abstract

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 6 minor

Summary. The paper introduces UI2App, a benchmark for evaluating interaction inference in executable web application generation from static screenshots. The benchmark comprises 327 screenshots organized into 45 state-coherent screenshot sets for multi-route web applications. The authors design a four-metric evaluation protocol: EXEC (executability), NRS (navigation reachability), VFS (visual fidelity), and IIS (interaction inference score). The key finding is a dissociation between visual fidelity and interaction inference: the VFS leader (Gemini 3.1 Pro Preview) scores only 7.5 on IIS, ranking fourth and trailing the IIS leader (Claude Sonnet 4.6 at 39.3) by 5.2x. The paper evaluates six frontier VLMs and a Qwen2.5-VL scaling ladder, finding that cross-route state persistence (S3) is a frontier-wide bottleneck.

Significance. The paper addresses a genuine gap in the literature: existing benchmarks for image-to-webpage generation focus on visual fidelity and do not measure whether generated artifacts are functionally interactive. The concept of interaction inference — recovering application behavior from screenshots alone without textual or behavioral specification — is well-motivated and distinct from specification-following. The benchmark design has several strengths: the VFS metric is judge-free (no LLM-as-judge), using DOM-level bipartite matching with four sub-metrics; the IIS taxonomy of seven interaction categories across three scope tiers (S1/S2/S3) provides structured diagnostic signal; the scope-weight sensitivity analysis (Table 4, Kendall τ=1.0 across three weightings) demonstrates robustness of model rankings to the one free parameter choice; and the dataset construction pipeline (four-stage automated filter from 2,013 repos to 164, then three-level expert selection to 45) is documented in detail. The finding that visual fidelity does not imply interaction inference capability is a concrete, falsifiable claim supported by the data. The case studies in Appendix G (audio synthesis, state-ma ch

major comments (1)
  1. §3.5, Eq. (2) and Appendix B: The Reference Interaction Inventory (RII) is constructed by two experts who also participated in app selection (§3.2, §3.5), meaning they know the actual application behavior. The paper instructs them to label only interactions 'implied by the screenshots,' but no inter-annotator agreement is reported for the RII construction itself — Table 5 reports agreement only for the generation-side fields (gen_appl, gen_result, gen_scope). If the RII includes interactions present in the real app but not genuinely inferable from screenshots alone, all models are equally penalized, which preserves the relative ranking (Claude vs. Gemini 5.2x gap) but could overstate the absolute difficulty of interaction inference. This is load-bearing for the claim that 'inferring complete interaction behavior from static screenshots remains a key challenge' (Abstract, §5), since some
minor comments (6)
  1. §4.2, Table 2: The 'VFS components' column appears to report route coverage (percentage of input screenshots matched to generated routes), but this is not explicitly defined in §3.4 or the table caption. Clarify what this column measures.
  2. §3.4, Eq. (1): The NRS formula uses min(1, n_reach/n_tot), but the text says 'reachable from the home route via visible navigation.' It is unclear whether routes reachable only through multi-hop navigation (e.g., A→B→C where C is not directly linked from home) are counted as reachable.
  3. Appendix F, Algorithm 1: The algorithm references L^matched_3 in line 36, but this variable is defined in Eq. (3) as L^matched_{3,a}. The subscript inconsistency (3 vs. 3,a) could confuse readers trying to trace the computation.
  4. §4.5, Figure 5a: The y-axis label 'Score (%)' combines EXEC@1, EXEC@3 (percentages) and VFS (0-100 scale). While the text notes these are on different scales, plotting them on the same axis without secondary labeling may mislead casual readers. Consider adding a note or using dual axes.
  5. Table 8: The subcategory 'Vertical Admin (food)' is the only subcategory with a parenthetical qualifier. Clarify whether this refers to a specific app or a subcategory type.
  6. References: Several model references (Claude Sonnet 4.6, Gemini 3.1 Pro Preview, GPT-5.4, Qwen3.5) cite only vendor blog posts or model cards. While this is standard for proprietary models, including version identifiers or API access dates would strengthen reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee correctly identifies that the Reference Interaction Inventory (RII) lacks a reported inter-annotator agreement metric, and that this is relevant to the absolute difficulty claim. We agree this should be addressed in revision and explain below how we will do so.

read point-by-point responses

  1. Referee: §3.5, Eq. (2) and Appendix B: The Reference Interaction Inventory (RII) is constructed by two experts who also participated in app selection (§3.2, §3.5), meaning they know the actual application behavior. The paper instructs them to label only interactions 'implied by the screenshots,' but no inter-annotator agreement is reported for the RII construction itself — Table 5 reports agreement only for the generation-side fields (gen_appl, gen_result, gen_scope). If the RII includes interactions present in the real app but not genuinely inferable from screenshots alone, all models are equally penalized, which preserves the relative ranking (Claude vs. Gemini 5.2x gap) but could overstate the absolute difficulty of interaction inference. This is load-bearing for the claim that 'inferring complete interaction behavior from static screenshots remains a key challenge' (Abstract, §5), since some

    Authors: The referee is correct that we report inter-annotator agreement for the generation-side annotation fields (Table 5) but not for the RII construction itself. This is a genuine gap in our validation evidence, and we will address it in the revision. revision: yes

  2. Referee: [continued] ...since some interactions present in the real app may not be genuinely inferable from screenshots alone, the absolute IIS scores could be inflated downward.

    Authors: We agree with the referee's logical point: if the RII includes interactions that are present in the real application but not genuinely inferable from the screenshots alone, all models would be equally penalized. This would preserve relative rankings (the 5.2x Claude vs. Gemini gap is robust to this concern) but could overstate the absolute difficulty of interaction inference. We note three mitigating factors in the current design. First, the RII annotators were explicitly instructed to label only interactions 'implied by the screenshots,' and the selection rubric (§3.2) already filters for applications whose screenshots contain sufficient visual evidence for interaction logic inference. Second, the IIS scoring formula (Eq. 2) is recall-oriented: the denominator sums only over RII-positive categories (the reference set G_a), so categories not marked as implied do not enter the score at all. Third, the rubric-based evaluation credits any valid implementation rather than matching a single reference, so the RII defines what interaction categories are implied, not how they must be implemented. Nevertheless, the referee's concern about potential annotator bias from prior knowledge of the real app is valid. In the revision we will: (1) report inter-annotator agreement (Krippendorff alpha) for the RII coverage and scope labels, computed under the same paired-annotation design used for the generation side; (2) add an explicit audit step where a third annotator, who did not participate in app selection, independently reviews a sample of RII items to check for interactions that are present in the real app but not inferable from screenshots alone; and (3) add a sentence to the abstract and §5 qualifying that the absolute IIS scores reflect difficulty under the RII's inferability标准, revision: yes

Circularity Check

0 steps flagged

No circularity found: the benchmark and metrics are defined independently of the models evaluated, with no self-citation chain or definitional reduction.

full rationale

The paper introduces UI2App, a benchmark for interaction inference from screenshots. The core metric IIS (Eq. 2) is defined via a Reference Interaction Inventory (RII) constructed from input screenshots by experts, and a generation-side annotation of model outputs. The RII is constant across all models; it is not derived from any model's output. The IIS formula is recall-oriented with a denominator determined by the RII and a numerator determined by independent human annotation of each model's generated artifact. No step in the derivation chain reduces to its own inputs by construction. There are no self-citations used to define the metrics, taxonomy, or dataset. The evaluation is grounded in external artifacts (GitHub repositories) and human annotation with reported inter-annotator agreement (Krippendorff α 0.72–0.84). The headline finding (VFS leader ≠ IIS leader) is an empirical observation from running six external models, not a tautological consequence of the metric definition. The skeptic's concern about RII over-inclusion is a correctness/validity concern about absolute score precision, not a circularity concern: the relative ranking is robust because the RII is model-independent, and the metric is not defined in terms of the quantity it claims to measure. The derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The paper introduces a benchmark and metrics. The 'invented entities' are the interaction taxonomy and the scoring formula, which are methodological contributions rather than postulated physical or mathematical objects. The free parameters are hyperparameters of the evaluation pipeline, justified by the design of the metrics.

free parameters (4)
  • Scope weights (wS1, wS2, wS3) = 1, 2, 3
    Chosen to reflect ordinal ranking of implementation cost. The paper shows rankings are insensitive to this choice (Table 4), but the specific values are a design decision.
  • VFS token-Jaccard bonus (β) = 0.08
    Set to break ties in DOM alignment scores without overriding strong evidence. Value is stated as a hyperparameter in Appendix F.
  • VFS text similarity threshold (θtext) = 0.3
    Threshold for retaining block assignments in VFS. Stated in Appendix F.
  • VFS DOM alignment floor (θDOM) = 0.20
    Absolute floor for accepting a screenshot-to-route match. Stated in Appendix F.
axioms (3)
  • domain assumption The seven interaction categories (toggle, expand/collapse, list operations, data CRUD, form validation, notification, cross-route state) comprehensively cover the interaction space inferable from web app screenshots.
    The IIS metric is built on this taxonomy (§3.5). If a significant interaction type is missing, the metric is incomplete.
  • domain assumption Human annotators can reliably judge whether a generated application realizes an inferred interaction according to the rubric.
    The IIS relies on this annotation (§3.5). Inter-annotator agreement is substantial (α 0.72–0.84) but not perfect.
  • domain assumption The 45 curated applications are representative of the diversity of multi-route web applications.
    The benchmark's conclusions about model capability depend on this representativeness (§3.2).

pith-pipeline@v1.1.0-glm · 33689 in / 2312 out tokens · 398723 ms · 2026-07-08T10:11:05.274642+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 52 canonical work pages · 9 internal anchors

  1. [1]

    Proceedings of the ACM on Web Conference 2025 , pages =

    Webcode2m: A real-world dataset for code generation from webpage designs , author =. Proceedings of the ACM on Web Conference 2025 , pages =

    work page 2025

  2. [2]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , month = nov, year =

    Wu, Yifan and Yan, Lutao and Shen, Leixian and Wang, Yunhai and Tang, Nan and Luo, Yuyu , editor =. Findings of the Association for Computational Linguistics: EMNLP 2024 , month = nov, year =. doi:10.18653/v1/2024.findings-emnlp.710 , pages =

    work page doi:10.18653/v1/2024.findings-emnlp.710 2024

  3. [3]

    2026 , eprint =

    AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines , author =. 2026 , eprint =

    work page 2026

  4. [4]

    Sketch2code: Evaluating vision-language models for interactive web design prototyping , author =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    work page 2025

  5. [5]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    Iw-bench: Evaluating large multimodal models for converting image-to-web , author =. Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    work page 2025

  6. [6]

    Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach

    Automatically generating ui code from screenshot: A divide-and-conquer-based approach , author =. arXiv preprint arXiv:2406.16386 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    Latcoder: Converting webpage design to code with layout-as-thought , author =. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages =

    work page

  8. [8]

    Proceedings of the ACM on Software Engineering , volume =

    Mllm-based ui2code automation guided by ui layout information , author =. Proceedings of the ACM on Software Engineering , volume =. 2025 , publisher =

    work page 2025

  9. [9]

    arXiv preprint arXiv:2506.06251 , year =

    Designbench: A comprehensive benchmark for mllm-based front-end code generation , author =. arXiv preprint arXiv:2506.06251 , year =

    work page arXiv

  10. [10]

    Advancing vision-language models in front-end development via data synthesis

    Advancing vision-language models in front-end development via data synthesis , author =. arXiv preprint arXiv:2503.01619 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2507.04952 , year =

    Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation , author =. arXiv preprint arXiv:2507.04952 , year =

    work page arXiv

  12. [12]

    Advances in Neural Information Processing Systems , year =

    Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch , author =. Advances in Neural Information Processing Systems , year =

    work page

  13. [13]

    arXiv preprint arXiv:2603.26648 , year =

    Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification , author =. arXiv preprint arXiv:2603.26648 , year =

    work page arXiv

  14. [14]

    arXiv preprint arXiv:2509.24709 , year =

    IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video? , author =. arXiv preprint arXiv:2509.24709 , year =

    work page arXiv

  15. [15]

    2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages =

    Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping , author =. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages =. 2025 , organization =

    work page 2025

  16. [16]

    arXiv preprint arXiv:2603.13391 , year =

    WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics , author =. arXiv preprint arXiv:2603.13391 , year =

    work page arXiv

  17. [17]

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author =

    work page

  18. [18]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    work page

  19. [19]

    Communications of the ACM , volume =

    Datasheets for datasets , author =. Communications of the ACM , volume =. 2021 , publisher =

    work page 2021

  20. [20]

    2026 , url =

    Claude 4.6 Sonnet , author =. 2026 , url =

    work page 2026

  21. [21]

    2026 , url =

    Kimi K2.5: Visual Agentic Intelligence , author =. 2026 , url =

    work page 2026

  22. [22]

    2026 , url =

    GPT 5.4 , author =. 2026 , url =

    work page 2026

  23. [23]

    2026 , url =

    Gemini 3.1 Pro Model Card , author =. 2026 , url =

    work page 2026

  24. [24]

    2026 , url =

    Qwen3.5: Towards Native Multimodal Agents , author =. 2026 , url =

    work page 2026

  25. [25]

    2025 , url =

    GLM-4.6V , author =. 2025 , url =

    work page 2025

  26. [26]

    Code Llama: Open Foundation Models for Code

    Code llama: Open foundation models for code , author =. arXiv preprint arXiv:2308.12950 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

    DeepSeek-Coder: when the large language model meets programming--the rise of code intelligence , author =. arXiv preprint arXiv:2401.14196 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author =. arXiv preprint arXiv:2501.12948 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    The Twelfth International Conference on Learning Representations , year =

    SWE-bench: Can Language Models Resolve Real-world Github Issues? , author =. The Twelfth International Conference on Learning Representations , year =

    work page

  30. [30]

    The twelfth international conference on learning representations , year =

    MetaGPT: Meta programming for a multi-agent collaborative framework , author =. The twelfth international conference on learning representations , year =

    work page

  31. [31]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages =

    Chatdev: Communicative agents for software development , author =. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages =

    work page

  32. [32]

    Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems , pages =

    pix2code: Generating code from a graphical user interface screenshot , author =. Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems , pages =

    work page

  33. [33]

    Design2code: Benchmarking multimodal code generation for automated front-end engineering , author =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    work page 2025

  34. [34]

    Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

    Unlocking the conversion of web screenshots into html code with the websight dataset , author =. arXiv preprint arXiv:2403.09029 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  35. [35]

    Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages =

    ScreenAI: a vision-language model for UI and infographics understanding , author =. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages =

    work page

  36. [36]

    Advances in neural information processing systems , volume =

    Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms , author =. Advances in neural information processing systems , volume =

    work page

  37. [37]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    Uicoder: Finetuning large language models to generate user interface code through automated feedback , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    work page 2024

  38. [38]

    MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs

    Mrweb: An exploration of generating multi-page resource-aware web code from ui designs , author =. arXiv preprint arXiv:2412.15310 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2512.20034 , year =

    VSA: Visual-Structural Alignment for UI-to-Code , author =. arXiv preprint arXiv:2512.20034 , year =

    work page arXiv

  40. [40]

    arXiv preprint arXiv:2511.04012 , year =

    PSD2Code: Automated Front-End Code Generation from Design Files via Multimodal Large Language Models , author =. arXiv preprint arXiv:2511.04012 , year =

    work page arXiv

  41. [41]

    DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models

    DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models , author =. arXiv preprint arXiv:2506.13663 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2512.19918 , year =

    Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs , author =. arXiv preprint arXiv:2512.19918 , year =

    work page arXiv

  43. [43]

    arXiv preprint arXiv:2507.22827 , year =

    Screencoder: Advancing visual-to-code generation for front-end automation via modular multimodal agents , author =. arXiv preprint arXiv:2507.22827 , year =

    work page arXiv

  44. [44]

    arXiv preprint arXiv:2509.22644 , year =

    Webgen-agent: Enhancing interactive website generation with multi-level feedback and step-level reinforcement learning , author =. arXiv preprint arXiv:2509.22644 , year =

    work page arXiv

  45. [45]

    arXiv preprint arXiv:2510.11498 , year =

    Relook: Vision-grounded rl with a multimodal llm critic for agentic web coding , author =. arXiv preprint arXiv:2510.11498 , year =

    work page arXiv

  46. [46]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    work page

  47. [47]

    EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation

    Efficientuicoder: Efficient mllm-based ui code generation via input and output token compression , author =. arXiv preprint arXiv:2509.12159 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  48. [48]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    Webmmu: A benchmark for multimodal multilingual website understanding and code generation , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    work page 2025

  49. [49]

    International Conference on Machine Learning , pages =

    Pix2struct: Screenshot parsing as pretraining for visual language understanding , author =. International Conference on Machine Learning , pages =. 2023 , organization =

    work page 2023

  50. [50]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages =

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi , author =. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages =

    work page

  51. [51]

    Advances in Neural Information Processing Systems , volume =

    Mind2web: Towards a generalist agent for the web , author =. Advances in Neural Information Processing Systems , volume =

    work page

  52. [52]

    First Conference on Language Modeling , year =

    VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? , author =. First Conference on Language Modeling , year =

    work page