{"id":"017ece6f-8610-4c12-9f5b-4253c1bc65f6","arxiv_id":"2607.06306","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"UI2App introduces a benchmark showing that vision-language models can reconstruct web page visuals but largely fail to infer the underlying interaction logic from screenshots alone.","lead":"UI2App is a benchmark that tests whether AI models can generate working, interactive web applications from screenshots alone, without any text instructions. It reveals that models good at recreating visuals often fail at inferring the underlying interactions, exposing a key weakness in current systems.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"RII construction by app-aware experts lacks reported agreement, but the relative IIS comparison between models is robust to this concern.","rationale":"The reader correctly identifies the human annotation protocol as the weakest point, focusing on taxonomy completeness and annotation bias in the generation-side scoring. My concern is more specific: the reference-side RII construction is done by experts with knowledge of the actual applications, and no agreement metric is reported for this step (unlike the generation side where α = 0.72–0.84 is reported in Table 5). This is a concrete gap in the validity evidence for IIS. However, this concern does not undermine the central claim. The RII is constant across all six models, so any over-inclusion of interactions affects all models equally. The 5.2x gap between Claude (39.3) and Gemini (7.5) is robust to this concern, as confirmed by the weighting sensitivity analysis (Table 4, Kendall τ = 1.0). The gap is also too large to be explained by annotation noise given the α values reported. The paper provides substantial supporting evidence: the VFS metric is fully automated and judge-free (Algorithm 1), the IIS formula is implementation-agnostic (crediting any valid implementation), the weighting is shown to be insensitive, and the conditional metrics (Table 6) confirm the top-3 IIS ordering persists when controlling for build success. The case studies (§G.5–G.6) provide concrete, reproducible examples of the inference gap. The main limitation — small dataset (45 apps) and reliance on human annotation — is honestly acknowledged. The concern about RII construction is worth addressing in a revision (reporting reference-side agreement, or having screenshot-only annotators reconstruct the RII), but it does not change the verdict. The paper makes a solid contribution to evaluation methodology and the central finding is well-supported.","tokens_in":30877,"tokens_out":7297,"duration_ms":498997,"concrete_test":"Have two additional front-end experts who have NOT seen the source repositories or running applications construct the RII independently from screenshots alone. Compare this screenshot-only RII with the original RII category-by-category. If more than 15% of RII items differ (categories added or removed, or scope levels changed), recompute IIS under the screenshot-only RII and check whether the absolute scores shift substantially while the model ranking is preserved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The IIS formula (Eq. 2) is recall-oriented with a denominator Σ(w_{s_i^ref}) determined entirely by the Reference Interaction Inventory (RII). The RII is constructed by two experts who also participated in app selection (§3.2, §3.5), meaning they know the actual application behavior. The paper instructs them to label only interactions 'implied by the screenshots,' but no inter-annotator agreement is reported for the reference-side RII construction — Table 5 reports agreement only for generation-side fields (gen_appl, gen_result, gen_scope). If the RII includes interactions present in the real app but not genuinely inferable from screenshots alone, all models are equally penalized, which preserves the relative ranking (Claude vs. Gemini 5.2x gap) but could overstate the absolute difficulty of interaction inference and the magnitude of the 'frontier challenge.' This concern does not undermine the headline claim about the VFS-IIS dissociation, since the RII is constant across models. However, it does affect the precision of the absolute IIS scores and the claim that 'inferring complete interaction behavior from static screenshots remains a key challenge' — some of that challenge may reflect RII over-inclusion rather than genuine inference difficulty.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces UI2App, a benchmark for evaluating interaction inference in executable web application generation from static screenshots. The benchmark comprises 327 screenshots organized into 45 state-coherent screenshot sets for multi-route web applications. The authors design a four-metric evaluation protocol: EXEC (executability), NRS (navigation reachability), VFS (visual fidelity), and IIS (interaction inference score). The key finding is a dissociation between visual fidelity and interaction inference: the VFS leader (Gemini 3.1 Pro Preview) scores only 7.5 on IIS, ranking fourth and trailing the IIS leader (Claude Sonnet 4.6 at 39.3) by 5.2x. The paper evaluates six frontier VLMs and a Qwen2.5-VL scaling ladder, finding that cross-route state persistence (S3) is a frontier-wide bottleneck.","tokens_in":31615,"tokens_out":1217,"duration_ms":322368,"significance":"The paper addresses a genuine gap in the literature: existing benchmarks for image-to-webpage generation focus on visual fidelity and do not measure whether generated artifacts are functionally interactive. The concept of interaction inference — recovering application behavior from screenshots alone without textual or behavioral specification — is well-motivated and distinct from specification-following. The benchmark design has several strengths: the VFS metric is judge-free (no LLM-as-judge), using DOM-level bipartite matching with four sub-metrics; the IIS taxonomy of seven interaction categories across three scope tiers (S1/S2/S3) provides structured diagnostic signal; the scope-weight sensitivity analysis (Table 4, Kendall τ=1.0 across three weightings) demonstrates robustness of model rankings to the one free parameter choice; and the dataset construction pipeline (four-stage automated filter from 2,013 repos to 164, then three-level expert selection to 45) is documented in detail. The finding that visual fidelity does not imply interaction inference capability is a concrete, falsifiable claim supported by the data. The case studies in Appendix G (audio synthesis, state-ma ch","major_comments":[{"comment":"§3.5, Eq. (2) and Appendix B: The Reference Interaction Inventory (RII) is constructed by two experts who also participated in app selection (§3.2, §3.5), meaning they know the actual application behavior. The paper instructs them to label only interactions 'implied by the screenshots,' but no inter-annotator agreement is reported for the RII construction itself — Table 5 reports agreement only for the generation-side fields (gen_appl, gen_result, gen_scope). If the RII includes interactions present in the real app but not genuinely inferable from screenshots alone, all models are equally penalized, which preserves the relative ranking (Claude vs. Gemini 5.2x gap) but could overstate the absolute difficulty of interaction inference. This is load-bearing for the claim that 'inferring complete interaction behavior from static screenshots remains a key challenge' (Abstract, §5), since some ","section":null}],"minor_comments":[{"comment":"§4.2, Table 2: The 'VFS components' column appears to report route coverage (percentage of input screenshots matched to generated routes), but this is not explicitly defined in §3.4 or the table caption. Clarify what this column measures.","section":null},{"comment":"§3.4, Eq. (1): The NRS formula uses min(1, n_reach/n_tot), but the text says 'reachable from the home route via visible navigation.' It is unclear whether routes reachable only through multi-hop navigation (e.g., A→B→C where C is not directly linked from home) are counted as reachable.","section":null},{"comment":"Appendix F, Algorithm 1: The algorithm references L^matched_3 in line 36, but this variable is defined in Eq. (3) as L^matched_{3,a}. The subscript inconsistency (3 vs. 3,a) could confuse readers trying to trace the computation.","section":null},{"comment":"§4.5, Figure 5a: The y-axis label 'Score (%)' combines EXEC@1, EXEC@3 (percentages) and VFS (0-100 scale). While the text notes these are on different scales, plotting them on the same axis without secondary labeling may mislead casual readers. Consider adding a note or using dual axes.","section":null},{"comment":"Table 8: The subcategory 'Vertical Admin (food)' is the only subcategory with a parenthetical qualifier. Clarify whether this refers to a specific app or a subcategory type.","section":null},{"comment":"References: Several model references (Claude Sonnet 4.6, Gemini 3.1 Pro Preview, GPT-5.4, Qwen3.5) cite only vendor blog posts or model cards. While this is standard for proprietary models, including version identifiers or API access dates would strengthen reproducibility.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is a solid benchmark contribution. The main concern (RII construction agreement) is addressable with a paragraph of discussion and ideally a sensitivity check, but does not undermine the core VFS-IIS dissociation finding. The dataset size (45 apps) is modest but defensible given the expert curation and human annotation cost. The paper would benefit from positioning relative to concurrent work on interaction-aware evaluation (e.g., Interaction2Code, IWR-Bench) more sharply in the intro — the distinction between 'interaction inferred from screenshots' vs. 'interaction specified by text/video' is the key novelty and could be stated more prominently."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee correctly identifies that the Reference Interaction Inventory (RII) lacks a reported inter-annotator agreement metric, and that this is relevant to the absolute difficulty claim. We agree this should be addressed in revision and explain below how we will do so.","responses":[{"response":"The referee is correct that we report inter-annotator agreement for the generation-side annotation fields (Table 5) but not for the RII construction itself. This is a genuine gap in our validation evidence, and we will address it in the revision.","revision_made":"yes","referee_comment":"§3.5, Eq. (2) and Appendix B: The Reference Interaction Inventory (RII) is constructed by two experts who also participated in app selection (§3.2, §3.5), meaning they know the actual application behavior. The paper instructs them to label only interactions 'implied by the screenshots,' but no inter-annotator agreement is reported for the RII construction itself — Table 5 reports agreement only for the generation-side fields (gen_appl, gen_result, gen_scope). If the RII includes interactions present in the real app but not genuinely inferable from screenshots alone, all models are equally penalized, which preserves the relative ranking (Claude vs. Gemini 5.2x gap) but could overstate the absolute difficulty of interaction inference. This is load-bearing for the claim that 'inferring complete interaction behavior from static screenshots remains a key challenge' (Abstract, §5), since some"},{"response":"We agree with the referee's logical point: if the RII includes interactions that are present in the real application but not genuinely inferable from the screenshots alone, all models would be equally penalized. This would preserve relative rankings (the 5.2x Claude vs. Gemini gap is robust to this concern) but could overstate the absolute difficulty of interaction inference. We note three mitigating factors in the current design. First, the RII annotators were explicitly instructed to label only interactions 'implied by the screenshots,' and the selection rubric (§3.2) already filters for applications whose screenshots contain sufficient visual evidence for interaction logic inference. Second, the IIS scoring formula (Eq. 2) is recall-oriented: the denominator sums only over RII-positive categories (the reference set G_a), so categories not marked as implied do not enter the score at all. Third, the rubric-based evaluation credits any valid implementation rather than matching a single reference, so the RII defines what interaction categories are implied, not how they must be implemented. Nevertheless, the referee's concern about potential annotator bias from prior knowledge of the real app is valid. In the revision we will: (1) report inter-annotator agreement (Krippendorff alpha) for the RII coverage and scope labels, computed under the same paired-annotation design used for the generation side; (2) add an explicit audit step where a third annotator, who did not participate in app selection, independently reviews a sample of RII items to check for interactions that are present in the real app but not inferable from screenshots alone; and (3) add a sentence to the abstract and §5 qualifying that the absolute IIS scores reflect difficulty under the RII's inferability标准,","revision_made":"yes","referee_comment":"[continued] ...since some interactions present in the real app may not be genuinely inferable from screenshots alone, the absolute IIS scores could be inflated downward."}],"tokens_in":30723,"tokens_out":752,"duration_ms":188523,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"UI2App introduces a genuinely new evaluation axis: can a VLM infer interaction behavior from static screenshots alone, without any textual or behavioral spec? The core finding — that the visual-fidelity leader (Gemini) scores near-zero on interaction inference while Claude leads by 5.2x — is real, well-supported, and the kind of result that changes how people think about the screenshot-to-code task. The IIS metric with its 7-category taxonomy and 3-scope tiers is a solid contribution to evaluation methodology. The scope-weight insensitivity check (Table 4, Kendall τ=1.0 across three weightings) is good practice. The case studies in the appendix are genuinely illustrative — the music synthesizer and e-commerce cart examples make the VFS-IIS gap concrete in a way that tables alone wouldn't. The Qwen2.5-VL scaling ladder showing a phase transition between 32B and 72B is a nice bonus. The VFS metric design (judge-free, DOM-alignment-based) is also worth crediting — it's reproducible and doesn't introduce LLM-judge circularity. The main soft spot is the one the stress-test flags: the Reference Interaction Inventory is constructed by two experts who already know the actual app behavior, and no inter-annotator agreement is reported for the RII side (Table 5 covers only generation-side fields). If the RII includes interactions that are present in the real app but not genuinely inferable from screenshots, the absolute IIS scores are deflated — all models are penalized equally, so relative rankings hold, but the magnitude of the 'frontier challenge' could be overstated. This is a real gap but not a fatal one. The paper should have addressed it, and a referee should ask for at least a sensitivity analysis or a second RII annotator. The dataset size (45 apps) is small but defensible given the curation effort and the human annotation cost. The α values (0.72–0.84) are moderate but acceptable for a rubric with this much judgment involved. This paper is for researchers working on VLM-based code generation and benchmark design. It deserves a serious referee — the contribution is novel, the methodology is largely sound, and the findings are actionable. The RII agreement gap is fixable in revision.","headline":"UI2App introduces a genuinely new evaluation axis — interaction inference from screenshots alone — and the core finding (visual fidelity ≠ interaction capability) is well-supported.","tokens_in":31572,"tokens_out":1139,"would_cite":true,"duration_ms":97685,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Looks right, does nothing: VLMs fail interaction inference","keywords":["interaction inference","visual fidelity","vision-language models","web application generation","benchmark","cross-route state","frozen facade","screenshot-to-code"],"falsifier":"If a model were trained to specifically optimize for the seven IIS categories without gaining general interaction-inference ability, IIS scores would rise without real progress on the underlying capability the benchmark intends to measure.","tokens_in":31020,"feed_emoji":"🖱️","tokens_out":1234,"duration_ms":2071126,"temperature":0.7,"pith_summary":"This paper introduces UI2App, a benchmark that tests whether vision-language models can infer the interactive behavior of a web application from static screenshots alone — no text instructions, no action demonstrations, just images. The central finding is a sharp decoupling between visual fidelity and interaction inference: the model that best reconstructs what a page looks like (Gemini 3.1 Pro Preview, VFS 78.1) is nearly the worst at making it actually work (IIS 7.5), trailing the interaction leader (Claude Sonnet 4.6, IIS 39.3) by 5.2×. The paper shows that current frontier models can produce pixel-faithful but behaviorally inert 'frozen façades' — apps where every button is a no-op and no state persists across page navigation. Cross-route state persistence (S3-scope interactions) is the hardest sub-capability: three of six models score exactly zero on it, and even the best reaches only 21.6 out of 100. The paper constructs this finding through a four-metric evaluation protocol (executability, navigation reachability, visual fidelity, and interaction inference score) applied to 327 screenshots organized into 45 multi-route web applications, with IIS grounded in a seven-category interaction taxonomy tiered by state-management complexity (S1 component-local, S2 data-state, S3 cross-route persistence).","feed_headline":"Visual fidelity does not imply interaction inference in VLMs","feed_subtitle":"The best-looking generated app is 5.2× worse at actually working than the interaction leader, revealing a frontier-wide gap in cross-route.","key_machinery":"The Interaction Inference Score (IIS) is the paper's central mechanism. It evaluates generated applications along a seven-category interaction taxonomy (toggle, expand/collapse, list operations, data CRUD, form validation, notification, cross-route state) and three scope tiers of increasing state-management complexity (S1: component-local UI state; S2: shared data-state across components; S3: state persisting across route changes). Each interaction is scored by human annotators on a three-level outcome rubric (working/partial/failed) rather than against a single reference implementation, accommodating the fact that static screenshots underdetermine behavior and admit multiple valid real","core_discovery":"The paper's central discovery is that visual reconstruction and interaction realization are independent capabilities in current vision-language models. A model can produce a pixel-perfect rendering of every page in a multi-route web application while simultaneously implementing zero functional interactions — buttons without handlers, cart pages that never receive data, forms that accept no input. This is invisible to any benchmark that measures only visual fidelity, because the rendered output at idle is identical whether or not the underlying interaction logic exists. The divergence surfaces only under live behavioral testing: clicking 'Add to Cart' on a product page and then navigating to ","pith_inferences":["If visual fidelity and interaction inference are genuinely independent axes, then training recipes that optimize for screenshot-to-code visual matching may actively trade against interaction reasoning capacity — a model trained to reproduce pixels might learn to prioritize static DOM structure over behavioral architecture, producing the frozen-façade pattern as a systematic training artifact rathe","The seven-category interaction taxonomy could serve as a diagnostic training signal: if models are fine-tuned with explicit interaction-inference objectives organized by scope tier, the S3 bottleneck might decompose into a data-management architecture problem (when to introduce a global store) separable from an interaction-detection problem (what the screenshots imply should happen).","The finding that different models excel at different latent-affordance inferences (Kimi infers audio runtimes, Claude infers state machines) suggests interaction inference may not be a single capability but a family of domain-specific reasoning patterns, and a model's IIS profile may be more diagnostic than its aggregate score for predicting where targeted improvement will occur."],"forward_implications":["Benchmarks that measure only visual fidelity of generated UIs are measuring a capability orthogonal to the one that matters for producing functional software from design artifacts.","Cross-route state persistence is a distinct and currently unsolved sub-problem: models that can manage component-local state still fail when state must survive route unmounting, suggesting the bottleneck is architectural reasoning about application-level state stores, not interaction logic per se.","The closed-vs-open model quality hierarchy observed on visual tasks does not transfer to interaction inference: two open-weight models outscore two closed-frontier models on IIS despite the reverse ordering on VFS, indicating that interaction inference may depend on different training signals or capabilities than visual reconstruction.","Self-debug (feeding build errors back for repair) improves executability but does not improve interaction inference, suggesting that the interaction gap is not a code-quality problem but a reasoning problem about what behavior the screenshots imply.","Within-family scaling shows a phase transition between 32B and 72B parameters for basic app buildability, but even at 72B, 37.8% of generations fail to build, indicating that parameter scaling alone does not close the interaction-inference gap."],"fun_headline_variants":["VLMs can draw web apps but fail to make them work","Visual fidelity in generated web apps does not equal functional interactions","Vision-language models struggle to infer interactions from UI screenshots","Pretty web apps fail basic interaction tests in vision-language models","Generating functional web apps from screenshots remains a VLM challenge"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark assumes that its seven-category interaction taxonomy and rubric-based human annotation protocol comprehensively and fairly capture what 'interaction inference' means. If significant interaction classes fall outside the taxonomy, or if the human annotators introduce systematic bias in how they judge partial versus working implementations, the IIS scores may not reflect true interaction-inference ability.","fun_headline_variants_meta":{"raw":{"variants":["VLMs can draw web apps but fail to make them work","Visual fidelity in generated web apps does not equal functional interactions","Vision-language models struggle to infer interactions from UI screenshots","Pretty web apps fail basic interaction tests in vision-language models","Generating functional web apps from screenshots remains a VLM challenge","VLMs generate pixel-perfect web apps that lack basic interactions","Visual reconstruction and interaction inference are decoupled in VLMs","Cross-page state management is a bottleneck for UI generation models","Best-looking generated web apps fail real interaction benchmarks","VLMs render web apps well but infer interactions poorly"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1207,"prompt_tokens":587,"completion_tokens":620,"prompt_tokens_details":null},"tokens_in":587,"tokens_out":620,"duration_ms":44157,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T10:11:05.274642+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a model were trained to specifically optimize for the seven IIS categories without gaining general interaction-inference ability, IIS scores would rise without real progress on the underlying capability the benchmark intends to measure.","supporting_citations":[],"review_version":1}