{"id":"1f346d1d-b474-4268-8288-249dcdee0503","arxiv_id":"2509.07334","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Structured, parameterized UI specifications extracted from screenshots improve LLM-based generation fidelity and iterative control compared with text prompting.","lead":"This paper presents SPEC, a structured, editable way to describe UI screenshots, and SpecifyUI, a tool that extracts SPEC from reference images and uses it to generate and refine web interfaces with large language models. It reports that professional designers rated SpecifyUI above Google Stitch on design quality, intent alignment, and control, suggesting a more predictable path for AI-assisted design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative fidelity claim is not yet attributable to SPEC: baseline models and RAG overlap are unspecified, and reported percentages conflict with Table 2.","rationale":"The reader's weakest_assumption focuses on extraction fidelity: whether Co-DETR and Gemini-2.5-Pro produce SPECs that faithfully represent the reference UI. That is a legitimate concern about absolute quality, but it is not the most load-bearing threat to the central claim. If extraction were imperfect, the system would be less faithful than it could be, attenuating rather than creating the reported gains; the claim is comparative, so imperfect extraction does not by itself explain why SPEC-based generation beats prompt baselines. The more decisive threat is attribution: the quantitative comparison does not control for the vision-language model used in the baseline prompts, and it never reports whether the RAG database overlaps with the test references. If the prompt baseline is a text-only code model or if RAG retrieves near-duplicates of the test UI, Table 2's improvements could be produced by model capability or leakage rather than by the SPEC intermediate representation. The Section 6.4 vs. Table 2 numeric mismatch reinforces that the quantitative foundation cannot currently be verified. The user study is a genuine strength: 16 professional designers, within-subjects comparison, expert ratings with reported inter-rater agreement, and objective typing-effort logs. But the abstract's headline claim rests on both the technical evaluation and the user study, and the technical evaluative deficiency is enough to keep the paper conditional rather than fully accepted. The reader's CONDITIONAL verdict remains appropriate, so I recommend UNCHANGED; the concrete test above would determine whether the concern actually lands.","tokens_in":22610,"tokens_out":8989,"duration_ms":102652,"concrete_test":"Specify the exact model and API for each Section 6.2 baseline, then re-run the comparison under matched conditions: (1) Direct Prompt using the same Gemini-2.5-Pro VLM that performs SPEC extraction, given the full reference screenshot and no SPEC scaffolding; (2) Integrated SPEC with retrieval restricted so that no test reference has a near-neighbor in the SPEC–UI Code Database (or with exact overlap reported). If Direct Prompt's MSE/CLIP/SSIM approaches Integrated SPEC within a small margin, or if Integrated SPEC degrades sharply once overlapping RAG examples are excluded, the Table 2 results do not demonstrate a SPEC-specific advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's core quantitative claim is that SPEC-based generation 'more faithfully captures reference intent than prompt-based baselines.' Establishing this requires that the comparison isolate SPEC as the causal variable. It does not. Section 6.2 never identifies which model powers Direct Prompt, Text-augmented Prompt, or Self-Revision Prompt, while Section 4.4 states the generator is Qwen3-Coder and Section 4.2 uses Gemini-2.5-Pro for SPEC extraction. If Direct Prompt is run on Qwen3-Coder with only a screenshot, the model cannot genuinely see the image, making the baseline structurally handicapped; if it is run on Gemini-2.5-Pro, the SPEC pipeline uses a different generator. Either way, the Table 2 gains (MSE 40.99 vs. 50.86; CLIP 0.887 vs. 0.755; SSIM 0.854 vs. 0.787) are confounded with model choice and with the 2,000-example RAG database whose overlap with the test set is never disclosed. The Section 6.4 text also reports improvements of 11.4% / +7.6% / +14.4%, which do not match the table (actual relative changes are about 19.4% / +17.5% / +8.5%). Without matched baselines, a stated test set, and RAG/test disjointness, the central claim that the SPEC representation itself drives fidelity is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SPEC, a structured, parameterized, hierarchical intermediate representation for UI design, and SpecifyUI, an interactive system that extracts SPEC from reference screenshots via a trained region detector and a vision-language model, allows designers to compose and edit SPECs, and renders them into React/Ant Design code through a multi-agent generator with retrieval augmentation. The authors report a quantitative benchmark showing that SPEC-based generation outperforms prompt-based baselines on MSE, CLIP similarity, and SSIM, and a within-subjects user study with 16 professional designers in which SpecifyUI is rated significantly higher than Google Stitch on intent alignment, design quality, controllability, and overall experience. The central claim is that a specification-driven intermediate representation improves both fidelity and controllability of LLM-based UI generation relative to prompt-based approaches.","tokens_in":22971,"tokens_out":4243,"duration_ms":52135,"significance":"If the empirical claims are supported, SPEC is a valuable and reusable intermediate representation for human-AI co-creation in UI design. The paper makes a concrete contribution by formalizing the schema (Eqs. 1–5), building a full extraction-generation pipeline, and evaluating it against both technical baselines and a commercial tool in a user study with professional designers. The user study is carefully designed with counterbalanced tasks, expert assessment of outputs, and objective interaction logs, and the qualitative findings align with the quantitative direction. The main caveat is that the technical benchmark, as reported, does not yet isolate SPEC as the causal factor: the baseline models are unspecified, the retrieval database may overlap with the test set, and several headline numbers do not match the tables. No code, data, or appendix prompts are provided, which limits reproducibility. Overall, the paper addresses an important problem and the paradigm is promising, but the evidence for the central quantitative claim needs to be tightened before publication.","major_comments":[{"comment":"The baselines are not sufficiently specified. Section 6.2 says Direct Prompt, Text-augmented Prompt, and Self-Revision Prompt are implemented by instructing 'an LLM' to generate React code, but never states which model. Section 4.4 states that the generator is Qwen3-Coder, while Section 4.2 uses Gemini-2.5-Pro for SPEC extraction. If Direct Prompt runs on a text-only code model with only a screenshot as input, the baseline is structurally handicapped; if it runs on a different multimodal model, then the comparison confounds the SPEC representation with the choice of generator. The paper must state the backbone and exact input modality for each baseline, and ideally run all conditions with the same generator, before the Table 2 gains can be attributed to SPEC.","section":"§6.2, §4.4"},{"comment":"The relative improvements reported in the text do not match Table 2. The abstract-level comparison to Self-Revision Prompt should be: MSE 40.99 vs 50.86 is a 19.4% reduction, not 11.4%; CLIP 0.887 vs 0.755 is a +17.5% increase, not +7.6%; SSIM 0.854 vs 0.787 is a +8.5% increase, not +14.4%. The numbers 11.4% and +7.6% appear to be computed against a different baseline or formula, while +14.4% does not match any direct ratio in the table. This needs to be corrected and the computation basis stated.","section":"§6.4, Table 2"},{"comment":"The RAG experiment is not cleanly defined. Section 4.4.1 describes a SPEC–UI code database of 2,000 human-validated SPEC–code pairs, but Section 6 never states what test set is used for the reconstruction benchmark or whether the test screenshots were excluded from the database. If any test image (or a near-duplicate) is among the 2,000 RAG examples, the SPEC+RAG and Integrated SPEC results could benefit from retrieval memorization rather than from the SPEC representation. The authors should describe the test set construction and confirm that it is disjoint from the RAG database, preferably also reporting retrieval overlap rates.","section":"§6.3, §4.4.1"},{"comment":"The inferential statistics for the expert assessment appear inconsistent with the reported sample size. The text says 16 participants produced 32 design samples (16 per system), which suggests a paired analysis with 16 pairs and d.f. = 15, yet all four t-tests report t(7). If the analysis was instead performed on 8 paired observations per task, this needs to be stated and justified; if the d.f. are misreported, all p-values in this subsection should be recomputed. As written, the reader cannot verify the significance claims for RQ1.","section":"§7.2.1"},{"comment":"The quantitative claim is that SPEC-based generation 'more faithfully captures reference intent.' This places weight on the SPEC extraction stage: if the Co-DETR region detector (trained on 8,000 self-annotated screenshots) or the Gemini-2.5-Pro VLM with few-shot prompting systematically misses or hallucinates regions/attributes, then the fidelity gains in Table 2 could reflect properties of the extraction setup rather than the SPEC representation's editability and structure. The paper reports no accuracy or error analysis for the extraction stage (e.g., region detection mAP, VLM attribute correctness). I recommend adding such an analysis, or at least a clear statement that the benchmark assumes near-correct extraction, so the reader can assess generalizability to inputs outside this pipeline.","section":"§4.2.1–§4.2.2"}],"minor_comments":[{"comment":"Typo: 'Deisgn' should be 'Design'.","section":"§5 title"},{"comment":"Typo: 'Basline' should be 'Baseline'.","section":"§7.1.2"},{"comment":"The object detection model name is garbled as 'Co-DI N O' in the figure; it should be 'Co-DETR'.","section":"Figure 3"},{"comment":"Spacing error: 'Specifysystem' should be 'Specify system' or 'SpecifyUI system'.","section":"§3.3"},{"comment":"The full extraction prompt is said to be 'provided in Appendix A,' but no appendix appears in the manuscript. Likewise, the prompting strategy in §4.4 is deferred to Appendix B. If these are in a supplementary document, this should be stated; otherwise the prompt templates are missing, which hinders reproducibility.","section":"§4.2.3"},{"comment":"The Qualitative Results subsection contains only a figure and no interpretive text. Please add a short discussion of what the qualitative comparison shows, including any failure cases.","section":"§6.5"},{"comment":"The last paragraph promises a discussion of 'scalability and limitations,' but no explicit limitations subsection is present. Adding a short limitations discussion would improve the paper's completeness.","section":"§8.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems paper with a credible user study, but the technical benchmark is not yet a clean comparison: the baseline models are unspecified, the reported percentages in §6.4 do not match Table 2, and the RAG/test overlap is not disclosed. I would not be comfortable accepting until these are resolved, ideally with matched baselines. The user study and qualitative findings are promising and could support a revised claim even if some quantitative details change."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is SPEC, a hierarchical, parameterized JSON-like representation that lets designers compose UIs from multiple references and make scoped edits without re-prompting from scratch. The system wraps that in a plausible pipeline: region detection, VLM extraction, RAG-conditioned code generation, and a user study against Google Stitch. That integration is genuinely new in one package, and the user study's direction is consistent: designers typed far less and reported better control. The expert ratings on output quality favor SpecifyUI across all four dimensions. That is real value for the HCI/UI-generation crowd.\n\nThe soft spots are mostly in the technical evaluation, and they are not minor. Section 6.2 never says which model powers the prompt baselines; the generator elsewhere is Qwen3-Coder, but if Direct Prompt is run on a model that cannot see images, it is structurally handicapped. The RAG database overlap with the test set is undisclosed, so retrieval could be memorizing. And the percentages in Section 6.4 do not match Table 2: the claimed 11.4% MSE reduction is actually 19.4%, the +7.6% CLIP claim is actually +17.5%, and the +14.4% SSIM claim is actually +8.5%. That kind of arithmetic error makes me distrust the numbers until corrected. The user study also reports df=7 for 32 samples, which needs clarification, and the abstract says 16 professional designers while the methods say university students plus professionals. The paper references Appendix A and B, but they are missing from the arXiv version; code and data are not released.\n\nNone of this kills the idea. The representation and system are plausible, and the direction matches concurrent work. But the central claim that SPEC itself drives fidelity is not yet supported because the comparisons are confounded. The paper is for HCI researchers building generative UI tools and intermediate representations; they will get a solid design rationale and a cautionary tale about evaluation. A serious referee should engage, but the paper needs major revision: specify the baselines, clarify the statistics, correct the arithmetic, and release the appendices or data.\n\nI would send it to peer review with a request for major revision. The concept deserves scrutiny; the current reporting does not support the strongest claims.","headline":"A promising SPEC-driven UI generation system whose idea and user study are worthwhile, but whose quantitative claims are not yet cleanly attributable to SPEC due to baseline and reporting problems.","tokens_in":23418,"tokens_out":2780,"would_cite":true,"duration_ms":30592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured specifications, not longer prompts, are what make AI-generated interfaces faithful and editable.","keywords":["user interface design","intermediate representation","design intent expression","controllable generation","large language models","vision-language models","human-AI co-creation","specification-driven design"],"falsifier":"Replace the automatically extracted SPEC in the fidelity benchmark with hand-verified SPECs for the same reference screenshots. If hand-verified SPECs do not beat the automatically extracted ones (or if extracted SPECs do not beat prompt baselines once the detector is fixed), then the reported gains come from the extraction stack, not from the SPEC representation. A second check: run the same user study while equalizing initial setup time and see whether the refinement-time advantage and intent-alignment ratings persist.","tokens_in":22487,"feed_emoji":"🎨","tokens_out":7694,"duration_ms":81294,"temperature":0.7,"pith_summary":"This paper claims that the main obstacle to useful AI-assisted interface design is not the power of large language models but the lack of a structured way for designers to say what they want and to keep it stable across revisions. To fix this it introduces SPEC, a layered, parameterized description of a page—global style and layout on top, then sections, then components—with each attribute stored as a mix of exact numbers (colors, spacing, grids) and semantic labels (roles, style tags). The system built around SPEC, SpecifyUI, reads reference screenshots, turns them into SPEC via region detection and a vision-language model, lets designers combine pieces from several references, and applies edits as small targeted updates to the specification instead of rewriting a prompt. The paper reports that SPEC-based generation reconstructs reference designs more faithfully than prompt-based baselines in pixel, semantic, and structural metrics, and that in a 16-designer comparison against a commercial prompt-based tool, SpecifyUI scored higher on intent alignment, design quality, controllability, and overall experience.","feed_headline":"Structured specs beat text prompts for UI design fidelity","feed_subtitle":"New spec layer: pull layout from one reference, style from another; edit one part without breaking the rest.","key_machinery":"The central object is the SPEC schema, a JSON-like intermediate representation whose two layers are the global specification (layout, color, shape, usage scenario as parameterized values plus semantic tags) and a page composition (page → section → component hierarchy with ids, positions, layouts, colors, and component sets). Its load-bearing property is the inheritance constraint: each section's attributes must be subsets of the global specification, so the representation is simultaneously hierarchical, editable, and globally consistent. The pipeline's other machinery is a Co-DETR region detector trained from scratch, a vision-language model with few-shot chain-of-thought for extraction, the","core_discovery":"SPEC is formalized as a two-level structure: a Global UI Specification G=(L,C,S,U), covering layout grid, color system, shape language, and usage scenario, and a Page Composition P in which a page decomposes into sections and each section into components. A constraint requires every section's attributes to specialize the global specification, which enforces coherence while permitting local variation. SpecifyUI obtains the full SPEC from a reference by segmenting the screenshot into regions with a transformer-based detector trained on 8,000 annotated pages, asking a vision-language model to emit a structured Region SPEC Unit per region, and asking it again on the whole page for a global desig","pith_inferences":["The same SPEC-style intermediate layer could transfer to other generative domains with strong visual structure—dashboards, slides, documents, data visualizations—where intent is hierarchical and prompt-based edits are unstable; the paper's design goals do not depend on UI-specific assumptions beyond the schema itself.","The quantitative fidelity gains are only as strong as the extraction step; a testable extension is to compare hand-authored SPECs against automatically extracted SPECs on the same screenshots, which would separate the representation's power from the detector and vision-language model's accuracy.","Sketch input is a natural next modality: because SPEC separates layout from style, a rough layout sketch could supply the page composition while a reference image supplies the global profile, giving designers an even more direct way to express intent.","Binding SPEC to mature component libraries and design tokens, which the paper mentions as future work, could make generated prototypes align with production design systems and reduce the last-mile handoff gap."],"forward_implications":["SPEC-based generation captures reference intent more faithfully than prompt-based baselines: in the paper's evaluation, integrated SPEC reaches an MSE of 40.99 versus 50.86 for the strongest prompt baseline, with higher CLIP and SSIM scores.","Designers can edit one part of a design—a region or a single component—without disturbing the rest, because edits update one node of the specification rather than regenerating from a changed prompt.","Multiple references can be composed at different levels: style from one image, layout from another, widgets from a third, while the global consistency constraint keeps the result coherent.","Because the specification drives code generation with retrieval and self-debugging, outputs are editable React prototypes rather than static images, shortening the path from ideation to handoff.","In the user study, participants typed about 73% fewer characters than with the prompt-based tool and finished refinement faster, suggesting the structured channel lowers iteration cost without increasing total task time."],"supporting_citations":[{"why":"Co-DETR, the transformer-based detector trained from scratch on 8,000 annotated screenshots; supplies the region segmentation that feeds SPEC extraction.","marker":"[81]"},{"why":"Stitch, the commercial prompt-based design tool used as the baseline in the formative study and the 16-designer user study.","marker":"[15]"},{"why":"Design2Code; provides the prompting baselines and the CLIP similarity metric used in the fidelity evaluation.","marker":"[57]"},{"why":"Chain-of-thought prompting; the few-shot CoT protocol that makes the vision-language model emit reliable Region SPEC Units.","marker":"[71]"},{"why":"Gestalt-based grouping; the post-processing that merges detection boxes into coherent regions while preserving full page coverage.","marker":"[78]"},{"why":"Jelly, a prior specification-driven UI system; defines the semi-formal specification baseline that SPEC extends with hierarchy and global constraints.","marker":"[4]"}],"fun_headline_variants":["Structured specs beat prompts for UI design fidelity","UI design: structured specs vs text prompts, specs win","Spec-based UI generation outperforms prompt-based baselines","Iterative UI design: structured specs enable control","Study: structured specs improve UI design intent alignment"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The advantage rests on the assumption that the region detector and vision-language model extract SPECs that faithfully and completely represent the reference UI's structure and global style; if segmentation misses or fragments regions, or the model invents attributes, the fidelity gains could come from the extraction setup rather than from SPEC's editability, and the controllability results would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Structured specs beat prompts for UI design fidelity","UI design: structured specs vs text prompts, specs win","Spec-based UI generation outperforms prompt-based baselines","Iterative UI design: structured specs enable control","Study: structured specs improve UI design intent alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":9.7e-05,"raw_usage":{"total_tokens":820,"prompt_tokens":695,"completion_tokens":125,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":51}},"tokens_in":439,"tokens_out":125,"duration_ms":2280,"temperature":1.0,"reasoning_tokens":51,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:23:24.976507+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the automatically extracted SPEC in the fidelity benchmark with hand-verified SPECs for the same reference screenshots. If hand-verified SPECs do not beat the automatically extracted ones (or if extracted SPECs do not beat prompt baselines once the detector is fixed), then the reported gains come from the extraction stack, not from the SPEC representation. A second check: run the same user study while equalizing initial setup time and see whether the refinement-time advantage and intent-alignment ratings persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Co-DETR, the transformer-based detector trained from scratch on 8,000 annotated screenshots; supplies the region segmentation that feeds SPEC extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Stitch, the commercial prompt-based design tool used as the baseline in the formative study and the 16-designer user study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Design2Code; provides the prompting baselines and the CLIP similarity metric used in the fidelity evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gestalt-based grouping; the post-processing that merges detection boxes into coherent regions while preserving full page coverage."}],"review_version":1}