Pith. sign in

REVIEW 5 major objections 7 minor 81 references

SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI

T0 review · 5 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured specifications, not longer prompts, are what make AI-generated interfaces faithful and editable.

desk verdict A promising SPEC-driven UI generation system whose idea and user study are worthwhile, but whose quantitative claims are not yet cleanly attributable to SPEC due to baseline and reporting problems. read the letter →

arxiv 2509.07334 v1 pith:HP7ENV6K submitted 2025-09-09 cs.HC

Yunnong Chen , Chengwei Shi , Liuqing Chen This is my paper · ORCID
classification cs.HC
keywords userinterfacedesignintermediaterepresentationintentexpressioncontrollablegenerationlargelanguagemodelsvision-languagehuman-AIco-creationspecification-driven
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main obstacle to useful AI-assisted interface design is not the power of large language models but the lack of a structured way for designers to say what they want and to keep it stable across revisions. To fix this it introduces SPEC, a layered, parameterized description of a page—global style and layout on top, then sections, then components—with each attribute stored as a mix of exact numbers (colors, spacing, grids) and semantic labels (roles, style tags). The system built around SPEC, SpecifyUI, reads reference screenshots, turns them into SPEC via region detection and a vision-language model, lets designers combine pieces from several references, and applies edits as small targeted updates to the specification instead of rewriting a prompt. The paper reports that SPEC-based generation reconstructs reference designs more faithfully than prompt-based baselines in pixel, semantic, and structural metrics, and that in a 16-designer comparison against a commercial prompt-based tool, SpecifyUI scored higher on intent alignment, design quality, controllability, and overall experience.

What carries the argument

The central object is the SPEC schema, a JSON-like intermediate representation whose two layers are the global specification (layout, color, shape, usage scenario as parameterized values plus semantic tags) and a page composition (page → section → component hierarchy with ids, positions, layouts, colors, and component sets). Its load-bearing property is the inheritance constraint: each section's attributes must be subsets of the global specification, so the representation is simultaneously hierarchical, editable, and globally consistent. The pipeline's other machinery is a Co-DETR region detector trained from scratch, a vision-language model with few-shot chain-of-thought for extraction, the

What would settle it

Replace the automatically extracted SPEC in the fidelity benchmark with hand-verified SPECs for the same reference screenshots. If hand-verified SPECs do not beat the automatically extracted ones (or if extracted SPECs do not beat prompt baselines once the detector is fixed), then the reported gains come from the extraction stack, not from the SPEC representation. A second check: run the same user study while equalizing initial setup time and see whether the refinement-time advantage and intent-alignment ratings persist.

Watch

Extended reading notes

Core claim

SPEC is formalized as a two-level structure: a Global UI Specification G=(L,C,S,U), covering layout grid, color system, shape language, and usage scenario, and a Page Composition P in which a page decomposes into sections and each section into components. A constraint requires every section's attributes to specialize the global specification, which enforces coherence while permitting local variation. SpecifyUI obtains the full SPEC from a reference by segmenting the screenshot into regions with a transformer-based detector trained on 8,000 annotated pages, asking a vision-language model to emit a structured Region SPEC Unit per region, and asking it again on the whole page for a global desig

Load-bearing premise

The advantage rests on the assumption that the region detector and vision-language model extract SPECs that faithfully and completely represent the reference UI's structure and global style; if segmentation misses or fragments regions, or the model invents attributes, the fidelity gains could come from the extraction setup rather than from SPEC's editability, and the controllability results would not generalize.

Editorial extensions

If this is right

  • SPEC-based generation captures reference intent more faithfully than prompt-based baselines: in the paper's evaluation, integrated SPEC reaches an MSE of 40.99 versus 50.86 for the strongest prompt baseline, with higher CLIP and SSIM scores.
  • Designers can edit one part of a design—a region or a single component—without disturbing the rest, because edits update one node of the specification rather than regenerating from a changed prompt.
  • Multiple references can be composed at different levels: style from one image, layout from another, widgets from a third, while the global consistency constraint keeps the result coherent.
  • Because the specification drives code generation with retrieval and self-debugging, outputs are editable React prototypes rather than static images, shortening the path from ideation to handoff.
  • In the user study, participants typed about 73% fewer characters than with the prompt-based tool and finished refinement faster, suggesting the structured channel lowers iteration cost without increasing total task time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same SPEC-style intermediate layer could transfer to other generative domains with strong visual structure—dashboards, slides, documents, data visualizations—where intent is hierarchical and prompt-based edits are unstable; the paper's design goals do not depend on UI-specific assumptions beyond the schema itself.
  • The quantitative fidelity gains are only as strong as the extraction step; a testable extension is to compare hand-authored SPECs against automatically extracted SPECs on the same screenshots, which would separate the representation's power from the detector and vision-language model's accuracy.
  • Sketch input is a natural next modality: because SPEC separates layout from style, a rough layout sketch could supply the page composition while a reference image supplies the global profile, giving designers an even more direct way to express intent.
  • Binding SPEC to mature component libraries and design tokens, which the paper mentions as future work, could make generated prototypes align with production design systems and reduce the last-mile handoff gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces SPEC, a structured, parameterized, hierarchical intermediate representation for UI design, and SpecifyUI, an interactive system that extracts SPEC from reference screenshots via a trained region detector and a vision-language model, allows designers to compose and edit SPECs, and renders them into React/Ant Design code through a multi-agent generator with retrieval augmentation. The authors report a quantitative benchmark showing that SPEC-based generation outperforms prompt-based baselines on MSE, CLIP similarity, and SSIM, and a within-subjects user study with 16 professional designers in which SpecifyUI is rated significantly higher than Google Stitch on intent alignment, design quality, controllability, and overall experience. The central claim is that a specification-driven intermediate representation improves both fidelity and controllability of LLM-based UI generation relative to prompt-based approaches.

Significance. If the empirical claims are supported, SPEC is a valuable and reusable intermediate representation for human-AI co-creation in UI design. The paper makes a concrete contribution by formalizing the schema (Eqs. 1–5), building a full extraction-generation pipeline, and evaluating it against both technical baselines and a commercial tool in a user study with professional designers. The user study is carefully designed with counterbalanced tasks, expert assessment of outputs, and objective interaction logs, and the qualitative findings align with the quantitative direction. The main caveat is that the technical benchmark, as reported, does not yet isolate SPEC as the causal factor: the baseline models are unspecified, the retrieval database may overlap with the test set, and several headline numbers do not match the tables. No code, data, or appendix prompts are provided, which limits reproducibility. Overall, the paper addresses an important problem and the paradigm is promising, but the evidence for the central quantitative claim needs to be tightened before publication.

major comments (5)
  1. [§6.2, §4.4] The baselines are not sufficiently specified. Section 6.2 says Direct Prompt, Text-augmented Prompt, and Self-Revision Prompt are implemented by instructing 'an LLM' to generate React code, but never states which model. Section 4.4 states that the generator is Qwen3-Coder, while Section 4.2 uses Gemini-2.5-Pro for SPEC extraction. If Direct Prompt runs on a text-only code model with only a screenshot as input, the baseline is structurally handicapped; if it runs on a different multimodal model, then the comparison confounds the SPEC representation with the choice of generator. The paper must state the backbone and exact input modality for each baseline, and ideally run all conditions with the same generator, before the Table 2 gains can be attributed to SPEC.
  2. [§6.4, Table 2] The relative improvements reported in the text do not match Table 2. The abstract-level comparison to Self-Revision Prompt should be: MSE 40.99 vs 50.86 is a 19.4% reduction, not 11.4%; CLIP 0.887 vs 0.755 is a +17.5% increase, not +7.6%; SSIM 0.854 vs 0.787 is a +8.5% increase, not +14.4%. The numbers 11.4% and +7.6% appear to be computed against a different baseline or formula, while +14.4% does not match any direct ratio in the table. This needs to be corrected and the computation basis stated.
  3. [§6.3, §4.4.1] The RAG experiment is not cleanly defined. Section 4.4.1 describes a SPEC–UI code database of 2,000 human-validated SPEC–code pairs, but Section 6 never states what test set is used for the reconstruction benchmark or whether the test screenshots were excluded from the database. If any test image (or a near-duplicate) is among the 2,000 RAG examples, the SPEC+RAG and Integrated SPEC results could benefit from retrieval memorization rather than from the SPEC representation. The authors should describe the test set construction and confirm that it is disjoint from the RAG database, preferably also reporting retrieval overlap rates.
  4. [§7.2.1] The inferential statistics for the expert assessment appear inconsistent with the reported sample size. The text says 16 participants produced 32 design samples (16 per system), which suggests a paired analysis with 16 pairs and d.f. = 15, yet all four t-tests report t(7). If the analysis was instead performed on 8 paired observations per task, this needs to be stated and justified; if the d.f. are misreported, all p-values in this subsection should be recomputed. As written, the reader cannot verify the significance claims for RQ1.
  5. [§4.2.1–§4.2.2] The quantitative claim is that SPEC-based generation 'more faithfully captures reference intent.' This places weight on the SPEC extraction stage: if the Co-DETR region detector (trained on 8,000 self-annotated screenshots) or the Gemini-2.5-Pro VLM with few-shot prompting systematically misses or hallucinates regions/attributes, then the fidelity gains in Table 2 could reflect properties of the extraction setup rather than the SPEC representation's editability and structure. The paper reports no accuracy or error analysis for the extraction stage (e.g., region detection mAP, VLM attribute correctness). I recommend adding such an analysis, or at least a clear statement that the benchmark assumes near-correct extraction, so the reader can assess generalizability to inputs outside this pipeline.
minor comments (7)
  1. [§5 title] Typo: 'Deisgn' should be 'Design'.
  2. [§7.1.2] Typo: 'Basline' should be 'Baseline'.
  3. [Figure 3] The object detection model name is garbled as 'Co-DI N O' in the figure; it should be 'Co-DETR'.
  4. [§3.3] Spacing error: 'Specifysystem' should be 'Specify system' or 'SpecifyUI system'.
  5. [§4.2.3] The full extraction prompt is said to be 'provided in Appendix A,' but no appendix appears in the manuscript. Likewise, the prompting strategy in §4.4 is deferred to Appendix B. If these are in a supplementary document, this should be stated; otherwise the prompt templates are missing, which hinders reproducibility.
  6. [§6.5] The Qualitative Results subsection contains only a figure and no interpretive text. Please add a short discussion of what the qualitative comparison shows, including any failure cases.
  7. [§8.3] The last paragraph promises a discussion of 'scalability and limitations,' but no explicit limitations subsection is present. Adding a short limitations discussion would improve the paper's completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central claims rest on independent empirical comparisons against prompt baselines and Google Stitch.

full rationale

The paper's central claims are empirical: SPEC-based generation is compared against prompt-based baselines on a reconstruction-fidelity benchmark, and SpecifyUI is compared against Google Stitch in a user study. Although SPEC is extracted from the same reference screenshot used as the fidelity ground truth, this is an intrinsic reconstruction task rather than a derivation where the output equals the input by construction: the generator is a separate Qwen3-Coder/Claude-4 code pipeline, and the baselines receive the same reference. No fitted parameter is renamed as a prediction: Co-DETR is trained on 8,000 annotated screenshots, but the benchmark measures UI generation fidelity rather than detector accuracy, and the RAG database (2,000 SPEC-code pairs) is an input component whose test-set overlap is a validity concern, not circularity. Self-citations (refs 5, 6, 8) appear only in related-work and task-scenario contexts and are not load-bearing for the core comparison. The inconsistency between the percentage gains reported in Section 6.4 and Table 2 is an internal-consistency/correctness issue, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

SPEC is the paper's central invented construct, and the entire system depends on the adequacy of its schema. Several hand-set or unreported parameters govern region extraction and retrieval; the evaluation proxies are domain assumptions rather than externally validated measures. No free parameters are fitted to data in the sense of a numerical model, but the design choices are load-bearing and not independently benchmarked.

free parameters (3)
  • Co-DETR region merge thresholds = not reported
    Hand-set criteria for merging boxes (alignment, proximity, enclosure) in Section 4.2.1; region extraction quality is upstream of every SPEC fidelity result.
  • Few-shot prompt templates for VLM extraction and edit generation = not reported
    Prompts referenced as Appendices A and B (Sections 4.2.3 and 4.4) are absent from the arXiv v1; these determine SPEC extraction and edit-triplet quality.
  • RAG retrieval top-k and similarity threshold = not reported
    The number of nearest-neighbor SPEC-code pairs injected into the code prompt is unspecified (Section 4.4.1); retrieval content directly affects generation fidelity.
assumptions (4)
  • ad hoc to paper The SPEC schema (global G = <L,C,S,U> plus page-to-section-to-component hierarchy) suffices to capture UI design intent.
    Equations (1)-(5) in Section 4.1 define the schema; the entire pipeline and evaluation assume this encoding is complete enough, without comparison to alternative schemas.
  • domain assumption MSE, CLIP similarity, and SSIM relative to the screenshot are valid proxies for reference intent.
    Section 6.1 defines these as fidelity measures; no user validation of the metrics is reported.
  • domain assumption Co-DETR detection plus Gestalt merging yields exhaustive, semantically coherent region segmentation.
    Section 4.2.1; the annotation rule that the union of bounding boxes fully covers the page and the post-processing merge assume perfect coverage.
  • domain assumption The 2,000 SPEC-code pairs built with Claude-4 and human validation are accurate, and the retrieval database is disjoint from the evaluation set.
    Section 4.4.1; the paper does not state that the evaluation screenshots are separate from the RAG database, so retrieval gains could be inflated by near-duplicates.
invented entities (1)
  • SPEC (structured, parameterized, hierarchical UI specification)
    purpose: Intermediate representation capturing layout, color, shape, usage scenario, and page/section/component hierarchy to guide LLM generation and edits.
    The only evidence for SPEC's usefulness is the authors' own pipeline evaluation and user study; no external benchmark or independent reproduction is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI." pith.science (2026). https://pith.science/paper/HP7ENV6K

@misc{pith2026250907334,
  author       = {Pith},
  title        = {Pith review of: SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HP7ENV6K}},
  note         = {Machine review of arXiv:2509.07334}
}
read the original abstract

Large language models (LLMs) promise to accelerate UI design, yet current tools struggle with two fundamentals: externalizing designers' intent and controlling iterative change. We introduce SPEC, a structured, parameterized, hierarchical intermediate representation that exposes UI elements as controllable parameters. Building on SPEC, we present SpecifyUI, an interactive system that extracts SPEC from UI references via region segmentation and vision-language models, composes UIs across multiple sources, and supports targeted edits at global, regional, and component levels. A multi-agent generator renders SPEC into high-fidelity designs, closing the loop between intent expression and controllable generation. Quantitative experiments show SPEC-based generation more faithfully captures reference intent than prompt-based baselines. In a user study with 16 professional designers, SpecifyUI significantly outperformed Stitch on intent alignment, design quality, controllability, and overall experience in human-AI co-creation. Our results position SPEC as a specification-driven paradigm that shifts LLM-assisted design from one-shot prompting to iterative, collaborative workflows.

Figures

Figures reproduced from arXiv: 2509.07334 by the authors.

Figure 1
Figure 1. SpecifyUI introduces a vision-centered intermediate representation to make design intent explicit and controllable in UI generation: (a) the user begins by extracting specifications, identifying design elements such as Region, Style, and Layout; (b) these elements are then composed into the intermediate representation, SPEC Embedding, to form a coherent whole; (c) the design can be iteratively refined by directly se… view at source ↗
Figure 2
Figure 2. Mapping user intent into structured UI representations via SPEC. Designers express high-level product goals, page-level composition, and stylistic preferences, which are translated into a multi-layered SPEC representation. This structured prompt then guides large language models to generate UI layouts aligned with the designer’s goals. Input Intent SPEC Initialization Reference Pages Reference Image 1 Reference Imag… view at source ↗
Figure 3
Figure 3. The technical pipeline of SpecifyUI transforms user intent into three stages: initializing SPEC, editing SPEC, and finally rendering it into a UI design. 4.2.1 Region Segmentation via Visual Parsing. To enable structured analysis of reference UIs, we first segment each interface into semantically coherent regions. This preprocessing step transforms a complex layout into manageable visual units, thereby reducing the … view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The user interface of SpecifyUI consists of two panels: (A) the SPEC Composition Panel, which supports extracting design specifications, arranging components, and generating UIs; and (B) the UI Editing Panel, which enables direct manipulation by selecting elements thro…
Figure 5
Figure 5. Figure 5: Qualitative results of the UI generation pipeline compared with the best baseline method for fidelity-oriented design generation. 6.5 Qualitative Results 7 User Study Building on the technical evidence of intent alignment, the user study evaluates whether these gains t…
Figure 6
Figure 6. Figure 6: User Study Procedure Stitch, with task order counterbalanced across participants. Each design was evaluated along four reference-related dimensions: (i) style consistency, (ii) layout consistency, (iii) component correctness, and (iv) task relevance. Experts rated each…
Figure 7
Figure 7. Figure 7: Examples of participant design processes using SpecifyUI and Stitch, illustrating user inputs of color, layout, and region component references. Style Consistency Layout Consistency Component Correctness Task Relevance 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 …
Figure 8
Figure 8. Figure 8: Box plots of expert ratings on style consistency, layout consistency, component correctness, and task relevance, showing consistently higher scores for SpecifyUI over Stitch. 7.2 Results 7.2.1 RQ1: Design Output Quality [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Subjective ratings for intent expression and control, comparing SpecifyUI and Stitch across five dimensions: controllability in generation, controllability in editing, intent adherence, articulation efficiency, and ease of intent communication. 7.2.2 RQ2: Intent Expres…
Figure 10
Figure 10. Figure 10: Task completion times for initial generation and iterative refinement across two systems [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Subjective ratings of interaction experience comparing SpecifyUI and Stitch across five dimensions. logs ( [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Objective typing effort across edit scopes (global, region, component) and content types (style, layout, functional￾ity). noted, “I know what style or layout I want from a reference, but I don’t know how to describe it. Seeing the extracted design features gave me exa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 52 canonical work pages

  1. [1]

    Tyler Angert, Miroslav Suzara, Jenny Han, Christopher Pondoc, and Hariharan Subramonyam. 2023. Spellburst: A node-based interface for exploratory creative coding with natural language prompts. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–22

    2023

  2. [2]

    Ian Arawjo, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: An open-source visual programming environment for prompt engineering. InAdjunct proceedings of the 36th annual ACM symposium on user interface software and technology. 1–3

    work page 2023

  3. [3]

    Judy Bowen and Steve Reeves. 2008. Formal models for user interface design artefacts.Innovations in Systems and Software Engineering4, 2 (2008), 125–141

    work page 2008

  4. [4]

    Yining Cao, Peiling Jiang, and Haijun Xia. 2025. Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–20

    2025

  5. [5]

    Liuqing Chen, Qianzhi Jing, Yixin Tsang, Qianyi Wang, Ruocong Liu, Duowei Xia, Yunzhan Zhou, and Lingyun Sun. 2024. AutoSpark: Supporting Automobile Appearance Design Ideation with Kansei Engineering and Generative AI. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–19. Manuscript submitted to ACM SpecifyUI: Sup...

    work page 2024

  6. [6]

    Liuqing Chen, Yaxuan Song, Ke Lyu, Shuhong Xiao, Yilang Shen, and Lingyun Sun. 2025. SCENIC: A Location-based System to Foster Cognitive Development in Children During Car Rides. arXiv:2508.17058 [cs.HC] https://arxiv.org/abs/2508.17058

    work page Pith review arXiv 2025

  7. [7]

    Xiang ’Anthony Chen, Tiffany Knearem, and Yang Li. 2025. The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyond. InProceedings of the 2025 ACM Designing Interactive Systems Conference (DIS ’25). Association for Computing Machinery, New York, NY, USA, 1179–1196. https://doi.org/10.1145/3715336.3735780

    arXiv 2025

  8. [8]

    Yunnong Chen, Shixian Ding, YingYing Zhang, Wenkai Chen, Jinzhou Du, Lingyun Sun, and Liuqing Chen. 2025. DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models.arXiv preprint arXiv:2506.13663(2025)

    arXiv 2025

Show all 81 references
  1. [9]

    John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19

    2022

  2. [10]

    WT Council and GT Heineman. 2001. Component-based software engineering putting the pieces together.Addison Weysley150 (2001)

    2001

  3. [11]

    Fred D Davis et al. 1989. Technology acceptance model: TAM.Al-Suqri, MN, Al-Aufi, AS: Information Seeking Behavior and Technology Adoption205, 219 (1989), 5

    1989

  4. [12]

    Antonio Delgado, Antonio Estepa, JA Troyano, and Rafael Estepa. 2016. Reusing UI elements with model-based user interface development. International Journal of Human-Computer Studies86 (2016), 48–62

    2016

  5. [13]

    Nishit Gajjar, Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. Akin: Generating ui wireframes from ui design patterns using deep learning. InCompanion Proceedings of the 26th International Conference on Intelligent User Interfaces. 40–42

    2021

  6. [14]

    Galileo AI. 2024. Galileo AI. https://www.usegalileo.ai/. Accessed: 2025-07-10

    2024

  7. [15]

    Google. 2024. Stitch by Google. https://stitch.withgoogle.com/. Accessed: 2025-07-10

    2024

  8. [16]

    Julian Jorge Andrade Guerreiro, Naoto Inoue, Kento Masui, Mayu Otani, and Hideki Nakayama. 2024. LayoutFlow: Flow Matching for Layout Generation. InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXXVI(Milan,...

    2024 doi

  9. [17]

    Yi Gui, Zhen Li, Zhongyi Zhang, Guohao Wang, Tianpeng Lv, Gaoyang Jiang, Yi Liu, Dongping Chen, Yao Wan, Hongyu Zhang, et al. 2025. LaTCoder: Converting Webpage Design to Code with Layout-as-Thought. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Da...

    2025

  10. [18]

    Yi Gui, Yao Wan, Zhen Li, Zhongyi Zhang, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, et al. 2025. UICoPilot: Automating UI synthesis via hierarchical code generation from webpage designs. InProceedings of the ACM on Web Conference 2025. 1846–1855

    2025

  11. [19]

    Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. 2021. Layouttransformer: Layout generation and completion with self-attention. InProceedings of the IEEE/CVF International Conference on Computer Vision. 1004–1014

    2021

  12. [20]

    Jeffrey Heer and Michael Bostock. 2010. Declarative language design for interactive visualization.IEEE transactions on visualization and computer graphics16, 6 (2010), 1149–1156

    2010

  13. [21]

    Josh Holinaty, Alec Jacobson, and Fanny Chevalier. 2021. Supporting reference imagery for digital drawing. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2434–2442

    2021

  14. [22]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278(2025)

    2025 arXiv

  15. [23]

    Scott E Hudson. 1994. User interface specification using an enhanced spreadsheet model.ACM Transactions on Graphics (TOG)13, 3 (1994), 209–239

    1994

  16. [24]

    Scott E Hudson and Shamim P Mohamed. 1990. Interactive specification of flexible user interface displays.ACM Transactions on Information Systems (TOIS)8, 3 (1990), 269–288

    1990

  17. [25]

    Robert JK Jacob. 1983. Using formal specifications in the design of a human-computer interface.Commun. ACM26, 4 (1983), 259–264

    1983

  18. [26]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–30

    2024

  19. [27]

    Yue Jiang, Changkong Zhou, Vikas Garg, and Antti Oulasvirta. 2024. Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18

    2024

  20. [28]

    Qianzhi Jing, Tingting Zhou, Yixin Tsang, Liuqing Chen, Lingyun Sun, Yankun Zhen, and Yichun Du. 2023. Layout generation for various scenarios in mobile shopping applications. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18

    2023

  21. [29]

    Ben Jonson. 2005. Design ideation: the conceptual sketch in the digital age.Design studies26, 6 (2005), 613–624

    2005

  22. [30]

    Tae Soo Kim. 2025. Interacting with AI by Manipulating Intents. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1–5

    2025

  23. [31]

    David Kirsh. 2010. Thinking with external representations.AI & society25, 4 (2010), 441–454

    2010

  24. [32]

    Janin Koch, Nicolas Taffin, Michel Beaudouin-Lafon, Markku Laine, Andrés Lucero, and Wendy E Mackay. 2020. Imagesense: An intelligent collaborative ideation tool to support diverse human-computer partnerships.Proceedings of the ACM on human-computer interaction4, CSCW1 (2020), 1–27

    2020

  25. [33]

    Carine Lallemand, Guillaume Gronier, and Vincent Koenig. 2015. User experience: A concept without consensus? Exploring practitioners’ perspectives through an international survey.Computers in human behavior43 (2015), 35–48

    2015

  26. [34]

    Yassine Lamine and Jinghui Cheng. 2022. Understanding and supporting the design systems practice.Empirical Software Engineering27, 6 (2022), 146. Manuscript submitted to ACM 26 Chen et al

    2022

  27. [35]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing sy...

    2020

  28. [36]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompting for code generation.ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–23

    2025

  29. [37]

    Jieru Lin, Danqing Huang, Tiejun Zhao, Dechen Zhan, and Chin-Yew Lin. 2024. Spot the error: Non-autoregressive graphic layout generation with wireframe locator. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 3413–3421

    2024

  30. [38]

    Yuwen Lu, Alan Leung, Amanda Swearngin, Jeffrey Nichols, and Titus Barik. 2025. Misty: Ui prototyping through interactive conceptual blending. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–17

    2025

  31. [39]

    Jenny GuangZhen Ma, Karthik Sreedhar, Vivian Liu, Pedro A Perez, Sitong Wang, Riya Sahni, and Lydia B Chilton. 2025. Dynex: Dynamic code synthesis with structured design exploration for accelerated exploratory programming. InProceedings of the 2025 CHI Conference on Human Fact...

    2025

  32. [40]

    Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for ...

    2024

  33. [41]

    mastergo AI. 2024. mastergo AI. https://mastergo.com/. Accessed: 2025-07-10

    2024

  34. [42]

    Marjan Mernik, Jan Heering, and Anthony M Sloane. 2005. When and how to develop domain-specific languages.ACM computing surveys (CSUR) 37, 4 (2005), 316–344

    2005

  35. [43]

    Pedro J Molina, Santiago Meliá, and Oscar Pastor. 2002. Just-ui: A user interface specification model. InComputer-Aided Design of User Interfaces III: Proceedings of the Fourth International Conference on Computer-Aided Design of User Interfaces 15–17 May 2002, Valenciennes, F...

    2002

  36. [44]

    Mohammad Amin Mozaffari, Xinyuan Zhang, Jinghui Cheng, and Jin LC Guo. 2022. GANSpiration: balancing targeted and serendipitous inspiration in user interface design with style-based generative adversarial network. InProceedings of the 2022 CHI conference on human factors in co...

    2022

  37. [45]

    Brad Myers, Scott E Hudson, and Randy Pausch. 2000. Past, present, and future of user interface software tools.ACM Transactions on Computer-Human Interaction (TOCHI)7, 1 (2000), 3–28

    2000

  38. [46]

    Arpit Narechania, Arjun Srinivasan, and John Stasko. 2020. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries.IEEE Transactions on Visualization and Computer Graphics27, 2 (2020), 369–379

    2020

  39. [47]

    Jeffrey Nichols, Brad A Myers, Michael Higgins, Joseph Hughes, Thomas K Harris, Roni Rosenfeld, and Mathilde Pignol. 2002. Generating remote control interfaces for complex appliances. InProceedings of the 15th annual ACM symposium on User interface software and technology. 161–170

    2002

  40. [48]

    Jeffrey Nichols, Brad A Myers, and Kevin Litwack. 2004. Improving automatic interface generation with smart templates. InProceedings of the 9th international conference on Intelligent user interfaces. 286–288

    2004

  41. [49]

    Jeffrey Nichols, Brad A Myers, and Brandon Rothrock. 2006. UNIFORM: automatically generating consistent remote control user interfaces. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 611–620

    2006

  42. [50]

    Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, and Jinwook Seo. 2025. Leveraging Multimodal LLM for Inspirational User Interface Search. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–22

    2025

  43. [51]

    Xiaohan Peng, Janin Koch, and Wendy E. Mackay. 2024. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. In Proceedings of the 2024 ACM Designing Interactive Systems Conference(Copenhagen, Denmark)(DIS ’24). Association for Computing Machinery...

    2024

  44. [52]

    Thiago Rocha Silva. 2022. Towards a domain-specific language to specify interaction scenarios for web-based graphical user interfaces. InCompanion of the 2022 ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 48–53

    2022

  45. [53]

    Vishnu Sarukkai, Lu Yuan, Mia Tang, Maneesh Agrawala, and Kayvon Fatahalian. 2024. Block and Detail: Scaffolding Sketch-to-Image Generation. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology(Pittsburgh, PA, USA)(UIST ’24). Association for...

    2024

  46. [54]

    Arvind Satyanarayan and Graham M Jones. 2024. Intelligence as agency: Evaluating the capacity of generative AI to empower or constrain human action. (2024)

    2024

  47. [55]

    Xinyu Shi, Yinghou Wang, Ryan Rossi, and Jian Zhao. 2025. Brickify: Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing...

    2025

  48. [56]

    Atefeh Shokrizadeh, Boniface Bahati Tadjuidje, Shivam Kumar, Sohan Kamble, and Jinghui Cheng. 2025. Dancing With Chains: Ideating Under Constraints With UIDEC in UI/UX Design. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–23

    2025

  49. [57]

    Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2025. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comp...

    2025

  50. [58]

    Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen, and Sean Rintel. 2025. Ironies of generative AI: understanding and mitigating productivity loss in Human-AI interaction.International Journal of Human–Computer Interaction41, 5 (2025), 2898–2...

    2025

  51. [59]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with llms. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–19

    2024

  52. [60]

    Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured generation and exploration of design space with large language models for human-ai co-creation. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26

    2024

  53. [61]

    Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The metacognitive demands and opportunities of generative AI. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–24

    2024

  54. [62]

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasub- ramanian. 2024. Appworld: A controllable world of apps and people for benchmarking interactive coding agents.arXiv preprint arXiv:2407.189...

    2024 arXiv

  55. [63]

    uizard. 2024. uizard. https://uizard.io/. Accessed: 2025-07-10

    2024

  56. [64]

    Priyan Vaithilingam, Elena L Glassman, Jeevana Priya Inala, and Chenglong Wang. 2024. Dynavis: Dynamically synthesized ui widgets for visualization editing. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–17

    2024

  57. [65]

    Priyan Vaithilingam and Philip J Guo. 2019. Bespoke: Interactively synthesizing custom GUIs from command-line applications by demonstration. In Proceedings of the 32nd annual ACM symposium on user interface software and technology. 563–576

    2019

  58. [66]

    Samangi Wadinambiarachchi, Ryan M Kelly, Saumya Pareek, Qiushi Zhou, and Eduardo Velloso. 2024. The effects of generative ai on design fixation and divergent thinking. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18

    2024

  59. [67]

    Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael Lyu. 2025. Divide-and-Conquer: Generating UI Code from Screenshots.Proceedings of the ACM on Software Engineering2, FSE (2025), 2099–2122

    2025

  60. [68]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17

    2023

  61. [69]

    Ruotong Wang, Xinyi Zhou, Lin Qiu, Joseph Chee Chang, Jonathan Bragg, and Amy X Zhang. 2025. Social-RAG: Retrieving from Group Interactions to Socially Ground AI Generation. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–25

    2025

  62. [70]

    Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2024. PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for C...

    2024

  63. [71]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  64. [72]

    Justin D Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer. 2024. Design principles for generative AI applications. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–22

    2024

  65. [73]

    Fan Wu, Cuiyun Gao, Shuqing Li, Xin-Cheng Wen, and Qing Liao. 2025. MLLM-Based UI2Code Automation Guided by UI Layout Information. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1123–1145

    2025

  66. [74]

    Jason Wu. 2024. UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Lon...

    2024

  67. [75]

    Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021. Screen parsing: Towards reverse engineering of ui models from screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology. 470–483

    2021

  68. [76]

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. InProceedings of the 2022 CHI conference on human factors in computing systems. 1–22

    2022

  69. [77]

    Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian. 2024. Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–14

    2024

  70. [78]

    Mulong Xie, Zhenchang Xing, Sidong Feng, Xiwei Xu, Liming Zhu, and Chunyang Chen. 2022. Psychologically-inspired, unsupervised inference of perceptual groups of GUI widgets from GUI images. InProceedings of the 30th ACM joint European software engineering conference and sympos...

    2022

  71. [79]

    Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013. The Auckland layout editor: An improved GUI layout specification process. InProceedings of the 26th annual ACM symposium on User interface software and technology. 343–352

    2013

  72. [80]

    Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. Guigan: Learning to generate gui designs using generative adversarial networks. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 748–760

    2021

  73. [81]

    Zhuofan Zong, Guanglu Song, and Yu Liu. 2023. Detrs with collaborative hybrid assignments training. InProceedings of the IEEE/CVF international conference on computer vision. 6748–6758. Manuscript submitted to ACM

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.