{"id":"a7d59ae1-251e-4250-b2b4-0df21b8404fe","arxiv_id":"2608.04293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A staged, inspectable UI generation framework with provenance-carrying DSL items reports higher perceived transparency and controllability for beginner designers than one-shot tools.","lead":"This paper presents LegoUI, a staged tool that turns a UI prompt into a series of editable design decisions written in a UI-specific language. A small user study reports that beginner designers found the staged process more transparent and controllable than one-shot generative tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Technical evaluation's 95% accuracy may be self-consistency: the ground truth is built by annotators using LegoUI's own requirement-analysis output on an LLM-standardized corpus, and the metric excludes omissions, so the transparency claim lacks an independent measure of faithful extraction.","rationale":"The reader's weakest assumption and my concern coincide: the technical evaluation's ground truth is generated with the system under test, the corpus is LLM-standardized, and the metrics are lenient. I focused on this rather than the user-study limitations because it is the least secure link in the argument for the central mechanism. The central claim is that staged, inspectable UI-DSL generation yields higher perceived transparency, controllability, and alignment. For transparency to be meaningful, the system must actually capture explicit requirements from user prompts, not merely reproduce its own DSL categories. The user study provides subjective evidence, but the quantitative claim of over 95% accuracy is what would independently establish faithful extraction. If that quantitative foundation is circular, the paper's claim that users can 'inspect how the system interprets their prompt' is not supported beyond the self-consistent behavior of the pipeline. The concern is concrete and testable: construct ground truth without exposing annotators to the system's output or DSL categories, use the original prompts, and apply a stricter accuracy metric with omissions in the denominator. I recommend keeping the reader's CONDITIONAL verdict because the user study and system concept retain value, but the headline accuracy number should not be treated as established until this check is run.","tokens_in":23050,"tokens_out":6847,"duration_ms":65880,"concrete_test":"Re-run the technical evaluation on the original 36 unmodified UIPrompt prompts (without GPT-4o standardization), with ground truth created independently: annotators who have never seen LegoUI's UI-DSL or its generated PromptChecklists first mark explicit requirements in free-form, and a separate mapping step then assigns these to DSL categories. Compute Accuracy with omitted items included in the denominator and report exact-match accuracy excluding Partial Matches. If Accuracy or Coverage drops substantially (for example, below 85% or with a large OM rate), the reported 95% reflects self-consistency rather than faithful extraction, and the transparency claim needs to be re-benchmarked.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative support for LegoUI's transparency claim is the requirement-analysis evaluation in Section 7. The reference PromptChecklist is constructed in Section 7.1.2 by annotators who 'used our system to perform the requirement analysis step' and organized it according to the system's DSL sections (intentSpec, globalSpec, componentPool, etc.). This creates a circular reference standard: the same DSL categories that define LegoUI's output also define what annotators look for and what they count as an explicit requirement. The corpus is also not raw real-world input: Section 7.1.1 rewrites the 36 collected prompts with GPT-4o to fixed length and complexity criteria and adds four GPT-4o-generated prompts. The scoring in Section 7.1.3 further counts Partial Match as correct and computes Accuracy with omissions (OM) excluded from the denominator, so '95% accuracy' can remain high even if many requirements are missed, and Coverage is measured against a GT that was scaffolded by the system. Under this protocol, high agreement between system output and GT mostly demonstrates that the system reproduces its own DSL categories, not that it faithfully extracts user intent from arbitrary prompts. If extraction fidelity is unvalidated, the claimed benefit that users can inspect how the system interpreted their prompt rests on an unreliable foundation: users would be inspecting a self-consistent but possibly unfaithful interpretation. This does not invalidate the user-study perceptions, but it removes the quantitative grounding that would make the transparency claim general.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LegoUI, a staged generative UI design framework that externalizes intermediate design decisions into a UI-specific DSL with provenance, allowing users to inspect, accept, reject, or add items across generation stages. It reports a technical evaluation of the requirement-analysis stage on 40 prompts, claiming over 95% accuracy, near-complete coverage, and zero redundancy, and a within-subject user study with 15 participants comparing LegoUI to one-shot tools on perceived transparency, controllability, intent alignment, and workload. The central claim is that staged, inspectable generation improves transparency and controllability compared to one-shot GenUI systems.","tokens_in":23472,"tokens_out":6391,"duration_ms":59957,"significance":"If the central claims hold, the paper makes a useful contribution to GenUI research: it offers a concrete mechanism for exposing intermediate reasoning in UI generation, with a persistent, provenance-aware DSL that supports iteration and cross-stage reuse. The system design is clearly described, the DSL grammar in Appendix D is a genuine design artifact, and the user study combines Likert questionnaires, preference rankings, and semi-structured interviews, giving qualitative insight into how participants used staged control. The paper also discusses trade-offs honestly, including the interaction cost of inspectability in Section 9.5. However, the quantitative foundation for the transparency claim is not yet independent: the technical evaluation's reference standard is partly generated by the system under test, the metrics are lenient, and the abstract overstates the section-level results. The user study findings are suggestive, but the statistical reporting needs strengthening before the significance claims can be fully assessed.","major_comments":[{"comment":"The reference PromptChecklist is constructed by annotators who used the system to perform the requirement-analysis step and organized it according to the system's top-level DSL structure. Because the same UI-DSL categories define both the extraction target and the annotation scheme, the reported agreement partly reflects self-consistency rather than independent extraction fidelity. I recommend constructing the ground truth from the raw standardized prompts without exposing annotators to system output, and reporting inter-annotator agreement against that independent standard.","section":"Section 7.1.2"},{"comment":"The Accuracy metric is (MT+PM)/(MT+PM+ML+EX), so Omitted items are excluded from the denominator and Partial Matches count as correct. As a result, the reported 95% accuracy can coexist with many omissions, and exact-match performance is not separately reported. Please report precision and recall with omissions included in the appropriate denominator, and give exact-match rates separately from partial-match rates.","section":"Section 7.1.3"},{"comment":"The claim of over 95% accuracy is not supported by Table 4, where globalSpec has 92.77% accuracy and styleSpec has 89.29%. The abstract and summary should be revised to present section-level results and should not state an aggregate figure that exceeds the observed section-level values.","section":"Abstract and Section 7.2"},{"comment":"layoutSpec (N=23) and relationSpec (N=14) rest on very small samples, and relation items were injected by adding four GPT-4o-generated prompts after the 36 collected prompts contained no relationSpec. The perfect scores for these sections should be de-emphasized, and the dataset should be described as an artificially balanced corpus rather than 40 real-world design prompts.","section":"Section 7.1.1 and Tables 3-4"},{"comment":"Tasks 2 and 3 compare systems within subjects without explicit order counterbalancing, and multiple pairwise t-tests are reported without correction for multiple comparisons. With N=15, this creates nontrivial Type I error risk; please report confidence intervals, correct for multiple comparisons or justify the uncorrected approach, and state how system order was handled in the within-subject design.","section":"Sections 8.2-8.4"}],"minor_comments":[{"comment":"The phrase after verifying that the 36 ground-truth prompts contained no relationSpec is unclear; please clarify that relationSpec was absent and explain the implications of adding relation-heavy prompts for the naturalness of the corpus.","section":"Section 7.1.1"},{"comment":"Several cells contain concatenated values, such as 4.474.07 and 4.534.40; these formatting errors should be fixed.","section":"Table 6"},{"comment":"Appendix D refers to Section 5.4 for provenance handling, but the relevant subsection is Section 5.3; the cross-reference should be corrected.","section":"Appendix D"},{"comment":"The technical evaluation claims would be easier to verify if the coding protocol, annotation instructions, and de-identified extracted checklists were released publicly rather than available only upon request.","section":"Data Availability"},{"comment":"The prompt-standardization criteria and the number and content of the four supplementary prompts are design choices that are not varied or justified; a brief discussion of their influence on the results would improve the evaluation.","section":"Section 7.1.1"},{"comment":"The statement that Claude Chat was reliably above the baseline one-shot tools uses reliable for differences that may not all be statistically significant; please align this wording with the reported significance tests.","section":"Section 8.4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core idea and a useful qualitative user study, but the technical evaluation needs to be substantially revised before publication. I do not recommend rejection because the central issues are fixable: the ground-truth protocol and metrics can be redone or re-reported, and the user-study claims can be supported with more careful statistical reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a paper worth engaging with. The staged UI-DSL with provenance is a concrete, reusable pattern for making LLM-driven UI generation inspectable, and the user study gives credible, well-qualified evidence for the qualitative claims. The headline 95% accuracy, however, should not be quoted — the technical evaluation's ground truth was built on the system's own DSL categories, and the metric is lenient.\n\nWhat's actually new: a five-stage pipeline (requirements, layout, interaction, relation, style) that writes design decisions into a persistent UI-DSL with provenance, letting users accept, reject, or add items that shape subsequent generation. The grammar in Appendix D is detailed enough to build on. The user study (n=15, within-subjects) compares against Bolt, Lovable, v0, and Claude Chat, and the interview data supports the framing of transparency as \"checking the system's interpretation\" and control as \"early intervention.\" They also honestly report the cost: inspection overload, and difficulty judging interaction/relation decisions from text alone. The P4 workflow is a compelling trace.\n\nSoft spots, in order: (1) Section 7.1.2 — annotators used the LegoUI requirement-analysis step to build the reference PromptChecklist, organized by the system's own DSL sections. That makes the 95%+ \"accuracy\" partly a measure of self-consistency. (2) The scoring counts partial matches as correct and excludes omissions from the accuracy denominator, so high accuracy can coexist with missed requirements. StyleSpec shows the signs: 80% matched, 8% omitted. (3) The corpus was rewritten with GPT-4o to fixed length/complexity and supplemented with four GPT-4o prompts; it is not raw real-world input. (4) The user study doesn't benchmark against the closest intermediate-representation systems (Athena, Squire, SpecifyUI, Guide) which the paper itself cites — so the incremental advantage of this particular DSL over other IR approaches is unestablished. N=15 and multiple t-tests without correction are minor in this context, but worth noting.\n\nNone of this kills the paper. The user-study perceptions are real, and the design pattern is useful regardless of the exact accuracy number. The paper is honest about its limitations in Section 9.6. The fixes are straightforward: report stricter metrics (recall with omissions in the denominator, exact-match rates), build the GT independently of the system, run at least one close IR baseline, and soften the accuracy claim.\n\nWho it's for: HCI/GenUI researchers, and anyone building prompt-to-UI systems. I'd bring it to a reading group to discuss the evaluation methodology. It deserves a serious referee; I'd send it out, with a request for the technical evaluation reanalysis.","headline":"Useful staged-generation pattern and a credible user study, but the 95% extraction accuracy is inflated by circular ground truth and a lenient metric; deserves serious review.","tokens_in":23951,"tokens_out":3368,"would_cite":true,"duration_ms":29886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LegoUI claims that exposing a UI generator's intermediate design decisions as editable, provenance-tagged specification items gives beginner users more transparency, control, and intent alignment than one-shot prompt-to-UI tools.","keywords":["UI generation","UI-DSL","transparency","controllability","provenance","staged generation","human-AI collaboration","iterative design"],"falsifier":"Build an independent ground truth for the same 40 prompts: annotators who have never seen LegoUI's outputs or category definitions extract explicit requirements against an ontology fixed in advance, and recompute accuracy with strict matches only, omissions inside the denominator, and partial matches treated as errors; if accuracy drops well below 95 percent, the extraction-fidelity foundation of the transparency claim is not established. For the user-study claim, a second check would run the identical staged items with provenance stripped and with accept/reject/add disabled, and test whether reported transparency and intent alignment fall back to one-shot levels.","tokens_in":22832,"feed_emoji":"🧱","tokens_out":12753,"duration_ms":97689,"temperature":0.7,"pith_summary":"Most prompt-to-UI tools produce a finished interface in a single pass, leaving the model's intermediate design decisions invisible; when the result misses the intent, users can only rewrite the prompt and regenerate. LegoUI claims a better structure: generation proceeds through staged reasoning steps, and every decision — components, layout regions, interaction rules, relations, and style — is recorded as an editable item in a UI-specific language, tagged with its provenance (from the prompt, inferred by the model, or added by the user). Users can inspect, accept, reject, or add items before any interface is rendered, and later stages generate from the revised specification, so intent accumulates across iterations rather than being reinterpreted each time. If established, this matters because beginner designers could catch and steer misinterpretations early instead of debugging through prompts, at the price of extra review work. The paper supports this with over 95% extraction accuracy on 40 real-world prompts and a 15-participant study in which LegoUI was rated above the one-shot baselines on transparency, controllability, and intent alignment.","feed_headline":"Show your work: editable design bricks make AI UI tools controllable","feed_subtitle":"A staged UI spec shows how the model reads your prompt and lets you accept, reject, or add each design decision.","key_machinery":"The load-bearing mechanism is the UI-DSL, a domain-specific language whose items are atomic design commitments organized into seven specifications: intent, global, component pool, layout, interaction, relation, and style. Each item is a structured tuple — for example, a component $C=(N_c,\\tau_c,A_c,P_c)$ with identifier, type, attribute assertions, and provenance; an interaction rule $E_c[\\Phi]\\Rightarrow A\\langle P_i\\rangle$; a relation edge $c_1\\to c_2:k(a)\\langle P_r\\rangle$; a style override $t\\mapsto\\{h_1,\\ldots,h_k\\}\\langle P_s\\rangle$ — and every item carries provenance entries recording whether it came from the user's prompt, from model inference, or from a user edit. The mechanism's work is to make the design state persistent and actionable: accepting an item preserves it for later stages, rejecting it removes it from the assumptions later inference reads, and adding an item injects an explicit user requirement into the same reasoning context. Because generation always proceeds from the current specification state, the same machinery supports exploring alternative versions without discarding accumulated intent, and each rendered interface can be traced back to the commitments that produced it.","core_discovery":"The paper's discovery claim is that prompt-to-interface generation can be treated as an explicit, evolving reasoning process rather than an end-to-end transformation, and that doing so improves how beginner users understand and steer generation. LegoUI externalizes each intermediate design commitment as a UI-DSL item — an atomic, provenance-tagged statement in one of seven specifications (intent, global, component pool, layout, interaction, relation, style) — and presents these items as negotiable commitments before they are rendered into an interface. Users accept, reject, or add items, and every subsequent stage consumes the resulting specification state, so rejected inferences drop out of later generation, added requirements become explicit assumptions, and previously accepted decisions persist across regenerations. The paper reports that this shifts design work from post-hoc correction of a finished artifact to earlier intervention in the assumptions behind it, and it grounds the claim in a technical evaluation (explicit requirements extracted at over 95% accuracy with zero redundancy) and a within-subject study of 15 participants who rated LegoUI higher than Claude, Bolt, Vercel v0, and Lovable on transparency, controllability, usability, and intent alignment. The effect is uneven: component and layout decisions were easy for participants to judge, while interaction and relation decisions, whose consequences spread across components and pages, were harder to evaluate and added inspection cost.","pith_inferences":["The inspect-accept-reject-add loop is a general template beyond UI: any generative tool where users must judge model-made commitments (document drafting, slide decks, data-analysis reports) faces the same black-box tension, and a provenance-tagged intermediate specification could let users separate their own input from model inference there as well.","A control condition the paper does not run — presenting the same staged items without provenance labels, or without accept/reject/add controls — would isolate whether the transparency benefit comes from the UI-DSL mechanism itself or from simply slowing generation into reviewable stages.","The paper's finding that interaction and relation stages are the hardest to judge suggests a concrete design target: linking each item to the UI regions and behaviors it affects should cut inspection cost where it is highest; measuring whether that trade-off pays off is a direct follow-up study."],"forward_implications":["Users can inspect how the system interpreted their prompt before that interpretation hardens into a rendered interface, and can reject unwanted inferences before they propagate into later stages.","Regenerated versions build on the accumulated specification state — accepted, rejected, and user-added items — instead of reinterpreting the original prompt from scratch, so stable commitments persist across iterations.","Control becomes stage-appropriate: component and layout decisions invite confident user judgment, while interaction and relation decisions need extra visual or behavioral context to be judged well.","The workflow cost shifts earlier: staged inspection adds review work, while one-shot tools keep the advantage of speed and first-pass polish; the two paradigms fit different phases, exploration versus refinement.","Future systems can preserve the staged structure while adding selective, consequence-oriented transparency, linking each decision to the UI regions and behaviors it affects rather than showing every item with equal weight."],"supporting_citations":[{"why":"The GenUI design study; documents the transparency, intent-assimilation, and editing gaps in one-shot generative UI tools that motivate the design goals.","marker":"[10]"},{"why":"Claude/Anthropic artifacts; chat-based one-shot baseline in the user study and the model family powering the backend.","marker":"[1]"},{"why":"Bolt; industry one-shot baseline selected for the user-study comparison tasks.","marker":"[5]"},{"why":"Vercel v0; industry one-shot baseline in the user study.","marker":"[39]"},{"why":"Lovable; industry one-shot baseline in the user study.","marker":"[28]"},{"why":"Athena; prior system using intermediate representations for iterative app generation, the nearest comparison for persistent staged specifications.","marker":"[2]"},{"why":"Prior interactive generative UI work with an intermediate data model that the paper positions against.","marker":"[7]"},{"why":"SpecifyUI; prior structured-specification approach for iterative UI intent expression, included in the intermediate-system comparison.","marker":"[11]"},{"why":"Squire; prior slot-query intermediate representation system for interactive UI authoring, included in the comparison.","marker":"[22]"}],"fun_headline_variants":["AI UI generation that shows its reasoning, lets you edit the bricks","Step-by-step AI UI design: accept, reject, or add each decision","UI-DSL bricks let you inspect and steer AI-generated interfaces","From black-box prompt to negotiable UI design steps","AI UI design that explains itself and takes your edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 95% extraction accuracy rests on a reference standard produced by annotators who used LegoUI's own requirement-analysis tooling to build the PromptChecklists, with partial matches counted as correct and omissions excluded from the accuracy denominator, so if the system's DSL categories shaped what the annotators counted as explicit requirements, part of the accuracy measures self-consistency rather than faithful extraction.","fun_headline_variants_meta":{"raw":{"variants":["AI UI generation that shows its reasoning, lets you edit the bricks","Step-by-step AI UI design: accept, reject, or add each decision","UI-DSL bricks let you inspect and steer AI-generated interfaces","From black-box prompt to negotiable UI design steps","AI UI design that explains itself and takes your edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3426,"prompt_tokens":974,"completion_tokens":2452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2366}},"tokens_in":590,"tokens_out":2452,"duration_ms":14302,"temperature":1.0,"reasoning_tokens":2366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:39:49.429640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an independent ground truth for the same 40 prompts: annotators who have never seen LegoUI's outputs or category definitions extract explicit requirements against an ontology fixed in advance, and recompute accuracy with strict matches only, omissions inside the denominator, and partial matches treated as errors; if accuracy drops well below 95 percent, the extraction-fidelity foundation of the transparency claim is not established. For the user-study claim, a second check would run the identical staged items with provenance stripped and with accept/reject/add disabled, and test whether reported transparency and intent alignment fall back to one-shot levels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Vercel v0; industry one-shot baseline in the user study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lovable; industry one-shot baseline in the user study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GenUI design study; documents the transparency, intent-assimilation, and editing gaps in one-shot generative UI tools that motivate the design goals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude/Anthropic artifacts; chat-based one-shot baseline in the user study and the model family powering the backend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Bolt; industry one-shot baseline selected for the user-study comparison tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior interactive generative UI work with an intermediate data model that the paper positions against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Squire; prior slot-query intermediate representation system for interactive UI authoring, included in the comparison."}],"review_version":2}