{"id":"fd269fd9-f394-4e61-8e91-7b035f19708b","arxiv_id":"2607.14830","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM-based Figma plugin with a human-editable feature list and RAG component retrieval produced more completed prototypes and higher expert quality ratings than manual Figma in a small Thai-language pilot.","lead":"AI Prototyper is an open-source Figma plugin that turns a typed screen description into an editable GUI prototype using an LLM to decompose the request, retrieve pre-built components, and render them as native Figma layers. The paper reports that students using the plugin finished more prototype tasks in a fixed session and experts rated the output higher than manual mockups, but the study is small and preliminary.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality claim is potentially confounded: manual-group ratings include 11 incomplete drafts, and no analysis is reported on completed manual prototypes only; the AI quality advantage could shrink or disappear.","rationale":"The reader's verdict is CONDITIONAL, which remains appropriate. However, the reader's 'weakest_assumption' focuses on task representativeness and the ceiling effect. While that is a valid generalizability concern, the more load-bearing issue is the completeness confound in the quality ratings. The manual group included incomplete drafts; since the quality ratings were made on all outputs, the AI advantage may be inflated by comparing finished artifacts to unfinished ones. The paper itself acknowledges this limitation but does not provide a corrected analysis. This is an internal-validity threat to the central claim. Because the paper is transparent about the limitation and labels the results as preliminary, the verdict should stay CONDITIONAL rather than move to REJECT. The proposed concrete test would settle whether the quality claim survives when completeness is controlled for.","tokens_in":6762,"tokens_out":3852,"duration_ms":35360,"concrete_test":"Re-analyze the Phase 2 rating data: subset the manual group to the 13 completed prototypes (as marked in Phase 1) and recompute the nine per-dimension Mann–Whitney U-tests (or equivalent) against the 20 AI prototypes. If fewer than 9 (or, under a stricter threshold, fewer than 7) dimensions remain significant at p<0.05, the quality claim is not robust to completeness; if all nine remain significant, the confound is unlikely to explain the result. Also compute the difference in effect sizes, e.g., rank-biserial r, before and after subsetting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim has two components: productivity and quality. The productivity component (more completions in fixed time) is internally valid but limited by the 4/4 ceiling. The quality component, however, is threatened by a direct confound in Phase 2: expert raters evaluated all Phase 1 outputs, including the 11/24 manual-group prototypes that were incomplete. Because incompleteness directly lowers ratings on dimensions such as 'meets requirements', 'necessary components', and 'minimal structural errors', the observed higher ratings for AI prototypes may simply reflect that they were finished while many manual ones were not. The paper acknowledges this in Section III-D ('the quality gap partly reflects completeness') but never re-runs the analysis on completed manual prototypes only. Without that comparison, the claim 'expert practitioners rated the AI-generated prototypes higher across nine quality dimensions' (Abstract) is not established for the attribute being tested — it may be an artifact of completeness rather than of intrinsic quality. This is more load-bearing than task representativeness because it undermines the internal validity of the key result, not just its generalizability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AI Prototyper, an open-source Figma plugin that generates editable GUI prototypes from natural-language descriptions through a four-stage decomposition and retrieval-augmented generation (RAG) pipeline, using Gemini 2.5 Flash as the LLM backend and a custom 32-component library. Its claimed contributions are the plugin itself, a human-in-the-loop feature-list editing step, multilingual input support (Thai, English, Mandarin Chinese, and, with noted limitations, Malayalam), and preliminary empirical evidence of productivity and quality gains over manual Figma prototyping. The evaluation has two phases: Phase 1 compares five AI-assisted participants with six manual participants on four mobile-screen tasks in a 45-minute session; Phase 2 has ten expert practitioners rate all outputs on nine quality dimensions. The abstract concludes that AI Prototyper users completed more prototypes in a fixed time window and that expert raters scored the AI-generated prototypes higher on all nine dimensions.","tokens_in":6911,"tokens_out":3943,"duration_ms":36173,"significance":"If the empirical claims hold, the paper provides useful evidence that a small, purpose-built component library combined with prompt decomposition and RAG can automate a meaningful fraction of routine mobile GUI prototyping. The artifact is open source, the pipeline is described in sufficient detail to be reimplemented, and the evaluation uses random assignment, an anonymous expert-rater protocol, and a genuine manual-work baseline. The paper is also admirably transparent about its limitations, including the small sample, the ceiling effect, the non-independence of expert ratings, and the completeness confound. As a tool demonstration with preliminary evidence, it is a reasonable contribution to the GUI-generation literature, provided the quality claim is either re-supported by a confound-free analysis or appropriately qualified.","major_comments":[{"comment":"The central quality claim is confounded. Phase 2 raters evaluated every Phase 1 output regardless of completion status, as stated in §III-B, so the 11/24 manual-group entries that were not completed ('partial drafts') were rated alongside finished AI prototypes. Dimensions such as 'meets requirements', 'necessary components', and 'minimal structural errors' directly penalize incompleteness. The paper acknowledges in §III-D that 'the quality gap partly reflects completeness,' but it never re-runs the analysis on the 13 completed manual prototypes only. Without that analysis, the abstract's claim that 'expert practitioners rated the AI-generated prototypes higher across nine quality dimensions' is not established for intrinsic quality; it may be an artifact of completeness. Please report the completed-only comparison (means, effect sizes, and tests) or visibly soften the quality claim to a","section":"§III-B, Table II, §III-D"},{"comment":"The productivity comparison suffers from a ceiling effect that limits the claim. Every AI-assisted participant completed all 4/4 tasks (mean = 4.00, SD = 0), while the manual group completed 13/24 (mean = 2.17). The design leaves no room to measure the size of the productivity gain, and the paper explicitly says time-to-completion and interaction/refinement time were not captured. The four task types are also the same patterns used to construct the 32-component library (§II-C), so the experiment may be particularly favorable to the tool. This does not invalidate the observed completion difference for these tasks, but it means the abstract's phrasing 'completed more prototypes in a fixed time window' cannot be generalized beyond a simple, library-aligned task set. Please add a per-task completion-time measure or additional, more complex tasks, and at minimum state the ceiling and co-desig","section":"§III-A, Table I, §III-D"},{"comment":"The Phase 2 significance tests are statistically non-independent. Each of the ten expert raters scored every prototype, so ratings are clustered by rater and by prototype; the p-values in Table II are therefore inflated, as the paper acknowledges in §III-D. Yet the text still reports 'statistically significant (p < .001)' without qualification. Please use a mixed-effects model with rater and prototype random effects, or otherwise account for clustering, and report effect-size estimates with confidence intervals. If the analysis remains a pilot-level exploration, the p-values should be presented as descriptive rather than inferential.","section":"§III-B, Table II, §III-D"},{"comment":"The evaluation tasks and the component library appear to be co-designed: §II-C says the 32-component set was constructed by analyzing 'login screens, product cards, social profiles, and settings pages,' which are exactly the four Phase 1 tasks. This does not by itself make the results circular, but it means the evaluation demonstrates feasibility for a narrow, tool-aligned workload rather than general GUI prototyping. The paper should explicitly acknowledge this co-design and discuss what evidence would be needed to show broader usefulness, e.g., tasks drawn from an external corpus or from professional design scenarios not used to build the library.","section":"§II-C, §III-A, §III-D"}],"minor_comments":[{"comment":"Typo: 'validated JASON' should be 'validated JSON'.","section":"§II-B, Stage 4"},{"comment":"The abstract states support for Thai, English, and Mandarin Chinese, while the main text also discusses Malayalam. Add Malayalam to the abstract or explicitly qualify the language support list.","section":"Abstract and §II-E"},{"comment":"The description of participants as 'computing undergraduates with prior Figma experience' is vague. Please report how Figma experience was assessed, the randomization procedure, and any demographic details relevant to the task.","section":"§III-A"},{"comment":"Cliff's delta = 0.83 is reported without a confidence interval. Given the small sample, a confidence interval would better convey uncertainty.","section":"Table I"},{"comment":"The table gives p-values but no test statistic or effect-size confidence intervals. Please report the Mann-Whitney U (or equivalent) value and CIs, and note the non-independence issue in the table caption.","section":"Table II"},{"comment":"The multilingual probes are described as single-speaker observations. This is fine as an exploratory note, but the wording 'each language was tested by a single native speaker' should be moved earlier and referenced in the abstract's multilingual claim to avoid overstatement.","section":"§III-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the artifact is genuinely open source, which is valuable. The key issue is the completeness confound in the quality claim; it is fixable by a completed-only analysis or by carefully qualifying the abstract. The ceiling effect and non-independence of ratings are also important but are already partly disclosed. If the authors supply the completed-only analysis and tighten the claims, I would be willing to accept a revision. No concerns about scope or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on arXiv:2607.14830 (AI Prototyper). It's an open-source Figma plugin implementing the decomposition+RAG idea from GUIDE, with a custom 32-component library, Gemini 2.5 Flash, a human-in-the-loop feature-editing step, and multilingual prompts (Thai, English, Mandarin, Malayalam). The code and component library are public, and the authors are upfront about the preliminary nature of the evaluation. That's more than many tool papers.\n\nWhat's new is modest but real: the editable feature list before rendering is a nice checkpoint, and the multilingual support is a practical extension. The Phase 1 productivity result is internally valid as far as it goes: 5/5 AI participants finished all 4 tasks (20/20), while the manual group managed 13/24, with U=27.5, p=.016. The ceiling effect (everyone at 4/4) means the gain isn't quantified in time, but the direction is credible.\n\nThe soft spot is load-bearing, and the stress-test note is right. Phase 2 rated every Phase 1 output, including 11 incomplete manual drafts, under dimensions like 'meets requirements' and 'minimal structural errors.' Incomplete outputs will score lower no matter the tool's intrinsic quality. The paper says in III-D that the quality gap 'partly reflects completeness,' but never re-analyzes completed manual prototypes only. So the abstract's claim about higher expert ratings across nine dimensions is not established for what it appears to claim. That's a fixable confound, but it's central.\n\nAlso minor: repeated-measures p-values are inflated (authors say effect sizes matter), multilingual claims rest on single-speaker probes, and the four tasks are simple mobile screens. All disclosed.\n\nBottom line: this is a legitimate, honest engineering contribution, not a breakthrough. It deserves a serious referee — someone should push for a completed-only analysis, a time-to-completion measure, and harder tasks. I'd accept for review with expectation of major revision.","headline":"Honest, well-scoped tool paper with a real completeness confound in the quality evaluation; worth a referee's time but not at face value.","tokens_in":7484,"tokens_out":3461,"would_cite":true,"duration_ms":30679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source Figma plugin that decomposes natural-language screen descriptions into editable components lets users finish more GUI prototypes in a fixed time than manual Figma work, with expert ratings higher on every quality dimension.","keywords":["GUI prototyping","LLM-based generation","decomposition","retrieval-augmented generation","Figma plugin","human-in-the-loop","multilingual UI generation","component library"],"falsifier":"Give both conditions a realistic multi-screen task with navigation flows, custom branding, and a form with validation states, and measure time-to-completion rather than task count; if manual Figma users match AI-assisted completion or expert ratings on that task, the paper's central claim of general productivity gain is not supported.","tokens_in":6552,"feed_emoji":"🧩","tokens_out":4010,"duration_ms":30705,"temperature":0.7,"pith_summary":"The paper tries to establish that GUI prototyping can be automated without sacrificing editability by decomposing a screen request into discrete features and retrieving matching components from a small purpose-built library. It claims the resulting open-source Figma plugin, AI Prototyper, lets users complete more prototypes in a fixed time than manual Figma work and produces prototypes experts rate higher across nine quality dimensions. If true, the main practical consequence is that designers can regenerate and revise prototypes from a revised description in seconds, which directly reduces a known bottleneck in requirements evolution and software maintenance. The paper frames the evaluation as preliminary evidence, not a definitive effectiveness claim.","feed_headline":"AI Prototyper completes every task; manual Figma finishes 54%","feed_subtitle":"Open-source plugin turns Thai-language prompts into editable Figma screens; experts rate them higher on all nine quality measures.","key_machinery":"The carrying mechanism is decomposition-based retrieval-augmented generation with a two-stage RAG design and role-assigned prompts. The system first casts the LLM as an expert product manager to decompose the request, then as an expert UI/UX designer to select and instantiate components. The central object is the custom 32-primitive component library, each defined by a JSON schema with required and optional properties; retrieval stays compact by including only one-line catalogue summaries in the selection prompt. The human-in-the-loop editing step is the checkpoint that distinguishes the tool and lets users correct misinterpretations before rendering.","core_discovery":"The central discovery is a four-stage pipeline: the LLM decomposes the user's natural-language screen request into a JSON list of GUI features; a retrieval step maps each feature to a small set of candidates from a custom 32-component library; an instantiation step emits schema-valid component instances; and the Figma plugin renders them as native, auto-layout, fully editable layers. A human-in-the-loop review of the feature list sits between decomposition and rendering. In the paper's empirical evaluation, all five AI-assisted participants completed all four tasks in 45 minutes, versus 13 of 24 for six manual participants, and ten expert practitioners rated the AI-generated prototypes highe","pith_inferences":["The ceiling effect in Phase 1 (all AI participants 4/4) implies the measured productivity gain is a lower bound for trivial tasks, not a quantified margin; a time-to-completion study with harder tasks would likely shrink the gap.","The human-in-the-loop editing step could be the most transferable idea: intercepting the model's interpretation before rendering turns the pipeline into a collaboration tool rather than a black-box generator.","The component retrieval approach suggests a natural extension: as libraries grow, the one-line-catalogue retrieval step could itself become learned or hierarchical, keeping context small while scaling coverage.","The Malayalam transliteration result hints that low-resource-language support may be improved by augmenting prompts with small translation glossaries rather than relying on the LLM alone."],"forward_implications":["Designers can regenerate an updated prototype from a revised description in seconds, removing a bottleneck in requirements evolution.","Decomposition plus a small, purpose-built library is enough to produce usable prototypes, suggesting larger catalogues are not required for common mobile screens.","Multilingual input works for Thai, English, and Mandarin Chinese on first attempt; Malayalam yields structurally valid but transliterated labels, so usability generalizes while label accuracy depends on LLM language proficiency.","Because output is fully editable Figma layers with auto-layout, generation is a starting point rather than a finished artifact; branded visuals and custom illustrations still require manual work.","The tool's four-stage pipeline with human-in-the-loop editing provides a template for other design-tool plugins."],"fun_headline_variants":["AI Prototyper: 100% task completion, manual hits 54%","Figma plugin automates GUI prototyping with LLM decomposition","Decompose-to-prototype: AI plugin beats manual in speed and quality","Open-source Figma AI: 4-step pipeline, multilingual, human-in-loop","LLM-driven prototyping: all tasks done, experts rate higher"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the four evaluated screens—login, product card, social profile, and settings—are representative of GUI prototyping workloads; since every AI-assisted participant finished all four easily, the measured productivity and quality advantages may not extend to larger, more complex, or multi-screen prototypes.","fun_headline_variants_meta":{"raw":{"variants":["AI Prototyper: 100% task completion, manual hits 54%","Figma plugin automates GUI prototyping with LLM decomposition","Decompose-to-prototype: AI plugin beats manual in speed and quality","Open-source Figma AI: 4-step pipeline, multilingual, human-in-loop","LLM-driven prototyping: all tasks done, experts rate higher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3115,"prompt_tokens":819,"completion_tokens":2296,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2200}},"tokens_in":563,"tokens_out":2296,"duration_ms":12155,"temperature":1.0,"reasoning_tokens":2200,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T00:54:58.227256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give both conditions a realistic multi-screen task with navigation flows, custom branding, and a form with validation states, and measure time-to-completion rather than task count; if manual Figma users match AI-assisted completion or expert ratings on that task, the paper's central claim of general productivity gain is not supported.","supporting_citations":[],"review_version":1}