{"id":"089da4f4-cdfe-4a85-b15a-523b83b4c26e","arxiv_id":"2505.19554","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ASR replaces the vision encoder of a multimodal LLM with graph-derived structural features to generate UI layouts, reporting better overlap and relation metrics than four prior methods.","lead":"The paper introduces a method called ASR that feeds graph-based structure of a user interface into a large language model to generate UI layouts with fewer overlaps and more reliable component relations. A smart generalist reader will want to see whether combining graph priors with multimodal LLMs actually produces the layout quality improvement the paper reports.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not secured by Table 2 because the FID metric is nonstandard and the graph-editing protocol gives ASR an interaction advantage that baselines do not receive.","rationale":"The reader's weakest_assumption concerns whether the graph feature can replace the ViT stream in Intern-VL without losing coordinate grounding. That is a genuine under-specification (no Intern-VL variant, no connector details), but the paper's experiments are the evidence that the substitution works. The more load-bearing risk is that the experiments themselves do not establish the comparative superiority on which the central claim rests. Section 4.2 explicitly admits an asymmetric graph-editing protocol, and Section 4.2's 'FID' definition is not the standard Inception-based FID that the baseline comparisons assume. Table 4 also fails to reproduce Table 2's with-RM numbers, casting doubt on internal reproducibility. If a controlled re-evaluation removes these artifacts and ASR still leads, the architecture concern can be revisited; until then the claim is conditional. I keep the reader's CONDITIONAL verdict but shift the emphasis from the architecture to the evaluation validity.","tokens_in":16596,"tokens_out":8714,"duration_ms":86586,"concrete_test":"Run a controlled re-evaluation of Table 2: for Graph Editing, disable ASR's interactive refinement and give every model identical one-shot coarse node/relation constraints; recompute FID with a standard Inception-v3 feature extractor on the same generated and ground-truth images; and re-run the with/without-RM ablation on the exact test split used for Table 2. If ASR's margins survive, the protocol concern is resolved; if they shrink or reverse, the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's central claim — 'ASR consistently outperforms other models in terms of mIoU, RE, and OL' — is a comparative statement, so the comparison protocol is load-bearing. Two features of Table 2 prevent it from carrying that weight. First, Section 4.2's Graph Editing protocol is explicitly asymmetric: 'our ASR allows for manual graph editing during the generation process, whereas other models require all settings to be predefined.' The authors say they 'coarsely set some nodes and relations before generation for the other models' — that is not the same task for ASR, which can interactively refine its relation matrix. Any gain in the Graph Editing rows may come from extra information or iterative refinement, not from the graph-representation substitution. Second, the FID column is not the standard FID used by the cited baselines; Section 4.2 defines it using features from 'a model... trained to classify whether an input layout is corrupted or not.' This is a task-specific feature extractor, so the FID values in Table 2 are not comparable across methods or to published numbers. Additionally, Table 4's with-RM row (mIoU 0.54, RE 0.30) does not match Table 2's ASR UI-Gen row (mIoU 0.63, RE 0.27) on RICO, so the internal consistency of the headline numbers is not established without clarification of the task setting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Aggregated Structural Representation (ASR), a layout-generation pipeline in which a GNN encoder extracts graph features and an editable relation matrix from UI layout graphs, and these representations are fed to the Intern-VL MLLM after replacing its ViT visual stream. The LLM is fine-tuned to output JSON describing node categories, coordinates, and relations, and the relation matrix can be edited by users or randomly perturbed to inject diversity. The method is evaluated on RICO, PubLayNet, and Magazine for UI generation, completion, and graph editing, using mIoU, relation error, overlap, FID, and a crowdsourced perceptual study. The authors report consistent quantitative improvements and claim the relation matrix is essential for structural fidelity.","tokens_in":16895,"tokens_out":5520,"duration_ms":46156,"significance":"The core idea—replacing the visual token stream of an MLLM with explicit graph-derived structural features and an editable relation matrix—is timely and, if validated, would provide a useful recipe for injecting hard structural constraints into LLM-based layout generation. The paper's breadth is a strength: three datasets, three tasks, an ablation, and a user study, and the relation-matrix editability is a genuinely useful human-centric feature. The main weaknesses are in the evaluation protocol: the graph-editing comparison is asymmetric, the FID metric is nonstandard, the ablation numbers do not match the main table, and no uncertainty estimates are reported. These issues do not invalidate the idea, but they currently prevent the quantitative claims from being accepted at face value.","major_comments":[{"comment":"The protocol is asymmetric: the text states that ASR allows manual graph editing during generation whereas other models require all settings to be predefined, and for baselines the authors 'coarsely set some nodes and relations before generation.' This means the Graph Editing rows in Table 2 compare ASR with interactive refinement against baselines given coarse initial conditions, so the reported advantages (e.g., RE 0.09 vs 0.12) may come from additional information or iterative adjustment rather than from the graph-representation substitution. Please either give baselines an equivalent interactive protocol or restrict the comparison to a single-shot setting for all methods.","section":"Section 4.2, Graph Editing protocol"},{"comment":"FID is computed with a model 'trained to classify whether an input layout is corrupted or not,' which is not the standard Inception-based FID used by the cited baselines. The resulting values are therefore not comparable across methods or to published FID numbers, and the FID column in Table 2 should either be replaced with the standard definition or removed. As written, the FID entries cannot support the claim that ASR 'produces more realistic and coherent layout components.'","section":"Section 4.2, FID definition"},{"comment":"The ablation reports 'ASR with RM' on RICO with mIoU 0.54 and RE 0.30, whereas Table 2's ASR UI-Gen row on RICO reports mIoU 0.63 and RE 0.27. If the ablation uses the same task and split, the numbers are inconsistent; if it uses a different setting, that setting is not described. This needs clarification before the ablation can be used to quantify the relation matrix's contribution.","section":"Table 4 vs Table 2"},{"comment":"The paper says Intern-VL is fine-tuned by replacing its ViT component with the graph representation module, but it does not specify which Intern-VL checkpoint is used, how the 1024-dimensional feature is projected into the LLM's token embedding space, or how the connector is initialized. Because the method's central premise is that Intern-VL can treat graph features as a substitute for visual tokens while retaining coordinate-grounding and language abilities, this missing detail is essential for reproducibility and for judging whether the approach is a genuine MLLM adaptation.","section":"Section 3.4, architecture details"},{"comment":"The claim that ASR 'consistently outperforms' all baselines in mIoU, RE, and OL is not fully supported by the table: in Graph Editing, L-DM has higher mIoU (0.69 vs 0.68), and in Completion and Graph Editing ASR has worse FID than LDGM or L-DM. No error bars or significance tests are reported, so small differences such as the UI-Gen mIoU gap (0.63 vs 0.62) could be within run-to-run variation. Please provide uncertainty estimates and restrict the claim to metrics where the comparison is actually consistent.","section":"Section 4.3 and Table 2"}],"minor_comments":[{"comment":"There are several typos and formatting inconsistencies: 'Defination' in Table 1, 'ultilized' in Section 4.1, 'Amanze Mechanical Turk' in Section 4.4, 'Rico Dataset' in the Table 2 caption, and 'langauge' in the PosterLLaMA reference.","section":"Throughout"},{"comment":"The denominator in Eq. (3) sums only over negative samples, whereas the standard SimSCE loss includes the positive pair in the denominator as well. If the implementation follows the equation as written, the loss is not the stated SimSCE loss; if it follows SimSCE, the equation should be corrected.","section":"Section 3.2, Eq. (3)"},{"comment":"Only mean scores are reported; no standard deviations, number of ratings per layout, or inter-annotator agreement are given, so it is difficult to assess whether differences such as 4.30 vs 4.28 on easy layouts are meaningful.","section":"Section 4.4, user study"},{"comment":"The dimensions of the relation matrices Msem and Mpos are not defined, and the handling of graphs with variable node counts in the GNN/LLM interface is not described; a worked example showing how node sets map to matrix rows and to the JSON output would improve clarity.","section":"Sections 3.2 and 3.3"}],"recommendation":"major_revision","confidential_remarks":"I see no circular reasoning in the paper: the relation matrix is a learned intermediate representation supervised by ground-truth graphs, and no reported metric is constructed from the target by definition. My main concern is evaluative rather than conceptual: the comparative claims rest on a protocol that gives ASR additional interactive information, a nonstandard FID, and inconsistent ablation numbers. If the authors fix the protocol and internal consistency, the paper could be a reasonable contribution; in its current form the quantitative claims are not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper is not the usual incremental combo. Replacing the ViT stream of an MLLM with a GNN-extracted graph representation for layout generation is a new configuration, and the editable relation matrix gives the user a real control channel during generation. That's worth taking seriously. The second thing: the evaluation currently doesn't prove the headline. Table 2 shows consistent wins on mIoU, RE, and OL across RICO, PubLayNet, and Magazine, but the metric definitions and task protocols have problems that could account for part of the gap.\n\nWhat's genuinely good: the contrastive training on triplets with masked positives is a sensible way to learn structure, and the relation-matrix decoder for the LLM is a clean idea that makes the interaction concrete. The paper also reports compute budgets and gives a straightforward ablation (with/without relation matrix) showing the matrix matters. They are transparent about limitations, including conflict handling. The user study, while small, is an attempt to go beyond automatic metrics.\n\nSoft spots: First, the FID is not FID. They train a corruption classifier and use its features, which means the values are not comparable to the baselines' published FID and not interpretable as a distributional distance. That's a serious comparability issue. Second, the Graph Editing task is asymmetric: ASR gets to edit its relation matrix interactively, while baselines get a one-shot coarse setting. The gains there could come from extra information rather than the architectural change. Third, there's an internal inconsistency you'll want clarified: the with-RM row in Table 4 (mIoU 0.54, RE 0.30) doesn't match the ASR UI-Gen row in Table 2 (0.63/0.27) on the same dataset. That has to be explained. Fourth, missing error bars and significance tests, and no code release, make the wins hard to verify. Finally, they skip MLLM baselines like LayoutPrompter and Graphist even though they cite them, which weakens the claim that replacing the ViT is the key.\n\nWho this is for: people working on layout generation, especially UI and graphic design, and anyone interested in injecting structural priors into multimodal LLMs. It's not a desk reject; the idea is strong enough to warrant a serious referee. But as submitted, the evidence is conditional. I'd send it to peer review with a request for major revision: fix the FID, make the baseline protocol fair, add significance testing, and clarify the ablation numbers. If the authors can do that, this could be a genuinely useful contribution.","headline":"A genuinely new GNN-for-ViT swap in an MLLM for layout generation, but the current evaluation does not yet support the headline claims.","tokens_in":17397,"tokens_out":3306,"would_cite":false,"duration_ms":93056,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that swapping the visual encoder of a multimodal LLM for GNN graph features plus a human-editable relation matrix yields UI layouts with less overlap and more accurate structure than graph-based, transformer, and…","keywords":["UI layout generation","graph neural networks","multimodal large language models","relation matrix","layout completion","graph editing","human-centric design","RICO dataset"],"falsifier":"A controlled comparison would feed the same layout prompt to two otherwise identical MLLMs, one with the original ViT image stream and one with the ASR graph stream, and check whether the graph stream's mIoU, relation-error, and overlap advantages survive across the RICO, PubLayNet, and Magazine test splits. If the graph-stream model loses to the image-stream model on any dataset, the paper's attribution of the gain to structural representation would be falsified.","tokens_in":16403,"feed_emoji":"🧩","tokens_out":8259,"duration_ms":74867,"temperature":0.7,"pith_summary":"This paper is trying to settle a two-sided failure mode in automatic UI layout generation. Graph-based methods keep structure but generate poorly, while vision-based generative models produce richer layouts but forget the explicit relationships between components, so boxes overlap. The proposed answer is to feed the large language model a structural prior instead of an image: a graph neural network extracts a 1024-dimensional feature, and a decoder produces a relation matrix encoding containment, parallelism, and relative position that the LLM must respect. The paper reports that on the RICO dataset this combination beats four baselines on mean Intersection over Union, relation error, and overlap across direct generation, completion, and graph editing, and that human editing of the relation matrix gives users a handle on the output. A sympathetic reader would take this as evidence that explicit structure can be injected into an LLM without relying on pixel-level vision.","feed_headline":"Graph features, not pixels, sharpen LLM UI layout generation","feed_subtitle":"Replacing the vision encoder with a GNN plus editable relation matrix cuts overlaps and relation errors.","key_machinery":"The central object is the relation matrix $M = \\{M_{sem}, M_{pos}\\}$, a discrete encoding of semantic relations ('contain', 'parallel') and positional relations ('top', 'left') between UI components. It is produced by a decoder from the 1024-dimensional graph feature $h_a$ computed by a five-layer GNN over node features $N_f$ and edge matrix $E_M$, and it is then fed into the LLM together with $h_a$. The matrix does two jobs: it supplies the inter-node structure that the LLM is not naturally good at, and because it is a table of discrete relations, a designer can edit it or sample random perturbations of it to steer generation. The same matrix also supervises the generator, so the LLM's output is pinned to the stated structure while its grounding ability supplies coordinates and categories.","core_discovery":"The paper's central claim is that graph structure and LLM generation are complementary, not competing, and that a relation matrix is the right interface between them. Concretely, the LLM is fine-tuned with its Vision Transformer replaced by the graph representation module; during training the GNN sees a masked layout graph while the relation matrix comes from the ground truth, so the LLM learns to finish missing nodes and even add new ones without breaking existing relations. On all three tasks in the main RICO table, ASR reports the best mIoU and relation error and the lowest overlap; for example, on direct UI generation it reaches mIoU 0.63 with relation error 0.27, and the ablation shows that removing the relation matrix drops Max-IoU from 0.54 to 0.27 and raises relation error from 0.30 to 0.61. The paper also reports better crowd-sourced reasonableness scores on easy, medium, and hard layouts, with the largest gap on hard layouts.","pith_inferences":["Not in the paper: treating the relation matrix as a constraint language for accessibility rules—for example, requiring touch targets to be at least a given size or keeping high-contrast text separated from backgrounds—since the matrix already exposes exactly those relations.","Not in the paper: a user-in-the-loop study that records what happens when designers actually edit the relation matrix, rather than only scoring final generated layouts, would test the human-centric claim more directly.","Not in the paper: ablating perturbations of positional relations versus semantic relations separately would isolate which part of the editable matrix drives the reported diversity in generation."],"forward_implications":["A designer can change one relation in the matrix and regenerate a full, structurally consistent layout without re-entering all coordinates.","The masked-graph training scheme lets the model complete partial layouts and add new components that do not violate the existing structure.","If the recipe transfers, any task expressible as a graph with typed nodes and relations—posters, documents, floorplans, scene graphs—can be generated by the same graph-to-LLM pipeline.","Because the image encoder is removed at inference, the required input is a compact graph feature rather than a rendered image, which may reduce inference cost and simplify conditioning."],"supporting_citations":[{"why":"Defines graph matching for layout similarity and is the graph-based baseline that ASR must outperform.","marker":"(Patil et al., 2021)"},{"why":"Provides the task settings for UI generation and completion and is the LDGM diffusion baseline.","marker":"(Hui et al., 2023)"},{"why":"Supplies the discrete diffusion baseline LayoutDM on all three tasks.","marker":"(Inoue et al., 2023)"},{"why":"Supplies the sequence-to-sequence transformer baseline UniLayout.","marker":"(Jiang et al., 2022)"},{"why":"Provides the SimCSE contrastive loss used to train the graph representation module.","marker":"(Gao et al., 2021)"},{"why":"Provides PubLayNet for cross-dataset generalization evaluation.","marker":"(Zhong et al., 2019)"},{"why":"Provides the Magazine dataset for cross-dataset generalization evaluation.","marker":"(Xinru Zheng and Lau, 2019)"}],"fun_headline_variants":["Graph features replace pixels to improve LLM layout generation","Editable relation matrix plus LLM cuts layout overlap","GNN plus LLM: structural priors beat vision for UI layout","Human-editable graph matrix boosts LLM layout quality","LLM layout generation gets a structural boost from GNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the language model still understands coordinates and component types when its image encoder is taken away and replaced by a graph feature plus relation matrix; the paper does not give details of how the graph feature is turned into tokens or which Intern-VL model is used.","fun_headline_variants_meta":{"raw":{"variants":["Graph features replace pixels to improve LLM layout generation","Editable relation matrix plus LLM cuts layout overlap","GNN plus LLM: structural priors beat vision for UI layout","Human-editable graph matrix boosts LLM layout quality","LLM layout generation gets a structural boost from GNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1396,"prompt_tokens":952,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":568,"tokens_out":444,"duration_ms":4955,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:11:56.338437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison would feed the same layout prompt to two otherwise identical MLLMs, one with the original ViT image stream and one with the ASR graph stream, and check whether the graph stream's mIoU, relation-error, and overlap advantages survive across the RICO, PubLayNet, and Magazine test splits. If the graph-stream model loses to the image-stream model on any dataset, the paper's attribution of the gain to structural representation would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines graph matching for layout similarity and is the graph-based baseline that ASR must outperform."},{"cited_title":"LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction","cited_arxiv_id":"2208.08037","evidence_quote":"Supplies the sequence-to-sequence transformer baseline UniLayout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PubLayNet for cross-dataset generalization evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Magazine dataset for cross-dataset generalization evaluation."}],"review_version":1}