{"id":"7642efc3-c9fe-4c05-b5fd-f67c59fc45c4","arxiv_id":"2412.15310","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A resource-list representation and a 500-site benchmark let multimodal LLMs generate web code with real links, images, and routes, lifting resource matching from ~0% to 66-80%.","lead":"Design-to-code tools usually produce single static pages with placeholder images and dead links. This paper introduces MRWeb, a benchmark where models get a screenshot plus a machine-readable resource list, and shows that this list lifts correctly matched resources from near zero to 66-80% while improving visual similarity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RER measures transcription of the provided resource list, not navigation: baseline near-zero is forced by withholding the list, and matched links are never checked for resolution or cross-page functionality.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the RER metric's definition, combined with the task input design, makes the baseline near-zero by construction and scores transcription rather than genuine multi-page navigation. This is a construct-validity problem for the central functional claim, but it does not undermine the paper's independent contributions: the dataset construction, the human-study comparison of image similarity metrics, and the released tool are valuable and are not affected by the RER circularity. A conditional acceptance with a request to reanalyze or reword the navigation claim is appropriate. No additional objection rises to the same level of importance; the visual-metric analysis and the per-page evaluation are secondary issues that could be clarified without changing the verdict.","tokens_in":17048,"tokens_out":3010,"duration_ms":27521,"concrete_test":"Sample 50 generated pages across models and methods that achieved nonzero RER. Render each page in a headless browser and, for every matched internal-link resource, (1) extract the href, (2) navigate to that URL within the generated site, (3) verify the target page returns a non-blank page and contains the expected page title or heading from the reference site, and (4) for external links and images, check that the URL returns HTTP 200. Compute the fraction of matched resources that pass all checks and compare it with the reported RER. If the click-through pass rate is materially below 66-80%, the claim 'resource lists boost navigation functionality' should be revised to 'resource lists improve reproduction of provided link/image URLs'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline functional claim ('resource lists boost navigation functionality from 0% to 66%-80%', Section 6.2) rests on the Resource Existence Ratio defined in Section 5.3.1. RER counts a resource as matched when the generated element points to the same URL/source as the reference element. This creates two problems. First, the self-contained (SC) baseline is given no resource list, so it cannot know the ground-truth URLs; a near-zero RER is forced by the input design, not by a failure of navigation. The 66-80% scores under ZS/CoT/SR therefore largely measure whether the model copies the provided URLs into the generated code, which is a transcription check rather than a navigation check. Second, RER says nothing about whether a matched link resolves, whether an internal link leads to a generated page, whether the target page has the expected content, or whether backend routing actually works. The task formulation Cg = M(I0, R0) makes the resource list an input, so comparing SC against list-based prompting changes two variables at once: the presence of R0 and the prompt. Consequently, the reported 0-to-66-80% jump conflates 'providing the answer' with 'improving the model'. The paper's other contributions, such as the IQA comparison in Section 6.1 and the released dataset/tool, are not affected by this issue, but the central navigation-functionality claim is not established by the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MRWeb, a task and benchmark for generating multi-page, resource-aware web code from UI screenshots and an accompanying 'resource list' that specifies the positions, types, and URLs of links and images. The authors collect 300 synthetic and 200 real-world websites, propose a suite of visual and functional metrics, and evaluate Gemini, GPT-4o, and Claude-3.5 under self-contained, zero-shot, CoT, and self-refine prompting. They also run a human study to identify which image-similarity metric best matches human judgments, concluding that MAE and NEMD correlate best (SROCC 0.542 and 0.508). The headline finding is that providing the resource list raises the Resource Existence Ratio from near zero to 66-80%, interpreted as a boost in navigation functionality.","tokens_in":17337,"tokens_out":5190,"duration_ms":38308,"significance":"If the headline claim were rigorously supported, the work would be a meaningful step beyond single-page design-to-code. The dataset, the resource-list formalism, and the IQA comparison are concrete contributions, and the release of the tool, dataset, and evaluation framework supports reproducibility. The experimental setup is transparent, with temperature 0 and seed 42 and exact prompts in the appendix. However, the functional metric and the task setup undermine the central navigation claim, so the significance currently rests on the benchmarking and IQA contributions rather than on the claimed functional advance.","major_comments":[{"comment":"The Resource Existence Ratio (RER) is defined as the proportion of reference resources matched to generated resources by exact URL/source equality. Because the self-contained (SC) baseline is given no resource list, it cannot know ground-truth URLs, so its near-zero RER is forced by the input design. The 66-80% scores under zero-shot, CoT, and self-refine largely measure whether the model copies the provided URLs from the resource list into the generated code, which is a transcription check rather than a navigation check. Moreover, RER does not verify that a matched link resolves, that an internal link leads to a generated page with the expected content, or that backend routing works. The abstract and Section 6.2 claim that resource lists 'boost navigation functionality from 0% to 66%-80%', but the reported numbers do not establish end-to-end navigability.","section":"Section 5.3.1 and Table 4"},{"comment":"The task formulation Cg = M(I0, R0) and the prompt designs in Appendix C change two variables at once when comparing SC (no resource list) with ZS/CoT/SR (resource list provided): the presence of R0 and the prompt content. The reported improvements in RER and visual metrics cannot be attributed solely to the resource list because the prompting strategies also differ in instruction and supplementary text. A controlled comparison, for example providing a resource list with incorrect or scrambled URLs, or placing the same URL information in a non-structured textual format, is needed to isolate the effect of the resource list itself.","section":"Section 3 and Section 5.2"},{"comment":"The case study reports a 100% success rate for the MRWeb tool on a three-page personal website, but the criteria for success are not defined. It is unclear whether 'success' means that the generated code contains the correct URL strings, that the links are actually clickable and resolve, that the internal pages are generated and consistent, or that the backend route was tested with an actual request. Without an explicit evaluation protocol, this anecdote does not substantiate end-to-end navigation functionality.","section":"Section 6.4 and Table 6"}],"minor_comments":[{"comment":"The terms 'resource list' and 'action list' are used interchangeably; unify the terminology throughout the paper.","section":"Section 3, Appendix C, Appendix I"},{"comment":"The string 'bonding_box' in the resource list example is a typo for 'bounding_box'.","section":"Section 3"},{"comment":"The model name 'Genimi-Pro' is a typo; it should be 'Gemini-Pro'.","section":"Section 6.2"},{"comment":"The sentence 'SROCC ... where a indicates perfect positive or negative correlation' is missing the value '1' after 'a'; it should read 'where a value of 1 indicates'.","section":"Appendix E"},{"comment":"The visual and functional metric tables report point estimates with no error bars or significance tests; given the small gaps between some models and methods, the reader cannot assess whether the differences are meaningful.","section":"Tables 3 and 4"},{"comment":"The binning of image sizes and action-list lengths into five bins is not described, including the number of samples per bin, which makes the complexity-trend analysis difficult to interpret.","section":"Figure 3 and Appendix H"},{"comment":"The citation key '(web, 2024)' in the Introduction does not match any reference entry; the corresponding TechJury entry should be cited with an author or consistent key.","section":"Introduction and References"}],"recommendation":"major_revision","confidential_remarks":"The reader's and skeptic's assessments are accurate: the headline RER improvement is largely by construction because the resource list contains the ground-truth URLs and the metric rewards copying them. The paper's other contributions—the dataset, the resource-list formalism, and the IQA comparison with human annotations—are solid and worth publishing after a major revision. The authors should either redefine RER to measure actual navigation (e.g., link resolution, page existence, routing), add a controlled comparison that isolates the resource-list effect, or substantially weaken the functional claim in the abstract and Section 6.2. I would not reject the paper, but the current version overstates its central contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a useful benchmark and dataset, but the 0-to-66-80% RER headline is a transcription check, not a navigation result. The resource list contains the answer; the self-contained baseline is starved of it.\n\nThe genuine new things here: MRWeb is the first design-to-code benchmark that includes internal/external links, real images, and backend routing, at least as inputs. The resource-list representation is simple and workable, and the 500-site dataset (synthetic + real) with an automated extraction pipeline is a concrete contribution. The IQA study is also real: 600 pairs, 14 raters, standard protocol, and the negative finding that CLIP/SSIM/LPIPS track humans poorly for low/medium-similarity pairs is useful for anyone building evaluation pipelines.\n\nNow the soft spots. The central functional claim rests on RER, which counts a resource as matched when the generated element points to the same URL/source as the reference. That makes the SC baseline near-zero by construction—no resource list means no ground-truth URLs. The 66-80% under ZS/CoT/SR mostly measures whether the model copies the provided URLs into code. The paper calls this \"navigation functionality\" but never checks that a link resolves, that an internal link leads to a generated page, or that a route works. In fact, the benchmark scores each page in isolation; there is no end-to-end multi-page test. So the abstract's \"0% to 66-80%\" should be read as \"0% to 66-80% of provided URLs are transcribed,\" which is a different and weaker claim.\n\nOther weaknesses: Tables 3-5 report single point estimates with no variance. Temperature zero helps but there is still model stochasticity across inputs. The case study in Section 6.4 is one three-page site with a 100% success rate; that's a demo, not evidence. The paper also conflates improving the prompt with adding the resource list, since the SC condition changes both variables.\n\nThat said, the paper is not sloppy. It has a Limitations section that acknowledges non-link functionality and context limits. The IQA methodology follows standard practice. The dataset and tool are released, which is reproducible evidence. The circularity is in the interpretation, not in the mechanics.\n\nWho is this for? Anyone working on design-to-code, UI generation benchmarks, or MLLM evaluation for front-end engineering. It's worth a serious referee, but I'd send it back for revisions: redefine or supplement RER with a check that matched links resolve and that internal links point to generated pages, report error bars, and rewrite the abstract so \"navigation functionality\" is not claimed from this metric. I'd also ask for one end-to-end navigation test, even on a small subset.\n\nRecommendation: conditional accept, after those changes.","headline":"Useful benchmark and IQA data, but the headline RER boost is an artifact of giving the model the answer; the navigation claim is not supported by the metric.","tokens_in":17860,"tokens_out":2973,"would_cite":true,"duration_ms":23358,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a machine-readable resource list is what lets multimodal models turn UI screenshots into multi-page, navigable websites—raising functional resource matching from near zero to 66–80%.","keywords":["multi-page website generation","UI design to code","resource list","multimodal large language models","web UI similarity metrics","design-to-code benchmark","navigation functionality","image quality assessment"],"falsifier":"Take a model that scores 66–80% RER, then click through every internal link in its generated multi-page site. If the proportion of links that lead to a real, rendered page with the expected content is far below 66–80%—or near zero—the paper's claim that resource lists \"boost navigation functionality\" would be falsified.","tokens_in":16852,"feed_emoji":"🧭","tokens_out":6621,"duration_ms":45233,"temperature":0.7,"pith_summary":"Existing design-to-code tools assume a webpage is a single, self-contained file: they generate static replicas with placeholder images and empty links. This paper introduces a harder task, MRWeb, in which a model must turn a UI screenshot into multi-page code with internal and external navigation, real images, and backend routing. The central proposal is a \"resource list\"—a structured dictionary of each page's links, images, and routes with their bounding boxes—fed to the model alongside the screenshot. The paper reports that with this list, the fraction of resources correctly reproduced jumps from near zero under self-contained prompting to 66–80% across three leading multimodal models, while visual similarity also improves. It also reports a human study showing that among six image-similarity metrics, the pixel-level MAE and NEMD track human judgment best (SROCC 0.542 and 0.508), suggesting that simple pixel comparisons are the right evaluation for web UI code.","feed_headline":"Resource lists lift web-code navigation from 0% to 80%","feed_subtitle":"Giving multimodal models a machine-readable map of links and images turns static replicas into navigable sites.","key_machinery":"The load-bearing mechanism is the resource list: a dictionary-like data structure in which each actionable element of a webpage is recorded with its position (bounding box), its type (link, image, or backend route), and its target URL or source. It is the input channel through which the model learns the exact URLs it must emit, which is why the self-contained baseline—which receives only a screenshot—has no way to reproduce any real resource. The paper's MRWeb dataset of 300 synthetic and 200 real-world websites, and its human-rated set of 600 UI-similarity pairs, are the supporting instruments that give the resource-list mechanism empirical force.","core_discovery":"On its own terms, the paper's central claim is that the missing ingredient in design-to-code is a machine-readable map of the page's resources. Providing the model with a resource list—entries of the form {position: bounding box, type: image/link, url}—makes it possible to reproduce the exact image sources and link targets that make a site functional. Empirically, this lifts the Resource Existence Ratio from 0.006–0.008 under self-contained prompting to 0.640–0.832 depending on model and prompting strategy, and improves MAE, NEMD, and CLIP visual scores across all three tested models. The paper further discovers that when choosing an automated similarity metric for web UIs, pixel-level MAE and NEMD correlate most strongly with human similarity ratings, and that semantic and structural metrics such as SSIM, CLIP, and LPIPS only align with humans in highly similar pairs. The authors treat these results as evidence that resource-aware prompting, not just larger models, is the lever that unlocks functional multi-page generation.","pith_inferences":["Beyond the paper: the RER metric defines \"exists\" as pointing to the same URL/source, so the 66–80% numbers mostly measure whether the model transcribes the supplied list into code; whether the resulting multi-page site actually navigates end-to-end is a separate question the paper does not test.","Beyond the paper: because MAE and NEMD excel in low-to-medium similarity while semantic metrics excel in high similarity, a hybrid metric—pixel error plus a learned semantic score—could match human judgment more uniformly than any single metric in the study.","Beyond the paper: the resource-list trick likely generalizes beyond HTML, e.g., to mobile UI code generation or accessible document generation, wherever the failure mode is that models invent rather than reproduce resource references.","Beyond the paper: a testable extension would be an end-to-end crawl of generated sites, counting what fraction of internal links lead to real rendered pages; the paper's tool case study suggests this could be high on simple sites, but the benchmark's 66–80% figure does not by itself establish it."],"forward_implications":["Design-to-code systems that adopt a resource list can generate pages with genuine image sources and link targets instead of placeholders, which is the difference between a static mockup and a usable site.","For evaluating generated web UIs, the paper's human study makes a case for using pixel-level MAE and NEMD as primary metrics, rather than the semantic or structural scores that prior work favored.","The reported degradation with input image size and resource-list length implies a concrete scaling target: models and prompts must handle large, dense pages before MRWeb generation works on complex real-world websites.","Self-refine prompting consistently outperformed direct and chain-of-thought prompting, while CoT actually hurt performance, giving a practical prompt-design rule for this task.","If reproduced, the 66–80% RER numbers suggest that resource lists are a cheap, data-free way to make multimodal models produce navigable multi-page code from UI designs."],"supporting_citations":[{"why":"Supplies the WebSight synthetic HTML-screenshot pairs that the synthetic dataset starts from and modifies.","marker":"(Laurençon et al., 2024)"},{"why":"Provides the URL list used for link insertion in synthetic data and the pool of real-world website URLs.","marker":"(Raffel et al., 2019)"},{"why":"Supplies the self-contained prompting baseline and the direct prompting approach the study adapts.","marker":"(Si et al., 2024)"},{"why":"Establishes the IQA methodology and the SSIM structural metric used in the human-alignment study.","marker":"(Wang et al., 2004)"},{"why":"Provides a basis for the Wasserstein/EMD formulation behind the NEMD metric.","marker":"(Arjovsky et al., 2017)"},{"why":"Gives the original Earth Mover's Distance metric for image comparison that NEMD normalizes.","marker":"(Rubner et al., 2000)"},{"why":"Provides the CLIP score used as the semantic-level similarity metric in the comparison.","marker":"(Radford et al., 2021)"},{"why":"Supplies the CIEDE2000 color-difference formula used in the fine-grained element evaluation.","marker":"(Luo et al., 2001)"}],"fun_headline_variants":["Resource lists lift web-code navigation from 0% to 80%","A link map turns AI-generated web designs into working multi-page sites","Give the AI a resource list and it builds real navigation, not just pixels","Pixel-perfect isn't enough: online UIs need a link and image map"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on defining a resource as \"matched\" when the generated element points to the same URL or image source as the reference, regardless of whether the link resolves or the target page exists; under that definition the near-zero baseline is almost forced, because the self-contained model is never told the ground-truth URLs.","fun_headline_variants_meta":{"raw":{"variants":["Resource lists lift web-code navigation from 0% to 80%","A link map turns AI-generated web designs into working multi-page sites","Give the AI a resource list and it builds real navigation, not just pixels","Pixel-perfect isn't enough: online UIs need a link and image map"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001208,"raw_usage":{"total_tokens":4989,"prompt_tokens":972,"completion_tokens":4017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":3936}},"tokens_in":588,"tokens_out":4017,"duration_ms":24846,"temperature":1.0,"reasoning_tokens":3936,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:47:16.017855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model that scores 66–80% RER, then click through every internal link in its generated multi-page site. If the proportion of links that lead to a real, rendered page with the expected content is far below 66–80%—or near zero—the paper's claim that resource lists \"boost navigation functionality\" would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the original Earth Mover's Distance metric for image comparison that NEMD normalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CIEDE2000 color-difference formula used in the fine-grained element evaluation."}],"review_version":1}