{"id":"7c51c0e9-a4cd-4ed6-890e-92e6a0c43f94","arxiv_id":"2608.12518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sixty-eight percent of AI-generated webpages fail to render correctly in at least one of nine tested browser or device environments, and a new DOM-plus-screenshot detector catches 91% of these failures.","lead":"What happens to a website when different AI tools build it from the same picture? This study renders 480 AI-generated webpages in nine browser and device setups and finds that 68% of them break in at least one environment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 68% rate is computed after filtering out 226 of 480 generated outputs, and the excluded pages are never analyzed; over all outputs the rate could be as low as 29%, so the unqualified prevalence claim is not yet supported.","rationale":"The reader's weakest assumption correctly identifies the filtering step in Section III-B3 as the main threat to the headline prevalence claim. I agree with that diagnosis. The 68% figure is not simply a noisy estimate over a well-defined population; the denominator itself is a curated subset of 254 valid static pages out of 480 generated outputs, and the 226 excluded pages are never analyzed. The abstract and RQ1 state the result as '68% of generated webpages,' without the static-valid qualifier. Simple bounding shows the true rate over all 480 outputs could be anywhere from 29% to 76%, and the lower endpoint is below the paper's own human-written baseline of 40%, which would reverse the central comparative finding. This makes the concern load-bearing, not a minor threat-to-validity footnote. Other issues, such as the strict wording of Algorithm 1 for cross-browser comparisons and the absence of confidence intervals, are real but secondary: they affect the decomposition and precision of the findings rather than the core prevalence estimate. The detector evaluation is largely independent of the prevalence estimate and remains convincing, with strong F1 on a held-out test set and a released artifact. The concern therefore supports, rather than changes, the reader's CONDITIONAL verdict: the paper should be accepted only with the prevalence claim restated as conditional on the filtering protocol and with the excluded pages either bounded or analyzed.","tokens_in":15422,"tokens_out":8605,"duration_ms":81151,"concrete_test":"Recompute the headline prevalence over all 480 generated outputs under the two extreme assumptions for the 226 filtered pages: all compatible (139/480 = 29.0%) and all incompatible (365/480 = 76.0%). If the lower bound falls below the human baseline or below 50%, revise the abstract and RQ1 to state the rate as conditional on the 254 valid static pages, and report the bounds for all generated outputs. As a follow-up, annotate a random sample of at least 50 of the 226 excluded pages with the same pairwise protocol, treating blank or malformed outputs as a separate outcome, to estimate the actual rate and narrow the interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central prevalence claim, '68% of generated webpages exhibit at least one compatibility issue,' is computed from 139/203 pages in the empirical split, after Stage 3 filtering removed 226 of 480 generated outputs (47%). Section III-B3 lists the excluded categories: blank pages, malformed HTML, rendering failures, and pages that depend on unsupported interactive behavior such as collapsible menus. The compatibility of these excluded outputs is never analyzed, so the denominator is a curated subset of 254 valid static pages, not the full set of generated outputs. This matters because the abstract presents the figure as a property of generated webpages without that qualifier. Bounding the excluded pages gives a wide interval: if all 226 were compatible, the rate over all 480 outputs would be 139/480 = 29.0%, below the paper's own 40% human-written baseline; if all were incompatible, it would be 365/480 = 76.0%. The lower bound reverses the direction of the paper's central comparative finding (Finding 2), so the filtering step is load-bearing rather than a minor scope limitation. The interactive-layout exclusion is particularly non-neutral: such pages are likely to come disproportionately from the harder Design2Code-Hard subset and may have a systematically different compatibility profile. The paper should therefore report prevalence as conditional on the static, valid-page protocol and either annotate or explicitly bound the excluded outputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents WebCompat, a dataset of 2,032 annotated cross-environment rendering pairs built from 480 candidate pages generated by eight AI tools from 60 source pages, each rendered in nine environments. The authors report that 68% of generated webpages have at least one compatibility issue, that cross-device failures dominate cross-browser failures, that 88.3% of failures are page-level, and that missing viewport meta tags and missing flexible wrapping are the dominant root causes. They also propose XCompat, a DOM- and screenshot-based detector, reporting an F1 of 0.903 on a held-out test set, and compare AI-generated pages with human-authored pages. A key design decision is that 226 of 480 generated pages are removed before annotation, and the prevalence analysis is conducted on 203 of the 254 remaining pages.","tokens_in":15673,"tokens_out":7535,"duration_ms":67877,"significance":"If the results hold, this is a timely and useful contribution: it defines a new evaluation dimension for UI-to-code generation, provides a reusable benchmark with high inter-annotator agreement, and ships an efficient detector with a held-out evaluation and a threshold sensitivity analysis. The taxonomy of symptoms and root causes is practically informative, and the comparison to human-authored pages is a reasonable baseline design. The main caveat is that the headline prevalence and the AI-versus-human comparison depend on the page-filtering protocol; this needs to be resolved before the quantitative claims can be taken at face value.","major_comments":[{"comment":"The headline '68% of generated webpages' is the rate on 139/203 pages from the empirical split, after Stage 3 removed 226 of 480 generated outputs (47%). The excluded categories include blank pages, malformed HTML, rendering/capture failures, and layouts that depend on unsupported interactive behavior. The compatibility of these excluded outputs is not analyzed, so the denominator is a curated subset of valid static pages, not the full set of generated outputs. Bounding the excluded outputs gives a range from 139/480 = 29.0% (if all excluded pages are compatible) to 365/480 = 76.0% (if all are incompatible); the lower bound is below the paper's own 40% human baseline, so comparative Finding 2 is not robust to the filtering step. The exclusion of interactive layouts is especially non-neutral because such pages likely come disproportionately from Design2Code-Hard and may have a different compatibility profile. Please report the prevalence as conditional on the static, valid-page protocol and either annotate the excluded pages or provide explicit bounds and a discussion of how the filtering affects Findings 1-3.","section":"§III-B3 / §IV-A, Finding 1 and Table III"},{"comment":"The per-tool incompatibility rates are based on very small page counts (v0 is 5/19, Cursor is 11/14, Direct-GPT5 is 43/43), and no confidence intervals or per-tool hypothesis tests are reported. The statement that 'the best-performing v0 stands at 0.26' is therefore not strongly supported; the exact binomial 95% interval for 5/19 is roughly 9-51%, which overlaps the rates of several other tools. Please report confidence intervals or exact binomial intervals for the per-cell rates and avoid categorical tool-ranking claims unless the sample sizes support them.","section":"§IV-A, Table III"},{"comment":"The detector is evaluated on a held-out test set, which is good practice, but the reported F1 and accuracy are point estimates over 408 pairs with no confidence intervals, and the sensitivity analysis varies each of the three heuristic thresholds over only two nearby values. Since the failure study in §V-E acknowledges that five false negatives are due to 'conservative thresholds,' the claim that threshold choice does not change predictions is too narrow. Please report confidence intervals for the main metrics and a broader threshold sweep, or state clearly which thresholds are fixed protocol choices rather than tuned parameters.","section":"§V-C, Table VI"}],"minor_comments":[{"comment":"The filtering step is described in one sentence; with 47% of the data removed, reproducibility requires a breakdown of the number of pages removed in each excluded category.","section":"§III-B3"},{"comment":"The sentence 'DCGen-GPT4 achieve the two highest fine-grained visual scores' should use 'achieves' to agree with the singular subject.","section":"§IV-A"},{"comment":"The phrase 'meaning it can detect 91% compatibility issues' is imprecise; F1 = 0.903 is not a recall rate. Use the reported recall (0.889) or phrase the sentence in terms of F1.","section":"§V-C"},{"comment":"The legend text 'XBI and XBI: 1 (1.7%)' should read 'XDI and XBI: 1 (1.7%)'.","section":"Figure 4"},{"comment":"There are minor grammar errors: 'Five cases involves' should be 'Five cases involve', and 'Three cases visually appeares' should be 'Three cases visually appear'.","section":"§V-E"},{"comment":"The tool name is spelled inconsistently as both 'REDECHECK' and 'ReDeCheck'; please use one form consistently.","section":"Table VI and elsewhere"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a useful released artifact. The main risk is the headline prevalence claim's sensitivity to the 47% filtering step; I would not accept until the authors either re-scope the prevalence claim to the valid static-page protocol or provide bounds and an analysis of the excluded outputs. The paper fits the journal's scope. I did not find evidence of circularity beyond the usual concern that the detector is designed from the same taxonomy; the held-out split mitigates this, though confidence intervals would strengthen the detector comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid empirical study of a real and underappreciated gap — AI-generated webpages can pass single-environment fidelity checks and still break across devices and browsers. The dataset, annotation protocol, and the XCompat detector are all real contributions. But the headline claim that 68% of generated webpages have compatibility issues is computed after filtering out 226 of 480 generated pages (47%), and the abstract presents it without that qualifier. That's not a minor scope note; it's load-bearing.\n\nWhat's new: first systematic look at cross-environment compatibility for MLLM-generated front-end code. They build a multi-generator, multi-environment dataset with human labels (Cohen's kappa ~0.95 on pairs, 0.94 on root causes) and characterize symptoms and root causes. The finding that visual fidelity scores correlate weakly with compatibility is useful for the UI-to-code evaluation community. The detector is lightweight, fast, and evaluated on a held-out 20% split with a sensitivity analysis on its thresholds; F1 0.903 on test is decent.\n\nSoft spots: the filtering is the big one. The excluded categories — blank pages, malformed HTML, rendering failures, and pages dependent on interactive behavior — are exactly the outputs where compatibility might be systematically different. If all excluded pages were compatible, the rate over all 480 outputs would be 29%, below the 40% human baseline, which would reverse Finding 2. The paper should report prevalence conditional on the valid static-page protocol, and either annotate a sample of excluded pages or explicitly bound them. The human-written comparison group also isn't perfectly matched: the human baseline drops the DCGen source pages entirely, while the AI set filters for static validity; that difference could bias the 1.7x comparison. Also, point estimates lack confidence intervals; with 203 pages, the binomial CI on 68% is roughly 61–74%, which is fine but should be reported. Baseline coverage is reasonable, but a simple \"missing viewport meta\" heuristic would be a useful sanity check.\n\nNone of this invalidates the core observation that compatibility issues are common in AI-generated static pages and that current fidelity benchmarks miss them. But the abstract's unqualified \"68% of generated webpages\" is not supported by the data as presented. The paper deserves a serious referee — it's a valuable dataset and study — but it needs a major revision on the prevalence framing before publication.","headline":"Worth engaging: a genuinely useful first empirical study of cross-environment compatibility in AI-generated webpages, but the headline 68% prevalence figure is computed after dropping 47% of generated outputs and should be qualified or bounded before publication.","tokens_in":16227,"tokens_out":3324,"would_cite":true,"duration_ms":30915,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI-generated webpages are not cross-environment reliable: 68% fail in at least one of nine rendering environments, mostly from missing viewport meta tags and inflexible wrapping.","keywords":["cross-environment compatibility","UI-to-code generation","multimodal large language models","web rendering reliability","responsive design","compatibility detection","empirical software engineering","AI-generated webpages"],"falsifier":"Render all 480 generated pages, including the 226 currently excluded, across the same nine environments and label them with the same annotation protocol; if the incompatibility rate over the full set falls near or below the 40% human baseline, the headline prevalence is an artifact of the filtering.","tokens_in":15198,"feed_emoji":"📱","tokens_out":8097,"duration_ms":64553,"temperature":0.7,"pith_summary":"The paper argues that current UI-to-code evaluation, which checks whether an AI-generated page visually matches a design in one fixed browser and device, misses a crucial deployment property: whether the page still renders correctly in other environments. It builds the first dataset of annotated cross-environment rendering pairs for AI-generated webpages, with 2,032 instances from 254 pages produced by eight generation tools and rendered in nine browser-and-device environments. The headline finding is that 68% of generated pages fail in at least one environment, roughly 1.7 times the failure rate of human-written pages, and that most failures are page-level breakages caused by a few recurring code omissions. The paper also shows that common visual-fidelity scores do not predict these failures, and it proposes a lightweight detector, XCompat, that achieves F1 0.903 on held-out pairs. A sympathetic reader would take the paper as establishing compatibility as a first-class quality dimension for AI-generated front-end code.","feed_headline":"68% of AI-made webpages break on another device or browser","feed_subtitle":"A 2,032-pair study pins most failures on missing viewport settings and catches them with F1 0.903.","key_machinery":"The paper's core object is the rendered-pair comparison: each generated page is rendered in a reference environment and in eight target environments, and each pair yields a screenshot and a post-render DOM tree, meaning the structural document model after layout. The dataset generated from these pairs, with human labels following a two-level taxonomy of page-level and element-level failures, is what carries the prevalence and symptom claims. XCompat carries the detection claim: a page-level module checks viewport metadata, content fill, horizontal overflow, whitespace, and DOM-element overlap, while an element-level module compares matched elements' bounding boxes, aspect ratios, and visibility states; the two module outputs are combined with a logical OR to flag a pair as incompatible.","core_discovery":"The paper's central claim is that cross-environment rendering compatibility is a largely independent quality dimension of AI-generated front-end code, one that single-environment visual-fidelity benchmarks do not capture. On a corpus of 2,032 annotated rendering pairs built from 254 generated pages across nine browser-and-device environments, 68% of AI-generated pages show at least one compatibility issue, compared with 40% of human-authored pages; cross-device failures dominate, with 67.5% of pages showing device-only failures versus 1.0% browser-only failures. The failure taxonomy places 88.3% of incompatible pairs in page-level classes—shrink-to-fit, initial scale mismatch, whitespace anomalies, overflow, and overlap—and root-cause analysis attributes most breakage to missing viewport meta tags (46%) and missing flexible wrapping (29.5%), even though responsive constructs are present in 97.5% of pages. The paper also claims that a lightweight detector combining screenshots and post-render DOM trees reaches F1 0.903 on a held-out test set, surpassing existing structural tools and LLM-based baselines.","pith_inferences":["If the same small set of code omissions causes most failures, a targeted repair model—not proposed in the paper—could be trained to insert viewport meta tags and wrapping rules, and its effect on the 68% rate would be a direct test of the root-cause analysis.","The study's static-page filter excludes interactive layouts; rendering those pages would likely raise the incompatibility rate, since collapsible menus and other dynamic behavior add environment-dependent states that the current corpus cannot measure.","Since neutral prompts were used, prompting generators to include viewport meta and responsive wrapping is a cheap, testable intervention that could close much of the gap to the human baseline."],"forward_implications":["Pages scored highly by single-environment fidelity metrics can still fail in other environments, so benchmark scores need to be supplemented by a multi-environment compatibility pass before deployment decisions are made.","Because cross-device failures outnumber cross-browser failures, generator improvements should target viewport adaptation first.","The dominant root causes are two simple omissions, so a repair step that inserts a viewport meta tag and adds wrapping rules could eliminate a large share of observed failures.","The offline detector's cost of about 0.13 seconds per comparison makes it practical to run compatibility checks across an entire generation corpus.","LLM-based detectors lose accuracy when given full-page screenshots in addition to DOM snapshots, suggesting that single-signal inputs may be preferable for automated triage."],"supporting_citations":[{"why":"Supplies the harder source-page subset, the human-authored baseline pages, and the visual-fidelity metrics used to compare generation quality with incompatibility.","marker":"[7]"},{"why":"Supplies the second source-page subset and the divide-and-conquer generation pipeline used behind three of the tool variants.","marker":"[10]"},{"why":"Provides the oracle-free responsive-layout failure baseline that XCompat is compared against and the initial failure taxonomy adapted for annotation.","marker":"[21]"},{"why":"Provides the differential-rendering approach and the initial symptom taxonomy used to structure the annotation guide.","marker":"[17]"},{"why":"Supplies the real-device cloud rendering service that produces the nine browser-and-device environments.","marker":"[27]"},{"why":"Supplies the judge model used to score overall UI quality, which the paper uses to show that fidelity scores do not predict compatibility.","marker":"[37]"}],"fun_headline_variants":["Most AI webpages break on a new browser or device","68% of AI webpages fail cross-environment checks","Viewport bugs plague 68% of AI-generated webpages","New detector flags AI webpage compat issues at F1 0.903"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study's prevalence figure rests on only the 254 static pages that rendered successfully, because the 226 excluded pages—blank, malformed, broken, or dependent on interactive behavior—were never analyzed.","fun_headline_variants_meta":{"raw":{"variants":["Most AI webpages break on a new browser or device","68% of AI webpages fail cross-environment checks","Viewport bugs plague 68% of AI-generated webpages","New detector flags AI webpage compat issues at F1 0.903"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1592,"prompt_tokens":1107,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":723,"tokens_out":485,"duration_ms":4175,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:06:29.889067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render all 480 generated pages, including the 226 currently excluded, across the same nine environments and label them with the same annotation protocol; if the incompatibility rate over the full set falls near or below the 40% human baseline, the headline prevalence is an artifact of the filtering.","supporting_citations":[{"cited_title":"Divide-and-conquer: Generating ui code from screenshots,","cited_arxiv_id":null,"evidence_quote":"Supplies the second source-page subset and the divide-and-conquer generation pipeline used behind three of the tool variants."},{"cited_title":"Design2code: Benchmarking multimodal code generation for automated front-end engineering,","cited_arxiv_id":null,"evidence_quote":"Supplies the harder source-page subset, the human-authored baseline pages, and the visual-fidelity metrics used to compare generation quality with incompatibility."},{"cited_title":"Automated layout failure detection for responsive web pages without an explicit oracle,","cited_arxiv_id":null,"evidence_quote":"Provides the oracle-free responsive-layout failure baseline that XCompat is compared against and the initial failure taxonomy adapted for annotation."},{"cited_title":"Webdiff: Automated iden- tification of cross-browser issues in web applications,","cited_arxiv_id":null,"evidence_quote":"Provides the differential-rendering approach and the initial symptom taxonomy used to structure the annotation guide."},{"cited_title":"Browserstack,","cited_arxiv_id":null,"evidence_quote":"Supplies the real-device cloud rendering service that produces the nine browser-and-device environments."}],"review_version":1}