Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Five AI coding models default to repeated UI patterns instead of the pixels they are shown.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful first benchmark for a real screenshot-to-code failure mode, but the headline bias rate mixes a code-prior with visual bias; internal controls rescue the core direction. the 4 major comments →

arxiv 2608.03691 v1 pith:HM35OTT5 submitted 2026-08-04 cs.SE cs.AIcs.CV

Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

Khai-Nguyen Nguyen , Oscar Chaparro , Antonio Mastropaolo This is my paper
classification cs.SE cs.AIcs.CV
keywords pattern-completion biasmultimodal code generationscreenshot-to-codevisual saliencyMLLM evaluationbenchmarkfront-end code generationvisual grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether screenshot-to-code AI models follow the actual pixels of a webpage screenshot or fall back to the repeated pattern visible in the surrounding HTML. To find out, the authors built a benchmark that perturbs exactly one element in a repeated UI pattern—one card wider, one text label with a different font size—masks that value in the HTML, and asks the model to recover it. Across 1,440 screenshots and five frontier multimodal models, all five default to the pattern-consistent value in roughly 70–80% of cases, and accuracy is far lower for fine-grained font-size changes (mean 7.89%) than for spatially obvious card-width changes (mean 21.17%). The bias grows with visual noise, subtler deviations, and boundary positions, and models sometimes compute the correct value and then discard it to match the pattern.

Core claim

The paper defines visual pattern-completion bias in screenshot-to-code: when one element in a repeated UI pattern is changed and its CSS value masked, models restore the repeated baseline (100%) rather than the value visible in the screenshot. Across five frontier models and 1,440 screenshots from 30 real webpages, mean bias is 69.78% for card-width perturbations and 80.22% for font-size perturbations, while accuracy is 21.17% and 7.89%. Bias tracks visual saliency: noise, subtler magnitudes, and boundary positions all raise it. The best model, Codex-5.3, drops from 68.61% accuracy on cards to 13.89% on text. Reasoning traces show models often compute the correct value, then override it to m

What carries the argument

The central object is the pattern-versus-visual-evidence benchmark instance: a rendered webpage screenshot paired with a masked HTML snippet in which one element of a repeated card or text pattern has its width or font-size blanked. Each model answer is scored as accurate (matches the perturbed ground truth), biased (matches the 100% pattern baseline), or other error; perturbation magnitude, position in the pattern, and a noise overlay serve as controlled saliency probes. A three-way failure-mode taxonomy (code-anchored, perceived-consistent, observed-then-overrode) classifies the verbalized reasoning behind each biased answer.

Load-bearing premise

The load-bearing premise is that answering '100%' counts as visual pattern completion, but the masked HTML already shows all sibling elements at 100%, so a model can arrive at the same answer from code text alone; the paper's own failure-mode analysis labels 26.2% of biased traces as code-anchored.

What would settle it

Run the same fill-in-the-blank prompt on the same 1,440 instances with the screenshot removed, leaving only the masked HTML snippet. If mean bias stays near the reported 70–80%, the effect is a text-prior phenomenon rather than a visual one; if it drops sharply, the visual input is genuinely driving the pattern-completion behavior.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fine-grained styling fidelity—font sizes, subtle spacing—is the least trustworthy part of current screenshot-to-code output, while coarse layout decisions are comparatively reliable.
  • Page-level quality scores can hide systematic local grounding failures, so evaluation suites should include controlled single-element perturbations.
  • Explicit per-element reasoning prompts can roughly halve bias for some models, but accuracy gains plateau, so prompting alone is not a fix.
  • Re-rendering generated code and comparing the target element's computed style against the source screenshot is a concrete verification step that could catch most pattern-completion errors.
  • Training with pixel-level visual feedback or with curricula of progressively subtler deviations could lower the saliency threshold at which grounding breaks down.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the 26.2% code-anchored share of biased traces implies that a text-only version of this benchmark would likely reproduce a large fraction of the bias, cleanly separating language-prior effects from visual grounding.
  • The observed 'compute the right value, then discard it' pattern suggests the failure may persist even if vision encoders improve, because the final answer is chosen against the model's own visual evidence.
  • The saliency–bias gradient could be measured as a per-model psychometric curve, letting teams know in advance which CSS properties fall below the trust threshold.
  • If pattern completion is a general prior, similar biases should appear for spacing, alignment, and color tokens, and in adjacent tasks such as design-to-markup generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Pattern2Code, a benchmark for visual pattern-completion bias in screenshot-to-code generation. From 484 Design2Code pages, the authors manually select 30 containing repeated card or text patterns, normalize repeated elements to 100%, perturb one element's width (cards) or font-size (text) to 80/90/110/120%, mask the target value in the HTML, and ask five MLLMs to fill in the blank from the screenshot and masked code. Across 1,440 screenshots per model (720 base instances x 2 conditions), they report that all models default to the 100% baseline at high rates (mean 69.78% card bias, 80.22% text bias; accuracy 21.17% and 7.89%, respectively). Noise, subtler perturbation magnitudes, and boundary positions increase bias. Reasoning analyses show longer reasoning correlates with lower bias, and qualitative traces show models sometimes observe the anomaly and still answer 100. The paper concludes that pattern-completion bias is strongly associated with visual saliency and that code specialization does not help.

Significance. This paper addresses a real and under-measured problem: whether screenshot-to-code models preserve localized visual deviations or default to repeated UI patterns. The controlled perturbation design, matched standard/noise rendering conditions, exact answer extraction from 7,200 responses with no parse failures, and public artifacts are solid methodological strengths. The failure-mode analysis (Section 4.5) is valuable, showing that models often engage the screenshot (74%) and sometimes explicitly override correct visual estimates (14.8%). If the central claim holds, the results have clear practical implications for front-end automation and benchmark design. However, the main metric conflates code-context guessing with visual bias, and the statistical support is thin; those issues need to be resolved before the result can be fully credited.

major comments (4)
  1. [§3.3, Eq. (2), Table 6] The headline bias rate (Eq. 2) counts every prediction of 100 as visual pattern completion, but the masked HTML snippet already exposes 100 in all unmasked siblings and the prompt explicitly names the repeated pattern. A model can output 100 from the code context without consulting the screenshot. The paper's own failure-mode analysis (Section 4.5, Table 6) places 26.2% of verbalized biased traces in mode A (Code-anchored), where the rationale never references rendered appearance. The RQ1 numbers (69.78% card bias, 80.22% text bias) therefore conflate text-prior guessing with visual bias. A text-only control (identical prompt/HTML, no screenshot) is needed to estimate the code-context baseline; without it, the 'pattern over pixels' attribution is not established.
  2. [§4.1, Table 2] The card-vs-text comparison confounds saliency with property type: cards vary width, text varies font-size. The claim that lower visual saliency drives higher bias is supported by the noise/magnitude/position analyses, but those analyses do not isolate saliency from property type in the headline RQ1 comparison. A within-property saliency manipulation (e.g., same font-size deviations with different contrast/background) or a property-matched design is needed to attribute the 69.78% vs. 80.22% gap specifically to saliency.
  3. [§3.1.3, Table 3] The noise-overlay condition places eight 80x40 opaque rectangles at random positions with a fixed seed, but the paper does not state whether rectangles may cover the perturbed element. If an overlay rectangle can occlude the target, the noise condition no longer holds the visual evidence for the target constant, and the matched-conditions claim in Section 3.1.3 is violated. The authors should specify that rectangles avoid the target region, or control for target occlusion (e.g., exclude overlap cases or report occlusion rates).
  4. [§4.1–§4.3, Tables 2–5] All headline comparisons are point estimates without confidence intervals or significance tests. Each model-pattern condition is based on 360 instances drawn from only 30 webpages (clustered), and each instance is run once; proprietary models may also have nondeterministic decoding. Bootstrap CIs stratified by webpage, and appropriate tests for the card-vs-text and noise differences, are needed before claims like 'all MLLMs exhibit' and 'strongly associated' are supported.
minor comments (5)
  1. [§3.1.1] The text 'i guess is 93.9% (93.7)% of rejections' is informal and contains an inconsistent double percentage. Fix the typo and reconcile the two percentages.
  2. [§3.4] Grouping Opus-4.6 as a 'code-specialized' model is questionable; Opus is a general flagship model. Reclassify or soften the RQ1 conclusion that code specialization does not help.
  3. [§3.3/§5] The answer constraint 'divisible by 10' is acknowledged as a limitation, but no sensitivity analysis is provided. Add a small ablation with unconstrained parsing or justify why the constraint cannot change bias measurements.
  4. [Figure 3] The x-axis tick labels appear garbled ('20 22 24 26 28 2100'); they should read as powers of two (2^0, 2^2, 2^4, 2^6, 2^8, 2^10) or similar.
  5. [§4.5] LLM-as-a-judge validation used only 30 traces. Report a confidence interval for the 96.7% agreement or add a second judge to strengthen reliability.

Circularity Check

0 steps flagged

No significant circularity: the benchmark measurements are independent external evaluations, and the only author-overlap citation ([38]) is motivational, not load-bearing. The main caveat is a construct-validity one—the bias metric can conflate text-prior guessing with visual pattern completion—but that is not a circular reduction.

full rationale

The paper's derivation chain is a controlled empirical benchmark, not a fitted model. Accuracy, bias rate, and other error (Eqs. 1?3) are defined directly and independently of the models' parameters: no parameter is fitted to the evaluation set, and no prediction is derived from a fitted input. The central RQ1/RQ2 comparisons (card vs text, standard vs noise, perturbation magnitude, position) are computed from model responses to external Design2Code-derived screenshots, so the numerical results are self-contained and falsifiable. The only author-overlap citation is Vo et al. [38], used in Sections 1 and 2.3 to motivate the 'pattern completer' concept; it supplies no equations, no fitted values, and no uniqueness claim, so it is not load-bearing. The paper itself acknowledges in Section 4.5 that 26.2% of verbalized biased responses are code-anchored (mode A) and in Section 5 that bias rate 'emphasizes one specific error direction'; this weakens the visual-causation wording of the headline claim, and a text-only control would sharpen the interpretation, but this is a construct-validity concern rather than a circular reduction. Because the empirical numbers do not reduce to their inputs by construction, the correct circularity score is low.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The benchmark rests on domain assumptions about representativeness and rendering, on the design constraint of divisible-by-10 answers, and on the interpretation that 100% predictions mean pattern bias. No fitted parameters and no invented entities are used.

axioms (3)
  • domain assumption The 30 manually curated Design2Code webpages are representative of real repeated UI patterns.
    Section 3.1.1 rejects 413 of 484 pages (93.9%) for lacking clear card patterns, so the benchmark is built on a small, hand-picked subset whose representativeness is asserted, not demonstrated.
  • domain assumption Playwright rendering at 1000x1400 with the fixed noise overlay preserves the visual information needed to recover the perturbation.
    Section 3.1.3 describes the rendering and noise conditions, but the paper does not verify that the perturbed value is actually recoverable from the rendered pixels by a human or an oracle.
  • ad hoc to paper Requiring the answer to be divisible by 10 does not materially change the measured bias.
    Section 3.3 instructs the model that the blank must be a percentage divisible by 10, which narrows the answer space and may inflate both accuracy and bias relative to unconstrained CSS generation.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation." pith.science (2026). https://pith.science/paper/HM35OTT5

@misc{pith2026260803691,
  author       = {Pith},
  title        = {Pith review of: Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HM35OTT5}},
  note         = {Machine review of arXiv:2608.03691}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. In this work, we test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from the Design2Code dataset, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. Codex-5.3 performs best but still drops from 68.61% accuracy on cards to 13.89% on text, while Flash-3.0 reaches 96.11% bias on text. Noise, subtler perturbations, and boundary positions further increase bias rate. Reasoning analysis further shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer. Our results identify a concrete failure mode in multimodal code generation and show that its severity is strongly associated with visual saliency

Figures

Figures reproduced from arXiv: 2608.03691 by Antonio Mastropaolo, Khai-Nguyen Nguyen, Oscar Chaparro.

Figure 1
Figure 1. Figure 1: Visual pattern-completion bias in action. Given [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pattern2Code benchmark construction. From 484 Design2Code webpages, we retain 30 with repeated UI structures, extract and perturb one element per pattern across three positions and four magnitudes, and render each under standard and noise-overlaid conditions (1,440 screenshots per model). model should recover the localized deviation from the screenshot, whereas a model that over-relies on learned design pr… view at source ↗
Figure 3
Figure 3. Figure 3: Reasoning effort vs accuracy and bias rates for the two OpenAI models. Each point is a model-specific effort bin positioned by the log2 of its mean reasoning-token count. More reasoning consistently correlates with lower bias [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Number of examples per reasoning effort: [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two examples of pattern-completion bias in model reasoning. In (a), [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 25 canonical work pages · 2 internal anchors

  1. [1]

    [n. d.]. Replication Package. https://doi.org/10.5281/zenodo.19341952

    work page doi:10.5281/zenodo.19341952

  2. [2]

    2026.Claude Code overview

    Anthropic. 2026.Claude Code overview. https://code.claude.com/docs/en/ overview Claude Code Docs. Accessed: 2026-03-09

    work page 2026

  3. [3]

    Anthropic. 2026. Introducing Claude Opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6. https://www.anthropic.com/news/claude-opus-4-6 Accessed: 2026-03-26

    work page 2026

  4. [4]

    Anthropic. 2026. Introducing Claude Sonnet 4.6. https://www.anthropic.com/ news/claude-sonnet-4-6. https://www.anthropic.com/news/claude-sonnet-4-6 Accessed: 2026-03-26

    work page 2026

  5. [5]

    Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christo- pher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A Rodriguez, et al. 2025. Webmmu: A benchmark for multimodal multilingual website under- standing and code generation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 25129–25156

    work page 2025

  6. [6]

    Tony Beltramelli. 2018. pix2code: Generating code from a graphical user inter- face screenshot. InProceedings of the ACM SIGCHI symposium on engineering interactive computing systems. 1–6

    2018

  7. [7]

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. InProceedings of the 26th annual international conference on machine learning. 41–48

    2009

  8. [8]

    Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2616– 2627

    work page 2023

  9. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  10. [10]

    2026.Gemini Code Assist overview

    Google. 2026.Gemini Code Assist overview. https://developers.google.com/ gemini-code-assist/docs/overview Google Developers Documentation. Accessed: 2026-03-09

    work page 2026

  11. [11]

    Google DeepMind. 2025. Gemini 3 Flash: Frontier Intelligence Built for Speed. https://blog.google/products/gemini/gemini-3-flash/. https://blog.google/ products/gemini/gemini-3-flash/ Accessed: 2026-03-26

    work page 2025

  12. [12]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    work page 2024

  13. [13]

    Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Bohua Chen, Siyuan Wu, Xing Zhou, Wenbin Jiang, Hai Jin, and Xiangliang Zhang. 2024. WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs.arXiv preprint arXiv:2404.06369(2024). https://arxiv.org/abs/2404.06369

    Pith/arXiv arXiv 2024

  14. [14]

    Yi Gui, Yao Wan, Zhen Li, Zhongyi Zhang, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, and Xiangliang Zhang. 2025. UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs.arXiv preprint arXiv:2505.09904(2025). https://arxiv.org/abs/ 2505.09904

    Pith/arXiv arXiv 2025

  15. [15]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  16. [16]

    Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Hem- ing Cui

    Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Hem- ing Cui. 2025. Bias Testing and Mitigation in LLM-based Code Generation. arXiv:2309.14345 [cs.SE] https://arxiv.org/abs/2309.14345

    Pith/arXiv arXiv 2025

  17. [17]

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Gong. 2024. Visual Hallu- cinations of Multi-modal Large Language Models. InFindings of the Associ- ation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (Eds.). Association for Computational Linguistics, 9614–96...

    doi:10.18653/v1/2024.findings-acl.573 2024

  18. [18]

    Lingjie Jiang, Shaohan Huang, Xun Wu, Yixia Li, Dongdong Zhang, and Furu Wei. 2025. Viscodex: Unified multimodal code generation via merging vision and coding models.arXiv preprint arXiv:2508.09945(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Yilei Jiang, Yaozhi Zheng, Yuxuan Wan, Jiaming Han, Qunzhong Wang, Michael R Lyu, and Xiangyu Yue. 2025. Screencoder: Advancing visual-to-code genera- tion for front-end automation via modular multimodal agents.arXiv preprint arXiv:2507.22827(2025)

    arXiv 2025

  20. [20]

    Hugo Laurençon, Léo Tronchon, and Victor Sanh. 2024. Unlocking the Conversion of Web Screenshots into HTML Code with the WebSight Dataset.arXiv preprint arXiv:2403.09029(2024). https://arxiv.org/abs/2403.09029

    Pith/arXiv arXiv 2024

  21. [21]

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Mar- tin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual lan- guage understanding. InInternational Conference on Machine Learning. PMLR, 18893–18912

    2023

  22. [22]

    Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. Vlind-bench: Measuring language priors in large vision-language models. InFindings of the Association for Computational Linguistics: NAACL 2025. 4129–4144

    work page 2025

  23. [23]

    Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, and Xuelong Li. 2025. WebUIBench: A Comprehensive Benchmark for Evalu- ating Multimodal Large Language Models in WebUI-to-Code.arXiv preprint arXiv:2506.07818(2025). https://arxiv.org/abs/2506.07818

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Lin Ling, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2025. Bias Unveiled: In- vestigating Social Bias in LLM-Generated Code. arXiv:2411.10351 [cs.SE] https://arxiv.org/abs/2411.10351

    Pith/arXiv arXiv 2025

  25. [25]

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2024. Phd: A chatgpt-prompted visual hallucination evaluation dataset.arXiv preprint arXiv:2403.11116(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Yan Liu, Xiaokang Chen, Yan Gao, Zhe Su, Fengji Zhang, Daoguang Zan, Jian- Guang Lou, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Uncovering and quantifying social biases in code generation.Advances in Neural Information Processing Systems36 (2023), 2368–2380

    2023

  27. [27]

    Microsoft. 2020. Playwright. https://github.com/microsoft/playwright

    work page 2020

  28. [28]

    2025.https://openai.com/index/introducing-codex/

    OpenAI. 2025.https://openai.com/index/introducing-codex/. https://openai.com/ index/introducing-codex/

    work page 2025

  29. [29]

    OpenAI. 2026. GPT-5.3 Instant: Smoother, More Useful Everyday Conversations. https://openai.com/index/gpt-5-3-instant/. https://openai.com/index/gpt-5-3- instant/ Accessed: 2026-03-26

    work page 2026

  30. [30]

    OpenAI. 2026. Introducing GPT-5.3-Codex. https://openai.com/index/ introducing-gpt-5-3-codex/. https://openai.com/index/introducing-gpt-5-3- codex/ Accessed: 2026-03-26

    work page 2026

  31. [31]

    2026.OpenRouter

    OpenRouter. 2026.OpenRouter. https://openrouter.ai/ Accessed: 2026-03-20

    work page 2026

  32. [32]

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2022. VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov,...

    work page doi:10.18653/v1/2022.acl-lo 2022

  33. [33]

    Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. 2024. IllusionVQA: A challenging optical illusion dataset for vision language models.arXiv preprint arXiv:2403.15952 (2024)

    Pith/arXiv arXiv 2024

  34. [34]

    Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang

  35. [35]

    2025.Technology — 2025 Stack Overflow Developer Survey

    Stack Overflow. 2025.Technology — 2025 Stack Overflow Developer Survey. https://survey.stackoverflow.co/2025/technology#most-popular- technologies-ai-models-ai-models-prof Accessed: 2026-03-20

    work page 2025

  36. [36]

    Haoyu Sun, Huichen Will Wang, Jiawei Gu, Linjie Li, and Yu Cheng. 2025. Full- Front: Benchmarking MLLMs Across the Full Front-End Engineering Workflow. arXiv preprint arXiv:2505.17399(2025). https://arxiv.org/abs/2505.17399

    Pith/arXiv arXiv 2025

  37. [37]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InCVPR

    work page 2024

  38. [38]

    An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. 2025. Vision Language Models are Biased.CoRRabs/2505.23941 (2025). https://doi.org/10.48550/ARXIV.2505.23941 arXiv:2505.23941

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23941 2025

  39. [39]

    Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael R. Lyu. 2024. Automatically Generating UI Code from Screen- shot: A Divide-and-Conquer-Based Approach.arXiv preprint arXiv:2406.16386 (2024). https://arxiv.org/abs/2406.16386

    Pith/arXiv arXiv 2024

  40. [40]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936

    2024

  41. [41]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt

  42. [42]

    Bigham, and Jeffrey Nichols

    Jason Wu, Eldon Schoop, Alan Leung, Titus Barik, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback.arXiv preprint arXiv:2406.07739 (2024). https://arxiv.org/abs/2406.07739

    Pith/arXiv arXiv 2024

  43. [43]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494

    2023

  44. [44]

    Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R. Lyu. 2024. Interaction2Code: Bench- marking MLLM-based Interactive Webpage Code Generation from Interactive Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation ASE ’26, October 12–16, 2026, Munich, Germany Proto...

    arXiv 2024

  45. [45]

    Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R. Lyu. 2025. DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation.arXiv preprint arXiv:2506.06251(2025). https://arxiv.org/abs/2506.06251

    arXiv 2025

  46. [46]

    Shuhong Xiao, Yunnong Chen, Jiazhi Li, Liuqing Chen, Lingyun Sun, and Tingting Zhou. 2024. Prototype2Code: End-to-end Front-end Code Generation from UI Design Prototypes.arXiv preprint arXiv:2405.04975(2024). https://arxiv.org/abs/ 2405.04975

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    Zhen Yang, Wenyi Hong, Mingde Xu, Xinyue Fan, Weihan Wang, Jiele Cheng, Xiaotao Gu, and Jie Tang. 2025. UI2Codeˆ N: A Visual Language Model for Test- Time Scalable Interactive UI-to-Code Generation.arXiv preprint arXiv:2511.08195 (2025)

    Pith/arXiv arXiv 2025

  48. [48]

    Moon Ye-Bin, Nam Hyeon-Woo, Wonseok Choi, and Tae-Hyun Oh. 2024. Beaf: Observing before-after changes to evaluate hallucination in vision-language models. InEuropean Conference on Computer Vision. Springer, 232–248

    work page 2024

  49. [49]

    Sukmin Yun, Rusiru Thushara, Mohammad Bhat, Yongxin Wang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, et al . 2024. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms.Advances in neural information processing systems37 (2024), 112134–112157

    work page 2024

  50. [50]

    Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang

    work page

  51. [51]

    Ting Zhou, Yanjie Zhao, Xinyi Hou, Xiaoyu Sun, Kai Chen, and Haoyu Wang

  52. [52]

    Hongda Zhu, Yiwen Zhang, Bing Zhao, Jingzhe Ding, Siyao Liu, Tong Liu, Dandan Wang, Yanan Liu, and Zhaojian Li. 2025. FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation.arXiv preprint arXiv:2506.13832(2025). https://arxiv.org/abs/2506.13832 Received 2026-03-27; accepted 2026-06-18

    Pith/arXiv arXiv 2025

  53. [55]

    https://arxiv.org/abs/2409

    Bridging Design and Development with Automated Declarative UI Code Generation.arXiv preprint arXiv:2409.11667(2024). https://arxiv.org/abs/2409. 11667

    Pith/arXiv arXiv 2024

  54. [2023]

    InFindings of the Association for Computational Linguistics: EMNLP

    ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense. InFindings of the Association for Computational Linguistics: EMNLP. https://openreview.net/forum?id=N6sXsHuWDE

    work page

  55. [2024]

    InGenerative ai for effective software development

    Chatgpt prompt patterns for improving code quality, refactoring, require- ments elicitation, and software design. InGenerative ai for effective software development. Springer, 71–108

    work page

  56. [2025]

    InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

    Design2code: Benchmarking multimodal code generation for automated front-end engineering. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 3956–3974

    2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.