REVIEW 12 cited by
Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Websites are critical in today's digital world, with over 1.11 billion currently active and approximately 252,000 new sites launched daily. Converting website layout design into functional UI code is a time-consuming yet indispensable step of website development. Manual methods of converting visual designs into functional code present significant challenges, especially for non-experts. To explore automatic design-to-code solutions, we first conduct a motivating study on GPT-4o and identify three types of issues in generating UI code: element omission, element distortion, and element misarrangement. We further reveal that a focus on smaller visual segments can help multimodal large language models (MLLMs) mitigate these failures in the generation process. In this paper, we propose DCGen, a divide-and-conquer-based approach to automate the translation of webpage design to UI code. DCGen starts by dividing screenshots into manageable segments, generating code for each segment, and then reassembling them into complete UI code for the entire screenshot. We conduct extensive testing with a dataset comprised of real-world websites and various MLLMs and demonstrate that DCGen achieves up to a 15% improvement in visual similarity and 8% in code similarity for large input images. Human evaluations show that DCGen can help developers implement webpages significantly faster and more similar to the UI designs. To the best of our knowledge, DCGen is the first segment-aware MLLM-based approach for generating UI code directly from screenshots.
Forward citations
Cited by 12 Pith papers
-
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
UI2App introduces a benchmark showing that vision-language models can reconstruct web page visuals but largely fail to infer the underlying interaction logic from screenshots alone.
-
Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.
-
WebDesignIter: Co-Evolving Design Knowledge for Repository-Level Front-End Code Generation
A co-evolving design-and-code knowledge graph improves multi-iteration front-end repository generation by ~9.5 Pass@2 points and cuts input tokens 25–30× versus agent baselines.
-
Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing
GUIRepair, a cross-modal LLM pipeline that converts issue screenshots into reproduction code and rendered patch screenshots into validation feedback, resolves 157/517 SWE-bench M instances with GPT-4o and 175 with o4-mini.
-
DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models
A hierarchy-aware, self-correcting LLM pipeline for generating React Native UI code improves visual fidelity and structural similarity over baselines on 300 mockups.
-
MLLM-Based UI2Code Automation Guided by UI Layout Information
LayoutCoder improves screenshot-to-code generation on real websites by parsing the layout into a tree, generating code per region, and fusing it deterministically, beating the best baseline by 10.14 BLEU and 3.95 CLIP...
-
P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark
P2P is a multi-agent framework that automatically generates HTML-rendered academic posters from papers, backed by a 30k instruction dataset and a 121-pair evaluation benchmark.
-
MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs
A resource-list representation and a 500-site benchmark let multimodal LLMs generate web code with real links, images, and routes, lifting resource matching from ~0% to 66-80%.
-
SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks
The full text describes LaTCoder, a block-wise chain-of-thought method for webpage design-to-code, reporting improved layout preservation on two benchmarks, while the title and abstract are for a different paper.
-
LaTCoder: Converting Webpage Design to Code with Layout-as-Thought
LaTCoder cuts a webpage screenshot into blocks, generates code for each block with chain-of-thought prompting, and assembles the blocks, improving layout fidelity over direct whole-page generation.
-
Frontend Diffusion: Empowering Self-Representation of Junior Researchers and Designers Through Multi-agent System
Frontend Diffusion turns sketches and prompts into websites with three AI agents, and interviews with 13 junior academics suggest it helps them express their professional identities online.
-
BitsAI-CR: Automated Code Review via LLM in Practice
An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.
Discussion (0). Continue with ORCID to comment.