Pith. sign in

REVIEW 2 major objections 5 minor 46 references

LLM Code Customization with Visual Results: A Benchmark on TikZ

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that current LLMs customize TikZ code to match a requested visual change only 13% of the time in one shot, and 28% when given five tries.

desk verdict A useful first-version benchmark for LLM code customization with visual validation, but the reported 13%/28% success rates are lower bounds because 13 known-valid solutions are missing from the v1.0 ground truths. read the letter →

arxiv 2505.04670 v2 pith:XMHGLLXL submitted 2025-05-07 cs.SE cs.AI

Charly Reux (DiverSe) , Mathieu Acher (DiverSe) , Djamel Eddine Khelladi (DiverSe) , Olivier Barais (DiverSe) , Clément Quinton (SPIRALS) This is my paper · ORCID
classification cs.SEcs.AI
keywords LLMcodecustomizationvisualintentTikZbenchmarkparameterizedgroundtruthmultimodalfeedbackeditingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces vTikZ, a benchmark built to measure whether large language models can customize existing TikZ code so that the rendered image matches a natural-language edit request. The authors assembled 100 TikZ customization scenarios drawn from real diagrams, with parameterized reference solutions to allow more than one correct edit, plus a reviewing tool that compares generated images against those solutions. Running current models on the benchmark, they report that LLMs produce the desired output in only 13% of cases in a single attempt and in 28% of cases when allowed five attempts. The result matters because code customization with visual output—adjusting a diagram, interface, or image by editing source—is exactly the task where users cannot easily verify code behavior, and the benchmark gives the community a reproducible way to measure and close that gap.

What carries the argument

The load-bearing object is the parameterized ground-truth framework. Each of the 100 scenarios pairs a TikZ program with an instruction and a list of solution templates written with four parameterization directives—§range(lower, higher, default), §rangei(value, interval), §choice([A,B,...], default), and §def(value)—so that multiple valid colors, radii, shifts, and variable names count as correct. Correctness is then decided by the SuccessCustomizationMetric, which accepts a generated variant if it matches one of these parameterized solutions or produces an exact rendered-image match, and ranking across reference solutions uses compile status, line-location, line-edit rate, and patch similarity in priority order. The framework is what allows the 13% and 28% numbers to be measured at all, and the paper's 300 human annotations of model outputs are used to validate and extend that ground truth.

What would settle it

Re-run the evaluation with an expanded ground-truth set that includes the 13 human-annotated additional solutions and permits optional-line and placement-insensitive edits, then check whether the one-shot success rate rises measurably above 13% and the best-of-five rate above 28%. A move of several percentage points would indicate the reported numbers underestimate what LLMs can do; a negligible change would confirm the gap.

Watch

Extended reading notes

Core claim

vTikZ is, to the authors' knowledge, the first benchmark explicitly targeting LLM-driven customization of code with visual results rather than code generation from scratch. Its central finding is that current models fail this task most of the time: against 100 carefully curated TikZ editing scenarios, the success rate for producing a fully correct modified diagram is about 13% in a one-shot setting and about 28% using best-of-five sampling with the strongest model evaluated. The benchmark also quantifies where failures occur: models often compile successfully and often find the right lines, but they frequently edit the wrong feature, edit too many features, or apply the right edit incompletely. The paper argues that this gap shows LLMs are effectively blind to the rendered output during editing and that integrating visual feedback mechanisms is a necessary next direction.

Load-bearing premise

The benchmark's correctness rule assumes that the parameterized reference solutions capture essentially all reasonable ways to satisfy each instruction; the paper itself notes that the four parameterization commands may miss cases where lines are optional or placement does not matter, and human review surfaced 13 additional correct solutions.

Editorial extensions

If this is right

  • If the reported rates hold, current LLM-based code assistants are not yet reliable enough for real-world use on visual code customization, since even the strongest model fails three out of four times when given five tries.
  • The vTikZ methodology transfers to other graphics languages: the same scenario, prompt, parameterized-ground-truth, and review-tool structure can be applied to SVG, P5.js, Turtle, or Matplotlib.
  • The 300 human-annotated variants become training or evaluation material for models that incorporate visual feedback or human preference signals.
  • Benchmark scores can serve as a quantitative target for agentic or self-refining systems that render intermediate output and iterate until the rendered image matches the instruction.
  • The explicit split between feature location and edit execution makes vTikZ useful for diagnosing whether failures come from understanding the prompt, finding the code region, or making the final change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely reason for the low success rates is that LLMs cannot see the rendered picture during editing; the paper's own oracle experiments, where vision models misclassified a substantial share of correct solutions, support the inference that simply adding an image input does not yet supply reliable feedback.
  • Because the ground truth penalizes valid edits it cannot express, the 13% and 28% figures are better read as lower bounds on model capability than as exact ability ceilings.
  • One testable extension of the paper's argument: an agent that renders its own draft, compares it with a rendered reference via a dedicated diffing tool, and re-prompts on mismatch should beat best-of-five sampling; if it does not, the bottleneck is not feedback but the model's code-editing competence itself.
  • The benchmark's focus on a single-file, single-diagram language means it likely understates the difficulty of visual customization in multi-file projects, where feature location spans files and edits must preserve cross-file consistency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces vTikZ, a benchmark for evaluating LLMs on TikZ code customization tasks in which a natural-language instruction requires modifying existing code to achieve a specific visual change. The benchmark contains 100 manually curated editing scenarios derived from 45 TikZ programs, parameterized ground-truth solutions, and a reviewing tool. The authors evaluate five open-source LLMs and GPT-4o (with and without image input) using compile, feature-location, success-customization, similarity, and line metrics, reporting success rates of 13% in one-shot and 28% with five attempts for the best model. The paper also contributes a dataset of 300 human-annotated LLM outputs and a failure-mode analysis.

Significance. If the benchmark is valid, it fills a genuine gap: existing software-engineering benchmarks ignore visual output, and existing visual benchmarks do not involve editing existing code. The released dataset, evaluation harness, and human annotations are concrete assets for future work. The qualitative finding that current LLMs are unreliable at such customization tasks is supported by the low success rates and the failure-mode analysis in Section 6.2. However, the precise numerical claims depend on the completeness of the parameterized ground truths, which the manuscript itself flags as incomplete; this makes the headline percentages lower bounds rather than point estimates.

major comments (2)
  1. [4.4, 6.3, 8] The ground-truth incompleteness acknowledged in Sections 4.4 and 6.3 directly affects the headline success rates reported in Section 8. Section 4.4 states that human annotation "surfaced 13 additional correct solutions that were previously missed," Section 6.3 concedes that the four parameterization commands "may not cover cases where certain lines are optional or line placement does not affect correctness," and Section 8 states that the results rely on version 1.0, with "potentially incorporating missing correct solutions" listed as a planned change. Because SuccessCustomizationMetric marks an output as successful only if it matches a parameterized reference solution or produces an exact image match, any model output that is one of these 13 valid solutions (or another valid solution outside the parameterization) is counted as a failure. The reported 13% one-shot and 28% best-of-5 figures are therefore lower bounds, and the magnitude of the resulting underestimation is unknown. I request that the authors either incorporate the 13 solutions into the benchmark and recompute all results, or provide a sensitivity analysis that quantifies how many of the evaluated outputs would be reclassified if these solutions were accepted.
  2. [4.1, 4.2] The definition of LocationMetric as "whether 100% of the lines of a patch was edited" is ambiguous: it is not clear whether it requires every line of the reference patch to appear in the generated patch, whether every edited line in the generated patch must be among the reference lines, or both. This ambiguity matters because LocationMetric is ranked second in the priority order used in Section 4.2 to select the best reference tuple per variant, so it can influence the final success classification. Please specify the exact computation (for example, precision, recall, or exact set equality) and the behavior when the generated patch contains extra or missing lines.
minor comments (5)
  1. [5.1, Table 3] Mixtral-8x7B appears in Table 3 but is missing from the enumeration of evaluated models at the start of Section 5.1; please add it to the model list or explain why it is only reported in the table.
  2. [5.3] The sentence in Section 5.3 stating that "the number of successful edits decreased marginally (by only one) when solely relying on code inputs" is confusing; please rephrase to report the actual counts for the text-only and text-plus-image conditions.
  3. [4.1, LineMetric] The formula for LineMetric divides by the total number of edited lines in the reference solution; if the generated patch edits more lines than the reference, the metric may exceed 100. Please clarify whether the metric is capped at 100 or normalized.
  4. [3.1] The character-length outlier criterion is described using an IQR formula that is not rendered cleanly in the submitted text; please format the formula properly with standard notation.
  5. [6.4] The vision-oracle experiments in Section 6.4 report misclassification counts without confidence intervals or statistical detail; these are clearly preliminary, so please label them explicitly as such and avoid drawing strong conclusions from the raw counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: vTikZ is an empirical benchmark whose correctness metric compares LLM outputs to pre-existing human-created reference solutions; the admitted incompleteness of those ground truths is a measurement-validity limitation, not a derivation forced by construction.

full rationale

This paper does not present a mathematical derivation or a fitted model whose predictions reduce to its inputs. The central claim is an empirical measurement: LLMs are evaluated against a benchmark of 100 human-curated TikZ customization tasks, and correctness is judged by the SuccessCustomizationMetric, which "evaluat[es] whether the generated code matches a parameterized solution or produces an exact image match." The parameterized ground truths were constructed by the authors from human-made variants, independently of the LLM outputs being scored. The evaluation is therefore external to the benchmark construction: the model responses are not used to define the reference solutions, nor are any benchmark parameters fitted to the evaluated models. The paper itself discloses the main weakness of this design in Section 6.3: the four parameterization commands "may not cover cases where certain lines are optional or line placement does not affect correctness," and Section 4.4 reports that human annotation "surfaced 13 additional correct solutions that were previously missed." Section 8 further states that the results rely on version 1.0 with "potentially incorporating missing correct solutions" listed as a planned update. This is a genuine ground-truth coverage limitation: the reported 13% and 28% success rates could underestimate true LLM performance if valid solutions were marked incorrect. However, an incomplete oracle is not circularity; the metric does not reduce to the input by construction, and the benchmark's correctness standard is an explicit, author-defined reference set rather than a hidden reuse of the evaluated outputs. There is no self-citation chain that carries the argument: the only self-citation visible in the text is contextual support for the growing interest in LLM-based code customization and is not load-bearing. There is also no uniqueness theorem, ansatz smuggled in via citation, or renaming of a known result. For these reasons, the appropriate finding is no significant circularity, with the ground-truth coverage issue treated as a benchmark-validity concern rather than a circularity defect.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central contribution is an empirical benchmark, not a derivation, so the ledger records the assumptions that support the measurement: representativeness of the 100 selected tasks, completeness of the parameterized reference solutions, oracle-style best-of-N selection as a fairness convention, and standard algorithms used for auxiliary metrics. The only hand-set numerical values are benchmark tolerances and curation thresholds, which influence how hard or lenient the evaluation is.

free parameters (2)
  • Parameterized ground-truth tolerance values = Not fitted; examples include §range(10, 30, 20) for a circle radius and §choice([600, 700], 600) for a color
    Human-chosen acceptance intervals define whether a generated variant counts as correct; wider or narrower ranges change the SuccessCustomizationMetric without changing the code.
  • Manual curation thresholds for source code selection = 700 to 3570 characters, more than 3 TikZ drawing commands, single \begin{tikzpicture}
    These thresholds cut the nllg/datikz pool before the authors manually picked 35 codes; different thresholds would yield a different benchmark.
assumptions (5)
  • domain assumption TikZ customization is a representative proxy for visual code customization in other domains such as SVG, Web design, and 3D modeling.
    Stated in Sections 1 and 6.3 to motivate generality; the benchmark only measures TikZ, so the transfer claim is assumed.
  • domain assumption The parameterized solution set covers the valid solution space well enough that a mismatch implies failure.
    Needed for the SuccessCustomizationMetric; the authors acknowledge in Section 6.3 that the four parameterization commands cannot cover all valid edits and that annotation found 13 additional correct solutions.
  • domain assumption The 100 scenarios and 45 source programs are representative of real customization workloads.
    The selection protocol in Section 3.1 mixes a StackExchange post, a public TikZ dataset, and informal manual exploration; representativeness is assumed, not statistically demonstrated.
  • domain assumption Best-of-N selection against ground-truth metrics provides an upper bound on what a user could expect from N tries.
    Section 4.2 selects the variant with the best ground-truth score; this is an oracle selection and assumes the metric is a valid reward model.
  • standard math Standard algorithms such as Zhang-Shasha tree edit distance and CrystalBLEU behave as published.
    Used to compute AST difficulty and patch similarity; standard, unverified in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Code Customization with Visual Results: A Benchmark on TikZ." pith.science (2026). https://pith.science/paper/XMHGLLXL

@misc{pith2026250504670,
  author       = {Pith},
  title        = {Pith review of: LLM Code Customization with Visual Results: A Benchmark on TikZ},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMHGLLXL}},
  note         = {Machine review of arXiv:2505.04670}
}
read the original abstract

With the rise of AI-based code generation, customizing existing code out of natural language instructions to modify visual results -such as figures or images -has become possible, promising to reduce the need for deep programming expertise. However, even experienced developers can struggle with this task, as it requires identifying relevant code regions (feature location), generating valid code variants, and ensuring the modifications reliably align with user intent. In this paper, we introduce vTikZ, the first benchmark designed to evaluate the ability of Large Language Models (LLMs) to customize code while preserving coherent visual outcomes. Our benchmark consists of carefully curated vTikZ editing scenarios, parameterized ground truths, and a reviewing tool that leverages visual feedback to assess correctness. Empirical evaluation with stateof-the-art LLMs shows that existing solutions struggle to reliably modify code in alignment with visual intent, highlighting a gap in current AI-assisted code editing approaches. We argue that vTikZ opens new research directions for integrating LLMs with visual feedback mechanisms to improve code customization tasks in various domains beyond TikZ, including image processing, art creation, Web design, and 3D modeling.

Figures

Figures reproduced from arXiv: 2505.04670 by the authors.

Figure 1
Figure 1. Contextualized example of the benchmark task. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. vTikZ dataset curation overview and inherent difficulty, making them challenging for end-users to understand at first sight. Such examples illustrate the complexity of TikZ while maintaining a well-defined structure. They also sup￾port high-level prompts specifying feature edits in the diagram, e.g., "Make the eye of the dog red." [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Annotation interface 4.3 Technical Details The benchmark is designed to evaluate both LLMs and Large Multi￾modal Models (LMMs) in a comparable manner. In addition to textual inputs, LMMs can also process images, making them suitable for TikZ-based image modifications. Both model types leverage system prompts, ensuring a better alignment with the benchmark objectives. LLM and LMM system prompts are structured as foll… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Classification of Variants using the Compile, Location, and SuccessCustomization metrics [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Prompt used for the evaluation of VLM/Multimodal [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 9 canonical work pages

  1. [1]

    Mathieu Acher, José Galindo Duarte, and Jean-Marc Jézéquel. 2023. On Program- ming Variability with Large Language Model-based Assistant (SPLC ’23, Vol. A). 8–14. https://dl.acm.org/doi/10.1145/3579027.3608972

    arXiv 2023

  2. [2]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992 version: 1

    arXiv 2024

  3. [3]

    Tyler Angert, Miroslav Suzara, Jenny Han, Christopher Pondoc, and Hariharan Subramonyam. 2023. Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts (UIST ’23). New York, NY, USA, 1–22

    work page 2023

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732

    arXiv 2021

  5. [5]

    Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. 2023. HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models. arXiv:2304.05390

    work page Pith review arXiv 2023

  6. [6]

    Ayan Banerjee, Nityanand Mathur, Josep Lladós, Umapada Pal, and Anjan Dutta

    work page

  7. [7]

    Jonas Belouadi, Anne Lauscher, and Steffen Eger. 2024. AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ. arXiv:2310.00367 [cs]

    arXiv 2024

  8. [8]

    Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. 2024. DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ. arXiv:2405.15306 [cs]

    arXiv 2024

Show all 46 references
  1. [9]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments ...

    2023 arXiv

  2. [10]

    Mark Chen and al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374

    2021 arXiv

  3. [11]

    Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. Visual Programming for Text- to-Image Generation and Evaluation. arXiv:2305.15328

    2023 arXiv

  4. [12]

    Yi Cui. 2024. WebApp1K: A Practical Code-Generation Benchmark for Web App Development. arXiv:2408.00019 version: 1

    2024 arXiv

  5. [13]

    Aryaz Eghbali and Michael Pradel. 2023. CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code (ASE ’22). New York, NY, USA, 1–12

    2023

  6. [14]

    Kanika Goswami, Puneet Mathur, Ryan Rossi, and Franck Dernoncourt. 2025. PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback. arXiv:2502.00988 version: 1

    2025 arXiv

  7. [15]

    Patrick Haller, Jonas Golde, and Alan Akbik. 2024. PECC: Problem Extraction and Coding Challenges. arXiv:2404.18766

    2024 arXiv

  8. [16]

    Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, and Chongyi Li. 2024. EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation. arXi...

    2024 arXiv

  9. [17]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv:2105.09938

    2021 arXiv

  10. [18]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8 (Dec. 2024), 220:1–220:79. https://dl...

    2024 doi

  11. [19]

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2I- CompBench: A Comprehensive Benchmark for Open-world Compositional Text- to-image Generation. arXiv:2307.06350

    2023 arXiv

  12. [20]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974

    2024 arXiv

  13. [21]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770

    2024 arXiv

  14. [22]

    Siegel, Nitya Nadgir, and Arvind Narayanan

    Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. AI Agents That Matter. arXiv:2407.01502 [cs]

    2024 arXiv

  15. [23]

    Beck LaBash, August Rosedale, Alex Reents, Lucas Negritto, and Colin Wiel. 2024. RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale. arXiv:2406.16801

    2024 arXiv

  16. [24]

    Shanchao Liang, Yiran Hu, Nan Jiang, and Lin Tan. 2024. Can Language Models Replace Programmers? REPOCOD Says ’Not Yet’. arXiv:2410.21647

    2024 arXiv

  17. [25]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. arXiv:2409.02977

    2024 arXiv

  18. [26]

    Vivian Liu, Rubaiat Habib Kazi, Li-Yi Wei, Matthew Fisher, Timothy Langlois, Seth Walker, and Lydia Chilton. 2024. LogoMotion: Visually Grounded Code Generation for Content-Aware Animation. arXiv:2405.07065

    2024 arXiv

  19. [27]

    Yongkun Liu, Jiachi Chen, Tingting Bi, John Grundy, Yanlin Wang, Jianxing Yu, Ting Chen, Yutian Tang, and Zibin Zheng. 2024. An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support. arXiv:2402.01156

    2024

  20. [28]

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203 [cs.CV] arXiv:2408.00203

    2024 arXiv

  21. [29]

    Rodriguez, Abhay Puri, Shubham Agarwal, Issam H

    Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Ro- driguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2024. StarVector: Generating Scalable Vector Graphics Code from Images and Text. arXiv:2312.11556

    2024 arXiv

  22. [30]

    Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024. Design2Code: How Far Are We From Automating Front-End Engineering? arXiv:2403.03163

    2024 arXiv

  23. [31]

    Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, and Marvin Purtorab

    George Thomas, Alex J. Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, and Marvin Purtorab. 2025. WebGames: Challenging General-Purpose Web-Browsing AI Agents. arXiv:2502.18356 [cs]

    2025 arXiv

  24. [32]

    Tiffany Tseng, Ruijia Cheng, and Jeffrey Nichols. 2024. Keyframer: Empowering Animation Design using Large Language Models. arXiv:2402.06071

    2024 arXiv

  25. [33]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2025 arXiv

  26. [34]

    Xu, and Graham Neubig

    Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023. MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages. arXiv:2203.08388

    2023 arXiv

  27. [35]

    Jingxuan Wei, Cheng Tan, Qi Chen, Gaowei Wu, Siyuan Li, Zhangyang Gao, Linzhuang Sun, Bihui Yu, and Ruifeng Guo. 2024. From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing. arXiv:2411.11916

    2024 arXiv

  28. [36]

    Simon Willison. 2025. Notes on Google’s Gemma 3. https://simonwillison.net/ 2025/Mar/12/gemma-3/

    2025

  29. [37]

    Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. 2023. IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers. ACM Trans. Graph. 42, 6 (Dec. 2023), 230:1–230:14

    2023

  30. [38]

    Ximing Xing, Juncheng Hu, Jing Zhang, Dong Xu, and Qian Yu. 2024. SVGFusion: Scalable Text-to-SVG Generation via Vector Space Diffusion. arXiv:2412.10437

    2024 arXiv

  31. [39]

    Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, and Qian Yu. 2024. SVGDreamer: Text Guided SVG Generation with Diffusion Model. arXiv:2312.16476

    2024 arXiv

  32. [40]

    Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. 2024. Diagram- merGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning. arXiv:2310.12128

    2024 arXiv

  33. [41]

    Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. 2024. HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks. arXiv:2410.12381

    2024 arXiv

  34. [42]

    Kaizhong Zhang and Dennis Shasha. 1989. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems. SIAM J. Comput. 18, 6 (Dec. 1989), 1245–1262. Publisher: Society for Industrial and Applied Mathematics

    1989

  35. [43]

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Hongyu Zhang, and Zibin Zheng. 2024. How Well Do LLMs Generate Code for Different Application Domains? Benchmark and Evaluation. arXiv:2412.18573 version: 1

    2024 arXiv

  36. [44]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al . 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854 (2023). https://webarena.dev

    2023 arXiv

  37. [45]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen- Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Y...

    2024 arXiv

  38. [2024]

    arXiv:2404.00412

    SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout. arXiv:2404.00412

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.