Symbolic Graphics Programming with Large Language Models

Haoquan Zhang; Kaipeng Zhang; Weiyang Liu; Yamei Chen; Yandong Wen; Yangyi Huang; Zeju Qiu

Symbolic Graphics Programming with Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Post-training with a cross-modal similarity reward lifts a 7B open LLM's SVG generation to frontier quality: its compositional score rises from 8.8 to 60.8 and its VQA score, 0.596, is the best of any model tested.

desk verdict Useful benchmark and a plausible RL recipe for SVG generation, but the headline 'on par with frontier' is not yet supported because the reward and evaluation share the same encoder family and no human evaluation is provided. read the letter →

arxiv 2509.05208 v2 pith:QO4FQPWQ submitted 2025-09-05 cs.CV cs.LG

Yamei Chen , Haoquan Zhang , Yangyi Huang , Zeju Qiu , Kaipeng Zhang , Yandong Wen , Weiyang Liu This is my paper · ORCID

classification cs.CVcs.LG

keywords symbolicgraphicsprogrammingSVGgenerationreinforcementlearningverifiablerewardcross-modalalignmentlargelanguagemodelscompositionalbenchmarkSGP-GenBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can do symbolic graphics programming — writing executable SVG code that renders into the image a prompt describes — and whether that ability can be trained in. To measure the skill, it builds SGP-GenBench, a benchmark testing object fidelity, scene fidelity, and compositionality (attribute binding, spatial relations, numeracy); on it, frontier proprietary models clearly beat open-weight models, and scores track general coding ability. To close the gap, the paper post-trains Qwen-2.5-7B with reinforcement learning whose only supervision is a verifiable reward: render the generated SVG, then measure how well the image matches the caption in SigLIP/CLIP embedding space and, when a reference image exists, DINO image-image similarity, gated by a format check that rejects non-renderable or text-rendering code. The result is a jump in the compositional score from 8.8 to 60.8, the best VQA score of any tested model (0.596), and behavior changes — finer object decomposition and added contextual details — that sampling from the base model would take roughly a million or more candidates to reproduce. The claim matters because, if right, it means visual knowledge can be injected into LLMs from pre-trained vision encoders alone, without expensive image-program annotation.

What carries the argument

One reward stack carries the whole argument. The total reward factorizes as r = r_fmt × (λ_text·r_text + λ_image·r_image): r_fmt is a binary format gate requiring the think-answer response structure and a successful render through CairoSVG, with SVG text-rendering tags banned to close a caption-verbatim loophole; r_text is the linear rescaling (cos+1)/2 of the cosine similarity between SigLIP text embeddings and image embeddings of the rendered SVG; r_image applies the same rescaling to DINOv2 cosine similarity between the render and a reference image when one exists. Policies are updated with GRPO, a critic-free variant of PPO used in rule-based RL. The load-bearing design move is that the

What would settle it

Prompt the trained model with captions that differ only by swapped attribute bindings ("a red cup to the left of a blue saucer" versus "a blue cup to the left of a red saucer"), render the SVGs, and check whether the high-reward outputs actually realize the binding. If the policy achieves similar similarity scores for swapped renders, the reward channel rewards semantic adjacency rather than the requested relation and the training signal is compromised; if it reliably draws the correct binding, the claim survives. A complementary check is Best-of-N sampling from the base model at N between 10^

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that symbolic graphics programming is an elicit-able skill: reinforcement learning with verifiable cross-modal rewards can turn a 7-billion-parameter open model that generates unrecognizable SVGs into one whose outputs are semantically on par with frontier proprietary systems. No ground-truth SVG programs are used; the policy learns purely from scalar similarity signals. Concretely, the authors report that RL lifts the SGP-CompBench average from 8.8 to 60.8 — going from the weakest open model to the strongest among all open models — and achieves the best VQA score across every tested model, frontier included, at 0.596. Training-dynam

Load-bearing premise

The method stands or falls on whether the cosine similarity between a rendered SVG and its caption in SigLIP/CLIP embedding space is a true, non-gameable measure of whether the drawing fulfills the prompt — the authors already had to ban text rendering to close one loophole in this channel.

Editorial extensions

If this is right

Open-weight models at 7B scale can be brought to frontier-level symbolic graphics with roughly a thousand RL steps and no ground-truth SVG programs, only captions plus pre-trained vision encoders.
The gap between open and closed models on this task is substantially closed by training, not scale: the tuned 7B model posts the best VQA score of any tested model (0.596) and outperforms all other open models on the compositional benchmark.
RL induces measurable drawing strategies — finer decomposition of objects into primitives, contextual optional details, and viewBox-cropping — so trained behavior goes beyond literal prompt fulfillment.
Best-of-N analysis implies that matching RL's gains by sampling more from the base model would require on the order of a million to a hundred million candidates, far beyond practical compute.
Benchmark ordering tracks general coding ability, so the authors argue SGP generation can serve as a diagnostic lens on cross-modal grounding and on how LLMs map language to structured visual programs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the global similarity reward is the binding constraint, then replacing or augmenting it with per-object or relation-checking rewards (for example, a VQA-based adjudicator) should push compositional scores further; the paper's fixed reward stack leaves this untested.
Editorial inference: the same loop could distill other visual priors into symbolic programs by swapping the reward encoder — say, a diffusion model's scorer — turning the paper's 'implicit distillation' framing into a general method for transferring visual knowledge into LLMs.
Editorial inference: the paper's own CLIP-vs-SigLIP color comparison shows that reward-encoder preferences are silently baked into the policy's style, so deployment of RL-tuned drawing models should include an audit of stylistic bias inherited from the chosen encoder.
Editorial inference: the learned viewBox-cropping behavior reveals that token-efficiency pressures shape drawing strategy; metrics that count only rendered pixels, alongside the reported program-level statistics, would capture this more directly.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a real contribution—a benchmark that tests SVG generation at scene and compositional level, plus an RL post-training recipe that clearly moves Qwen-2.5-7B. But the paper's central claim, that the RL model reaches frontier quality, is stronger than the evidence. The training reward (SigLIP cosine) and the main evaluation metrics (CLIP-Score, DINO-Score) are from the same family, so part of the reported gain is expected even if the model just learns to exploit encoder biases.

What it does well: SGP-GenBench is a useful extension of prior work. It covers object fidelity, scene fidelity, and compositionality (attribute binding, spatial relations, numeracy) with 3200 prompts. The systematic comparison of frontier and open models is informative, and the finding that SVG generation correlates with coding ability is plausible. The RL method is a sensible adaptation of RLVR: format gate plus cross-modal rewards. The ablations are thorough (embedding models, CoT, data mixture, GRPO vs PPO), and the training-dynamics analysis—decomposition into primitives, emergence of optional details, viewBox cropping—is genuinely interesting. The authors also took care to ban text rendering, which they found could game the reward; that is honest.

Soft spots, in proportion: (1) The reward-evaluation overlap is the biggest issue. The final model is trained with SigLIP text-image cosine (Eq. 4.7) and evaluated with CLIP-Score and DINO-Score. CLIP and SigLIP are closely related, and DINO was an optional part of the reward (though the final stack seems to use only SigLIP, which the paper should state clearly—the abstract and method suggest both). The documented text-rendering hack and the viewBox cropping show the reward is gameable in principle. So the headline gains on CLIP/DINO may be inflated. (2) The compositional scores come from an unvalidated Gemini judge. No human agreement or validation of the judge itself. (3) No human evaluation of semantic fidelity anywhere. For a task about whether the SVGs actually match captions, that is a notable omission. (4) No code, data, or checkpoints released; reproducibility is limited. (5) The "on par with frontier" claim is not supported across all metrics: VQA is best, but CLIP, DINO, and HPS are below Claude 3.7, and the compositional average (60.8 vs 84.8) is far behind.

None of these is fatal alone, but together they mean the paper's headline is ahead of its evidence. The method is sound in principle and the benchmark is useful. I'd send it to review, but with a request for artifacts, a human evaluation (even small), an independent judge or careful validation, and a more measured claim.

Recommendation: engage with it, treat the results as promising but conditional. For your reading group it's worth a discussion—especially the reward-gaming angle.

Referee Report

4 major / 5 minor

Summary. The paper studies symbolic graphics programming: generating SVG code from natural-language captions. It introduces SGP-GenBench, with three evaluation components (COCO-val scenes, SGP-Object-val objects, SGP-CompBench composition) totaling about 5,100 prompts, and reports that frontier closed-source models outperform open-source models. It then proposes an RLVR-style post-training recipe (GRPO) in which the reward is a format-validity gate plus a rescaled SigLIP text-image cosine (Eq. 4.7), optionally augmented with a DINO image-image cosine (Eq. 4.9). Applied to Qwen-2.5-7B, the method reportedly raises SGP-CompBench average from 8.8 to 60.8 and achieves the best VQA score in Table 1 (0.596). The paper also analyzes training dynamics, including object decomposition, optional contextual details, viewBox-clipping behavior, and Best-of-N scaling.

Significance. If the reported gains are genuine, the paper makes a useful contribution: it provides a sizable benchmark for SVG generation, demonstrates a simple reward pipeline that improves a 7B open model substantially, and offers an interesting Best-of-N analysis suggesting RL is not trivially replaceable by decoding-time scaling. The training-dynamics analysis (object decomposition, optional contextual elements, and color-choice differences between CLIP and SigLIP rewards) is also informative. The main caveat is that the headline evaluation relies heavily on automated metrics that overlap with the reward family, and the compositional scores come from an unvalidated commercial judge; without independent verification, the central 'on par with frontier systems' claim remains plausible but not established.

major comments (4)

[§5.2.1 / Eq. (4.7)] The reward used for RL is a rescaled SigLIP text-image cosine (Eq. 4.7), and the headline fidelity metric CLIP-Score is the same family of contrastive text-image similarity. The final model is trained with SigLIP Base/16-384 (§5.3.1), while Table 1 reports CLIP-Score averaged over two CLIP models. Thus a substantial part of the reported CLIP increase is expected even if the model only learns to exploit encoder-specific statistics rather than to draw semantically better. VQA-Score and HPS are less directly aligned with the reward, but VQA-Score itself depends on an unspecified VLM question/answer pipeline (§B.2). I recommend reporting a truly external semantic fidelity metric (e.g., human ratings on a held-out sample, or a VLM judge not in the contrastive-reward family), and reporting results separately for reward-family and non-reward-family metrics.
[§B.1.3, Tables 2 and 8] The entire SGP-CompBench evaluation is produced by a single model, Gemini-2.5-Flash-Preview, with no validation against human judgments, no confidence intervals, and no analysis of judge agreement. The headline claim that the RL model improves compositional scores from 8.8 to 60.8 rests on this unvalidated judge. Because the judge is a commercial model whose behavior can change, and because the scoring rubrics (100/50/30/0) are coarse, I cannot assess whether the compositional improvements are real. I recommend validating the judge on a human-annotated subset (or using a fixed, openly available judge with published agreement statistics) and reporting per-item errors.
[§D.3 and §E.4] The paper itself provides evidence that the SigLIP/CLIP reward family is gameable: it had to ban <text>, <tspan>, and <textPath> tags because rendering the caption verbatim inflated rewards (§D.3). Appendix E.4 then shows the trained model exploiting viewBox clipping in a non-obvious way. The viewBox behavior is not necessarily a flaw, but together these examples show that the policy can find reward-maximizing shortcuts that are not the intended drawing semantics. This strengthens the need for independent evaluation and for an explicit analysis of remaining reward-hacking strategies, rather than relying on automated similarity metrics alone.
[Reproducibility / artifacts] The paper does not provide code, data, or model checkpoints; the project name 'SphereLab.ai/SGP-Gen' appears without a usable URL. For a benchmark paper and an RL recipe, reproducibility is load-bearing: the claims cannot be checked, and the exact evaluation prompts, judge versions, and generated SVGs are not inspectable. I recommend releasing the evaluation harness, the training pipeline, the generated outputs at the reported checkpoints, and (where licensing permits) the benchmark data.

minor comments (5)

[§B.2] The VQA-Score description is underspecified: which VLM is used, how many questions are generated per image, and who creates the question-answer pairs? This should be documented to make the metric reproducible.
[§5.3.1 / Table 3] The final reward stack is described as 'SigLIP Base/16-384' but it is not fully clear whether lambda_image is zero in the final model or whether a DINO term is also used. Please state the exact final reward coefficients.
[§6.1 / Figure 6] The claim that Best-of-N would need 10^6–10^8 samples is based on a linear extrapolation of curves that appear to be measured only for N up to about 10^2. The out-of-range extrapolation should be clearly labeled as an assumption, not a measured result.
[Table 1] Some entries in Table 1 are run together without spacing (e.g., '0.2620.3050.284'), which makes the table hard to read. Please format the table for clarity.
[§A.2] The sentence 'This methodical approach allowed us to comprehensively evaluate the model’s ability to handle different quantities.to' contains a stray 'to'. Please fix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RL reward (SigLIP cosine) and the headline evaluation metrics (CLIP-Score, DINO-Score, VQA-Score, HPS v2) are distinct; the central claim does not reduce to the training objective.

full rationale

The paper's derivation chain is: (i) define a rule-based RL objective with reward r(s,c,x) = r_fmt(s) * (λ_text r_Text(s,c) + λ_image r_Image(s,x)) (Eq. 4.4); (ii) r_Text is a linearly rescaled SigLIP cosine similarity between caption and rendered image (Eqs. 4.6–4.7); (iii) r_Image is a DINO cosine similarity when a reference image is available (Eqs. 4.8–4.9); (iv) the final model is trained with SigLIP Base/16-384 only, as stated in Sec. 5.3.1: 'we report all final results using a fixed reward stack of SigLIP Base/16-384'. The evaluation metrics are CLIP-Score (averaged over CLIP ViT-B/32 and ViT-L/14), DINO-Score (averaged over four DINOv2 variants), VQA-Score, and HPS v2 (Sec. 3.2, Sec. B.2). Although CLIP-Score and the SigLIP reward are both cross-modal cosine similarities in contrastive embedding spaces, they use different pre-trained encoders; optimizing SigLIP does not by construction maximize CLIP-Score. DINO-Score is computed with DINOv2 models that were not part of the final reward stack, and VQA-Score uses an LLM-generated QA set answered by a VLM, which is independent of the SigLIP reward. HPS v2 is a learned human-preference predictor, also outside the reward stack. Thus the central claims—'substantially improves SVG generation quality and semantics' and 'performance on par with frontier systems'—are supported by metrics that are not merely the training objective renamed. The only self-citation with overlapping authors is [QLF+25] in the Introduction ('[QLF+25, ZCZL24a] have shown that LLMs possess semantic understanding of SGPs'), used as background motivation, not as the load-bearing argument for the RL improvement. The documented text-rendering ban (App. D.3) and the observed viewBox clipping behavior (App. E.4) indicate the perceptual reward is gameable in principle, and the paper closes one exploit and reports it; this is a reward-validity concern, not a circular derivation. No equation in the paper is shown to be equivalent to another by construction, and no fitted parameter is renamed as a prediction. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim leans on the validity of embedding-similarity rewards, the reliability of a VLM judge, and the sufficiency of the COCO/MMSVG caption mixture. No fitted free parameters are involved; the listed hyperparameters are hand-set. No invented entities are introduced; SGP-GenBench is a dataset, not an entity.

free parameters (3)

Numeracy score weights (Total 0.2, Item 0.2, CPI 0.6) = 0.2/0.2/0.6
Hand-set in Section 3.2; changes aggregate numeracy ranking and is used to report the headline 60.8 average.
Reward coefficients lambda_text / lambda_image = 1.0 / 1.0 (or 0 when no reference image)
Hand-set in Section 4.2; no tuning reported; affects reward balance and all reported results.
Asymmetric GRPO clip bounds = clip_low=0.20, clip_high=0.28
Chosen in Appendix D.2 following DAPO to prevent entropy collapse; a training hyperparameter, not fitted to evaluation.

assumptions (5)

domain assumption Rendering via CairoSVG is deterministic and captures visual content faithfully.
Section 4.2.1 uses renderer success as the format-validity gate; if rendering were not faithful, the reward images would not reflect the SVG semantics.
ad hoc to paper SigLIP/CLIP cross-modal cosine is a valid continuous reward for semantic correctness of a drawing.
Eq. 4.6-4.7 define the text-image reward; the entire RL pipeline depends on this similarity being a trustworthy proxy for drawing quality.
ad hoc to paper DINOv2 cosine similarity between rendered image and reference image is a valid fidelity reward.
Eq. 4.8-4.9 define the image-image reward; used when reference images are available, and in ablations.
domain assumption Gemini-2.5-Flash-Preview as judge reliably scores compositionality in place of human ratings.
Appendix B.1.3 uses this model for all SGP-CompBench scores; no human calibration or inter-annotator agreement is reported.
domain assumption Pretrained LLMs already have SVG priors that RL can elicit without ground-truth program annotations.
Section 4.3 argues RL can distill visual knowledge into the LLM; the method provides no ground-truth SVG targets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Symbolic Graphics Programming with Large Language Models." pith.science (2026). https://pith.science/paper/QO4FQPWQ

@misc{pith2026250905208,
  author       = {Pith},
  title        = {Pith review of: Symbolic Graphics Programming with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QO4FQPWQ}},
  note         = {Machine review of arXiv:2509.05208}
}

read the original abstract

Large language models (LLMs) excel at program synthesis, yet their ability to produce symbolic graphics programs (SGPs) that render into precise visual content remains underexplored. We study symbolic graphics programming, where the goal is to generate an SGP from a natural-language description. This task also serves as a lens into how LLMs understand the visual world by prompting them to generate images rendered from SGPs. Among various SGPs, our paper sticks to scalable vector graphics (SVGs). We begin by examining the extent to which LLMs can generate SGPs. To this end, we introduce SGP-GenBench, a comprehensive benchmark covering object fidelity, scene fidelity, and compositionality (attribute binding, spatial relations, numeracy). On SGP-GenBench, we discover that frontier proprietary models substantially outperform open-source models, and performance correlates well with general coding capabilities. Motivated by this gap, we aim to improve LLMs' ability to generate SGPs. We propose a reinforcement learning (RL) with verifiable rewards approach, where a format-validity gate ensures renderable SVG, and a cross-modal reward aligns text and the rendered image via strong vision encoders (e.g., SigLIP for text-image and DINO for image-image). Applied to Qwen-2.5-7B, our method substantially improves SVG generation quality and semantics, achieving performance on par with frontier systems. We further analyze training dynamics, showing that RL induces (i) finer decomposition of objects into controllable primitives and (ii) contextual details that improve scene coherence. Our results demonstrate that symbolic graphics programming offers a precise and interpretable lens on cross-modal grounding.