Vector-Bench: Can Models Surgically Edit SVG Code?

Prannay Hebbar; Yug Aditi Gupta

REVIEW 2 major objections 5 minor 8 references

Instruction-based SVG editing is failing the preservation half of the contract: the best of 34 model endpoints passes all three gates on only 15.0% of tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:34 UTC pith:JBOJU43J

load-bearing objection Vector-Bench is a sound, honestly reported benchmark: the three-gate reward is a real measurement contribution, and the repair-vs-specification gap is robust even if the absolute pass rates depend on author-set thresholds. the 2 major comments →

arxiv 2607.19056 v1 pith:JBOJU43J submitted 2026-07-21 cs.AI

Vector-Bench: Can Models Surgically Edit SVG Code?

Yug Aditi Gupta , Prannay Hebbar This is my paper

classification cs.AI

keywords SVG editinginstruction followingbenchmarkevaluationprogram preservationLLM capabilitiesvector graphicsspecification reward

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Instruction-based SVG editing demands two things at once: apply the requested visible change, and leave every unmentioned part of the program unchanged and valid. This paper builds a 40-task benchmark that makes both requirements measurable, using a deterministic three-gate reward: requested repairs must land within attribute-aware perceptual tolerances, the masked document must remain semantically identical, and the output must be valid SVG. Across 34 model endpoints and 1,360 attempts, the strongest endpoint passes all three gates on only 15.0% of tasks, despite 43.7% mean repair progress; overall only 2.35% of outcomes pass. The paper argues that visible repair progress and specification-faithful editing are different capabilities, and that the shortfall is broad across model classes, not a quirk of one endpoint. It releases the corpus, scoring code, per-task reports, and all model outputs so the gap can be inspected at the object and source level.

Core claim

The central discovery is that current large language models, when asked to repair a corrupted SVG from a natural-language instruction, routinely perform recognizable requested edits yet violate the implicit contract that everything else must stay intact. The paper demonstrates this with a benchmark where every task includes a hidden target program, explicit protected objects, and a binary reward that requires all requested edits to pass perceptual tolerances, the unrequested semantic structure to remain unchanged, and the output to be valid SVG. On 1,360 scored outcomes, only 32 satisfy the full specification, while roughly half make at least one annotated repair and validity-gated repair pr

What carries the argument

The load-bearing artifact is the deterministic binary specification reward, evaluated as valid(S) AND all requested edits ok_e(S) AND V(masked(S)) = V(masked(S*)). The three gates play distinct roles: requested-edit checks use attribute-aware tolerances (CIE Lab ΔE76 for color, 129-point sampled path distance with RMS/95th-percentile thresholds, corruption-relative bounds capped by viewport-relative limits); the preservation gate uses a semantic normalizer V(·) that permits only representation-level variation such as consistent ID renaming or equivalent style storage while holding element order, geometry, paint, text, and references strict; and the validity gate requires well-formed XML, uni

Load-bearing premise

The evaluator's hand-set perceptual tolerances and the semantic normalizer's definition of 'application-relevant structure' are assumed to match what a human editor accepts; the paper reports in §4.2 and §8 that no human agreement study was run and that CIE76 and finite path sampling are approximations, so the absolute rates are calibration-dependent even though the qualitative direction survives tolerance rescaling.

What would settle it

Conduct a human-agreement study on the 40 tasks: have human editors judge each released model output against the instruction for repair correctness, preservation of unmentioned content, and validity, then compare their verdicts to the binary reward. A systematic mismatch (for example, human-accepted repairs receiving zero reward) would recalibrate the central numbers; a strong match would confirm the gap is real. Alternatively, an intervention that adds a raster preview to the prompt and moves full-spec success substantially above 15% would indicate the gap is partly protocol-bound rather than

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the result is correct, repair-progress metrics overstate real editing capability for structured artifacts; benchmarks for editing agents need preservation and validity gates to be informative.
Editing agents must treat preservation and validity as hard constraints: a model that cannot leave unmentioned objects alone will fail real asset-editing workflows even when its raster output looks plausible.
The shortfall is not confined to cheap models: both inexpensive and frontier endpoints cluster near low full-spec success, with a large fraction of outcomes truncated or structurally invalid, pointing to output-length and validity as limiting factors.
The released corpus and evaluator provide a reproducible target for tracking whether future editing models improve on the combined repair-and-preserve contract, rather than only on visible repair completion.
The binary rate is calibration-dependent, but the qualitative gap survives rescaled tolerances, so the main conclusion—partial repair is common, full specification-faithful repair is rare—is robust to moderate threshold changes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's own caution, the most consequential open question is human calibration: if a panel of human editors accepts outputs the evaluator rejects, the tolerance constants would need adjustment; the authors report in §4.2 and §8 that no human agreement study was run and that CIE76 and finite path sampling are approximate.
The preservation gate's semantic equivalence could be decomposed further—separating element reordering, path rewrites, and deleted definitions as failure subtypes—which would tell developers where to invest (e.g., localization vs. restraint vs. validity).
The same three-gate construction transfers naturally to other structured program-editing tasks (JSON config repair, code refactoring, CSS fixes) where 'make this change, break nothing else' is the core contract; the masking and semantic-normalization design is a reusable template.
A testable extension is to give models a raster preview or interactive editor alongside the SVG source; the benchmark is text-and-source only, and the authors note multimodal interaction is out of scope, so a modality intervention could reveal how much of the gap is protocol-bound.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Vector-Bench is a sound, honestly reported benchmark: the three-gate reward is a real measurement contribution, and the repair-vs-specification gap is robust even if the absolute pass rates depend on author-set thresholds.

read the letter

Short version: this is a good benchmark paper and deserves refereeing. The new thing is the three-gate deterministic reward — requested edits, semantic preservation, validity — applied to dense multi-repair SVG editing with hidden targets. That fills a real gap: most prior SVG editing benchmarks score raster similarity or loose edit success and miss preservation. The corpus (40 tasks, 202 repairs, ~60 protected objects each) is small but dense, and the published traces let you inspect exactly why each outcome passes or fails.

What I like: the evaluator is executable, deterministic, and audited with four controls; the tolerance model is transparent and bounded by corruption distance so the known defect can't pass; and the sensitivity analysis (Table 4) shows the qualitative result — 2.35% overall pass, 15.0% best endpoint, vs 43.7% repair progress — survives 0.5×–1.5× rescaling. The paper also ships code, outputs, and per-task reports. The absence of a human-agreement study and human baseline is disclosed in §8, not hidden. The tolerance constants were not fitted to make models pass or fail, so the central claim is not circular.

Where I'd push back, mildly: the stress-test concern is real but it's a calibration concern, not a refutation. The preservation gate V(·) is strict — element order, style-storage location, geometry — so a model that does the visually right repair but reorders non-overlapping elements or refactors styles gets zero. That makes the absolute success rates spec-relative. The paper says this, which is good, but the framing in the abstract ("15.0% full specification success") could be read as a model capability number when it's really success under this particular executable contract. The single-sample-per-model-task design is fine for a first pass but the Wilson intervals only capture task sampling, not decoding variance; the paper says this too.

Two smaller things: the provenance issue for the 20 scenic source files is a real artifact risk and correctly flagged; and the operation-family analysis is useful but under-powered with 40 tasks.

Bottom line: the central empirical gap — models do visible repairs while failing repair-and-preserve — is convincingly supported. The absolute rates will shift with human calibration, but the direction will not. I'd send this to a serious referee, mainly to pressure-test the evaluator design and the claims about semantic preservation. I'd bring it to a reading group, and I'd cite it if I were working on vector editing or program-synthesis evaluation.

Referee Report

2 major / 5 minor

Summary. The paper introduces Vector-Bench, a benchmark of 40 SVG repair tasks. Each task provides a corrupted SVG, an author-written instruction, a hidden target SVG, and a target/anchor-based deterministic specification reward. The reward requires (i) requested edits to pass attribute-aware perceptual tolerances, (ii) unrequested rendering- and application-relevant structure to remain semantically unchanged under a canonical semantic form V(·), and (iii) the output to be a valid SVG. The authors evaluate 34 model endpoints over 1,360 requests and report that the best endpoint (Claude Sonnet 5) achieves only 15.0% full specification success, while its mean repair progress is 43.7%. They also report aggregate outcomes: 2.35% full specification success, 49.9% with at least one repair, and 27.5% validity-gated repair progress overall. The paper interprets this gap as evidence that repair-progress metrics substantially overstate models' surgical editing capability. The release includes prompts, outputs, evaluator code, per-task reports, and deterministic controls.

Significance. If the evaluator is accepted as a reasonable operationalization of 'surgical editing,' Vector-Bench is a valuable, transparent stress test for instruction-based SVG editing. The strengths are substantial: the evaluator is deterministic and does not rely on a learned judge; four controls verify its behavior; the paper publishes all model outputs and scoring code, supporting exact reruns; and Table 4 shows the qualitative conclusion is robust to rescaling of requested-edit distance tolerances. The finding that even frontier endpoints rarely satisfy the full repair-and-preserve contract, while making partial visible repairs, is a measurable and practically important result for LLM-based code editing evaluation. The paper is appropriately cautious in framing binary reward as success under the released executable specification, but the broader conclusion about model editing capability rests on the perceptual validity of that specification.

major comments (2)

[Sec. 4.1, Sec. 8, Table 4] The central claim—that repair progress (43.7%) far exceeds specification-faithful success (15.0%)—depends on the semantic preservation gate V(M_E(S)) = V(M_E(S*)). The authors concede (§8) that no human agreement study was run and that the comparator does not model browser layout, CSS, animation, or accessibility. Table 4 perturbs only the distance-based requested-edit tolerances; it never varies the preservation equivalence (element-order sensitivity, strict geometry/paint equality, style-storage merging, or the treatment of consistent ID renaming). If V(·) is stricter than a typical human editor, the reported success rates are underestimates; if more lenient, overestimates. Because the gap magnitude is the headline result, this is load-bearing. I recommend adding (a) a small human-annotation calibration on a subset of tasks, or (b) a sensitivity analysis that varies V(·) while keeping
[Sec. 6, Table 3, Sec. 5.2] The 'strongest endpoint' claim in the abstract (15.0% for Claude Sonnet 5 vs 12.5% for the next endpoint) is based on a difference of one task out of 40. With one scored outcome per model-task pair and overlapping Wilson intervals, the ordering of the top endpoints is not statistically robust. The broader conclusion that all endpoints perform poorly is well supported, but the precision attached to 'the strongest endpoint' overstates the certainty. Please soften the language or add a significance test for the top-endpoint comparison.

minor comments (5)

[Sec. 4.1] The term 'application-relevant structure' for V(S) is not formally defined. What exactly does 'application-relevant' include beyond rendering (e.g., ID references, script hooks, aria attributes)? Please clarify the algorithmic scope of the comparator.
[Table 4 caption] The caption says 'Scaling applies only to distance-based requested-edit tolerances.' Please clarify whether color ΔE76 and opacity absolute-error distances are considered 'distance-based' and thus scaled. If not, list exactly which checks are scaled and which are not.
[Sec. 5.2] The Wilson intervals are described under a Bernoulli task-sampling interpretation. It would help to state the effective sample size for the top endpoints (40 tasks) and to note that, for models with 0 passes, the interval is one-sided in practice.
[Sec. 6] The sentence 'Valid-output UCR is 2.4% overall' is ambiguous because UCR is conditional on valid outputs per model. Please specify whether this is a macro-average across models, a pooled average, or something else.
[Appendix A] The 'legacy_final_record' labeling is honest about missing retry envelopes, but it means the exact behavior of the model in the presence of transient failures is not fully reconstructable for the reported study. Please state how many tasks were affected by retries in the legacy run, if known.

Circularity Check

0 steps flagged

No significant circularity: the benchmark result is an executable measurement with hidden targets and unfitted tolerances.

full rationale

Vector-Bench is an evaluation artifact rather than a derivation. The central result—that repair progress (43.7% for Claude Sonnet 5) exceeds full specification success (15.0%)—is an empirical measurement under a published deterministic reward R_spec(S) = 1[valid(S)] · 1[∀e∈E: ok_e(S)] · 1[V(M_E(S)) = V(M_E(S*))]. The tolerance map t_e = min(c_e, max(f_e, r_e·δ_e), 0.9δ_e) uses author-set constants, but they were not fitted to any endpoint's outputs, and §6 Table 4 shows the qualitative conclusion survives 0.5×–1.5× rescaling, so the headline is not forced by calibration. The hidden target S* and expected repairs E are withheld from models (§3.1), so success is not constructed from model responses. No load-bearing self-citation appears; the cited prior work is external and contextual. The closest concern is external validity: §8 explicitly states 'no independent annotator-agreement or human-performance study was run' and 'binary reward should be interpreted as success under this released executable specification, not universal perceptual approval.' That is a calibration/limitation issue, not circularity, and the paper discloses it while providing deterministic controls (Table 2) and sensitivity analysis. No step in the paper's chain reduces by definition or by self-citation to its own inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The reward's absolute pass rates are controlled by hand-chosen tolerance constants (three free-parameter groups, §4.1) that are not human-calibrated. The benchmark's validity rests on five assumptions listed above: target-relative correctness, the adequacy of the purpose-built V(·) normalizer, CIE76/path-sampling as perceptual proxies, single-sample endpoint estimation, and OpenRouter as a fair test harness. No invented entities are postulated; the new objects — the corpus, the three-gate reward, the C(·)/V(·) normalizers — are measurement artifacts with released code.

free parameters (3)

Requested-edit tolerance scale factors r_e = 0.50 (scalar), 0.45 (color), 0.55 (point lists), 0.70 (paths)
§4.1 defines t_e = min(c_e, max(f_e, r_e·δ_e), 0.9·δ_e). These hand-chosen fractions interpolate between corruption distance and fixed caps. Table 4 shows rescaling them by 0.5×–1.5× moves aggregate full-spec pass from 0.81% to 3.38%, so the exact headline rates are calibration-dependent.
Per-check caps c_e and floor f_e = position 2.5% of viewport; color 18 ΔE76; opacity 0.12; path/point 3% of smaller viewport dimension; stroke width 1 user
§4.1 hand-set bounds that the acceptance windows sit under; no human agreement study validates them (§4.2, §8). The actual f_e floor values are not printed in the paper.
Path sampling density and distance statistic = 129 length-normalized samples; RMS + 95th percentile symmetric nearest-point distance
§4.1; finite sampling "can miss narrow or highly localized differences" (§8). The 129 samples and the 95th percentile are implementation choices that partly define which path shapes count as repaired.

axioms (5)

domain assumption The hidden target S* is the canonical correct output for each task; correctness of a repair is equivalence-within-tolerance to S*.
§3.1: τ = (x, S0, S*, E, U) with S* hidden. The benchmark's notion of a correct edit is entirely target-relative; if a target encodes an idiosyncratic author choice (e.g., a specific door position), a perceptually valid alternative gets zero reward.
ad hoc to paper The semantic form V(S) captures all rendering- and application-relevant structure; anything V ignores is out of scope for preservation.
§4.1: V merges consistent ID renames and style-storage rewrites but keeps element order, geometry, paint, unresolved references strict. §8 concedes: "The semantic comparator does not execute arbitrary CSS, browser layout, animation, scripting, accessibility, or editor-specific behavior." The V(·) equivalence relation is purpose-built for this paper.
domain assumption CIE Lab ΔE76 and sampled geometric distances are adequate perceptual proxies for color and shape equality.
§4.1 uses CIE76 and finite path sampling; §8 concedes "CIE76 is only an approximate color metric" and path comparison "can miss narrow or highly localized differences"; no human agreement study was run (§4.2).
domain assumption A single decoded completion per model–task pair suffices to rank endpoints; Bernoulli task-sampling underlies the intervals.
§5.2: "each model–task pair has one scored outcome"; Wilson intervals "quantify uncertainty under a Bernoulli task-sampling interpretation only" and do not estimate decoding variance.
domain assumption Evaluating via OpenRouter endpoints with provider-default temperature is a fair proxy for endpoint capability.
§5.1: temperature omitted rather than imposing a provider-incompatible value, retries only against the same endpoint, truncation counted as failure; OpenRouter pricing/routing/version drift is acknowledged as an audit caveat (§8).

pith-pipeline@v1.3.0-alltime-deepseek · 10769 in / 19435 out tokens · 184921 ms · 2026-08-01T13:34:28.288092+00:00 · methodology

0 comments

read the original abstract

Instruction-based vector editing requires two capabilities: making a requested change and leaving everything else alone. The second is easy to miss when an output is judged only as a raster image. We introduce Vector-Bench, a compact, difficult benchmark of 40 SVG repair tasks. Each task pairs a corrupted SVG program with an author-written visual instruction, a hidden target program, 5.05 annotated repairs on average, and an average of 60.55 protected objects. Instructions describe visible defects without exposing element identifiers, coordinates, color codes, or path data. We define a deterministic binary specification reward: requested repairs use attribute-aware perceptual tolerances, while unrequested rendering- or application-relevant structure must remain semantically unchanged and the result must be a valid SVG. Canonical target equality and stricter source fidelity are retained as diagnostics. Validity-gated repair progress, a near-complete tier, and valid-output Unintended Change Rate (UCR) explain partial outcomes. We evaluate 34 model endpoints (25 listed as open-weight, 5 inexpensive controls, and 4 frontier closed endpoints) over 1360 requests. The strongest endpoint reaches only 15.0% full specification success, despite 43.7% mean repair progress, showing that apparent repair progress and specification-faithful editing remain substantially different. All prompts, outputs, scoring code, costs, and per-task reports are released.

Figures

Figures reproduced from arXiv: 2607.19056 by Prannay Hebbar, Yug Aditi Gupta.

**Figure 1.** Figure 1: Representative corrupted inputs and hidden targets. Public prompts describe visible mistakes in ordinary [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Repair operations and per-task edit density. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: SVG size and preservation surface. Marker [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: Binary specification success for all endpoints. Orange bars are inexpensive controls, teal bars are endpoints [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Mutually exclusive decomposition of all 1360 outcomes. The first two rows distinguish clean specification [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Complementary model-level diagnostics. In (a), marker area scales with specification pass rate and lower [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Mean expected-edit repair progress by operation family. Rows follow overall specification-pass rank. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

**Figure 8.** Figure 8: One scored trace, end to end. Top: corrupted input, model output, hidden target. Bottom: the three-gate [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 5 linked inside Pith

[1]

2404.13710 , archivePrefix=

Kunato Nishina and Yusuke Matsui , year =. 2404.13710 , archivePrefix=

Pith/arXiv arXiv
[2]

2502.19453 , archivePrefix=

Kunato Nishina and Yusuke Matsui , year =. 2502.19453 , archivePrefix=

Pith/arXiv arXiv
[3]

2025 , eprint =

Josef Kucha. 2025 , eprint =

2025
[4]

2304.14400 , archivePrefix=

Ronghuan Wu and Wanchao Su and Kede Ma and Jing Liao , year =. 2304.14400 , archivePrefix=

Pith/arXiv arXiv
[5]

Empowering

Ximing Xing and Juncheng Hu and Guotao Liang and Jing Zhang and Dong Xu and Qian Yu , year =. Empowering. 2412.11102 , archivePrefix=

Pith/arXiv arXiv
[6]

Rodriguez and Abhay Puri and Shubham Agarwal and Issam H

Juan A. Rodriguez and Abhay Puri and Shubham Agarwal and Issam H. Laradji and Pau Rodriguez and Sai Rajeswar and David Vazquez and Christopher Pal and Marco Pedersoli , year =. 2312.11556 , archivePrefix=

Pith/arXiv arXiv
[7]

API Reference and Model Catalog , year =
[8]

2018 , url =

Scalable Vector Graphics (. 2018 , url =

2018

[1] [1]

2404.13710 , archivePrefix=

Kunato Nishina and Yusuke Matsui , year =. 2404.13710 , archivePrefix=

Pith/arXiv arXiv

[2] [2]

2502.19453 , archivePrefix=

Kunato Nishina and Yusuke Matsui , year =. 2502.19453 , archivePrefix=

Pith/arXiv arXiv

[3] [3]

2025 , eprint =

Josef Kucha. 2025 , eprint =

2025

[4] [4]

2304.14400 , archivePrefix=

Ronghuan Wu and Wanchao Su and Kede Ma and Jing Liao , year =. 2304.14400 , archivePrefix=

Pith/arXiv arXiv

[5] [5]

Empowering

Ximing Xing and Juncheng Hu and Guotao Liang and Jing Zhang and Dong Xu and Qian Yu , year =. Empowering. 2412.11102 , archivePrefix=

Pith/arXiv arXiv

[6] [6]

Rodriguez and Abhay Puri and Shubham Agarwal and Issam H

Juan A. Rodriguez and Abhay Puri and Shubham Agarwal and Issam H. Laradji and Pau Rodriguez and Sai Rajeswar and David Vazquez and Christopher Pal and Marco Pedersoli , year =. 2312.11556 , archivePrefix=

Pith/arXiv arXiv

[7] [7]

API Reference and Model Catalog , year =

[8] [8]

2018 , url =

Scalable Vector Graphics (. 2018 , url =

2018