{"id":"6d0185ae-5ae1-433d-9a92-05e08cc13c40","arxiv_id":"2505.04670","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"vTikZ, a 100-task benchmark for visual code customization, shows LLMs solve at most 28 percent of TikZ editing scenarios even with best-of-five sampling.","lead":"This paper introduces vTikZ, a benchmark with 100 curated TikZ editing tasks that tests whether large language models can modify code to produce a requested visual change. It finds that current models succeed on at most 28 percent of tasks even with five attempts, indicating that visual code customization remains largely unsolved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth coverage is incomplete: the 13 human-discovered correct solutions were not folded into the parameterized references, so the reported 13%/28% success rates likely understate LLM performance.","rationale":"The paper's central contribution is a benchmark whose absolute scores depend on the completeness of its ground truths. The reader's weakest assumption identifies exactly this point, and the manuscript itself supplies concrete evidence of incompleteness: Section 4.4 reports that human annotation surfaced 13 previously missed correct solutions, while Section 6.3 admits the parameterization commands may not cover all valid edits. Section 8 further clarifies that the reported results come from version 1.0 and that incorporating missing correct solutions is only a planned update. Thus the published 13% and 28% figures are lower bounds, with an unknown but potentially material upward correction. This is a load-bearing concern because the headline claim is a quantitative statement about LLM capability, and the benchmark's utility as a measurement instrument depends on the metric recognizing correct outputs. The qualitative conclusion that LLMs are not yet reliable at TikZ customization would likely survive even a correction, but the specific success rates and the fairness of the benchmark need qualification. The proposed recomputation is feasible because the dataset, code, and annotation data are released. This concern does not change the reader's CONDITIONAL verdict; it reinforces it. No other concern examined (model selection, temperature settings, single-annotator curation, or scope limits) is as directly tied to the validity of the central numerical claim.","tokens_in":14809,"tokens_out":5326,"duration_ms":56103,"concrete_test":"Use the released vTikZ dataset and evaluation code (github.com/IV2C/VTikZ; huggingface.co/datasets/CharlyR/vtikz). Add the 13 human-annotated correct solutions from the annotation data (Section 4.4) into the ground-truth set for their corresponding tasks, either as additional parameterized variants or as exact-image reference solutions. Re-run the Section 5.1 evaluation pipeline for all model configurations in Table 3 (including GPT-4o at N=5). Compare the recomputed SuccessCustomizationMetric against the published 13% and 28% figures. If the corrected best-of-5 rate exceeds the published rate by more than about 10 points, the paper's absolute success rates require revision and the benchmark's ground-truth coverage must be reported as a known limitation rather than a planned update.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 8: 13% one-shot, 28% at 5 tries) depends on SuccessCustomizationMetric correctly recognizing all valid outputs. The authors concede in Section 6.3 that the four parameterization commands 'may not cover cases where certain lines are optional or line placement does not affect correctness,' and Section 4.4 reports that human annotation 'surfaced 13 additional correct solutions that were previously missed.' Section 8 states the results rely on version 1.0, with 'potentially incorporating missing correct solutions' listed as a planned change. This means the version 1.0 ground truths omit at least 13 valid solutions that the metric would have marked wrong. Since the final evaluation re-uses the same metric, the reported success rates are lower bounds. The direction is clear: if even one of those 13 solutions corresponds to a model-generated variant in the evaluation, that task flips from failure to success. The magnitude is unknown but could be material: 13 additional successful tasks out of 100 could raise the best-of-5 figure from 28% to as high as 41%. The qualitative conclusion that LLMs are unreliable would survive, but the headline numbers and the benchmark's fairness as a measurement instrument are at stake.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces vTikZ, a benchmark for evaluating LLMs on TikZ code customization tasks in which a natural-language instruction requires modifying existing code to achieve a specific visual change. The benchmark contains 100 manually curated editing scenarios derived from 45 TikZ programs, parameterized ground-truth solutions, and a reviewing tool. The authors evaluate five open-source LLMs and GPT-4o (with and without image input) using compile, feature-location, success-customization, similarity, and line metrics, reporting success rates of 13% in one-shot and 28% with five attempts for the best model. The paper also contributes a dataset of 300 human-annotated LLM outputs and a failure-mode analysis.","tokens_in":15031,"tokens_out":5650,"duration_ms":53920,"significance":"If the benchmark is valid, it fills a genuine gap: existing software-engineering benchmarks ignore visual output, and existing visual benchmarks do not involve editing existing code. The released dataset, evaluation harness, and human annotations are concrete assets for future work. The qualitative finding that current LLMs are unreliable at such customization tasks is supported by the low success rates and the failure-mode analysis in Section 6.2. However, the precise numerical claims depend on the completeness of the parameterized ground truths, which the manuscript itself flags as incomplete; this makes the headline percentages lower bounds rather than point estimates.","major_comments":[{"comment":"The ground-truth incompleteness acknowledged in Sections 4.4 and 6.3 directly affects the headline success rates reported in Section 8. Section 4.4 states that human annotation \"surfaced 13 additional correct solutions that were previously missed,\" Section 6.3 concedes that the four parameterization commands \"may not cover cases where certain lines are optional or line placement does not affect correctness,\" and Section 8 states that the results rely on version 1.0, with \"potentially incorporating missing correct solutions\" listed as a planned change. Because SuccessCustomizationMetric marks an output as successful only if it matches a parameterized reference solution or produces an exact image match, any model output that is one of these 13 valid solutions (or another valid solution outside the parameterization) is counted as a failure. The reported 13% one-shot and 28% best-of-5 figures are therefore lower bounds, and the magnitude of the resulting underestimation is unknown. I request that the authors either incorporate the 13 solutions into the benchmark and recompute all results, or provide a sensitivity analysis that quantifies how many of the evaluated outputs would be reclassified if these solutions were accepted.","section":"4.4, 6.3, 8"},{"comment":"The definition of LocationMetric as \"whether 100% of the lines of a patch was edited\" is ambiguous: it is not clear whether it requires every line of the reference patch to appear in the generated patch, whether every edited line in the generated patch must be among the reference lines, or both. This ambiguity matters because LocationMetric is ranked second in the priority order used in Section 4.2 to select the best reference tuple per variant, so it can influence the final success classification. Please specify the exact computation (for example, precision, recall, or exact set equality) and the behavior when the generated patch contains extra or missing lines.","section":"4.1, 4.2"}],"minor_comments":[{"comment":"Mixtral-8x7B appears in Table 3 but is missing from the enumeration of evaluated models at the start of Section 5.1; please add it to the model list or explain why it is only reported in the table.","section":"5.1, Table 3"},{"comment":"The sentence in Section 5.3 stating that \"the number of successful edits decreased marginally (by only one) when solely relying on code inputs\" is confusing; please rephrase to report the actual counts for the text-only and text-plus-image conditions.","section":"5.3"},{"comment":"The formula for LineMetric divides by the total number of edited lines in the reference solution; if the generated patch edits more lines than the reference, the metric may exceed 100. Please clarify whether the metric is capped at 100 or normalized.","section":"4.1, LineMetric"},{"comment":"The character-length outlier criterion is described using an IQR formula that is not rendered cleanly in the submitted text; please format the formula properly with standard notation.","section":"3.1"},{"comment":"The vision-oracle experiments in Section 6.4 report misclassification counts without confidence intervals or statistical detail; these are clearly preliminary, so please label them explicitly as such and avoid drawing strong conclusions from the raw counts.","section":"6.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about its limitations, but the ground-truth incompleteness is substantial enough that the benchmark's headline success numbers should not appear as final point estimates in a published form. The authors should either fix the benchmark or bound the effect. I would also ask the handling editor to confirm the novelty claim of being the first code-customization benchmark with visual validation, given that the authors themselves cite Wei et al. (2024) as partially evaluating code editing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is a new benchmark, vTikZ, for evaluating whether LLMs can modify existing TikZ code to achieve a specified visual change. That is genuinely new: prior work mostly tests generation from scratch or text-only edits, and the paper makes the separation cleanly. The parameterized ground truths, the visual reviewing tool, and the 300 human-annotated variants are real artifacts, and the data and code are public. That reproducibility deserves credit.\n\nThe central finding—that current LLMs are unreliable at this task—is credible. The failure-mode analysis (feature not found, wrong edit, too many features edited) lines up with the low raw scores and gives the paper diagnostic value. I would trust the qualitative conclusion without hesitation.\n\nWhere the paper is soft is exactly where the stress-test note lands. The version 1.0 ground truths omit the 13 additional correct solutions that the authors' own annotation surfaced (Section 4.4). Since the SuccessCustomizationMetric matches against parameterized solutions or exact image equality, those 13 missing solutions mean the reported 13% one-shot and 28% best-of-5 figures are lower bounds. The magnitude is unknown—if any model-generated variant corresponds to one of those 13, that task flips from fail to pass. The stress-test's arithmetic (up to 41% best-of-5) is speculative but not misleading; the point is that the headline numbers should be read as floor estimates, not precise measurements. The authors acknowledge this in Section 8, listing \"potentially incorporating missing correct solutions\" as a planned change, so it is not a hidden flaw—but it does mean the benchmark is a v1.0, not a settled instrument.\n\nOther soft spots, in proportion: 100 tasks is small; the dataset was built by one annotator (mitigated somewhat by expert review); and the VLM-oracle experiments are preliminary, which the paper mostly presents as such.\n\nWho gets value: benchmark builders, people evaluating coding agents, and anyone working on multimodal or self-refining code generation. The paper deserves a serious referee. I would send it to review, ask for the ground truths to be updated (or at least for the 13 solutions to be folded in and the numbers recomputed), and note that the absolute scores will likely shift upward but the qualitative conclusion will survive.\n\nEnd with a clear recommendation: engage with it as a promising benchmark contribution, not as a definitive measurement.","headline":"A useful first-version benchmark for LLM code customization with visual validation, but the reported 13%/28% success rates are lower bounds because 13 known-valid solutions are missing from the v1.0 ground truths.","tokens_in":15600,"tokens_out":1354,"would_cite":true,"duration_ms":14885,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLMs customize TikZ code to match a requested visual change only 13% of the time in one shot, and 28% when given five tries.","keywords":["LLM code customization","visual intent","TikZ","benchmark","parameterized ground truth","multimodal feedback","code editing evaluation"],"falsifier":"Re-run the evaluation with an expanded ground-truth set that includes the 13 human-annotated additional solutions and permits optional-line and placement-insensitive edits, then check whether the one-shot success rate rises measurably above 13% and the best-of-five rate above 28%. A move of several percentage points would indicate the reported numbers underestimate what LLMs can do; a negligible change would confirm the gap.","tokens_in":14638,"feed_emoji":"🎨","tokens_out":4959,"duration_ms":46256,"temperature":0.7,"pith_summary":"This paper introduces vTikZ, a benchmark built to measure whether large language models can customize existing TikZ code so that the rendered image matches a natural-language edit request. The authors assembled 100 TikZ customization scenarios drawn from real diagrams, with parameterized reference solutions to allow more than one correct edit, plus a reviewing tool that compares generated images against those solutions. Running current models on the benchmark, they report that LLMs produce the desired output in only 13% of cases in a single attempt and in 28% of cases when allowed five attempts. The result matters because code customization with visual output—adjusting a diagram, interface, or image by editing source—is exactly the task where users cannot easily verify code behavior, and the benchmark gives the community a reproducible way to measure and close that gap.","feed_headline":"LLMs nail visual code edits only 13% of the time","feed_subtitle":"A new benchmark, vTikZ, tests whether AI can customize TikZ diagrams to match a requested change.","key_machinery":"The load-bearing object is the parameterized ground-truth framework. Each of the 100 scenarios pairs a TikZ program with an instruction and a list of solution templates written with four parameterization directives—§range(lower, higher, default), §rangei(value, interval), §choice([A,B,...], default), and §def(value)—so that multiple valid colors, radii, shifts, and variable names count as correct. Correctness is then decided by the SuccessCustomizationMetric, which accepts a generated variant if it matches one of these parameterized solutions or produces an exact rendered-image match, and ranking across reference solutions uses compile status, line-location, line-edit rate, and patch similarity in priority order. The framework is what allows the 13% and 28% numbers to be measured at all, and the paper's 300 human annotations of model outputs are used to validate and extend that ground truth.","core_discovery":"vTikZ is, to the authors' knowledge, the first benchmark explicitly targeting LLM-driven customization of code with visual results rather than code generation from scratch. Its central finding is that current models fail this task most of the time: against 100 carefully curated TikZ editing scenarios, the success rate for producing a fully correct modified diagram is about 13% in a one-shot setting and about 28% using best-of-five sampling with the strongest model evaluated. The benchmark also quantifies where failures occur: models often compile successfully and often find the right lines, but they frequently edit the wrong feature, edit too many features, or apply the right edit incompletely. The paper argues that this gap shows LLMs are effectively blind to the rendered output during editing and that integrating visual feedback mechanisms is a necessary next direction.","pith_inferences":["A likely reason for the low success rates is that LLMs cannot see the rendered picture during editing; the paper's own oracle experiments, where vision models misclassified a substantial share of correct solutions, support the inference that simply adding an image input does not yet supply reliable feedback.","Because the ground truth penalizes valid edits it cannot express, the 13% and 28% figures are better read as lower bounds on model capability than as exact ability ceilings.","One testable extension of the paper's argument: an agent that renders its own draft, compares it with a rendered reference via a dedicated diffing tool, and re-prompts on mismatch should beat best-of-five sampling; if it does not, the bottleneck is not feedback but the model's code-editing competence itself.","The benchmark's focus on a single-file, single-diagram language means it likely understates the difficulty of visual customization in multi-file projects, where feature location spans files and edits must preserve cross-file consistency."],"forward_implications":["If the reported rates hold, current LLM-based code assistants are not yet reliable enough for real-world use on visual code customization, since even the strongest model fails three out of four times when given five tries.","The vTikZ methodology transfers to other graphics languages: the same scenario, prompt, parameterized-ground-truth, and review-tool structure can be applied to SVG, P5.js, Turtle, or Matplotlib.","The 300 human-annotated variants become training or evaluation material for models that incorporate visual feedback or human preference signals.","Benchmark scores can serve as a quantitative target for agentic or self-refining systems that render intermediate output and iterate until the rendered image matches the instruction.","The explicit split between feature location and edit execution makes vTikZ useful for diagnosing whether failures come from understanding the prompt, finding the code region, or making the final change."],"supporting_citations":[{"why":"Supplies the tree edit distance algorithm used to compute the AST difficulty of each scenario.","marker":"[42]"},{"why":"Defines CrystalBLEU, the patch similarity metric used to score and rank generated variants.","marker":"[13]"},{"why":"Represents recent TikZ code generation from sketches that the paper contrasts with its editing task.","marker":"[8]"},{"why":"Text-to-diagram generation and editing benchmark used as the closest prior work on TikZ editing.","marker":"[35]"},{"why":"Repository-level code edit benchmark that anchors the difference between textual editing and visual-output editing.","marker":"[21]"},{"why":"TikZ generation model whose fine-tuned variant the authors tried and found unsuitable for code editing.","marker":"[7]"},{"why":"Object-detection parser considered as a candidate visual oracle for verifying edits, with limitations discussed.","marker":"[28]"}],"fun_headline_variants":["LLMs ace only 13% of visual code edits","Visual code customization: LLMs succeed 13% of time","New benchmark vTikZ reveals LLM visual editing gap","LLMs blind to output: 13% success on TikZ edits","TikZ benchmark: LLMs score 13% on visual edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's correctness rule assumes that the parameterized reference solutions capture essentially all reasonable ways to satisfy each instruction; the paper itself notes that the four parameterization commands may miss cases where lines are optional or placement does not matter, and human review surfaced 13 additional correct solutions.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace only 13% of visual code edits","Visual code customization: LLMs succeed 13% of time","New benchmark vTikZ reveals LLM visual editing gap","LLMs blind to output: 13% success on TikZ edits","TikZ benchmark: LLMs score 13% on visual edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1516,"prompt_tokens":907,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":523,"tokens_out":609,"duration_ms":5438,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:34:15.651907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with an expanded ground-truth set that includes the 13 human-annotated additional solutions and permits optional-line and placement-insensitive edits, then check whether the one-shot success rate rises measurably above 13% and the best-of-five rate above 28%. A move of several percentage points would indicate the reported numbers underestimate what LLMs can do; a negligible change would confirm the gap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the tree edit distance algorithm used to compute the AST difficulty of each scenario."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CrystalBLEU, the patch similarity metric used to score and rank generated variants."},{"cited_title":"From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing","cited_arxiv_id":"2411.11916","evidence_quote":"Text-to-diagram generation and editing benchmark used as the closest prior work on TikZ editing."}],"review_version":1}