{"id":"da4b0130-7b0f-47c2-96d6-6b721bd6d782","arxiv_id":"2608.10408","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new benchmark of 1,395 human-annotated visualization code-editing tasks shows that even the best vision-language model (Claude-4.6-Sonnet) only passes 74.46% of edits, while render-grounded iterative refinement improves GPT-4o from 55.75% to 67.99%.","lead":"VisEditBench gives AI models 1,395 chart-editing tests where they must fix or restyle visualization code using a chart image and a text instruction. Across 20 vision-language models, even the best model fails roughly a quarter of the time, and a render-checking agent shows that looking at the output chart helps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automatic judge calibration gap: human pass rates run 4–9 points below the GPT-4o/Gemini judge, so the headline ranking and the VisEditAgent gain are not yet established.","rationale":"The paper is carefully constructed and the automatic evaluator is unusually transparent (Table 9), with a genuine human validation effort on 2,790 outputs. The data-pipeline and annotation quality are real strengths, and a render-validate-refine loop improving visual editing is plausible. However, the strongest claimed quantitative results all pass through the automatic judge. The reader's weakest assumption identifies this correctly. My stress-test sharpens it: the existing Table 7 vs Table 4 discrepancy is not just a uniform offset; the judge is more lenient on VisEditAgent outputs (8.89 points) than on zero-shot outputs (4.05 points), which means the headline agent improvement is partly an artifact of judge leniency unless calibrated. This does not refute the paper—the human eval still shows an improvement—but it does mean the absolute pass rates, the sub-50% characterization of open models, and the exact agent gain should be treated as provisional. The proposed human-sample check is the minimal experiment that would settle whether the ranking and gain survive bias correction.","tokens_in":16911,"tokens_out":4119,"duration_ms":39076,"concrete_test":"Human-score a stratified 500-example sample of outputs from Claude-4.6-Sonnet, GPT-5, Qwen3-VL-32B, and GPT-4o+VisEditAgent using the Table 9 rubric; then recompute pass rates and rankings after applying the per-model automatic-vs-human offset. If Claude-4.6-Sonnet remains top and VisEditAgent's corrected gain over GPT-4o zero-shot is still positive, the central claims hold; if the offset moves any model across the 50% line or shrinks the agent gain to near zero, the claims need weakening.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number—Claude-4.6-Sonnet 74.46%, \"most open-source models below 50%\", VisEditAgent 55.75→67.99—is produced by a rubric-based VLM judge (GPT-4o, with Gemini 2.5 Pro for GPT-4o outputs) validated only on partial human data. Table 8 reports Pearson correlations of 83.38–87.00 and 81% pass/fail agreement, but Table 7 shows the judge is systematically more lenient: for GPT-4o zero-shot, automatic pass rate is 55.75% vs human 51.70% (−4.05); for GPT-4o+VisEditAgent, 67.99% vs 59.10% (−8.89). The leniency is not constant across conditions, so the automatic agent gain (+12.24 points) exceeds the human-measured gain (+7.40 points). Because no calibration analysis, confidence intervals, or threshold sensitivity analysis are reported, the central claim that the render-validate-refine loop \"improves overall pass rate from 55.75% to 67.99%\"—and the model ranking behind it—depends on unmeasured judge leniency that varies by output condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisEditBench, a benchmark of 1,395 human-annotated visualization code-editing tasks designed to test whether vision-language models can revise existing visualization code from multimodal feedback. It covers two settings (feedback-guided repair and reference-guided restyling), defines an eight-intent taxonomy, evaluates 20 VLMs in a zero-shot setting, and proposes VisEditAgent, a render-validate-refine framework. The headline results are that Claude-4.6-Sonnet achieves the best overall pass rate of 74.46%, most open-source models remain below 50%, and VisEditAgent improves GPT-4o's pass rate from 55.75% to 67.99%. Human validation on 2,790 outputs shows moderate-to-strong agreement with the automatic judge but reveals a systematic leniency bias in the automatic pass rates.","tokens_in":16972,"tokens_out":8661,"duration_ms":72263,"significance":"VisEditBench addresses a genuine gap: iterative visualization editing from multimodal feedback is underexplored relative to one-shot chart generation and chart-to-code reconstruction. The paper has notable strengths, including a careful annotation pipeline with dual annotation and cross-review (84.65% agreement), a substantial human-validation effort, a structured evaluation rubric, and a plan to release the benchmark. If the evaluation calibration issues are resolved, the benchmark would be a valuable resource for studying grounded multimodal code editing. VisEditAgent is a sensible and clearly described baseline that demonstrates the potential of render-grounded refinement, although its quantitative gains currently depend on an uncalibrated automatic judge.","major_comments":[{"comment":"The automatic judge is systematically more lenient than human raters, and the leniency is condition-dependent: for GPT-4o zero-shot the automatic pass rate is 55.75% versus 51.70% human, while for GPT-4o+VisEditAgent it is 67.99% versus 59.10% human. As a result, the reported agent gain (+12.24 points) exceeds the human-measured gain (+7.40 points), so the abstract's headline improvement from 55.75% to 67.99% is not robust to judge calibration. The paper should report human-calibrated pass rates (or apply a calibration transform) for the main comparisons, include confidence intervals, and perform a threshold-sensitivity analysis over the final pass thresholds (task accuracy ≥ 4.5, readability/clarity ≥ 4.0, visual quality ≥ 4.0, visual similarity ≥ 90) to verify that the ranking and the agent gain are stable.","section":"Section 5.1, Tables 7–8"},{"comment":"Human validation covers only two of the twenty evaluated models: GPT-4o on all 1,395 tasks and Qwen3-VL-4B on a 500-example subset. The claim that Claude-4.6-Sonnet is the best model at 74.46% pass rate, and the comparisons among the other seventeen models, rest entirely on the automatic judge, whose calibration for those models is unknown. At minimum, human spot-checks on the top-scoring model and on a few open-source models should be reported, or the paper should explicitly qualify the ranking as automatic-judge-based until calibration evidence is provided.","section":"Section 5.2.2, Table 8"}],"minor_comments":[{"comment":"The editing-intent counts sum to 1,384, not the stated 1,395; either an 'Other' intent is missing or the counts/percentages need correction.","section":"Table 1"},{"comment":"Pearson correlations are reported as 83.38, 87.00, etc., which reads as percentages; standard notation would be r = 0.8338 and r = 0.8700 to avoid ambiguity.","section":"Table 8"},{"comment":"Since 1,275 of the 1,395 tasks derive from model-generated failures on Text2Vis rather than naturally occurring user edit requests, the paper should briefly discuss how this source distribution affects the realism and difficulty representativeness of the benchmark.","section":"Section 3.1"},{"comment":"The difficulty labels are assigned by GPT-5, which sits uneasily with the statement in the Ethical Considerations that LLM assistance was used only for language editing and polishing; the role of the LLM in difficulty labeling should be stated more prominently.","section":"Section 3.3"},{"comment":"The phrase 'most open-source models remain below 50%' is accurate, but since Qwen3-VL-32B achieves 51.72%, naming the exception would improve precision.","section":"Abstract"},{"comment":"The sentence reporting Pearson correlations should present them as decimal coefficients (0.8338–0.8700) and should separate the 81% pass/fail agreement from the correlation figures.","section":"Section 5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The calibration gap is the key blocking point. I believe the benchmark is a useful contribution and the human-validation effort is commendable, but the headline quantitative claims need to be recalibrated or re-reported with human-based numbers. This is fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it fills a real gap: no prior benchmark evaluates editing existing visualization code from multimodal feedback with reference-guided restyling. The construction is careful — dual annotation, cross-review, 84.65% agreement, human validation of 2,790 outputs, and an explicit eight-intent taxonomy. Second, every headline number is produced by a rubric-based VLM judge that is systematically more lenient than humans, and the leniency is not constant across conditions. That is the main thing to keep in mind before you quote any pass rate.\n\nWhat the paper does well: the benchmark design is honest and well-documented. The authors report human evaluation of the same outputs, with Pearson correlations from 83 to 87 and 81% pass/fail agreement. They also show that executability is not the bottleneck — Claude-4.6-Sonnet executes 96% of tasks but only scores 89.4 visual similarity. That is a solid, plausible finding. The VisEditAgent framework is a reasonable render-validate-refine baseline, and the ablation isolating candidate generation and refinement is useful.\n\nThe soft spots are real but not fatal. The automatic judge gives GPT-4o zero-shot a 55.75% pass rate versus 51.70% from humans, and for VisEditAgent it gives 67.99% versus 59.10%. So the paper's headline improvement of +12.24 points is likely inflated; the human-measured gain is +7.40. The paper never addresses this calibration gap. It also reports no confidence intervals or significance tests for any ranking or ablation, and the final pass thresholds are hand-set with no sensitivity analysis. These are addressable weaknesses, and the qualitative conclusion — that render-grounded feedback helps — survives the human check.\n\nWho this is for: anyone building chart copilots or evaluating VLMs on visualization tasks. It deserves a serious referee, but the referee should ask for calibration of the automatic judge, uncertainty estimates, and the actual dataset before the numbers become citable. My recommendation: engage with it, but treat the absolute numbers as preliminary until the calibration question is settled.","headline":"A genuinely useful first benchmark for visualization code editing from multimodal feedback, but the headline numbers rest on an uncalibrated automatic judge; human checks confirm the direction of the agent gain, not its magnitude.","tokens_in":17723,"tokens_out":1519,"would_cite":true,"duration_ms":15180,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes visualization code editing from multimodal feedback as a distinct, measurable, mostly unsolved capability, with the best of 20 models at a 74.46% pass rate and a render-grounded agent adding 12.2 points over its base…","keywords":["visualization code editing","vision-language models","multimodal feedback","benchmark","chart repair","style adaptation","render-grounded feedback","iterative refinement"],"falsifier":"Take a stratified sample of outputs the automatic judge marks pass and fail (including all style-adaptation cases), have human raters judge them by blind pairwise preference instead of the rubric, and recompute the 20-model ranking and the GPT-4o zero-shot versus VisEditAgent gap from the human labels alone. If the human gap is less than half of the reported 12.2 points, or if the human preference ordering differs on any model family, the central claim collapses. A cheaper version: rerun the full evaluation with the visual-similarity threshold raised from 90 to 95 and check whether any headline ranking changes.","tokens_in":16546,"feed_emoji":"📊","tokens_out":9123,"duration_ms":76812,"temperature":0.7,"pith_summary":"Real-world visualization work is iterative—users fix broken charts and restyle them, not generate from scratch once—yet benchmarks have measured one-shot generation. The paper argues that editing existing visualization code from a rendered chart plus a natural-language instruction is a separate capability, and that current vision-language models are weak at it. To support the argument it contributes VisEditBench, 1,395 human-annotated repair and restyling tasks, plus zero-shot results for 20 models: Claude-4.6-Sonnet passes 74.46%, most open models fall below 50%, and visually grounded style adaptation is the hardest intent at 55.71%. It then shows VisEditAgent, a generate-render-validate-refine loop, raising GPT-4o from 55.75% to 67.99%, concluding that render-grounded feedback, not larger models alone, is the lever for faithful chart editing.","feed_headline":"Chart-editing AI tops out at 74% pass rate","feed_subtitle":"1,395-task benchmark: style restyling stumps most models; render-check loop lifts GPT-4o from 56% to 68%.","key_machinery":"The load-bearing object is VisEditBench's task tuple—input visualization code, a rendered chart that is either buggy/marked or a style reference, and a natural-language instruction—with an eight-intent taxonomy and a strict pass criterion requiring executability, task accuracy at least 4.5/5, readability and visual quality at least 4.0/5, and visual similarity at least 90/100. The mechanism that carries the argument is VisEditAgent's render-grounded loop: plan the edit, generate multiple candidate revisions, execute and render each candidate, visually validate candidates against the instruction and reference chart, select the best, then refine it using the validation feedback. That loop is what converts the idea that a model should see its own output into a measured gain.","core_discovery":"On the paper's own terms, the central discovery is that executing code is not the bottleneck in visualization editing—visual fidelity is. Top models render successfully on around 96% of VisEditBench tasks yet miss the strict fidelity bar on most of them, and the discrepancy is largest for style adaptation, where editing means matching a reference image's colors, layout, fonts, and encodings without changing the data's meaning. A second discovery is that a loop which renders candidate edits and validates them against the target image converts that visual gap into a usable training signal: VisEditAgent lifts GPT-4o's overall pass rate from 55.75% to 67.99%, with style adaptation jumping from 10.00% to 62.85%. The authors read this as evidence that visualization authoring should be treated as an iterative multimodal editing problem rather than a one-shot code-generation problem.","pith_inferences":["Extension beyond the paper: since human–automatic pass/fail agreement is 81%, an independent blind pairwise human-preference study could reveal whether the rubric over-rewards clean-but-generic charts, which would affect the reported 12.2-point agent gain.","Extension: VisEditBench covers Matplotlib and Vega-Lite only; porting a stratified subset to Plotly or D3 would show whether render-grounded validation transfers across imperative and declarative ecosystems.","Extension: manipulating rendering resolution inside VisEditAgent would isolate the active ingredient—if coarse rendering shrinks the gain, visual detail, not candidate count or planning, is what matters.","Extension: the taxonomy invites a targeted test—retrieving a visually similar solved edit should help style adaptation and transformation more than correctness repair, since the latter already names the fix in text."],"forward_implications":["Executability alone cannot serve as a proxy for visualization skill; any benchmark or agent that stops at 'the code runs' will overstate capability.","Style adaptation and reference-guided restyling are the hardest intents, so progress claims should report them separately rather than as part of an aggregate pass rate.","Render-grounded iterative refinement improves editing across closed- and open-source base models, and ablations show both multi-candidate generation and refinement contribute, with the largest effect on style adaptation.","Because the benchmark's pass threshold is strict, scores near 50–70% mean the task is far from solved even for frontier models; claims of human-level chart editing would need to clear the same bar.","The eight-intent taxonomy gives future systems a target map: correctness repair and quality improvement are comparatively tractable, while style-aware repair and transformation remain open."],"supporting_citations":[{"why":"Supplies the real-world tables and queries whose failed model-generated charts became many of VisEditBench's repair tasks, and serves as the primary text-to-visualization comparison.","marker":"Rahman et al., 2025b"},{"why":"Prior multimodal software-repair benchmark used as the contrast point for what visualization-specific editing adds beyond general visual bug fixing.","marker":"Yang et al., 2024b"},{"why":"Earlier text-to-visualization benchmark that VisEditBench extends by adding code editing from multimodal feedback instead of generation from scratch.","marker":"Chen et al., 2024"},{"why":"Chart-to-code reconstruction benchmark against which VisEditBench distinguishes its editing setting; cited in the gap argument.","marker":"Wu et al., 2025"},{"why":"Provides the iterative visualization workflow principle that motivates VisEditAgent's render-validate-refine design.","marker":"Munzner, 2014"},{"why":"Documents GPT-4o, the paper's default zero-shot baseline, the base model of VisEditAgent, and the default automatic evaluator.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Render, don't just code: AI chart editors need visual check","Style beats code: AI chart editing fails at visual fidelity","AI can draw charts, but can't see style: 55% on restyling","From 56% to 68%: render feedback lifts GPT-4o chart editing","Seeing is editing: AI improves chart fixes with render feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking and the reported agent gain rest on the automatic VLM judge's scores and hand-set thresholds matching what a human would accept as a faithful edited chart, and the paper's own human check shows only partial agreement (81% pass/fail agreement; Pearson correlations of 83.38–87.00).","fun_headline_variants_meta":{"raw":{"variants":["Render, don't just code: AI chart editors need visual check","Style beats code: AI chart editing fails at visual fidelity","AI can draw charts, but can't see style: 55% on restyling","From 56% to 68%: render feedback lifts GPT-4o chart editing","Seeing is editing: AI improves chart fixes with render feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3162,"prompt_tokens":1016,"completion_tokens":2146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2049}},"tokens_in":632,"tokens_out":2146,"duration_ms":13722,"temperature":1.0,"reasoning_tokens":2049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:21:14.576560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified sample of outputs the automatic judge marks pass and fail (including all style-adaptation cases), have human raters judge them by blind pairwise preference instead of the rubric, and recompute the 20-model ranking and the GPT-4o zero-shot versus VisEditAgent gap from the human labels alone. If the human gap is less than half of the reported 12.2 points, or if the human preference ordering differs on any model family, the central claim collapses. A cheaper version: rerun the full evaluation with the visual-similarity threshold raised from 90 to 95 and check whether any headline ranking changes.","supporting_citations":[],"review_version":1}